Retrieval device, retrieval method and program

Abstract

A retrieval device comprising: an acquisition means for acquiring first query data in which a relationship between physical bodies is represented by using relational text representing a relationship between physical bodies; a generation means for generating second query data representing a relationship similar to the relationship represented by the first query data; a calculation means for calculating, from each of the first query data and the second query data, a relationship feature amount representing the feature of a relationship between the physical bodies; and a detection means for using the relationship feature amounts calculated for each of the first query data and the second query data to detect physical body relationship information matching the first query data from a plurality of pieces of physical body relationship information in which a relationship feature amount is indicated for the relationship between the physical bodies included in image data.

Description

Search device, search method, and program

[0001] The present disclosure relates to a search device, a search method, and a program.

[0002] Technologies for searching information about images have been developed. For example, Patent Literature 1 discloses a technology for managing a plurality of images by associating each image with a feature amount related to the posture of a person included in the image. The search device in Patent Literature 1 calculates a feature amount related to the posture of a person included in a query image, and searches an image database for images associated with a feature amount that matches the calculated feature amount. This makes it possible to search the image database for images containing a person in a desired posture.

[0003] Japanese Patent Application Laid-Open No. 2019-091138

[0004] The technology of Patent Document 1 does not anticipate retrieval of information based on anything other than the posture of a person. The present disclosure has been made in light of this problem, and one of its purposes is to provide a new technology for retrieving information obtained from an image.

[0005] A search device according to the present disclosure includes an acquisition means for acquiring first query data in which a relationship between objects is expressed using relational text representing the relationship between the objects; a generation means for generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation means for calculating relation features that represent characteristics of the relationship between the objects from each of the first query data and the second query data; and a detection means for detecting the object relationship information that matches the first query data from a plurality of object relationship information pieces in which the relation features are indicated for the relationship between objects included in image data, using the relation features calculated for each of the first query data and the second query data.

[0006] A search method according to the present disclosure includes an acquisition step of acquiring first query data that expresses a relationship between objects using relational text that expresses the relationship between the objects; a generation step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation step of calculating, from each of the first query data and the second query data, a relation feature that expresses a characteristic of the relationship between the objects; and a detection step of detecting, from a plurality of object relation information pieces that indicate the relation feature about the relationship between objects included in image data, the object relation information pieces that match the first query data, using the relation feature calculated for each of the first query data and the second query data.

[0007] The program according to the present disclosure causes a computer to execute the following steps: an acquisition step of acquiring first query data that expresses a relationship between objects using relation text that expresses the relationship between the objects; a generation step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation step of calculating, from each of the first query data and the second query data, a relation feature that expresses a characteristic of the relationship between the objects; and a detection step of detecting, from a plurality of object relation information pieces that indicate the relation feature about the relationship between objects included in image data, the object relation information pieces that match the first query data, using the relation feature calculated for each of the first query data and the second query data.

[0008] According to the present disclosure, new techniques are provided for retrieving information from images.

[0009] FIG. 1 is a diagram illustrating an overview of the operation of a search device. FIG. 2 is a block diagram illustrating an example of the functional configuration of a search device. FIG. 3 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the search device. FIG. 4 is a flowchart illustrating an example of the flow of processing executed by the search device. FIG. 5 is a diagram illustrating an example of the configuration of object relation information. FIG. 6 is a diagram illustrating an example of the configuration of object relation information including information about source image data. FIG. 7 is a diagram illustrating an example of the configuration of object information. FIG. 8 is a diagram illustrating an example of the configuration of a vision-and-language model. FIG. 9 is a diagram illustrating an example of a relationship extraction model.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an example of an overview of the operation of the search device 2000. Here, Fig. 1 is a diagram for facilitating understanding of the overview of the search device 2000, and the operation of the search device 2000 is not limited to the operation shown in Fig. 1.

[0012] The search device 2000 is used to search for desired object relation information 40 from a plurality of pieces of object relation information 40. The object relation information 40 is stored in advance in an arbitrary storage unit (for example, a storage unit of a database server) in a form that can be acquired by the search device 2000. Hereinafter, the storage unit in which the object relation information 40 is stored will be referred to as an object relation information storage unit 90.

[0013] The object relation information 40 indicates relation features for a pair of objects. The relation features of an object pair are features that represent the characteristics of the relationship between two objects. One of the two objects in an object pair is a subject and the other is an object. Hereinafter, an object pair will also be referred to as an "object pair." Hereinafter, the relationship between objects will also be referred to as an "inter-object relationship."

[0014] Various relationships can be handled as inter-object relationships. For example, inter-object relationships represent actions performed by a subject against an object. For example, in a situation where "a person lifts a ball," the subject, the person, performs the action of "lifting" on the object, the ball. Therefore, there is a relationship of "lifting" between the two objects, the person and the ball. Therefore, object relationship information 40 generated for the situation where "a person lifts a ball" shows the pair of objects, the person and the ball, in association with a relationship feature that represents the relationship of "lifting."

[0015] In addition, for example, inter-object relations represent the positional relationship between a subject and an object. For example, in a situation where "a bird is on the sea," there is a positional relationship of "above" between two objects, the bird as the subject and the sea as the object. Therefore, object relation information 40 generated for the situation where "a bird is on the sea" represents the object pair of the bird and the sea in association with a relation feature that represents the relationship of "above."

[0016] The object-related information 40 is generated, for example, by analyzing one or more source image data, which may be still image data generated by a still camera or video frames obtained from video data generated by a video camera.

[0017] For example, suppose source image data includes a person sitting on a chair and a dog holding a ball in its mouth. This source image data shows the situations "the person sitting on the chair" and "the dog holding the ball in its mouth." Two pieces of object relation information 40 can be generated from this source image data. The first piece of object relation information 40 is associated with the object pair of the person and the chair and indicates a relation feature representing the relationship "sitting." The second piece of object relation information 40 is associated with the object pair of the dog and the ball and indicates a relation feature representing the relationship "holding."

[0018] Here, the relation feature may include information about the subject, information about the object, or both. When the relation feature includes information about the subject, the relation feature generated for the situation "a person sits on a chair" represents the information that "a person sits on something." When the relation feature includes information about the object, the relation feature generated for the situation "a person sits on a chair" represents the information that "something sits on a chair." When the relation feature includes information about both the subject and the object, the relation feature generated for the situation "a person sits on a chair" represents the information that "a person sits on a chair."

[0019] A search by the search device 2000 is performed, for example, as follows. First, the search device 2000 acquires first query data 10. The first query data 10 includes text representing a relationship between objects. Specifically, the first query data 10 includes text representing a subject object (subject text 12), text representing an object object (object text 14), and text representing the relationship (relationship text 16).

[0020] For example, the first query data 10 is a sentence (hereinafter referred to as query text) including subject text 12, object text 14, and relation text 16. For example, the query text "The bird is above the sea" includes "bird" as the subject text 12, "sea" as the object text 14, and "above" as the relation text 16. Note that, as will be described later, the first query data 10 is not limited to query text.

[0021] The search device 2000 uses the second query data 20 in addition to the first query data 10 to search for object relation information 40 that matches the first query data 10. To this end, the search device 2000 generates the second query data 20 based on the first query data 10.

[0022] The second query data 20 represents an inter-object relationship similar to the inter-object relationship represented by the first query data 10. The second query data 20 includes subject text 22, object text 24, and relation text 26.

[0023] For example, the search device 2000 generates second query data 20 by replacing the related text 16 in the first query data 10 with text that is similar in meaning to the related text 16 (hereinafter, similar related text). In this case, the second query data 20 includes text that is identical to the subject text 12 as the subject text 22, text that is identical to the object text 14 as the object text 24, and similar related text as the related text 26. The similar related text is, for example, a synonym of the related text 16 or a paraphrase of the related text 16. For example, a similar related text for the related text 16 "on top" would be "above it."

[0024] The search device 2000 calculates a relation feature representing an inter-object relation represented by the first query data 10 and a relation feature representing an inter-object relation represented by the second query data 20. Furthermore, for each of one or more pieces of object relation information 40, the search device 2000 calculates a similarity between the object relation information 40 and the first query data 10 (hereinafter referred to as a first similarity) and a similarity between the object relation information 40 and the second query data 20 (hereinafter referred to as a second similarity). The similarity between the object relation information 40 and the query data is represented by the similarity between the relation feature represented in the object relation information 40 and the relation feature calculated for the query data.

[0025] The search device 2000 detects object relation information 40 that matches the first query data 10 from among the plurality of pieces of object relation information 40, based on the first similarity and the second similarity calculated for each piece of object relation information 40. "Object relation information 40 that matches the first query data 10" refers to object relation information 40 that should be acquired as a result of a search using the first query data 10. For example, if the statistical values ​​of the first similarity and the second similarity calculated for a certain piece of object relation information 40 are equal to or greater than a threshold, the search device 2000 detects the object relation information 40 as object relation information 40 that matches the first query data 10.

[0026] The search device 2000 may generate a plurality of pieces of second query data 20 from the first query data 10. For example, when the first query data 10 is a query text, the search device 2000 identifies a plurality of paraphrases of the query text and generates the second query data 20 for each of the plurality of paraphrases.

[0027] When a plurality of pieces of second query data 20 are generated, the search device 2000 may calculate a relation feature for each of the plurality of pieces of second query data 20 to calculate a plurality of second similarities between the plurality of pieces of second query data 20 and the object relation information 40. In this case, the search device 2000 determines whether the object relation information 40 and the first query data 10 match, using one first similarity and a plurality of second similarities.

[0028] Although the object relation information 40 is represented in the form of a table in FIG. 1 , the representation format of the object relation information 40 is not limited to a table. Another example of a data structure representing the relationship between objects is a scene graph. In a scene graph, each object is represented by a node. Furthermore, the relationship between two objects is represented by an edge connecting those objects. Each object relation information 40 can also be represented as a scene graph in which a subject and an object are represented by a node and relationship features are attached to the edges. From this, the processing performed by the search device 2000 can also be considered as processing to detect a desired node pair from a scene graph in which relationship features are attached to edges representing the relationship between objects.

[0029] <Example of Operation and Effect> The search device 2000 can search for information based on the relationship between objects. Specifically, it is possible to search for a relationship between objects that matches the relationship between objects in the object pair represented by the first query data 10 from among the relationships between objects in each object pair extracted from the source image data. In this way, the search device 2000 provides a new technique for searching for information obtained from images, which is a search using the relationship between objects as a key.

[0030] Here, the term expressing a specific relationship between objects is not limited to one, and there are often multiple terms. For example, the relationship "the bird is over the sea" can also be expressed as "the bird is above the sea" or "the bird is flying over the sea." Therefore, the expression of the inter-object relationship in the first query data 10 and the expression of the inter-object relationship used to generate the object relation information 40 (e.g., the expression of the inter-object relationship used to train the machine learning model used to generate the object relation information 40) may differ from each other. Due to this difference, a search for the object relation information 40 performed using only the first query data 10 may fail to find object relation information 40 that matches the first query data 10.

[0031] Therefore, the search device 2000 generates, based on the first query data 10, second query data 20 that represents an inter-object relationship similar to that of the first query data 10. Then, the search device 2000 uses both the first query data 10 and the second query data 20 to search for object relationship information 40 that matches the first query data 10. According to this method, the search for object relationship information 40 that matches the first query data 10 can be performed with higher accuracy.

[0032] The search device 2000 of this embodiment will be described in more detail below.

[0033] <Example of Functional Configuration> FIG. 2 is a block diagram illustrating an example of the functional configuration of the search device 2000. For example, the search device 2000 includes an acquisition unit 2020, a generation unit 2040, a calculation unit 2060, and a detection unit 2080. The acquisition unit 2020 acquires first query data 10. The generation unit 2040 generates second query data 20 based on the first query data 10. The calculation unit 2060 calculates relation feature quantities for each of the first query data 10 and the second query data 20. The detection unit 2080 calculates, for each of one or more pieces of object relation information 40, a first similarity that is the similarity with the first query data 10 and a second similarity that is the similarity with the second query data 20. Furthermore, the detection unit 2080 detects object relation information 40 that matches the first query data 10 based on the first similarity and the second similarity calculated for each of the one or more pieces of object relation information 40.

[0034] <Example of Hardware Configuration> Each functional component of the search device 2000 may be realized by hardware that realizes the respective functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the search device 2000 is realized by a combination of hardware and software will be further described.

[0035] 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the search device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the search device 2000, or may be a general-purpose computer.

[0036] For example, by installing a predetermined application on the computer 1000, the computer 1000 realizes each function of the search device 2000. The application is configured with a program for realizing each functional component of the search device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD (Digital Versatile Disc) or a USB (Universal Serial Bus) memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0037] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0038] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0039] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0040] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a local area network (LAN) or a wide area network (WAN). For example, the computer 1000 is communicably connected to the object relationship information storage unit 90 or a server device that manages the object relationship information storage unit 90 via the network interface 1120.

[0041] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the search device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the search device 2000.

[0042] The search device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0043] 4 is a flowchart illustrating the flow of processing executed by the search device 2000. The acquisition unit 2020 acquires first query data 10 (S102). The generation unit 2040 generates second query data 20 based on the first query data 10 (S104). The calculation unit 2060 calculates relation feature amounts for each of the first query data 10 and the second query data 20 (S106).

[0044] Steps S108 to S116 constitute a loop process L1. The process in the loop process L1 is executed once for each of the plurality of pieces of object-relationship information 40.

[0045] In S108, the search device 2000 determines whether a predetermined termination condition is satisfied. If the predetermined termination condition is satisfied, the search device 2000 terminates the execution of the loop process L1. If the predetermined termination condition is not satisfied, the search device 2000 selects one piece of object relation information 40 that has not yet been subjected to the loop process L1. The object relation information 40 selected here is referred to as object relation information i.

[0046] The detection unit 2080 calculates a first similarity and a second similarity for the object relation information i (S110). The detection unit 2080 determines whether the object relation information i matches the first query data 10 based on the first similarity and the second similarity calculated for the object relation information i (S112). If the object relation information i matches the first query data 10 (S112: YES), the detection unit 2080 detects the object relation information i as object relation information 40 that matches the first query data 10 (S114). Then, the detection unit 2080 ends the loop process L1.

[0047] If the object relation information i does not match the first query data 10 (S112: NO), the process proceeds to S116. Since S116 is the end of the loop process L1, S108 is executed next.

[0048] There are various conditions for ending the loop process L1. For example, the condition for ending the loop process L1 is that "the loop process L1 has been executed for all of the object relation information 40 stored in the object relation information storage unit 90." Another example of the condition for ending the loop process L1 is that "a predetermined time has elapsed since the start of processing by the search device 2000."

[0049] The flow of the process executed by the search device 2000 is not limited to the flow shown in Fig. 4. For example, in Fig. 4, the execution of the loop process L1 ends in response to detection of object relation information 40 that matches the first query data 10. Therefore, at most one piece of object relation information 40 that matches the first query data 10 is detected.

[0050] In this regard, the search device 2000 may be configured to detect two or more pieces of object relation information 40 that match the first query data 10. In this case, in the flowchart of FIG. 4 , even if the object relation information i matches the first query data 10, the loop process L1 is not terminated. Specifically, S116 is executed after S114 is executed. In this way, even after the object relation information 40 that matches the first query data 10 is detected, the detection of the object relation information 40 that matches the first query data 10 (i.e., the loop process L1) continues.

[0051] <About the Object Relation Information 40> The object relation information 40 indicates an object pair and a relation feature amount in association with each other. The relation feature amount corresponding to a certain object pair represents the characteristics of the relationship between the objects in the object pair.

[0052] Fig. 5 is a diagram illustrating an example of the configuration of the object relation information 40. In Fig. 5, the object relation information 40 associates object pairs 42 with relation feature quantities 43. In Fig. 5, a plurality of pieces of object relation information 40 are listed in a table format.

[0053] The object pair 42 indicates an object pair. More specifically, the object pair 42 indicates a correspondence between a subject identifier 44 and an object identifier 45. The subject identifier 44 is an identifier of an object that is a subject. The object identifier 45 is an identifier of an object that is an object.

[0054] The relation feature 43 indicates a relation feature that represents the characteristics of the relationship between a subject and an object for a corresponding object pair. The relation feature is expressed, for example, as a numeric vector of a predetermined length (a vector in which each element indicates a numeric value) or a tensor of a predetermined size.

[0055] The object-relationship information 40 may further indicate information about the source image used to generate the object-relationship information 40. Fig. 6 is a diagram illustrating an example of the configuration of the object-relationship information 40 that includes information about the source image data.

[0056] 6, the object-related information 40 further indicates a source identifier 41. The source identifier 41 indicates an identifier of source image data. The identifier of the source image data is expressed, for example, by the file name or path of a file representing the source image data. Alternatively, for example, if the source image data is a video frame constituting video data, the identifier of the source image data may be expressed by a frame number. Furthermore, if multiple video data are used to generate the object-related information 40, the identifier of the source image data is expressed, for example, by a combination of an identifier of the video data (such as a file name or path) and a frame number. Furthermore, if source images are generated from multiple cameras, the identifier of the camera may further be added to the identifier of the source image data.

[0057] Here, when the object relation information 40 is generated using video data, the relation between a certain object pair may continue across multiple video frames. In this case, the source identifier 41 may indicate multiple identifiers of the source image data.

[0058] The object relationship information 40 may indicate time information together with the source identifier 41 or instead of the source identifier 41. For example, each record in the object relationship information 40 indicates the date and time of generation of the source image data used to generate that record. Furthermore, if the relationship between a certain object pair continues, the source identifier 41 may indicate a combination of the start and end points of the relationship indicated by that record. By including date and time information in the object relationship information 40, it becomes possible to determine the date and time when the inter-object relationship indicated in the object relationship information 40 was observed by referring to the object relationship information 40.

[0059] 5 and 6, each object is represented by an identifier. However, to be able to search for a relationship such as "a person sitting on a chair," further information such as the type of each object is required.

[0060] Information about each object may be included in the object-relationship information 40, or may be managed separately from the object-relationship information 40. Hereinafter, information in which an object identifier and information about the object are associated with each other will be referred to as object information.

[0061] 7 is a diagram illustrating an example of the configuration of object information. The object information 50 indicates an object identifier 51 in association with a type 52 and an area 53. In FIG. 7, a plurality of pieces of object information 50 are listed in a table format.

[0062] The object identifier 51 indicates the identifier of the object. The type 52 indicates the type of the object. The area 53 indicates the position of the image area (hereinafter referred to as the object area) representing the corresponding object in the source image data. For example, if the bounding rectangle of the object is treated as the object area, the area 53 indicates information that can identify the bounding rectangle (for example, the coordinates of the upper left corner and the lower right corner of the bounding rectangle).

[0063] Here, if the multiple source image data used to generate the object relation information 40 is time-series data such as video data, the same object may be detected from the multiple source image data. In this case, the region 53 indicates multiple pairs of a time point and an object region at that time point. The time point may be expressed as an absolute value or a relative value. In the former case, the time point is expressed, for example, by a date and time. In the latter case, the time point is expressed, for example, by a frame number.

[0064] <Acquisition of First Query Data 10: S102> The acquisition unit 2020 acquires the first query data 10 (S102). The first query data 10 is, for example, data including a query text, as described above. Alternatively, the first query data 10 may be a tuple (hereinafter referred to as a query tuple) including a subject text 12, an object text 14, and a relation text 16. For example, a query tuple "(subject, object, relation) = (person, chair, sit)" indicates that the subject text 12 is "person," the object text 14 is "chair," and the relation text 16 is "sit."

[0065] There are various methods for the acquisition unit 2020 to acquire the first query data 10. For example, the acquisition unit 2020 provides a screen for inputting the first query data 10 to the user of the search device 2000. The acquisition unit 2020 acquires the data input on this screen as the first query data 10. As another example, the acquisition unit 2020 acquires the first query data 10 by receiving the first query data 10 transmitted from another device such as a terminal used by the user (e.g., a PC or a smartphone). As another example, the acquisition unit 2020 may acquire the first query data 10 by reading the first query data 10 stored in a storage unit accessible from the search device 2000 from the storage unit.

[0066] The acquisition unit 2020 may acquire information for generating the first query data 10 and use the information to generate the first query data 10, thereby acquiring the first query data 10. For example, the acquisition unit 2020 acquires image data generated by capturing an image of an object pair and uses the image data to generate the first query data 10.

[0067] For example, image data generated by capturing an image of a person sitting on a chair represents the relationship "sitting" between the object pair of the person and the chair. Therefore, from this image data, it is possible to generate a query text such as "a person sits on a chair" and a query tuple such as (person, chair, sit).

[0068] Alternatively, for example, the acquisition unit 2020 acquires ranging data generated by sensing the object pair with a ranging sensor, and uses the ranging data to generate the first query data 10. Various sensors such as a LiDAR (Light Detection and Ranging) sensor or a depth camera can be used as the ranging sensor. The ranging data may be, for example, a distance image or point cloud data.

[0069] Alternatively, for example, the acquiring unit 2020 may acquire voice data representing a statement that indicates the relationship between the object pair, and generate the first query data 10 using the voice data.

[0070] <Generation of Second Query Data 20: S104> The generation unit 2040 generates the second query data 20 based on the first query data 10 (S104). The second query data 20 is query data that represents an inter-object relationship similar to the inter-object relationship represented by the first query data 10.

[0071] For example, as described above, the generating unit 2040 identifies similarly related texts having a similar meaning to the related text 16, and uses the similarly related texts to generate the second query data 20. In this case, for example, as described above, the second query data 20 indicates a text identical to the subject text 12 as the subject text 22, a text identical to the object text 14 as the object text 24, and a similarly related text as the related text 26.

[0072] For example, suppose the first query data 10 indicates a query text "A bird is on the sea." In this case, "bird," "sea," and "on" are the subject text 12, the object text 14, and the related text 16, respectively. The generation unit 2040 identifies a related text similar to "on" which is the related text 16.

[0073] For example, suppose the generation unit 2040 has identified "Above" as a similar related text to "Above" which is related text 16. In this case, the generation unit 2040 generates a query text "The bird is above the sea" as the second query data 20. In the generated query text, the subject text 22 is "bird" which is the same as the subject text 12, and the object text 24 is "sea" which is the same as the object text 14. On the other hand, in the generated query text, the related text 26 is different from the related text 16 and is "Above" which is a similar related text to the related text 16.

[0074] As described above, the similarity-related text is, for example, a synonym or paraphrase of the related text 16 (hereinafter, referred to as "synonyms"). There are various methods for identifying the similarity-related text based on the related text 16. For example, the generation unit 2040 is configured to have access to dictionary data that allows searching for synonyms. In this case, the generation unit 2040 uses the dictionary data to search for synonyms of the related text 16, and uses the text obtained by the search as the similarity-related text. For example, a thesaurus or the like can be used as a dictionary that allows searching for synonyms.

[0075] Alternatively, for example, the generation unit 2040 may identify similarity-related text using a language model configured to return answer text for question text. The language model may be, for example, a model classified as a large language model (LLM). However, the language model used by the generation unit 2040 may not be classified as a large language model.

[0076] When using a language model, for example, the generation unit 2040 inputs a question text such as "Please tell me synonyms for X" or "Please tell me alternative expressions for X" into the language model. Here, X is assigned the related text 16. The generation unit 2040 obtains similar related text by extracting words representing synonyms or alternative expressions for X from the answer text obtained from the language model in response to the question text.

[0077] Here, it is possible that a plurality of similar related texts are obtained for the related text 16. In this case, the generating unit 2040 may generate a plurality of second query data 20 from the first query data 10.

[0078] The generating unit 2040 may further use text having a similar meaning to the subject text 12 (hereinafter, similar subject text) or text having a similar meaning to the object text 14 (hereinafter, similar object text) to generate the second query data 20. In this case, for example, the second query data 20 indicates a similar subject text in the subject text 22, a similar object text in the object text 24, and a similar related text in the related text 26.

[0079] The similar subject text may be, for example, a synonym of the subject text 12 or a paraphrase of the subject text 12. Similarly, the similar object text may be, for example, a synonym of the object text 14 or a paraphrase of the object text 14.

[0080] The method of obtaining a similar subject text from the subject text 12 and the method of obtaining a similar object text from the object text 14 can use the above-mentioned method of obtaining a similar related text from the related text 16.

[0081] Alternatively, for example, the generation unit 2040 may request the language model to rephrase the entire first query data 10. For example, as in the above example, assume that the first query data 10 is a query text of "The bird is over the sea." In this case, the generation unit 2040 inputs a question text of "Please rephrase 'The bird is over the sea' in a different way" into the language model. As a result, the generation unit 2040 can obtain, from the language model, text in which the entire query text is rephrased in a different way. Therefore, the generation unit 2040 generates the second query data 20 using the text obtained from the language model as new query text.

[0082] In addition, in the case where a paraphrase of the entire first query data 10 is requested from the language model, the first query data 10 does not necessarily have to be a query text. For example, the generation unit 2040 may generate a sentence from the query tuples indicated in the first query data 10 and request a paraphrase of the sentence from the language model.

[0083] For example, assume that the first query data 10 indicates a query tuple (bird, sea, above). In this case, the generation unit 2040 generates a sentence "The bird is above the sea" from this query tuple. The generation unit 2040 then inputs a question text requesting a paraphrase of the generated sentence to the language model.

[0084] The generation unit 2040 may include the query tuple in the question text without converting the query tuple into a sentence. In this case, for example, the generation unit 2040 may include, in the question text, the query tuple indicated in the first query data 10 as well as how to interpret the query tuple. This allows the language model to understand the meaning of the query tuple and then rephrase the query tuple with another query tuple.

[0085] The question text including the query tuple and the interpretation method of the query tuple is, for example, a question query such as "Please replace (X, Y, Z) with another tuple. In this tuple, the first element is the subject, the second element is the object, and the third element represents the relationship between the subject and the object." Note that the query tuple shown in the first query data 10 is assigned to (X, Y, Z). By inputting the question text to the language model, the generation unit 2040 can obtain from the language model a tuple in which at least one of the subject, object, and relationship is replaced with a synonym or another expression.

[0086] When using a language model, it is preferable to prepare a question template for the language model in advance. The question template is, for example, a sentence containing characters to be replaced, such as X or (X, Y, Z), as described above. Specifically, it may be "Please tell me a synonym for X." Note that a question template such as "Please rephrase X in a different way" may be used to rephrase a part of a query, such as the related text 16, or may be used to rephrase the entire query text. The question template is, for example, stored in advance in any storage unit accessible from the search device 2000.

[0087] <<Regarding the Case Where Multiple Second Query Data 20 Are Generated>> As described above, the generation unit 2040 may generate multiple second query data 20. In this case, an upper limit may be set on the number of second query data 20 to be generated. The upper limit on the number of second query data 20 may be determined in advance or may be specified by the user. Note that when a language model is used, the generation unit 2040 may specify the desired number of synonyms, etc. in the question text, thereby obtaining a desired number of synonyms, etc. from the language model.

[0088] The generating unit 2040 may select some of the obtained synonyms, etc., according to some criteria. For example, the generating unit 2040 selects, as the similar related text, synonyms, etc., from among the synonyms, etc., of the related text 16 in descending order of similarity to the related text 16. The same applies to the similar subject text and the similar object text.

[0089] Here, preferentially using synonyms and the like that have a lower similarity to the relational text 16 has the advantage of increasing the probability of detecting object relation information 40 that matches the first query data 10 for the following reason. That is, when the similarity between the relational text 16 and the relational text 26, which is a synonym or the like of the relational text 16, is low, the relational features obtained from the first query data 10 and the second query data 20 are located at positions farther apart in the feature space. This means that the range in the feature space that can be covered by the relational features of the first query data 10 and the second query data 20 becomes wider. As a result, the similarity between the relational features indicated in the object relation information 40 and the relational features obtained from the first query data 10 or the second query data 20 is likely to be high, making it easier to detect object relation information 40 that matches the first query data 10.

[0090] However, the generating unit 2040 may be configured to select, as the similar related text, synonyms, etc., from among the multiple synonyms, etc. of the related text 16 in descending order of similarity with the related text 16 .

[0091] There are various methods for grasping the similarity between the related text 16 and the synonyms, etc. For example, the generation unit 2040 calculates the feature amounts of each of the related text 16 and the synonyms, etc., and treats the similarity of the calculated feature amounts as the similarity between the related text 16 and the synonyms, etc. For calculating the feature amounts, for example, a method using a machine learning model such as a neural network can be adopted. This model is trained in advance to output the feature amounts of a word or phrase (hereinafter, "word, etc.") in response to the input of the word, etc.

[0092] Various methods can be used to calculate the similarity between feature quantities. For example, the generation unit 2040 uses the distance between feature quantities as an index representing the similarity between feature quantities. Here, it can be said that the shorter the distance between feature quantities, the higher the similarity between feature quantities. Therefore, for example, the generation unit 2040 calculates, as the similarity between feature quantities, a value that increases as the distance between feature quantities decreases, such as the inverse of the distance between feature quantities.

[0093] Alternatively, for example, the generation unit 2040 may calculate the cosine similarity between the feature quantities as the similarity between the feature quantities. Alternatively, for example, the similarity between the feature quantities may be calculated using a machine learning model such as a neural network. This model is trained in advance to output the similarity between the two input feature quantities in response to input of two feature quantities.

[0094] Alternatively, for example, when synonyms of the related text 16 are identified using dictionary data, the generation unit 2040 may use the dictionary data to identify the similarity between the related text 16 and the synonyms. For example, assume that the dictionary data indicates multiple synonyms for a certain word, etc., in order of similarity to the word, etc. In this case, the generation unit 2040 can represent the similarity between the related text 16 and each of the multiple synonyms of the related text 16 by the ranking of the multiple synonyms in the dictionary data.

[0095] The method of selecting some synonyms from the plurality of synonyms in the relation text 16 is not limited to the method of selecting based on the similarity between the relation text 16 and the synonyms, as described above. For example, the generation unit 2040 selects synonyms based on text data (hereinafter, training text) used to train a machine learning model (hereinafter, relationship extraction model) used to generate the object relationship information 40. Details of the relationship extraction model will be described later.

[0096] For example, suppose that the number of occurrences of each word, etc. is recorded for a set of training texts used in training the relationship extraction model. This record is called occurrence count data. The generation unit 2040 uses the occurrence count data to identify the number of occurrences in training the relationship extraction model for each synonym, etc. identified for the related text 16. The generation unit 2040 then selects as similar related texts synonyms, etc. in descending order of appearance count in training the relationship extraction model (synonyms, etc. with the highest number of appearances indicated in the occurrence count data).

[0097] Here, when the query data includes text that appears frequently in training of the relation extraction model, there is a high probability that the relation feature obtained from the query data will be similar to the relation feature indicated in the object relation information 40 generated using the relation extraction model. For this reason, by including text that appears frequently in training of the relation extraction model in the second query data 20, the probability that the object relation information 40 can be detected using the second query data 20 increases.

[0098] <Calculation of Relation Feature Amount> The calculation unit 2060 calculates the relation feature amount for each of the first query data 10 and the second query data 20 (S108). To calculate the relation feature amount, for example, a trained model (hereinafter, referred to as a relation feature amount calculation model) is used. For example, a machine learning model such as a neural network can be used as the relation feature amount calculation model.

[0099] The calculation unit 2060 inputs the first query data 10 to the relation feature calculation model. The calculation unit 2060 acquires the relation feature output from the relation feature calculation model as the relation feature of the first query data 10. Similarly, the calculation unit 2060 inputs the second query data 20 to the relation feature calculation model. The calculation unit 2060 acquires the relation feature output from the relation feature calculation model as the relation feature of the second query data 20.

[0100] For example, a feature calculation model included in a vision-and-language model can be used as the relation feature calculation model. The vision-and-language model is a model that can execute processing that handles both visual data and linguistic data. Examples of visual data include image data and distance measurement data. Examples of linguistic data include text data.

[0101] An example of a process that handles both visual and linguistic data is a process that compares the content of image data with the content of text data, such as determining whether the relationships between objects contained in the image data match the relationships between objects expressed in the text data.

[0102] 8 is a diagram illustrating the configuration of a vision-and-language model. The vision-and-language model 60 acquires visual data 70 and linguistic data 80. The vision-and-language model 60 then determines whether the content of the visual data 70 matches the content of the linguistic data 80.

[0103] In order to compare the content of visual data 70 with the content of linguistic data 80, the vision-and-language model 60 calculates mutually comparable features from the visual data 70 and the linguistic data 80. To this end, the vision-and-language model 60 has a first feature calculation model 62 and a second feature calculation model 64. The first feature calculation model 62 calculates the feature of the visual data 70. The second feature calculation model 64 calculates the feature of the linguistic data 80.

[0104] The vision-and-language model 60 further includes a determination model 66. The determination model 66 compares the feature values ​​obtained from the visual data 70 with the feature values ​​obtained from the linguistic data 80 to determine whether the content of the visual data 70 matches the content of the linguistic data 80.

[0105] For example, suppose that the vision-and-language model 60 determines whether the relationship between objects represented by the visual data 70 matches the relationship between objects represented by the linguistic data 80. In this case, by training the vision-and-language model 60, the first feature calculation model 62 is trained to calculate the feature of the relationship between the objects represented by the visual data 70. Similarly, the second feature calculation model 64 is trained to calculate the feature of the relationship between the objects represented by the linguistic data 80.

[0106] For this reason, the relation feature calculation model included in the calculation unit 2060 can use the trained second feature calculation model 64 that handles the same type of linguistic data as the first query data 10.

[0107] For example, suppose that the first query data 10 expresses a relationship between objects using query text. In this case, the second feature calculation model 64 of the trained vision-and-language model 60, in which text data expressing the relationship between the objects in sentences is treated as linguistic data 80, can be used as the relationship feature calculation model to be included in the calculation unit 2060.

[0108] Alternatively, for example, the first query data 10 may represent a relationship between objects using a query tuple. In this case, the second feature calculation model 64 of the trained vision-and-language model 60, in which the text tuples representing the relationship between the objects are treated as linguistic data 80, may be used as the relationship feature calculation model to be included in the calculation unit 2060.

[0109] <<Regarding Training of the Relation Extraction Model>> Here, the above-described vision-and-language model 60 can be used to train the relationship extraction model used to generate the object relationship information 40. FIG. 9 is a diagram illustrating an example of the relationship extraction model. In FIG. 9, image regions (image regions 110 and 120) of an object pair for which the object relationship information 40 is to be generated are input to the relationship extraction model 100. In response to the input of these image regions, the relationship extraction model 100 is configured to output a relationship feature to be associated with the object pair. Note that the relationship extraction model 100 may also be configured to receive the entire source image data in addition to the image regions 110 and 120.

[0110] The training data used for training the relationship extraction model 100 indicates image regions of object pairs in correspondence with ground truth relationship features. The ground truth relationship features are generated from training text that represents the inter-object relationships in the object pairs. Specifically, the training text that represents the inter-object relationships in the object pairs can be input to the second feature calculation model 64 in FIG. 8 to obtain the ground truth relationship features.

[0111] <Calculation of Similarity: S112> The detection unit 2080 calculates a first similarity and a second similarity for the object relation information 40 (S112). Specifically, the detection unit 2080 calculates, as the first similarity, the similarity between the relation feature indicated in the object relation information 40 and the relation feature calculated for the first query data 10. The detection unit 2080 also calculates, as the second similarity, the similarity between the relation feature indicated in the object relation information 40 and the relation feature calculated for the second query data 20. Here, the method of calculating the similarity between the features is as described above.

[0112] <Determining Whether or Not the Object Relation Information 40 Matches the First Query Data 10: S112> The detection unit 2080 determines whether or not the object relation information 40 matches the first query data 10 based on the first similarity and the second similarity calculated for the object relation information 40 (S112). For example, the detection unit 2080 calculates a statistical value of the first similarity and the second similarity (hereinafter, similarity statistical value) and compares the similarity statistical value with a threshold value to determine whether or not the object relation information 40 matches the first query data 10.

[0113] For example, if the similarity statistical value calculated for the object relation information 40 is equal to or greater than a threshold, the detection unit 2080 determines that the object relation information 40 matches the first query data 10 (S112: YES). On the other hand, if the similarity statistical value calculated for the object relation information 40 is less than the threshold, the detection unit 2080 determines that the object relation information 40 does not match the first query data 10 (S112: NO).

[0114] Various statistical values ​​can be used as the similarity statistics. For example, the maximum value is used as the similarity statistics. In this case, even if the first similarity is less than the threshold, if the second similarity is equal to or greater than the threshold, it is determined that the object relation information 40 and the first query data 10 match. Therefore, if any one of the paraphrases obtained from the first query data 10 represents the inter-object relationship indicated in the object relation information 40, the object relation information 40 can be detected as the object relation information 40 that matches the first query data 10.

[0115] Alternatively, for example, a simple average or a weighted average can be used as the similarity statistical value. Here, in a case where a weighted average is used as the similarity statistical value, it is assumed that a plurality of second query data 20 are generated, and therefore a plurality of second similarities are calculated. In this case, a weight is set for each of the plurality of second similarities. The weights set for the plurality of second similarities may be the same or different from each other.

[0116] There are various methods for determining the weights to be set for the multiple second similarities. For example, the detection unit 2080 calculates the similarity between the related text 16 and each related text 26 in the second query data 20. The detection unit 2080 sets a higher weight for the related text 26 that has a lower similarity to the related text 16. Note that if the generation unit 2040 has already calculated the similarity between the related text 16 and a synonym or the like of the related text 16, the detection unit 2080 can reuse this calculation result.

[0117] For example, the inverse of the similarity between the related text 26 and the related text 16 is set as the weight of the related text 26. Then, the detection unit 2080 determines the weight set for the related text 26 as the weight of the second similarity calculated for the second query data 20 including the related text 26. The weight determined here is a relative weight between the second similarities.

[0118] The weight set for the first similarity is arbitrary. For example, a weight α is assigned to the first similarity, and a total weight (1-α) is assigned to the multiple second similarities. α is a predetermined real number satisfying 0<α<1. The weight of each second similarity is determined based on the relative weights between the second similarities and the total weight (1-α).

[0119] When multiple second similarities are calculated, the detection unit 2080 may calculate the similarity statistics using only some of the multiple second similarities. For example, the detection unit 2080 uses the top N second similarities (N is an arbitrary integer) in descending order of magnitude among the multiple second similarities to calculate the similarity statistics. For example, when N is 2, the detection unit 2080 calculates the similarity statistics using the largest second similarity, the second largest second similarity, and the first similarity.

[0120] The method for determining whether the first query data 10 matches the object-relation information 40 based on the first similarity and the second similarity is not limited to the method using similarity statistics. For example, the detection unit 2080 may make the determination using a machine learning model (hereinafter, a determination model) such as a neural network. For example, the determination model is trained in advance to output a determination result as to whether the first query data 10 matches the object-relation information 40 in response to input of the first similarity and one or more second similarities. By using the machine learning model, the importance levels and determination thresholds of the first similarity and the second similarity can be automatically determined by machine learning.

[0121] <Cases in which the second query data 20 is not generated> When a specific condition is satisfied, the search device 2000 may detect object relation information 40 that matches the first query data 10 without generating the second query data 20. In this case, the search device 2000 determines whether or not it is necessary to generate the second query data 20. Assume that it is determined that it is necessary to generate the second query data 20. In this case, as described above, the search device 2000 generates the second query data 20 from the first query data 10, and detects the object relation information 40 that matches the first query data 10 using the first query data 10 and the second query data 20.

[0122] On the other hand, suppose that it is determined that there is no need to generate the second query data 20. In this case, the search device 2000 does not generate the second query data 20 from the first query data 10. The search device 2000 calculates a first similarity between each piece of object relation information 40 and the first query data 10. If the first similarity calculated for the object relation information 40 is equal to or greater than a threshold, the search device 2000 detects the object relation information 40 as object relation information 40 that matches the first query data 10.

[0123] Whether or not it is necessary to generate the second query data 20 is determined, for example, based on whether or not a word or the like that frequently appeared in the training of the relationship extraction model is included in the first query data 10. For example, the search device 2000 uses the above-mentioned occurrence count data to identify the number of occurrences of each word or the like included in the first query data 10 in the training of the relationship extraction model.

[0124] If the first query data 10 includes a word or the like whose number of appearances in the training of the relationship extraction model is equal to or greater than a threshold, the search device 2000 determines that there is no need to generate the second query data 20. On the other hand, if the first query data 10 does not include a word or the like whose number of appearances in the training of the relationship extraction model is equal to or greater than a threshold, the search device 2000 determines that there is a need to generate the second query data 20.

[0125] When the first query data 10 contains words and the like that appear frequently in the training of the relationship extraction model, the words and the like contained in the first query data 10 are sufficiently taken into consideration in the training of the relationship extraction model. Therefore, the relationship feature of the object relationship information 40 that matches the first query data 10 is considered to be sufficiently similar to the relationship feature calculated from the first query data 10. Therefore, when the first query data 10 contains words and the like that appear frequently in the training of the relationship extraction model, the object relationship information 40 that matches the first query data 10 can be detected without using the second query data 20. Furthermore, in this case, the process of generating the second query data 20 is not required, and therefore the time required to detect the object relationship information 40 that matches the first query data 10 can be reduced.

[0126] <<Regarding the object relationship information 40 to be detected>> The detection of the object relationship information 40 that matches the first query data 10 may be performed on all of the object relationship information 40 stored in the object relationship information storage unit 90, or on only a portion of the object relationship information 40.

[0127] The object relation information 40 to be detected is limited using, for example, time information. In this case, the first query data 10 indicates a time period condition in addition to a query text or a query tuple. The time period condition can be expressed as a pair of a start point and an end point of the time period, such as "June 2023 to September 2023." Furthermore, the object relation information 40 indicates the time (such as the date and time of generation of the source image data) when the inter-object relation represented by the object relation information 40 was observed.

[0128] The detection unit 2080 detects only object relation information 40 that satisfies the period condition indicated in the first query data 10, among the object relation information 40 stored in the object relation information storage unit 90. For example, in S108 of the flowchart in FIG. 4 , object relation information i is selected from the object relation information 40 that satisfies the period condition indicated in the first query data 10.

[0129] <Output of Search Results> The search device 2000 outputs information based on the results of detection by the detection unit 2080. Hereinafter, the information output based on the search results will be referred to as output information.

[0130] The content of the output information varies. For example, the detection unit 2080 includes, in the output information, information indicated in the object relation information 40 determined to match the first query data 10. Specifically, the output information indicates information on the subjects, objects, and relationships between the subjects and objects indicated in the object relation information 40 determined to match the first query data 10. By obtaining this output information, the user of the retrieval device 2000 can learn the subjects, objects, and relationships between the subjects and objects for inter-object relations that match the inter-object relations represented by the first query data 10, among the inter-object relations obtained from the source image data. Note that the retrieval device 2000 preferably uses object information 50 in addition to the object relation information 40 and includes more specific information, such as the type of object, for each of the subjects and objects in the output information.

[0131] The detection unit 2080 may output source image data used to generate the object relation information 40 in addition to or instead of the content of the object relation information 40 determined to match the first query data 10. For example, as shown in FIG. 6 , the object relation information 40 indicates a source identifier 41. In this case, for the object relation information 40 determined to match the first query data 10, the detection unit 2080 includes the source image data identified by the source identifier 41 of the object relation information 40 in the output information.

[0132] For example, suppose the first query data 10 is text data representing the relationship "a person sits on a chair." In this case, a user of the search device 2000 can obtain source image data including a scene in which a person sits on a chair by performing a search using the first query data 10. Furthermore, if video data is used to generate the object relation information 40, the scene represented by the first query data 10 can be detected from the video data by performing a search using the first query data 10 to obtain source image data.

[0133] The output information may include image data that includes only the object of interest, instead of the entire source image data. For example, suppose the first query data 10 is text data that reads, "A person sits on a chair." Furthermore, suppose a certain source image data includes both a scene of a person sitting on a chair and a scene of a dog holding a ball. In this case, the output information may include image data that represents only the area of ​​the person and the chair from the source image data.

[0134] The output information may be output in various ways. For example, the detection unit 2080 stores the output information in an arbitrary storage unit. Alternatively, for example, the detection unit 2080 displays the output information on a display device. Alternatively, for example, the detection unit 2080 transmits the output information to another device. For example, when the first query data 10 is transmitted from another device, the detection unit 2080 transmits the output information to the device that transmitted the first query data 10.

[0135] <Regarding Queries Indicating Relationships Between Two or More Objects> Here, the search device 2000 may handle search queries that express relationships between two or more object pairs. For example, text data such as "a person is sitting in a chair and a dog is holding a ball" may be used as a search query.

[0136] In this way, when one search query can represent inter-object relations between two or more object pairs, the acquisition unit 2020 is configured to acquire the search query and generate a plurality of first query data 10 from the acquired search query. Furthermore, the search device 2000 detects object relation information 40 that matches each of the generated plurality of first query data 10. Then, the search device 2000 identifies source image data that matches the search query, for example, using the object relation information 40 detected for each of the first query data 10.

[0137] For example, suppose a search query specifies that all of the relationships between multiple objects must be satisfied (i.e., the AND of the relationships between multiple objects). An example of such a search query is "a person sitting in a chair and a dog holding a ball." In this case, source image data that includes both an image of a person sitting in a chair and an image of a dog holding a ball is detected as source image data that matches the search query.

[0138] Specifically, the search device 2000 detects one or more pieces of object relation information 40 that match the first query data 10 representing "a person sitting on a chair." The search device 2000 also detects one or more pieces of object relation information 40 that match the first query data 10 representing "a dog holding a ball in its mouth." The search device 2000 then detects object relation information 40 that matches "a person sitting on a chair" and object relation information 40 that matches "a dog holding a ball in its mouth" that are generated from the same source image data. The source image data detected as a result of this is the one that was used to generate the object relation information 40 that represents "a person sitting on a chair" and the one that was used to generate the object relation information 40 that represents "a dog holding a ball in its mouth." In other words, this source image data includes both an image of a person sitting on a chair and an image of a dog holding a ball.

[0139] A search query may specify a condition that one or more of multiple inter-object relations must be satisfied (i.e., OR of multiple inter-object relations). An example of such a search query is "a person sitting in a chair or a dog holding a ball." In this case, source image data that includes either or both of an image of a person sitting in a chair and an image of a dog holding a ball is detected as source image data that matches the search query.

[0140] Here, if the search query indicates search conditions using sentences, the acquisition unit 2020 preferably includes a parser that interprets the sentences indicated in the search query. If the search query includes multiple sentences, the parser divides the search query into sentences. The parser further interprets each sentence to identify the subjects, objects, and relationships between the subjects and objects indicated in each sentence. As a result, multiple pieces of first query data 10 are generated from the search query.

[0141] Furthermore, when multiple sentences in a search query are connected by words representing logical operators such as "and" or "or," the parser interprets the logical relationship between the sentences to identify search conditions. For example, assume that the search query indicates the sentence "A person is sitting on a chair, and a dog is holding a ball in its mouth." In this case, the parser divides the search query into two sentences, "A person is sitting on a chair" and "A dog is holding a ball," and identifies that these sentences are connected by "and." The parser then interprets each of the two sentences to generate first query data 10, "(subject, object, relationship) = (person, chair, sitting)," and second query data 10, "(subject, object, relationship) = (dog, ball, holding)." Based on the logical relationship between the two sentences, the parser then identifies that the source image data to be detected is source image data that includes both of the inter-object relationships represented by the two first query data 10.

[0142] If the logical relationship between multiple sentences is not specified in the search query, the search device 2000 may assume that a predetermined logical relationship (for example, AND) exists between these sentences.

[0143] <Example Use Case of Search Device 2000> There are various possible use cases for the search device 2000. For example, the search device 2000 can be used to search for a desired event or collect information about an accident from video footage captured by a drive recorder. In this case, the object-relation information 40 is generated using the video footage captured by the drive recorder. The source image data is each video frame that constitutes the video footage captured by the drive recorder.

[0144] In addition, for example, the search device 2000 can be used to monitor a person using video. Specifically, the search device 2000 can be used to detect a desired behavior of a person being monitored from video obtained by capturing the person being monitored (hereinafter referred to as monitoring video). In this case, the object-relationship information 40 is generated using the monitoring video. The source image data is each video frame that constitutes the monitoring video.

[0145] Alternatively, for example, the search device 2000 can be used to detect criminal activity from surveillance video. In this case, the surveillance video is used to generate the object-related information 40. The source image data is each video frame that constitutes the surveillance video.

[0146] In addition, for example, the search device 2000 can be used to analyze customer purchasing behavior. In this case, the object-relation information 40 is also generated using surveillance video.

[0147] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0148] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0149] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A search device comprising: an acquisition means for acquiring first query data in which a relationship between objects is expressed using relation text representing the relationship between the objects; a generation means for generating second query data in which a relationship similar to the relationship expressed by the first query data is expressed; a calculation means for calculating, from each of the first query data and the second query data, a relation feature amount representing a characteristic of the relationship between the objects; and a detection means for detecting, from a plurality of object relation information pieces in which the relation feature amount is indicated for the relationship between objects included in image data, the object relation information pieces that match the first query data, using the relation feature amount calculated for each of the first query data and the second query data. (Supplementary Note 2) The search device according to Supplementary Note 1, wherein the generation means identifies a synonym or paraphrase of the relation text included in the first query data, and generates, as the second query data, data in which the relation text in the first query data is replaced with the synonym or the paraphrase of the relation text. (Supplementary Note 3) The search device according to Supplementary Note 2, wherein the generation means inputs question text, asking for synonyms or paraphrases of the related text, to a language model trained to output answer text in response to input question text, and extracts synonyms or paraphrases of the related text from the answer text. (Supplementary Note 4) The search device according to Supplementary Note 2 or 3, wherein the generation means generates the second query data by preferentially using, among a plurality of synonyms or paraphrases of the related text, synonyms or paraphrases that have a lower similarity to the related text. (Supplementary Note 5) The search device according to Supplementary Note 2 or 3, wherein the relationship feature indicated in the object relationship information is calculated using a relationship extraction model trained to output the relationship feature for two objects in response to input of image regions of the two objects, and the generation means generates the second query data by preferentially using, among a plurality of synonyms or paraphrases of the related text, synonyms or paraphrases that appeared less frequently in training of the relationship extraction model.(Supplementary Note 6) The search device according to Supplementary Note 5, wherein the generating means does not generate the second query data when the first query data includes text whose appearance count in training of the relation extraction model is equal to or greater than a threshold, the calculating means calculates the relation feature of the second query data when the second query data is generated, and the detecting means detects the object relation information matching the first query data based on a similarity between the relation feature indicated in the object relation information and the relation feature calculated for the first query data when the second query data is not generated. (Supplementary Note 7) The search device according to Supplementary Note 1, wherein the generating means inputs question text asking for a paraphrase of the first query data to a language model trained to output an answer text in response to input question text, and uses the paraphrase indicated in the answer text as the second query data. (Supplementary Note 8) The search device according to Supplementary Note 1, wherein the detection means calculates a first similarity that is a similarity between the relation feature indicated in the object relation information and the relation feature calculated for the first query data, and a second similarity that is a similarity between the relation feature indicated in the object relation information and the relation feature calculated for the second query data, and determines whether the first query data and the object relation information match based on statistical values ​​of the first similarity and the second similarity. (Supplementary Note 9) The search device according to Supplementary Note 7, wherein the generation means generates a plurality of the second query data, and the detection means calculates the second similarity for each of the plurality of second query data, and calculates a weighted average of the first similarity and the plurality of second similarities as the statistical value of the first similarity and the second similarity, and a weight assigned to the second similarity calculated for the second query data is determined based on a similarity between a text included in the second query data and a text included in the first query data.(Supplementary Note 10) A search method executed by a computer, comprising: an acquisition step of acquiring first query data in which a relationship between objects is expressed using relation text that expresses the relationship between the objects; a generation step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation step of calculating, from each of the first query data and the second query data, a relation feature amount that expresses a characteristic of the relationship between the objects; and a detection step of detecting, from a plurality of object relation information in which the relation feature amount is indicated for the relationship between objects included in image data, the object relation information that matches the first query data, using the relation feature amount calculated for each of the first query data and the second query data. (Supplementary Note 11) A program causing a computer to execute the following steps: an acquiring step of acquiring first query data in which a relationship between objects is expressed using a relation text that expresses the relationship between the objects; a generating step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculating step of calculating, from each of the first query data and the second query data, a relation feature amount that expresses a characteristic of the relationship between the objects; and a detecting step of detecting, from a plurality of object relation information in which the relation feature amount is indicated for the relationship between objects included in image data, the object relation information that matches the first query data, using the relation feature amount calculated for each of the first query data and the second query data.

[0150] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 9 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 10 and 11 in the same dependency relationship as Supplementary Notes 2 to 9. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods.

[0151] This application claims priority based on Japanese Patent Application No. 2024-012265, filed January 30, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0152] 10 First query data 12 Subject text 14 Object text 16 Relation text 20 Second query data 22 Subject text 24 Object text 26 Relation text 40 Object relation information 41 Source identifier 42 Object pair 43 Relation feature 44 Subject identifier 45 Object identifier 50 Object information 51 Object identifier 52 Type 53 Region 60 Vision-and-language model 62 First feature calculation model 64 Second feature calculation model 66 Decision model 70 Visual data 80 Linguistic data 90 Object relation information storage unit 100 Relation extraction model 110 Image region 120 Image region 120 Image region 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Search device 2020 Acquisition unit 2040 Generation unit 2060 Calculation unit 2080 Detection unit

Claims

1. A search device comprising: an acquisition means for acquiring first query data that expresses a relationship between objects using relational text that expresses the relationship between the objects; a generation means for generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation means for calculating relation features that express characteristics of the relationship between the objects from each of the first query data and the second query data; and a detection means for detecting object relation information that matches the first query data from a plurality of object relation information pieces that indicate the relation features regarding the relationship between objects included in image data, using the relation features calculated for each of the first query data and the second query data.

2. The search device described in claim 1, wherein the generation means identifies synonyms or paraphrases of the related text contained in the first query data, and generates data in which the related text in the first query data is replaced with synonyms or paraphrases of the related text as the second query data.

3. The search device described in claim 2, wherein the generation means inputs a question text asking for synonyms or paraphrases of the related text into a language model trained to output answer text in response to input question text, and extracts synonyms or paraphrases of the related text from the answer text.

4. A search device as described in claim 2 or 3, wherein the generation means generates the second query data by preferentially using, from among a plurality of synonyms or paraphrases of the related text, synonyms or paraphrases that have a lower degree of similarity to the related text.

5. The search device described in claim 2 or 3, wherein the relation feature indicated in the object relation information is calculated using a relation extraction model trained to output the relation feature for the two objects in response to input of image regions of the two objects, and the generation means generates the second query data by preferentially using, from among multiple synonyms or paraphrases of the relation text, synonyms or paraphrases that appear less frequently in training of the relation extraction model.

6. The search device described in claim 1, wherein the generation means inputs the question text, which asks for a paraphrase of the first query data, to a language model trained to output an answer text in response to input of the question text, and uses the paraphrase indicated in the answer text as the second query data.

7. The search device described in claim 1, wherein the detection means calculates a first similarity which is the similarity between the relation feature indicated in the object relation information and the relation feature calculated for the first query data, and a second similarity which is the similarity between the relation feature indicated in the object relation information and the relation feature calculated for the second query data, and determines whether the first query data and the object relation information match based on statistical values of the first similarity and the second similarity.

8. The search device described in claim 7, wherein the generating means generates a plurality of the second query data, the detecting means calculates the second similarity for each of the plurality of the second query data, calculates a weighted average of the first similarity and the plurality of second similarities as a statistical value of the first similarity and the second similarity, and a weight assigned to the second similarity calculated for the second query data is determined based on the similarity between text included in the second query data and text included in the first query data.

9. A search method executed by a computer, comprising: an acquisition step of acquiring first query data that expresses a relationship between objects using relation text that expresses the relationship between the objects; a generation step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation step of calculating relation features that express characteristics of the relationship between the objects from each of the first query data and the second query data; and a detection step of detecting object relation information that matches the first query data from a plurality of object relation information that indicates the relation features for the relationship between objects included in image data, using the relation features calculated for each of the first query data and the second query data.

10. A program that causes a computer to execute the following steps: an acquisition step of acquiring first query data that expresses a relationship between objects using relation text that expresses the relationship between the objects; a generation step of generating second query data that expresses a relationship similar to the relationship expressed by the first query data; a calculation step of calculating relation features that express characteristics of the relationship between the objects from each of the first query data and the second query data; and a detection step of detecting object relation information that matches the first query data from a plurality of object relation information pieces that indicate the relation features regarding the relationships between objects included in image data, using the relation features calculated for each of the first query data and the second query data.

Images