Search device, relationship information generation device, search method, relationship information generation method, and program

Abstract

This search device acquires, for a pair of objects, query data representing the relationship between the objects. The search device calculates, from the query data, a relationship feature quantity that represents a feature of the relationship between the objects. The search device detects a relationship feature quantity that matches the calculated relationship feature quantity from object relationship information in which a relationship feature quantity is associated with each of a plurality of pairs of objects.

Description

Search device, relationship information generation device, search method, relationship information generation method, and program

[0001] The present disclosure relates to a search device, a relationship information generation device, a search method, a relationship information generation method, and a program.

[0002] Technologies for searching information about images have been developed. For example, Patent Literature 1 discloses a technology for managing a plurality of images by associating each image with a feature amount related to the posture of a person included in the image. The search device in Patent Literature 1 calculates a feature amount related to the posture of a person included in a query image, and searches an image database for images associated with a feature amount that matches the calculated feature amount. This makes it possible to search the image database for images containing a person in a desired posture.

[0003] Japanese Patent Application Laid-Open No. 2019-091138

[0004] The technology of Patent Document 1 does not anticipate retrieval of information based on anything other than the posture of a person. The present disclosure has been made in light of this problem, and one of its purposes is to provide a new technology for retrieving information obtained from an image.

[0005] A search device according to the present disclosure includes an acquisition unit that acquires query data representing a relationship between a pair of objects, a calculation unit that calculates a relation feature representing a characteristic of the relationship between the objects from the query data, and a detection unit that detects a relation feature that matches the calculated relation feature from object relation information in which the relation feature is associated with each of a plurality of pairs of objects. The object relation information is generated using one or more image data.

[0006] The relationship information generation device disclosed herein includes an acquisition means for acquiring source image data, a detection means for detecting one or more pairs of objects from the source image data, a calculation means for calculating a relationship feature that represents a characteristic of the relationship between the objects for the pair, and a generation means for generating object relationship information that shows the pair in correspondence with the relationship feature calculated for the pair.

[0007] The search method of the present disclosure includes an acquisition step of acquiring query data representing a relationship between a pair of objects, a calculation step of calculating a relation feature representing a characteristic of the relationship between the objects from the query data, and a detection step of detecting a relation feature that matches the calculated relation feature from object relation information in which the relation feature is associated with each of a plurality of pairs of objects. The object relation information is generated using one or more image data.

[0008] The relationship information generation method disclosed herein includes an acquisition step of acquiring source image data, a detection step of detecting one or more pairs of objects from the source image data, a calculation step of calculating, for the pairs, relationship features that represent characteristics of the relationship between the objects, and a generation step of generating object relationship information that shows the pairs in correspondence with the relationship features calculated for the pairs.

[0009] A first program according to the present disclosure causes a computer to execute the following steps: acquiring query data representing a relationship between a pair of objects; calculating, from the query data, a relation feature representing a characteristic of the relationship between the objects; and detecting, from object relation information in which the relation feature is associated with each of a plurality of pairs of objects, a relation feature that matches the calculated relation feature. The object relation information is generated using one or more image data.

[0010] A second program of the present disclosure causes a computer to execute an acquisition step of acquiring source image data, a detection step of detecting one or more pairs of objects from the source image data, a calculation step of calculating, for the pairs, relation features that represent characteristics of the relationship between the objects, and a generation step of generating object relation information that shows the pairs in correspondence with the relation features calculated for the pairs.

[0011] According to the present disclosure, new techniques are provided for retrieving information from images.

[0012] FIG. 1 is a diagram illustrating an example of an outline of the operation of a search device. FIG. 2 is a block diagram illustrating an example of a functional configuration of a search device. FIG. 3 is a block diagram illustrating an example of a hardware configuration of a computer that realizes the search device. FIG. 4 is a flowchart illustrating an example of a flow of processing executed by the search device. FIG. 5 is a diagram illustrating an example of a configuration of object relation information. FIG. 6 is a diagram illustrating an example of a configuration of object relation information in which an identifier of source image data is indicated. FIG. 7 is a diagram illustrating an example of a configuration of object information. FIG. 8 is a diagram illustrating an example of a configuration of a vision-and-language model. FIG. 9 is a diagram illustrating an example of an outline of the operation of a relationship information generation device. FIG. 10 is a diagram illustrating an example of a functional configuration of a relationship information generation device. FIG. 11 is a flowchart illustrating an example of a flow of processing executed by the relationship information generation device. FIG. 12 is a diagram conceptually illustrating processing performed by a calculation unit. FIG. 13 is a diagram illustrating an example of an outline of the operation of a training device. FIG. 14 is a diagram illustrating an example of a functional configuration of a training device. FIG. 15 is a flowchart illustrating an example of a flow of processing executed by the training device.

[0013] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0014] <Overview> Fig. 1 is a diagram illustrating an example of an overview of the operation of the search device 2000. Here, Fig. 1 is a diagram for facilitating understanding of the overview of the search device 2000, and the operation of the search device 2000 is not limited to the operation shown in Fig. 1.

[0015] The search device 2000 is used to search for desired information from object relation information 20. The object relation information 20 indicates a relation feature for each of a plurality of pairs of objects. The relation feature of an object pair is a feature that represents a characteristic of the relationship between two objects. One of the two objects is a subject and the other is an object.

[0016] A relationship between a subject and an object represents an action performed by the subject toward the object. For example, in a situation where "a person is lifting a ball," the subject, the person, performs the action of "lifting" toward the object, the ball. Therefore, there is a relationship of "lifting" between the two objects, the person and the ball. Therefore, object relationship information 20 generated for the situation where "a person is lifting a ball" shows a pair of objects, the person and the ball, in association with a relationship feature that represents the relationship of "lifting." Hereinafter, a pair of objects will also be referred to as an "object pair."

[0017] The object-related information 20 is generated, for example, by analyzing one or more source image data, which may be still image data generated by a still camera or video frames obtained from video data generated by a video camera.

[0018] For example, suppose source image data includes a person sitting on a chair and a dog holding a ball in its mouth. This source image data indicates the situation "the person sitting on the chair" and the situation "the dog holding the ball in its mouth." In this case, the object relation information 20 generated from this source image data indicates a relation feature representing the relationship "sitting" in association with the object pair of the person and the chair. The object relation information 20 also indicates a relation feature representing the relationship "holding" in association with the object pair of the dog and the ball.

[0019] Here, the relation feature may include information about the subject, information about the object, or both. When the relation feature includes information about the subject, the relation feature generated for the situation "a person is sitting on a chair" represents the information that "a person is sitting on something." When the relation feature includes information about the object, the relation feature generated for the situation "a person is sitting on a chair" represents the information that "something is sitting on the chair." When the relation feature includes information about both the subject and the object, the relation feature generated for the situation "a person is sitting on a chair" represents the information that "a person is sitting on a chair."

[0020] A search by the search device 2000 is performed, for example, as follows. First, the search device 2000 acquires query data 10. The query data 10 is information representing one or more object pairs and the relationship between the objects in each object pair. For example, the query data 10 represents image data including one or more object pairs. In other examples, the query data 10 represents text data in which the relationship between the objects is expressed in words for one or more object pairs. The text data in which the relationship between the objects is expressed in words is, for example, data representing text such as "a person is sitting on a dog."

[0021] The search device 2000 uses the query data 10 to calculate a relation feature for each object pair indicated in the query data 10. Furthermore, the search device 2000 uses the relation feature calculated from the query data 10 to detect, from the object relation information 20, a relation between objects that matches the relation between objects indicated by the query data 10.

[0022] For example, the search device 2000 outputs information based on the detection result. The information output by the search device 2000 indicates, for example, information on object pairs associated with relational features detected from the object relation information 20, and source image data used to generate records of the object relation information 20 that indicate the relational features. Details of the information output by the search device 2000 will be described later.

[0023] <Example of Operation and Effect> The search device 2000 can search for information based on the relationship between objects. Specifically, it is possible to search for a relationship between objects that matches the relationship between objects in the object pair represented by the query data 10 from among the relationships between objects in each object pair extracted from the source image data. In this way, the search device 2000 provides a new technique for searching for information obtained from images, which is a search using the relationship between objects as a key.

[0024] Here, in the object relation information 20 searched by the search device 2000, relations between objects are expressed by relation feature amounts. The search device 2000 then performs a search using the relation feature amounts.

[0025] In this regard, it is conceivable to use labels such as "sitting" or "holding" instead of relational features as data representing relationships between objects. However, this method requires that the variations in relationships between objects that may be included in the source image data be grasped in advance and a set of available labels (hereinafter referred to as a label group) be defined. As a result, the relationships between objects that can be extracted from the source image data are limited to those included in the predetermined label group. This limits the types of relationships between objects that can be handled.

[0026] According to the search device 2000, the relationships between objects are expressed using relationship features, and therefore it is not necessary to predefine the types of relationships between objects that can be handled by the search device 2000. Therefore, the search device 2000 can handle various relationships between objects.

[0027] The search device 2000 of this embodiment will be described in more detail below.

[0028] 2 is a block diagram illustrating an example of the functional configuration of the search device 2000. For example, the search device 2000 includes an acquisition unit 2020, a calculation unit 2040, and a detection unit 2060. The acquisition unit 2020 acquires query data 10. The calculation unit 2040 uses the query data 10 to calculate a relation feature for an object pair indicated in the query data 10. The detection unit 2060 uses the relation feature calculated from the query data 10 to detect, from the object relation information 20, a relation between objects that matches the relation between objects indicated by the query data 10.

[0029] <Example of Hardware Configuration> Each functional component of the search device 2000 may be realized by hardware that realizes the respective functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the search device 2000 is realized by a combination of hardware and software will be further described.

[0030] 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the search device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the search device 2000, or may be a general-purpose computer.

[0031] For example, by installing a predetermined application on the computer 1000, each function of the search device 2000 is realized on the computer 1000. The application is configured as a program for realizing each functional component of the search device 2000. Note that any method for acquiring the program is possible. For example, the program can be acquired from a storage medium (such as a DVD disc or USB memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0032] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0033] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0034] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0035] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0036] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the search device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the search device 2000.

[0037] The search device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0038] 4 is a flowchart illustrating the flow of processing executed by the search device 2000. The acquisition unit 2020 acquires query data 10 (S102). The calculation unit 2040 calculates a relation feature for an entity pair indicated in the query data 10 (S104). The detection unit 2060 uses the relation feature calculated from the query data 10 to detect, from the entity relation information 20, an entity relation that matches the entity relation indicated by the query data 10 (S106).

[0039] <Acquisition of Query Data 10: S102> The acquisition unit 2020 acquires query data 10 (S102). For example, the query data 10 indicates image data generated by capturing images of one or more object pairs. For example, image data generated by capturing an image of a person sitting on a chair represents a relationship of "sitting" between the object pair of the person and the chair.

[0040] Alternatively, the query data 10 may represent distance measurement data generated by sensing one or more object pairs using a distance measurement sensor. Various distance measurement sensors, such as a LiDAR (Light Detection and Ranging) sensor or a depth camera, may be used. The distance measurement data may be, for example, a distance image or point cloud data.

[0041] Alternatively, for example, the query data 10 may indicate text data representing a relationship between one or more pairs of objects. For example, such text data may be text data representing a sentence such as "a person is sitting on a chair." Alternatively, for example, the text data representing a relationship between objects may be a tuple listing a subject, an object, and a relationship between them. For example, a tuple listing a subject, an object, and a relationship between them may be (person, chair, sit).

[0042] When the relationship between two or more entity pairs is expressed by text data, the query data 10 includes, for example, two or more of the above-mentioned sentences or tuples.

[0043] Alternatively, the query data 10 may be voice data representing the relationship between the objects in an object pair. The voice data may be, for example, a voice representation of the above-mentioned text data.

[0044] There are various methods for the acquisition unit 2020 to acquire the query data 10. For example, the acquisition unit 2020 provides a screen for inputting the query data 10 to the user of the search device 2000. The acquisition unit 2020 acquires the data input on this screen as the query data 10. As another example, the acquisition unit 2020 acquires the query data 10 by receiving the query data 10 transmitted from another device such as a terminal used by the user (e.g., a PC or a smartphone). As another example, the acquisition unit 2020 may acquire the query data 10 by reading the query data 10 stored in a storage unit accessible from the search device 2000 from the storage unit.

[0045] <About the Object Relation Information 20> The object relation information 20 indicates an object pair and a relation feature amount in association with each other. The relation feature amount corresponding to a certain object pair represents the characteristics of the relationship between the objects in the object pair.

[0046] 5 is a diagram illustrating an example of the configuration of the object relation information 20. In FIG. 5, the object relation information 20 includes columns named object pairs 22 and relation features 23.

[0047] The object pair 22 indicates an object pair. More specifically, the object pair 22 indicates a correspondence between a subject identifier 24 and an object identifier 25. The subject identifier 24 is an identifier of an object that is a subject. The object identifier 25 is an identifier of an object that is an object.

[0048] The relation feature 23 indicates a relation feature that indicates the characteristics of the relationship between the subject and the object for the corresponding entity pair. The relation feature is expressed, for example, as a numeric vector of a predetermined length (a vector in which each element indicates a numeric value).

[0049] When the object relation information 20 is generated using multiple source image data, it is preferable that each record of the object relation information 20 indicates the identifier of the source image data used to generate that record. Fig. 6 is a diagram illustrating an example of the configuration of the object relation information 20 in which the identifier of the source image data is indicated.

[0050] 6, the object-related information 20 further includes a column named source identifier 21. The source identifier 21 indicates an identifier of source image data. The identifier of source image data is expressed, for example, by the file name or path of a file representing the source image data. Alternatively, for example, if the source image data is a video frame constituting video data, the identifier of the source image data may be expressed by a frame number. Furthermore, if multiple video data are used to generate the object-related information 20, the identifier of the source image data is expressed, for example, by a combination of the identifier of the video data (such as the file name or path) and the frame number.

[0051] Here, when the object relation information 20 is generated using video data, the relation between a certain object pair may continue across multiple video frames. In this case, the source identifier 21 may indicate multiple identifiers of the source image data.

[0052] The object relationship information 20 may indicate time information together with the source identifier 21 or instead of the source identifier 21. For example, each record in the object relationship information 20 indicates the date and time of generation of the source image data used to generate that record. Furthermore, if the relationship between a certain object pair continues, the source identifier 21 may indicate a combination of the start and end points of the relationship indicated by that record.

[0053] 5 and 6, each object is represented by an identifier. However, to be able to search for a relationship such as "a person is sitting on a chair," further information such as the type of each object is required.

[0054] Information about each object may be included in the object-relationship information 20, or may be managed separately from the object-relationship information 20. Hereinafter, information in which an object identifier and information about the object are associated with each other will be referred to as object information.

[0055] 7 is a diagram illustrating an example of the configuration of object information. Object information 30 has columns named object identifier 31, type 32, and area 33. Object identifier 31 indicates the identifier of the object. Type 32 indicates the type of the object. Area 33 indicates the position of an image area (hereinafter referred to as object area) representing the corresponding object in the source image data. For example, if the bounding rectangle of an object is treated as the object area, area 33 indicates information that can identify the bounding rectangle (e.g., the coordinates of the upper left corner of the bounding rectangle and the coordinates of the lower right corner of the bounding rectangle).

[0056] Here, if the multiple source image data used to generate the object relation information 20 is time-series data such as video data, the same object may be detected from the multiple source image data. In this case, the region 33 indicates multiple pairs of a time point and an object region at that time point. The time point may be expressed as an absolute value or a relative value. In the former case, the time point is expressed, for example, by a date and time. In the latter case, the time point is expressed, for example, by a frame number.

[0057] Here, a scene graph is an example of a data structure representing the relationships between objects. In a scene graph, each object is represented by a node. Furthermore, the relationship between two objects is represented by an edge connecting those objects. Each record of the above-mentioned object relationship information 20 can also be represented as a scene graph in which a subject and an object are represented by a node and relationship features are attached to the edges. From this, the processing performed by the search device 2000 can also be considered as processing to detect a desired node pair from a scene graph in which relationship features are attached to edges representing relationships between objects.

[0058] <Calculation of Relation Feature Amount: S104> The calculation unit 2040 calculates the relation feature amount using the query data 10 (S104). To calculate the relation feature amount, for example, a trained model (hereinafter, relation feature amount calculation model) is used. For example, a machine learning-based model such as a neural network is used as the relation feature amount calculation model.

[0059] The calculation unit 2040 inputs the query data 10 to the relation feature calculation model. The calculation unit 2040 acquires the relation feature output from the relation feature calculation model.

[0060] For example, a feature calculation model included in a vision-and-language model can be used as the relation feature calculation model. The vision-and-language model is a model that can execute processing that handles both visual data and linguistic data. Examples of visual data include image data and point cloud data. Examples of linguistic data include text data.

[0061] An example of a process that handles both visual and linguistic data is a process that compares the content of image data with the content of text data, such as determining whether the relationships between objects contained in the image data match the relationships between objects expressed in the text data.

[0062] 8 is a diagram illustrating the configuration of a vision-and-language model. The vision-and-language model 40 acquires visual data 50 and linguistic data 60. The vision-and-language model 40 then determines whether the content of the visual data 50 matches the content of the linguistic data 60.

[0063] In order to compare the content of visual data 50 with the content of linguistic data 60, the vision-and-language model 40 calculates mutually comparable features from the visual data 50 and the linguistic data 60. To this end, the vision-and-language model 40 has a first feature calculation model 42 and a second feature calculation model 44. The first feature calculation model 42 calculates the feature of the visual data 50. The second feature calculation model 44 calculates the feature of the linguistic data 60.

[0064] The vision-and-language model 40 further includes a determination model 46. The determination model 46 compares the feature values ​​obtained from the visual data 50 with the feature values ​​obtained from the linguistic data 60 to determine whether the content of the visual data 50 matches the content of the linguistic data 60.

[0065] For example, suppose the vision-and-language model 40 determines whether the relationship between objects represented by the visual data 50 matches the relationship between objects represented by the linguistic data 60. In this case, by training the vision-and-language model 40, the first feature calculation model 42 is trained to calculate the feature of the relationship between the objects represented by the visual data 50. Similarly, the second feature calculation model 44 is trained to calculate the feature of the relationship between the objects represented by the linguistic data 60.

[0066] Therefore, the relation feature calculation model to be included in the calculation unit 2040 can be obtained from the vision-and-language model 40 that handles the same type of data as the type of query data 10. For example, assume that the query data 10 is image data. In this case, the first feature calculation model 42 of the trained vision-and-language model 40 that handles image data as visual data 50 can be used as the relation feature calculation model to be included in the calculation unit 2040.

[0067] Alternatively, for example, assume that the query data 10 is text data in which the relationship between objects is expressed in sentences. In this case, the second feature calculation model 44 of the trained vision-and-language model 40, in which the text data expressing the relationship between objects in sentences is treated as linguistic data 60, can be used as the relationship feature calculation model to be included in the calculation unit 2040.

[0068] Here, it is assumed that the query data 10 is visual data such as image data and also includes information other than the object pair. In this case, the calculation unit 2040 detects an area representing the object pair from the visual data and inputs the detected area to the relation feature calculation model.

[0069] When the query data 10 is image data, a relationship extraction model included in a relationship information generation device (described later) may be used for the relationship feature calculation model. In this case, the calculation unit 2040 detects, from the query data 10, an object region of the subject of the object pair, an object region of the object of the object pair, and a region including both the subject and the object (e.g., a region circumscribing the subject and the object). Furthermore, the calculation unit 2040 generates combined data of these three regions (e.g., a concatenation of data representing these three regions). Then, the calculation unit 2040 inputs the combined data into the relationship extraction model. Details of the relationship extraction model will be described later.

[0070] <Search: S106> The detection unit 2060 uses the relation feature obtained from the query data 10 to detect, from the object relation information 20, a relation between objects that matches the relation between objects represented by the query data 10 (S106). For example, the detection unit 2060 calculates the similarity between each relation feature 23 indicated in the object relation information 20 and the relation feature obtained from the query data 10, and determines whether the calculated similarity is equal to or greater than a predetermined threshold. If the similarity calculated for a certain relation feature 23 is equal to or greater than the threshold, the detection unit 2060 detects the relation between objects represented by the relation feature 23 as matching the relation between objects represented by the query data 10.

[0071] When there are multiple relation feature quantities 23 whose similarity is equal to or greater than a threshold, for example, the detection unit 2060 treats the relation between the objects represented by the relation feature quantity 23 with the greatest similarity as matching the relation between the objects represented by the query data 10. Alternatively, for example, the detection unit 2060 may treat each of the relations between the objects represented by the top N (N is a predetermined natural number) relation feature quantities 23 in descending order of similarity as matching the relation between the objects represented by the query data 10.

[0072] The similarity between the feature quantities can be expressed using various metrics that can represent the similarity between two vectors. For example, the detection unit 2060 calculates the similarity between the two feature quantities using the distance between the two feature quantities. For example, the similarity between the two feature quantities can be expressed as the reciprocal of the distance between the feature quantities. Alternatively, the detection unit 2060 can calculate the cosine similarity between the two feature quantities as the similarity between the two feature quantities.

[0073] <Output of Search Results> The search device 2000 outputs information based on the results of detection by the detection unit 2060. Hereinafter, information output based on the results of the search will be referred to as output information.

[0074] The content of the output information varies. For example, the detection unit 2060 extracts, from the object relation information 20, relations between objects that match the relations between objects represented by the query data 10, and includes the extracted information in the output information. More specifically, for a relation feature 23 that is determined to match the relation feature calculated from the query data 10, the detection unit 20 includes, in the output information, a record of the object relation information 20 in which the relation feature 23 is indicated. For example, assume that the relation feature calculated from the query data 10 matches Fr2 in FIG. 5 . In this case, the output information includes the information indicated in the second record of the object relation information 20 in FIG. 5 .

[0075] By obtaining the above-described output information, the user of the search device 2000 can know the subjects, objects, and relationships between the subjects and objects for the object relationships among the object relationships obtained from the source image data that match the object relationships represented by the query data 10. Note that the search device 2000 preferably uses object information 30 in addition to the object relationship information 20, and includes information about the type of object, etc., for each of the subject and object in the output information.

[0076] In addition to or instead of the records of the object relation information 20, the detection unit 2060 may output source image data for a relationship between objects that matches the relationship between objects represented by the query data 10. For example, assume that a relationship feature 23 that matches the relationship feature calculated from the query data 10 is indicated in the second record of the object relation information 20 in FIG. 6. In this case, the search device 2000 includes source image data with identifier 002 identified by the source identifier 21 of the second record in the output information. This allows the user of the search device 2000 to obtain source image data including the relationship between objects represented by the query data 10.

[0077] For example, assume that the query data 10 is text data expressing the relationship "a person is sitting on a chair." In this case, a user of the search device 2000 can obtain source image data including a scene in which a person is sitting on a chair by searching using the query data 10.

[0078] The output information may include image data that includes only the object of interest, instead of the entire source image data. For example, suppose the query data 10 is text data such as "a person sitting on a chair." Furthermore, suppose a certain source image data includes both a scene of a person sitting on a chair and a scene of a dog holding a ball. In this case, the output information may include image data representing the areas of the person and the chair from the source image data.

[0079] Here, the query data 10 may represent a relationship between two or more pairs of objects. For example, the query data 10 may be text data such as "a person is sitting in a chair and a dog is holding a ball."

[0080] In this way, when the query data 10 represents the relationship between two or more object pairs, the calculation unit 2040 calculates a relationship feature for each object pair. Furthermore, the detection unit 2060 searches the object relationship information 20 for a relationship feature 23 that matches each of the calculated relationship feature features.

[0081] For example, suppose the query data 10 is text data of “a person is sitting on a chair, and a dog is holding a ball in its mouth.” In this case, the detection unit 2060 1) detects from the object relation information 20 a relation feature 23 that matches a relation feature obtained from the text data of “a person is sitting on a chair,” and 2) detects from the object relation information 20 a relation feature 23 that matches a relation feature obtained from the text data of “a dog is holding a ball.”

[0082] Here, when query data 10 indicating a sentence is handled, it is preferable that the calculation unit 2040 has a parser that interprets the sentence indicated in the query data 10. When the query data 10 includes multiple sentences, the parser divides the query data 10 into each sentence. Furthermore, the parser interprets each sentence to identify the subject, object, and relationship between the subject and object indicated in each sentence.

[0083] Furthermore, when multiple sentences in the query data 10 are connected by words representing logical operators such as "and" or "or," the parser interprets the logical relationship between the sentences to identify search criteria. For example, assume that the query data 10 represents the sentence "A person is sitting in a chair, and a dog is holding a ball in its mouth." In this case, the parser divides the query data 10 into two sentences, "A person is sitting in a chair" and "A dog is holding a ball," and identifies that these sentences are connected by "and." The parser then interprets each of the two sentences and identifies the subjects, objects, and relationships between the subjects and objects, such as "Subject: person, Object: chair, Relationship: sitting" and "Subject: dog, Object: ball, Relationship: holding." Based on the interpretation of the two sentences and the logical relationship between the two sentences, the parser then identifies the search criteria "(Subject: person, Object: chair, Relationship: sitting) and (Subject: dog, Object: ball, Relationship: holding)."

[0084] If the logical relationship between multiple sentences is not specified, the calculation unit 2040 assumes that a predetermined logical relationship (e.g., "and") exists between these sentences. For example, assume that the query data 10 indicates "1) a person is sitting on a chair, 2) a dog is holding a ball in its mouth." Also, assume that the predetermined logical relationship is "and." In this case, as in the previous example, the search criteria "(subject: person, object: chair, relationship: sitting) and (subject: dog, object: ball, relationship: holding)" are identified.

[0085] When a plurality of source image data are used to generate the object relation information 20, the detection unit 2060 may determine whether the relation feature 23 detected from the object relation information 20 for each object pair corresponds to the same source image data. Then, the detection unit 2060 may determine that the object relation information 20 indicates a relation between objects that matches the relation between objects represented in the query data 10 only when the relation feature 23 detected from the object relation information 20 for each object pair corresponds to the same source image data.

[0086] For example, suppose the query data 10 is text data of "a person sitting on a chair and a dog holding a ball in its mouth." In this case, as described above, the search criteria are specified as "(subject: person, object: chair, relationship: sitting) and (subject: dog, object: ball, relationship: holding)." Therefore, only when a single source image data contains both a scene of a person sitting on a chair and a scene of a dog holding a ball in its mouth, the desired relationship is considered to have been detected from the object relation information 20.

[0087] The output information may be output in various ways. For example, the detection unit 2060 stores the output information in an arbitrary storage unit. Alternatively, for example, the detection unit 2060 displays the output information on a display device. Alternatively, for example, the detection unit 2060 transmits the output information to another device. For example, when query data 10 is transmitted from another device, the detection unit 2060 transmits the output information to the device that transmitted the query data 10.

[0088] <Example Use Case of Search Device 2000> There are various possible use cases for the search device 2000. For example, the search device 2000 can be used to search for a desired event or collect information about an accident from video footage captured by a drive recorder. In this case, the object-relation information 20 is generated using the video footage captured by the drive recorder. The source image data is each video frame that constitutes the video footage captured by the drive recorder.

[0089] In addition, for example, the search device 2000 can be used to monitor a person using video. Specifically, the search device 2000 can be used to detect a desired behavior of a person being monitored from video obtained by filming the person being monitored (hereinafter referred to as monitoring video). In this case, the object-relation information 20 is generated using the monitoring video. The source image data is each video frame that constitutes the monitoring video.

[0090] Alternatively, for example, the search device 2000 can be used to detect criminal activity from surveillance video. In this case, the surveillance video is used to generate the object-related information 20. The source image data are each video frame that constitutes the surveillance video.

[0091] In addition, for example, the search device 2000 can be used to analyze customer purchasing behavior. In this case, the object-relation information 20 is also generated using surveillance video.

[0092] <Method for generating object relation information 20> Here, a method for generating object relation information 20 will be exemplified. A device that generates object relation information 20 is called a relation information generation device. The relation information generation device may be provided integrally with the search device 2000, or may be provided separately from the search device 2000. In the latter case, the hardware configuration of the relation information generation device is expressed in FIG. 3 , similar to the hardware configuration of the search device 2000, for example.

[0093] 9 is a diagram illustrating an example of an outline of the operation of the relationship information generation device 3000. Note that FIG. 9 is a diagram for facilitating understanding of the outline of the relationship information generation device 3000, and the operation of the relationship information generation device 3000 is not limited to the operation shown in FIG.

[0094] The relationship information generation device 3000 acquires source image data 100. The relationship information generation device 3000 detects object pairs from the source image data 100. The relationship information generation device 3000 calculates a relation feature for the detected object pairs. The relationship information generation device 3000 generates object relationship information 20 by associating the object pairs with the relation feature calculated for the object pairs. Note that two or more source image data 100 may be used to generate the object relationship information 20.

[0095] The relationship information generating device 3000 can automatically generate object relationship information 20 indicating relationship features that represent the characteristics of the relationship between objects for each of multiple object pairs included in one or more image data.

[0096] <<Example of Functional Configuration of Relationship Information Generation Device 3000>> Fig. 10 is a diagram illustrating an example of the functional configuration of the relationship information generation device 3000. For example, the relationship information generation device 3000 has an acquisition unit 3020, a detection unit 3040, a calculation unit 3060, and a generation unit 3080. The acquisition unit 3020 acquires source image data 100. The detection unit 3040 detects object pairs from the source image data 100. The calculation unit 3060 calculates a relation feature amount for the detected object pair. The generation unit 3080 generates object relationship information 20 by associating the object pair with the relation feature amount calculated for the object pair.

[0097] <<Processing Flow>> Fig. 11 is a flowchart illustrating the flow of processing executed by the relationship information generation device 3000. The acquisition unit 3020 acquires source image data 100 (S202). The detection unit 3040 detects object pairs from the source image data 100 (S204). The calculation unit 3060 calculates relational features for multiple pairs detected from the source image data 100 (S206). The generation unit 3080 associates the object pairs with the relational features calculated for the object pairs to generate object relationship information 20 (S208).

[0098] <<Acquisition of Source Image Data 100: S202>> The acquisition unit 3020 acquires the source image data 100 (S202). There are various methods for the acquisition unit 3020 to acquire the source image data 100. For example, the source image data 100 is stored in advance in a storage unit accessible by the relationship information generation device 3000. In this case, the acquisition unit 3020 acquires the source image data 100 by reading the source image data 100 from this storage unit. Alternatively, for example, the acquisition unit 3020 may acquire the source image data 100 by receiving source image data 100 transmitted from another device. The device that transmits the source image data 100 to the relationship information generation device 3000 is, for example, the device that generated the source image data 100 (such as a camera or a ranging device).

[0099] <<Detection of Object Pairs: S204>> The detection unit 3040 detects object pairs from the source image data 100 (S204). To do this, the detection unit 3040 first detects objects from the source image data 100. More specifically, the detection unit 3040 detects objects from the source image data 100 and generates the object information 30 described above.

[0100] For example, the detection unit 3040 detects objects by detecting object regions from the source image data 100. An identifier is assigned to each detected object. The detection unit 3040 also identifies the type of object represented by each object region. The detection unit 3040 generates object information 30 by associating the identifier, type, and object region for each object.

[0101] Here, for example, a trained model is used in the process of detecting an object region from the source image data 100 and identifying the type of object represented by the object region. For example, a machine learning-based model such as a neural network can be used as such a model.

[0102] When multiple time-series source image data 100 are used, such as video data, the detection unit 3040 generates object information 30 using information about objects detected from each of the multiple source image data 100. In this case, the same object may be detected from different source image data 100. Therefore, the detection unit 3040 identifies objects between the different source image data 100 by processing such as tracking. The detection unit 3040 then assigns the same identifier to the same objects detected from different source image data 100. Therefore, information about the same objects detected from different source image data 100 is stored in the same record in the object information 30.

[0103] The detection unit 3040 generates object pairs for multiple objects detected from the source image data 100. For example, the calculation unit 3060 generates all possible pairs for multiple objects detected from the source image data 100. For example, if there are three detected objects, A, B, and C, three object pairs are generated: (A, B), (B, A), (A, C), (C, A), (B, C), and (C, B).

[0104] In the above notation of object pairs, the first object is the subject and the second object is the object. For example, the notation of an object pair (A,B) represents an object pair whose subject and object are object A and object B, respectively.

[0105] The detection unit 3040 may generate only object pairs that are presumed to have a relationship between the objects. For example, the detection unit 3040 ensures that objects of a specific type are included in the object pairs. In other words, object pairs that do not include an object of a specific type are not treated as object pairs. Furthermore, when an object of a specific type is always included in an object pair in this way, the detection unit 3040 treats the object of the specific type as the subject.

[0106] For example, the specific type is a person. That is, a person is always included in the object pair. In this way, each record of the object relation information 20 represents the relationship between a person and other objects, in other words, the behavior of a person with respect to other objects.

[0107] For example, suppose a person is treated as a specific type. Also, suppose that object A representing a person, object B representing a person, object C representing a ball, and object D representing a chair are detected from the source image data 100. In this case, the detection unit 3040 generates object pairs (A, B), (A, C), (A, D), (B, A), (B, C), and (B, D). (C, D) and (D, C) are not treated as object pairs because they do not include a person.

[0108] Furthermore, the detection unit 3040 may determine whether or not there is a relationship between two objects based on the positional relationship between the object regions of these two objects. More specifically, the detection unit 3040 determines that a relationship exists between these objects when the object regions of the two objects overlap or touch each other. Therefore, the detection unit 3040 generates an object pair for these two objects. On the other hand, when the object regions of the two objects do not overlap or touch each other, the detection unit 3040 determines that there is no relationship between these objects. Therefore, the detection unit 3040 does not generate an object pair for these two objects.

[0109] <<Calculation of Relation Feature Amount: S208>> The calculation unit 3060 calculates a relation feature amount for each object pair (S208). The relation feature amount between two objects included in the object pair is calculated using, for example, feature amounts obtained from the image regions of the object pair.

[0110] 12 is a diagram conceptually illustrating the processing performed by the calculation unit 3060. The calculation unit 3060 has a relationship extraction model 200. The relationship extraction model 200 is composed of a first feature quantity calculation model 210 and a second feature quantity calculation model 220. The first feature quantity calculation model 210 and the second feature quantity calculation model 220 are each realized by a machine learning-based model such as a neural network, for example.

[0111] The calculation unit 3060 generates input data 130 from each object pair. The input data 130 is composed of image regions obtained from the object pair. For example, the calculation unit 3060 combines the object region 132, the object region 134, and the image region 136 obtained from the object pair to generate the input data 130 for the object pair. The object region 132 is the object region of the subject included in the object pair. The object region 134 is the object region of the object included in the object pair. The image region 136 is an image region representing the circumscribed rectangle of the object region 132 and the object region 134.

[0112] Data enumerating one or more input data 130 obtained from the source image data 100 (in the case of FIG. 12 , data enumerating input data 130-1 to 130-M) is input to the relationship extraction model 200. The first feature calculation model 210 is configured to output object pair features 140 in response to the input of the input data 130.

[0113] The calculation unit 3060 inputs each piece of input data 130 into the first feature calculation model 210, thereby obtaining an object pair feature 140 for each object pair. Here, in the example of FIG. 12 , M object pairs are generated as a premise. From these M object pairs, M pieces of input data 130, namely, input data 130-1 to 130-M, are generated. Then, M object pair features 140, namely, object pair feature 140-1 to object pair feature 140-M, are calculated from the input data 130-1 to 130-M.

[0114] The second feature calculation model 220 is configured to output, in response to input of a plurality of object pair features 140, the same number of relation features 150 as the input object pair features 140. The calculation unit 3060 inputs the object pair features 140 obtained for each object pair collectively to the second feature calculation model 220, thereby obtaining the relation feature 150 for each object pair. In the example of FIG. 12 , M relation features 150 are obtained. For example, the second feature calculation model 220 outputs data in which a plurality of relation features 150 are listed.

[0115] However, among the object pairs, there may be object pairs in which the objects are not related to each other. Therefore, for object pairs in which the objects are not related to each other, it is preferable that the second feature calculation model 220 outputs the relation feature 150 in a manner that makes it possible to determine that there is no relation between the objects. Therefore, for example, the second feature calculation model 220 is configured to output a predetermined reference vector (e.g., a zero vector) or a vector whose distance from the reference vector is equal to or less than a predetermined threshold value as the relation feature 150 for an object pair in which the objects are not related to each other.

[0116] The first feature quantity calculation model 210 and the second feature quantity calculation model 220 are trained in advance to perform the above-mentioned operations. The training method for the first feature quantity calculation model 210 and the second feature quantity calculation model 220 will be described later.

[0117] 12 is configured to acquire a plurality of object pair features 140 at once and calculate a plurality of relational features 150 from these plurality of object pair features 140. With this configuration, the second feature calculation model 220 can calculate the relational feature of an object pair by taking into consideration not only the features of that object pair but also the features of other object pairs. Therefore, the relational feature of each object pair can be calculated by taking into consideration the context of the entire scene recorded in the source image data 100.

[0118] However, the configuration of the second feature calculation model 220 is not limited to a configuration in which a plurality of object pair features 140 are acquired as input. For example, the second feature calculation model 220 may be configured to output one relation feature 150 in response to one input object pair feature 140. In this case, the calculation unit 3060 calculates the object pair feature 140 using the second feature calculation model 220 for each object pair individually.

[0119] <<Generation of Object Relation Information 20: S208>> The generation unit 3080 generates the object relation information 20 using the relation feature 150 calculated by the calculation unit 3060 (S208). Specifically, the calculation unit 3060 generates a record of the object relation information 20 by associating, for each object pair, the identifier of the subject included in the object pair, the identifier of the object included in the object pair, and the relation feature 150 calculated for the object pair.

[0120] As described above, the object pairs detected by the detection unit 3040 may include object pairs with no relationship between the objects. Therefore, for example, the generation unit 3080 determines whether or not there is a relationship between the objects for each object pair using the relationship feature 150 calculated for that object pair. Then, the generation unit 3080 generates records of the object relationship information 20 only for object pairs determined to have a relationship between the objects.

[0121] Assume that a reference vector is calculated as the relation feature 150 of an object pair that has no relation between the objects. In this case, the generation unit 3080 determines whether the relation feature 150 matches the reference vector for each object pair. Then, the generation unit 3080 generates records of the object relation information 20 only for object pairs corresponding to relation feature 150 whose relation feature 150 does not match the reference vector.

[0122] Assume that a vector whose distance from the reference vector is equal to or less than a threshold is calculated as the relation feature 150 for an object pair having no relation between the objects. In this case, the generation unit 3080 determines whether the distance between the relation feature 150 and the reference vector is equal to or less than the threshold for each object pair. Then, the generation unit 3080 generates records of the object relation information 20 only for object pairs corresponding to relation feature 150 whose distance from the reference vector is not equal to or less than the threshold.

[0123] <<Output of Object Relation Information 20>> The relationship information generation device 3000 outputs the generated object relation information 20. The object relation information 20 may be output in various ways. For example, the relationship information generation device 3000 stores the object relation information 20 in a storage unit accessible from the search device 2000. Alternatively, for example, the relationship information generation device 3000 may transmit the object relation information 20 to the search device 2000.

[0124] <Training Method of Relation Extraction Model 200> The first feature quantity calculation model 210 and the second feature quantity calculation model 220 that constitute the relation extraction model 200 are trained in advance. Here, a method for training the relation extraction model 200 will be described. A device that trains the relation extraction model 200 is called a training device.

[0125] 13 is a diagram illustrating an example of an outline of the operation of training device 4000. Note that Fig. 13 is a diagram for facilitating understanding of the outline of training device 4000, and the operation of training device 4000 is not limited to the operation shown in Fig. 9.

[0126] The training device 4000 acquires training data 170 and uses the training data 170 to train the relationship extraction model 200. The training data 170 indicates a pair of input data 172 and ground truth data 174 for each of one or more object pairs. The input data 172 has a structure similar to that of the input data 130. Therefore, for example, the input data 172 represents data in which a subject object region, an object object region, and circumscribed rectangular image regions of the subject object region and the object object region are concatenated (e.g., concatenation). The ground truth data 174 represents relationship features to be output by the trained relationship extraction model 200 for the corresponding input data 172.

[0127] The training device 4000 inputs all of the input data 172 shown in the training data 170 to the relationship extraction model 200. The input data 172 input to the relationship extraction model 200 is input to the first feature calculation model 210. The first feature calculation model 210 outputs the same number of object pair features 140 as the number of input data 172.

[0128] The object pair features 140 output from the first feature calculation model 210 are input to the second feature calculation model 220. As a result, the same number of relation features 150 as the number of object pair features 140 (i.e., the same number as the input data 172) are output.

[0129] For each object pair, it is preferable that the difference between the relationship feature 150 of that object pair output from the relationship extraction model 200 and the ground truth data 174 of that object pair is small. Therefore, for example, the training device 4000 calculates the similarity between the relationship feature 150 and the ground truth data 174 for each object pair, and updates the relationship extraction model 200 based on the calculated similarity. The relationship extraction model 200 is updated by updating trainable parameters of the relationship extraction model 200 (e.g., weights and biases assigned to each edge of a neural network).

[0130] The training device 4000 trains the relationship extraction model 200 by repeatedly updating the relationship extraction model 200 using a plurality of training data 170. The training device 4000 can automatically generate the relationship extraction model 200 that outputs the relationship feature quantities of each object pair in response to input of one or more images of object pairs.

[0131] <<Example of Functional Configuration of Training Apparatus 4000>> Fig. 14 is a diagram illustrating an example of the functional configuration of the training apparatus 4000. For example, the training apparatus 4000 includes an acquisition unit 4020, a calculation unit 4040, and an update unit 4060. The acquisition unit 4020 acquires training data 170. The calculation unit 4040 inputs input data 172 of one or more entity pairs to the relationship extraction model 200, thereby calculating a relationship feature 150 for each entity pair. The update unit 4060 updates the relationship extraction model 200 based on the similarity between the relationship feature 150 and ground truth data 174.

[0132] <<Example of Hardware Configuration of Training Device 4000>> The hardware configuration of the training device 4000 is shown in Fig. 3, similar to the hardware configuration of the search device 2000, for example. Here, a program that realizes each functional component of the training device 4000 is stored in a storage device of a computer that realizes the training device 4000.

[0133] <<Processing Flow>> Figure 15 is a flowchart illustrating the flow of processing executed by the training device 4000. The acquisition unit 4020 acquires training data 170 (S302). The calculation unit 4040 inputs the input data 172 of each entity pair to the relationship extraction model 200 to calculate the relationship feature 150 for each entity pair (S304). The update unit 4060 calculates the similarity between the relationship feature 150 and the ground truth data 174 (S306). The update unit 4060 updates the relationship extraction model 200 based on the calculated similarity (S308).

[0134] The processing flow executed by the training device 4000 is not limited to that shown in Fig. 15. For example, the training device 4000 may perform batch learning using a plurality of training data 170. Specifically, the training device 4000 may use each of the plurality of training data 170 to calculate a loss based on the similarity between the relation feature 150 and the ground truth data 174, and update the relation extraction model 200 using the calculated loss statistics.

[0135] <<Acquisition of Training Data 170: S302>> The acquisition unit 4020 acquires the training data 170 (S302). There are various methods for the acquisition unit 4020 to acquire the training data 170. For example, the training data 170 is stored in advance in a storage unit accessible from the training device 4000. In this case, the acquisition unit 4020 acquires the training data 170 by reading the training data 170 from this storage unit. Alternatively, for example, the training device 4000 may acquire the training data 170 by receiving training data 170 transmitted by another device. The device that transmits the training data 170 is, for example, the device that generated the training data 170.

[0136] Here, each input data 172 included in the training data 170 is generated from one image data. As described above, the relational information generation device 3000 detects one or more object pairs from the source image data 100 and generates input data 130 for each object pair. The multiple input data 172 included in the training data 170 can also be generated in a similar manner.

[0137] The process of generating input data 172 from image data may be performed by the training device 4000 or by another device. In the former case, the training device 4000 acquires image data and generates input data 172 for each of one or more object pairs detected from the image data.

[0138] Each piece of ground truth data 174 included in the training data 170 is calculated using, for example, data in which the relationship between the objects in the object pair is expressed in text (hereinafter referred to as relational text data). The process of calculating the ground truth data 174 representing the relational feature from the relational text data can be performed using, for example, a model included in the vision-and-language model. Specifically, the second feature calculation model 44 shown in FIG. 8 can be used.

[0139] The process of generating the ground truth data 174 for each object pair may be performed by the training device 4000 or by another device. In the former case, the training device 4000 acquires related text data for each of the multiple object pairs for the source image data used to generate the input data 172. The training device 4000 converts each related text data into a related feature by inputting the related text data to the second feature calculation model 44. In this way, the training device 4000 generates the ground truth data 174 for each object pair detected from the source image data.

[0140] Here, when multiple entity pairs are detected from one image data, multiple pieces of input data 172 and multiple pieces of ground truth data 174 are generated. Therefore, it is necessary to associate the input data 172 with the ground truth data 174. The method for associating the input data 172 with the ground truth data 174 (in other words, the method for identifying the input data 172 and the ground truth data 174 that correspond to the same entity pair) may be arbitrary. For example, information indicating which part of the image data the entity pair is located in is associated in advance with each piece of related text data used to generate the ground truth data 174. The device that generates the training data 170 uses this information to associate the input data 172 with the ground truth data 174.

[0141] <<Calculation of Similarity: S306>> The update unit 4060 calculates the similarity between the related feature 150 and the ground truth data 174 (S306). The similarity between the feature amounts can be calculated using, for example, the cosine similarity or the distance between the feature amounts, as described above.

[0142] The similarity between the relation feature 150 and the ground truth data 174 may be calculated for each entity pair, or may be calculated for multiple entity pairs collectively. In the former case, the update unit 4060 calculates the similarity between the relation feature 150 and the ground truth data 174 for each entity pair.

[0143] When the similarity between the relation feature 150 and the ground truth data 174 is calculated collectively for a plurality of object pairs, for example, the updating unit 4060 generates combined data (e.g., concatenation) of the relation feature 150 by combining the plurality of relation feature 150. Similarly, the updating unit 4060 generates combined data (e.g., concatenation) of the plurality of ground truth data 174 by combining the plurality of ground truth data 174 indicated in the training data 170. Then, the updating unit 4060 calculates the similarity between the combined data of the relation feature 150 and the combined data of the ground truth data 174. The similarity between the combined data can also be calculated using the cosine similarity or distance between the combined data.

[0144] As described above, the relation feature does not need to include information about the subject or the object. For example, it is assumed that the relation feature does not include information about the object. In this case, the ground truth data 174 is generated using a plurality of relation text data in which the subject and object relationships match but the objects do not. Specifically, the statistical values ​​of the relation feature obtained from each of the plurality of relation text data are used as the ground truth data 174.

[0145] For example, suppose the relationship extraction model 200 is trained to be able to calculate a relationship feature that represents the relationship between objects, such as "a person is holding something." In this case, a plurality of relational text data in which the subject is a person and the relationship is "has" is used. Such relational text data includes "a person is holding a ball," "a person is holding a cup," and "a person is holding a smartphone."

[0146] Furthermore, it is preferable that the ground truth data 174 generated from a plurality of relational text data in which the relationships between the subject and the object match but the objects do not match, be associated with a plurality of input data 172 in which the relationships between the subject and the object match but the objects do not match. For example, as described above, assume that the ground truth data 174 is generated from relational text data such as "a person is holding a ball," "a person is holding a cup," and "a person is holding a smartphone." In this case, a plurality of training data 170 is generated, such as training data 170 in which this ground truth data 174 is associated with input data 172 obtained from image data of a scene in which a person is holding a ball, or training data 170 in which this ground truth data 174 is associated with input data 172 obtained from image data of a scene in which a person is holding a cup.

[0147] Alternatively, for example, if the relation feature does not include information about the object, the training data 170 may represent a pair of input data 172 and ground truth data 174 that match each other in terms of the relationship between the subject and the object but do not match each other in terms of the object. For example, assume that the relation extraction model 200 is trained to be able to calculate a relation feature that represents the relation "a person is holding something." In this case, the relation extraction model 200 is trained using multiple sets of training data 170 in which input data 172 representing a scene in which a person is holding something is associated with ground truth data 174 obtained from text data that represents a person holding something.

[0148] As an example, the training data 170 associates input data 172 obtained from image data of a scene in which a person is holding a ball with ground truth data 174 obtained from related text data of "a person is holding a cup." The training data 170 also associates input data 172 obtained from image data of a scene in which a person is holding a cup with ground truth data 174 obtained from related text data of "a person is holding a smartphone."

[0149] The method of training the relationship extraction model 200 to calculate relationship features that do not include information about the subject can be realized by a method of training the search device 2000 to calculate relationship features that do not include information about the object.

[0150] <<Updating the Relationship Extraction Model 200: S308>> The updating unit 4060 updates the relationship extraction model 200 based on the similarity between the relationship feature 150 and the ground truth data 174 (S308). For example, the updating unit 4060 calculates the loss by inputting the similarity between the relationship feature 150 and the ground truth data 174 into a loss function. The loss function is designed so that the greater the similarity between the relationship feature 150 and the ground truth data 174, the smaller the loss.

[0151] The update unit 4060 updates the trainable parameters included in the relationship extraction model 200 (in other words, the trainable parameters included in each of the first feature amount calculation models 210 and 210) based on the calculated loss. Various methods such as gradient descent can be used to update the model parameters based on the loss.

[0152] Here, it is assumed that the similarity between the relation feature 150 and the ground truth data 174 is calculated for each object pair. In this case, the update unit 4060 calculates the loss using the calculated similarities. For example, the update unit 4060 calculates the loss by inputting the statistical values ​​of the calculated similarities into a loss function.

[0153] <<Training Using Negative Example Training Data>> The above explanation is based on the assumption that the training data 170 is positive example training data. That is, the input data 172 is associated with a relation feature that should be output when the input data 172 is input to the relation extraction model 200.

[0154] In this regard, negative example training data may also be used in training the relationship extraction model 200. The negative example training data associates, with the input data 172, relationship features that should not be output when the input data 172 is input to the relationship extraction model 200 (hereinafter, referred to as error data).

[0155] The error data can be generated using, for example, relational text data that expresses a relationship different from the relationship between object pairs. For example, suppose the input data 172 is obtained from an image region of a scene in which a person is sitting on a chair. In this case, the error data can be generated from relational text data such as "the person is holding a chair," "the person is looking at a smartphone," or "the dog is holding a ball."

[0156] The update unit 4060 calculates the similarity between the relation feature 150 obtained by inputting the input data 172 into the relationship extraction model 200 and the error data corresponding to the input data 172. Here, it is preferable that the similarity between the relation feature 150 and the error data is small. Therefore, in training using negative example training data, the loss function is designed so that the loss increases as the similarity between the relation feature 150 and the error data decreases. The update unit 4060 calculates the loss by inputting the similarity between the relation feature 150 and the error data into this loss function. Then, the update unit 4060 updates the relationship extraction model 200 based on the calculated loss.

[0157] The training data of positive examples and the training data of negative examples are configured so that they can be distinguished from each other. For example, these training data are stored in different storage units. Alternatively, for example, a label indicating whether the training data is a positive example or a negative example is attached in advance to the training data.

[0158] <<Output of the Relationship Extraction Model 200>> The training device 4000 outputs the generated relationship extraction model 200. The relationship extraction model 200 may be output in various forms. For example, the training device 4000 stores the relationship extraction model 200 in a storage unit accessible from the relationship information generation device 3000. Alternatively, for example, the training device 4000 may transmit the relationship extraction model 200 to the relationship information generation device 3000.

[0159] Here, the training device 4000 may output only a set of trainable parameters included in the relationship extraction model 200, or may output the entire relationship extraction model 200. In the latter case, the training device 4000 also outputs a program that realizes the relationship extraction model 200 and parameters that are not updated during training (such as hyperparameters).

[0160] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0161] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0162] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A search device comprising: an acquisition means for acquiring query data representing a relationship between a pair of objects; a calculation means for calculating, from the query data, a relationship feature representing a characteristic of the relationship between the objects; and a detection means for detecting, from object relationship information in which the relationship feature is associated with each of a plurality of pairs of objects, a relationship feature that matches the calculated relationship feature, wherein the object relationship information is generated using one or more image data. (Supplementary Note 2) The search device according to Supplementary Note 1, wherein the query data is text data representing the relationship between the objects for the pair. (Supplementary Note 3) The calculation means calculates the relationship feature for the pair of objects represented by the query data by inputting the query data to a relationship feature calculation model, and the relationship feature calculation model is generated using a model that calculates features of the input text data in a vision-and-language model to which image data including the pair of objects and text data representing the relationship between the objects for the pair of objects are input. (Supplementary Note 4) The search device according to any one of Supplementary Notes 1 to 3, wherein the detection means detects the relation feature indicated in the object relation information, the degree of similarity with the calculated relation feature being equal to or greater than a threshold, as the relation feature that matches the calculated relation feature. (Supplementary Note 5) The search device according to any one of Supplementary Notes 1 to 3, wherein the query data represents a relationship between each of a plurality of pairs of objects, the calculation means calculates the relation feature for each of the plurality of pairs indicated in the query data, and the detection means detects the relation feature that matches each of the plurality of relation feature calculated for the plurality of pairs indicated in the query data from the relation feature features associated with the same image data in the object relation information.(Supplementary Note 6) A relationship information generation device comprising: an acquisition means for acquiring source image data; a detection means for detecting one or more pairs of objects from the source image data; a calculation means for calculating, for the pairs, a relationship feature that indicates a characteristic of the relationship between the objects; and a generation means for generating object relationship information in which the pairs and the calculated relationship feature for the pairs are associated with each other. (Supplementary Note 7) The relationship information generation device according to Supplementary Note 6, wherein the calculation means uses the source image data to generate combined data obtained by combining an image region of a first object included in the pair, an image region of a second object included in the pair, and an image region including the first object and the second object, and calculates the relationship feature using the generated combined data. (Supplementary Note 8) The relationship information generation device according to Supplementary Note 7, wherein the calculation means outputs data in which the relationship feature for each of a plurality of pairs is listed in response to input of data in which the combined data for each of a plurality of pairs is listed. (Supplementary Note 9) The relationship information generation device according to any one of Supplementary Notes 6 to 8, wherein the calculation means calculates a predetermined reference vector or a vector whose distance from the reference vector is equal to or less than a predetermined threshold as the relationship feature for a pair of objects that are not related to each other. (Supplementary Note 10) A search method comprising: an acquisition step of acquiring query data that represents a relationship between the pair of objects; a calculation step of calculating, from the query data, a relationship feature that represents a characteristic of the relationship between the objects; and a detection step of detecting, from object relationship information in which the relationship feature is associated with each of a plurality of pairs of objects, a relationship feature that matches the calculated relationship feature, wherein the object relationship information is generated using one or more image data. (Supplementary Note 11) The search method according to Supplementary Note 10, wherein the query data is text data that represents the relationship between the objects for the pair.(Supplementary Note 12) The search method according to Supplementary Note 11, wherein in the calculation step, the query data is input to a relation feature calculation model to calculate the relation feature for the object pair represented by the query data, and the relation feature calculation model is generated using a model that calculates features of the input text data in a vision-and-language model to which image data including the object pair and text data representing a relationship between the objects for the object pair are input. (Supplementary Note 13) The search method according to any one of Supplementary Notes 10 to 12, wherein in the detection step, the relation feature indicated in the object relation information, whose similarity with the calculated relation feature is equal to or greater than a threshold, is detected as the relation feature that matches the calculated relation feature. (Supplementary Note 14) The search method according to any one of Supplementary Notes 10 to 12, wherein the query data represents a relationship between each of a plurality of pairs of objects, the calculation step calculates the relationship feature for each of the plurality of pairs indicated in the query data, and the detection step detects the relationship feature that matches each of the plurality of relationship feature calculated for the plurality of pairs indicated in the query data from the relationship feature associated with identical image data in the object relationship information. (Supplementary Note 15) A relationship information generation method comprising: an acquisition step of acquiring source image data, a detection step of detecting one or more pairs of objects from the source image data, a calculation step of calculating a relationship feature that represents a feature of the relationship between the objects for the pairs, and a generation step of generating object relationship information in which the pairs are indicated in correspondence with the relationship feature calculated for the pairs. (Supplementary Note 16) The relationship information generating method according to Supplementary Note 15, wherein in the calculation step, combined data is generated by combining an image region of a first object included in the pair, an image region of a second object included in the pair, and an image region including the first object and the second object, using the source image data, and the relationship feature is calculated using the generated combined data.(Supplementary Note 17) The relationship information generating method according to Supplementary Note 16, wherein, in the calculating step, data listing the relation feature for each of the plurality of pairs is input, and data listing the combined data for each of the plurality of pairs is output. (Supplementary Note 18) The relationship information generating method according to any one of Supplements 15 to 17, wherein, in the calculating step, a predetermined reference vector or a vector whose distance from the reference vector is equal to or less than a predetermined threshold is calculated as the relation feature for a pair of objects that has no relationship between them. (Supplementary Note 19) A program that causes a computer to execute the following steps: an acquisition step of acquiring query data that represents a relationship between the objects for the pair of objects; a calculation step of calculating, from the query data, a relation feature that represents a characteristic of the relationship between the objects; and a detection step of detecting, from object relationship information in which the relation feature is associated with each of a plurality of pairs of objects, a relation feature that matches the calculated relation feature; and the object relationship information is generated using one or more image data. (Supplementary Note 20) The program according to Supplementary Note 19, wherein the query data is text data that represents the relationship between the objects for the pair. (Supplementary Note 21) The program according to Supplementary Note 20, wherein in the calculating step, the query data is input to a relation feature calculation model to calculate the relation feature for the object pair represented by the query data, and the relation feature calculation model is generated using a model that calculates features of the input text data in a vision-and-language model to which image data including the object pair and text data indicating a relationship between the objects for the object pair are input. (Supplementary Note 22) The program according to any one of Supplementary Notes 19 to 21, wherein in the detecting step, the relation feature indicated in the object relation information, whose similarity with the calculated relation feature is equal to or greater than a threshold, is detected as the relation feature that matches the calculated relation feature.(Supplementary Note 23) The program according to any one of Supplementary Notes 19 to 21, wherein the query data represents a relationship between each of a plurality of pairs of objects, the calculation step calculates the relation feature for each of the plurality of pairs indicated in the query data, and the detection step detects the relation feature that matches each of the plurality of relation feature calculated for the plurality of pairs indicated in the query data from the relation feature associated with identical image data in the object relation information. (Supplementary Note 24) The program causes a computer to execute: an acquisition step of acquiring source image data; a detection step of detecting one or more pairs of objects from the source image data; a calculation step of calculating a relation feature that represents a feature of the relationship between the objects for the pairs; and a generation step of generating object relation information in which the pairs are indicated in correspondence with the relation feature calculated for the pairs. (Supplementary Note 25) The program according to Supplementary Note 24, wherein, in the calculation step, using the source image data, combined data is generated by combining an image region of a first object included in the pair, an image region of a second object included in the pair, and an image region including the first object and the second object, and the relation feature is calculated using the generated combined data. (Supplementary Note 26) The program according to Supplementary Note 25, wherein, in the calculation step, in response to input of data listing the combined data for each of the plurality of pairs, data listing the relation feature for each of the plurality of pairs is output. (Supplementary Note 27) The program according to any one of Supplements 24 to 26, wherein, in the calculation step, a predetermined reference vector or a vector whose distance from the reference vector is equal to or less than a predetermined threshold is calculated as the relation feature for a pair of objects that are not related to each other.

[0163] This application claims priority based on Japanese Patent Application No. 2023-118634, filed on July 20, 2023, the disclosure of which is incorporated herein by reference in its entirety.

[0164] 10 Query data 20 Object relation information 21 Source identifier 22 Object pair 23 Relation feature 24 Subject identifier 25 Object identifier 30 Object information 31 Object identifier 32 Type 33 Region 40 Vision-and-language model 42 First feature calculation model 44 Second feature calculation model 46 Decision model 50 Visual data 60 Linguistic data 100 Source image data 130 Input data 132 Object region 134 Object region 136 Image region 140 Object pair feature 150 Relation feature 170 Training data 172 Input data 174 Ground truth data 200 Relation extraction model 210 First feature calculation model 220 Second feature calculation model 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Search device 2020 Acquisition unit 2040 Calculation unit 2060 Detection unit 3000 Relationship information generation device 3020 Acquisition unit 3040 Detection unit 3060 Calculation unit 3080 Generation unit 4000 Training device 4020 Acquisition unit 4040 Calculation unit 4060 Update unit

Claims

1. A means for obtaining query data that represents the relationship between objects in a pair of objects, A calculation means for calculating relational features representing the characteristics of the relationships between objects from the aforementioned query data, The system includes a detection means for detecting relational features that match the calculated relational features from object relational information, in which the relational features are associated with each of several pairs of objects. A search device that generates the aforementioned object relationship information using one or more image data.

2. The search device according to claim 1, wherein the query data is text data representing the relationship between objects in the pair.

3. The calculation means inputs the query data into a relational feature calculation model to calculate the relational features for the pair of objects represented by the query data. The search device according to claim 2, wherein the relation feature calculation model is generated using a model for calculating features of the input text data in a vision-and-language model in which image data containing pairs of objects and text data representing the relationships between the pairs of objects are input.

4. The search device according to any one of claims 1 to 3, wherein the detection means detects the relational feature shown in the object relational information, whose similarity to the calculated relational feature is greater than or equal to a threshold, as the relational feature that matches the calculated relational feature.

5. The aforementioned query data represents the relationships between objects for each of multiple pairs of objects. The calculation means calculates the relational feature for each of the multiple pairs shown in the query data, The search device according to any one of claims 1 to 3, wherein the detection means detects the relational feature that matches each of the plurality of relational feature quantities calculated for each of the plurality of pairs shown in the query data from the relational feature quantities that are associated with the same image data in the object relation information.

6. A means for obtaining source image data, A detection means for detecting one or more pairs of objects from the source image data, A calculation means for calculating relational features that represent the characteristics of the relationship between the objects in the aforementioned pair, A relationship information generation device having a generation means for generating object relationship information in which the pair and the relationship feature quantities calculated for that pair are shown in correspondence.

7. A retrieval step to obtain query data representing the relationship between objects in a pair of objects, A calculation step of calculating relational features that represent the characteristics of the relationships between objects from the aforementioned query data, The method includes a detection step of detecting a relational feature that matches the calculated relational feature from object relational information, in which the relational feature is associated with each of several pairs of objects, A search method in which the aforementioned object relationship information is generated using one or more image data.

8. The acquisition step involves obtaining source image data, A detection step of detecting one or more pairs of objects from the source image data, For the aforementioned pair, a calculation step is performed to calculate relational features that represent the characteristics of the relationship between the objects, A relationship information generation method comprising: a generation step of generating object relationship information in which the pair and the relationship feature quantities calculated for that pair are shown in association.

9. A retrieval step to obtain query data representing the relationship between objects in a pair of objects, A calculation step of calculating relational features that represent the characteristics of the relationships between objects from the aforementioned query data, The computer is made to perform a detection step of detecting a relational feature that matches the calculated relational feature from object relational information, in which the relational feature is associated with each of several pairs of objects. A program that generates the aforementioned object relationship information using one or more image data.

10. The acquisition step involves obtaining source image data, A detection step of detecting one or more pairs of objects from the source image data, For the aforementioned pair, a calculation step is performed to calculate relational features that represent the characteristics of the relationship between the objects, A program that causes a computer to perform a generation step of generating object relationship information in which the pair and the relational feature quantities calculated for that pair are shown in correspondence.