Search device, search method, and program

Abstract

This search device: acquires query text data and video data; calculates, for each of a plurality of video frames included in the video data, a first association feature amount representing a feature of association between the relationship among objects included in the video frame and the relationship among the objects represented by the query text data; and, by using the first association feature amount of each of the video frames, detects, from the video data, an interest section that matches the content of the query text data.

Description

Search device, search method, and program

[0001] The present disclosure relates to a search device, a search method, and a program.

[0002] Techniques have been developed for obtaining information from video data using text data. For example, Non-Patent Document 1 discloses a technique for detecting segments of video data that represent scenes that match the content of text data.

[0003] In Non-Patent Document 1, features obtained from text data by using a text encoder and features obtained from images by using an image encoder are used to detect sections that represent scenes that match the content of text data. The image features obtained using the image encoder represent the appearance features of each object included in the image.

[0004] Shen Yan and seven others, "UnLoc: A Unified Framework for Video Localization Tasks," [online], arXiv, August 21, 2023, [accessed April 10, 2024], Internet <URL: https: / / arxiv.org / pdf / 2308.11062.pdf>

[0005] The image features that can be used to detect sections that represent scenes that match the content of text data are not limited to features that represent the appearance of objects. This disclosure has been made in light of this problem, and one of its objectives is to provide a new technique for obtaining information from video data using text data.

[0006] A search device according to the present disclosure includes an acquisition means for acquiring query text data and video data, a first calculation means for calculating, for each of a plurality of video frames included in the video data, a first related feature representing a characteristic of the relationship between the relationship between objects included in the video frame and the relationship between the objects represented by the query text data, and a detection means for detecting an interesting section from the video data that matches the content of the query text data using the first related feature of each of the video frames.

[0007] A search method according to the present disclosure, which is executed by a computer, includes an acquisition step of acquiring query text data and video data, a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first related feature representing a characteristic of a relationship between a relationship between objects included in the video frame and a relationship between the objects represented by the query text data, and a detection step of detecting an interesting section from the video data that matches content of the query text data using the first related feature of each of the video frames.

[0008] The program according to the present disclosure causes a computer to execute an acquisition step of acquiring query text data and video data, a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first related feature that represents a characteristic of the relationship between the relationship between objects included in the video frame and the relationship between the objects represented by the query text data, and a detection step of detecting an interesting section from the video data that matches the content of the query text data using the first related feature of each of the video frames.

[0009] According to the present disclosure, new techniques are provided for utilizing text data to derive information from video data.

[0010] FIG. 1 is a diagram illustrating an overview of the operation of a search device. FIG. 2 is a block diagram illustrating an example of the functional configuration of a search device. FIG. 3 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the search device. FIG. 4 is a flowchart illustrating an example of the flow of processing executed by the search device. FIG. 5 is a diagram illustrating an example of a relationship estimation model. FIG. 6 is a diagram illustrating an example of a three-dimensional matrix M representing a relationship feature. FIG. 7 is a diagram illustrating an example of a method for training a relationship estimation model. FIG. 8 is a diagram illustrating an example of a method for generating a first related feature string. FIG. 9 is a first example of the configuration of first linked data. FIG. 10 is a second example of the configuration of first linked data. FIG. 11 is a diagram illustrating an example of a training method for training a first related feature calculation model and an interest section detection model. FIG. 12 is a diagram illustrating an example of a training method for training an association score calculation model. FIG. 13 is a second diagram illustrating an overview of the operation of a search device. FIG. 14 is a second diagram illustrating the configuration of a search device. FIG. 15 is a second flowchart illustrating an example of the flow of processing executed by the search device. FIG. 16 is a diagram illustrating an example of a method for generating a second related feature string. FIG. 17 is a diagram illustrating an example of a training method for training a first related feature calculation model, a second related feature calculation model, and an interest section detection model. FIG. 18 is a second diagram illustrating an example of a training method for training an association score calculation model.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0012] [First Embodiment] <Overview> Fig. 1 is a diagram illustrating an example of an outline of the operation of a search device 2000. Fig. 1 is a diagram for facilitating understanding of the outline of the search device 2000, and the operation of the search device 2000 is not limited to the operation shown in Fig. 1.

[0013] The search device 2000 detects a section that matches the search query from the video data 20. A section of the video data 20 that matches the search query means a section of the video data 20 that represents a scene that meets the conditions specified in the search query. Hereinafter, the section that matches the search query detected by the search device 2000 is also referred to as an interesting section.

[0014] The search device 2000 acquires query text data 10, which is text data representing a search query. The query text data 10 represents one or more relationships between objects. Herein, the "relationship between objects" refers to any relationship between a pair of objects, i.e., a subject and an object. Hereinafter, the relationship between objects will also be referred to as an "inter-object relationship."

[0015] An example of an inter-object relationship represented by the query text data 10 is "a person is sitting on a chair." In this inter-object relationship, the object pair is a pair of a subject, "person," and an object, "chair." The relationship between the subject and the object is "sitting." The search device 2000, which has acquired this query text data 10, detects an interesting section from the video data 20 that represents a scene in which a person is sitting on a chair.

[0016] Relationships between objects can also be expressed as a triplet (subject, object, relationship). A triplet here means a combination of three elements arranged in a specific order (in other words, a tuple with three elements). For example, the relationship between objects "a person is sitting on a chair" can be expressed as a triplet (person, chair, sitting).

[0017] For example, the search device 2000 operates as follows. The search device 2000 acquires query text data 10 and video data 20. The search device 2000 calculates a first related feature 52 for each of a plurality of video frames 22 included in the video data 20. The first related feature 52 calculated for each video frame 22 is data representing a feature of the relationship between the inter-object relationship included in the video frame 22 and the inter-object relationship represented by the query text data 10. The search device 2000 detects an interesting section 60 using the first related feature 52 calculated for each video frame 22.

[0018] A more specific operation of the search device 2000 is as follows, for example. The search device 2000 calculates text features 30, which are features of the query text data 10. The search device 2000 also generates a relation feature sequence 40 by calculating a relation feature 42 for each of a plurality of video frames 22. The relation feature 42 calculated for a video frame 22 is data representing the features of the inter-object relationships included in the video frame 22. The relation feature sequence 40 is a sequence of the calculated plurality of relation feature values ​​42.

[0019] The search device 2000 calculates a first related feature 52 for each of the multiple video frames 22 using the text feature 30 and the related feature sequence 40. This results in a first related feature sequence 50, which is a sequence of the first related features 52. The search device 2000 detects an interesting section 60 from the video data 20 using the first related feature sequence 50.

[0020] Here, the first related feature 52 may be calculated for all video frames 22 included in the video data 20, or may be calculated for some of the video frames 22 included in the video data 20. The same applies to the related feature 42.

[0021] <Example of Action and Effect> For each video frame 22 included in the video data 20, the search device 2000 calculates a relation feature 42 that represents a feature of an inter-object relation included in the video frame 22. Then, using the relation feature 42 calculated for each video frame 22 and the text feature 30 obtained from the query text data 10, an interesting section 60 that represents a scene that matches the content of the query text data 10 is detected from the video data 20. In this way, the search device 2000 provides a new technology that "utilizes the feature of an inter-object relation included in a video frame in order to obtain information from video data using text data."

[0022] Here, the system of Non-Patent Document 1 uses features that represent the appearance features of each object included in the image (hereinafter referred to as image features), but does not use features that represent the characteristics of the relationship between the objects. Therefore, it is considered difficult for the system of Non-Patent Document 1 to accurately distinguish between two images in which "the pair of objects captured is the same but the relationships between the objects are different."

[0023] For example, suppose that image A captures a scene in which a person is sitting on a chair, while image B captures a scene in which a person is standing on the chair.

[0024] Image A and image B have in common the fact that they both capture a person and a chair positioned close to each other. Therefore, there is a high probability that the image features obtained from image A and image B will be similar to each other.

[0025] However, image A and image B represent different scenes, so it is preferable to be able to accurately distinguish between image A and image B in order to accurately detect a scene from video data that matches the content of the text data.

[0026] In this regard, the search device 2000 calculates a relation feature 42 representing the characteristics of the inter-object relationship from the video frame 22, and uses the relation feature 42 to detect the interesting section 60. Here, the inter-object relationship contained in the aforementioned images A and B is different from each other. Specifically, image A represents the inter-object relationship of "a person sitting on a chair," while image B represents the inter-object relationship of "a person standing on a chair." From this, it is considered that the relation feature 42 obtained from image A and the relation feature 42 obtained from image B are not similar to each other. Therefore, the search device 2000 can accurately distinguish between images such as image A and image B, which "include the same pair of captured objects but have different inter-object relationships." Therefore, the search device 2000 can accurately detect, from video data, a scene section that matches the content of the query text data.

[0027] The search device 2000 of this embodiment will be described in more detail below.

[0028] 2 is a block diagram illustrating an example of the functional configuration of the search device 2000. For example, the search device 2000 includes an acquisition unit 2020 and a first calculation unit 2040. The acquisition unit 2020 acquires query text data 10 and video data 20. The first calculation unit 2040 uses the text feature 30 to calculate a first related feature 52 for each of a plurality of video frames 22 included in the video data 20, thereby generating a first related feature sequence 50. The detection unit 2060 uses the first related feature sequence 50 to detect an interesting section 60 from the video data 20.

[0029] <Example of Hardware Configuration> Each functional component of the search device 2000 may be realized by hardware that realizes the respective functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the search device 2000 is realized by a combination of hardware and software will be further described.

[0030] 3 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the search device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the search device 2000, or may be a general-purpose computer.

[0031] For example, by installing a predetermined application on the computer 1000, the computer 1000 realizes each function of the search device 2000. The application is configured with a program for realizing each functional component of the search device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD (Digital Versatile Disc) or a USB (Universal Serial Bus) memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0032] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0033] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0034] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0035] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0036] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the search device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the search device 2000.

[0037] The search device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0038] <Processing Flow> Figure 4 is a flowchart illustrating the flow of processing executed by the search device 2000. The acquisition unit 2020 acquires query text data 10 (S102). The acquisition unit 2020 acquires video data 20 (S104). The first calculation unit 2040 generates a first related feature sequence 50 by calculating a first related feature amount 52 for each of a plurality of video frames 22 (S106). The detection unit 2060 detects an interesting section 60 from the video data 20 using the first related feature sequence 50 (S108).

[0039] <Acquisition of Query Text Data 10: S102> The acquisition unit 2020 acquires the query text data 10 (S102). There are various methods for the acquisition unit 2020 to acquire the query text data 10. For example, the acquisition unit 2020 provides a screen for inputting the query text data 10 to the user of the search device 2000. The acquisition unit 2020 acquires the text input on this screen as the query text data 10. Alternatively, for example, the acquisition unit 2020 acquires the query text data 10 by receiving the query text data 10 transmitted from another device such as a terminal used by the user (for example, a PC or a smartphone).

[0040] Alternatively, for example, the query text data 10 may be stored in advance in a storage unit in a form that allows it to be acquired from the search device 2000. In this case, the acquisition unit 2020 acquires the query text data 10 by reading the query text data 10 from the storage unit.

[0041] As described above, the query text data 10 indicates one or more inter-object relationships. There are various ways to express the inter-object relationships in the query text data 10. For example, the query text data 10 may express the inter-object relationship with a sentence such as "a person is sitting on a chair." Alternatively, the query text data 10 may express the inter-object relationship with a triplet such as (person, chair, sitting).

[0042] The query text data 10 may indicate a plurality of inter-object relationships. When the query text data 10 indicates a plurality of inter-object relationships, the search condition indicated by the query text data 10 is expressed as a logical sum or logical product of the plurality of inter-object relationships.

[0043] When the search condition is expressed as the logical sum of a plurality of inter-object relations, the section of interest 60 is a section of a scene in the video data 20 that includes one or more of the inter-object relations indicated by the query text data 10. For example, assume that the search condition is the logical sum of the inter-object relation "a person is sitting on a chair" and the inter-object relation "a bag is placed on the floor." In this case, the section of interest 60 is a section of a scene that satisfies at least one of the conditions "a person is sitting on a chair" and "a bag is placed on the floor."

[0044] When the search condition is expressed as the logical product of multiple inter-object relations, the section of interest 60 is a section of a scene in the video data 20 that includes all of the multiple inter-object relations indicated by the query text data 10. For example, assume that the search condition is the logical product of the inter-object relation "a person is sitting on a chair" and the inter-object relation "a bag is placed on the floor." In this case, the section of interest 60 is a section of a scene that satisfies both the conditions "a person is sitting on a chair" and "a bag is placed on the floor."

[0045] Whether the logical sum or the logical product of the relationships between multiple objects is to be adopted as a search condition may be predetermined and fixed in the search device 2000, or may be specified in the query text data 10. In the latter case, the logical sum of the relationships between multiple objects may be expressed, for example, as "a person is sitting on a chair, or a bag is placed on the floor." Also, the logical product of the relationships between multiple objects may be expressed, for example, as "a person is sitting on a chair, and a bag is placed on the floor."

[0046] When three or more inter-entity relationships are indicated in the query text data 10, the logical sum and the logical product can be arbitrarily combined.

[0047] Here, the acquiring unit 2020 may acquire data (hereinafter, referred to as query data) that includes the query text data 10 together with other search criteria. In this case, the query text data 10 is acquired as part of the query data. The method for acquiring the query data is the same as the method for acquiring the query text data 10 described above.

[0048] Various search conditions can be specified by the query data. For example, the search conditions are conditions for specifying the video data 20 to be searched. The video data 20 can be specified by, for example, the file name of the video data.

[0049] Alternatively, for example, the search condition may indicate a time range to be searched. When a time range is indicated as the search condition, the acquisition unit 2020 acquires each piece of video data within the time range as the video data 20. For example, the acquisition unit 2020 acquires the video data 20 by extracting only video frames generated within the time range from the video data stored in the storage unit or the video data input by the user.

[0050] Alternatively, for example, the search condition may indicate a spatial range (geographical range) to be searched. When a spatial range is indicated as the search condition, for example, the acquisition unit 2020 acquires, as the video data 20, video data generated by a camera located within the spatial range.

[0051] The spatial range is specified by information indicating a location, such as an address or GPS (Global Positioning System) coordinates. In this case, information indicating the installation location of the camera (hereinafter referred to as installation location information) is prepared in advance. The installation location information is stored in advance in an arbitrary storage unit in a manner accessible from the search device 2000. The acquisition unit 2020 uses the installation location information to identify the camera installed at the location specified in the search criteria.

[0052] The spatial range may be specified by a camera identifier. In other words, the search condition may be a condition specifying a camera. In this case, the acquisition unit 2020 acquires, as the video data 20, video data generated by the camera specified by the search condition.

[0053] <Acquisition of Video Data 20: S104> The acquisition unit 2020 acquires the video data 20 (S104). There are various methods for the acquisition unit 2020 to acquire the video data 20. For example, the video data 20 is stored in advance in an arbitrary storage unit in a format that allows it to be acquired from the search device 2000. In this case, the acquisition unit 2020 acquires the video data 20 from the storage unit.

[0054] The acquisition unit 2020 may acquire all of the video data 20 stored in the storage unit, or may acquire a portion of the video data 20. In the latter case, for example, the acquisition unit 2020 acquires, as the video data 20, video data that matches the search conditions, as described above.

[0055] The acquisition unit 2020 may receive video data transmitted from another device such as a terminal used by a user (for example, a PC or a smartphone), and use the received video data as the video data 20 .

[0056] Here, the acquisition unit 2020 may acquire a plurality of pieces of video data 20. In this case, the acquisition unit 2020 detects the interesting section 60 for each of the plurality of pieces of video data 20. Specifically, the processes of S106 and S108 in the flowchart of FIG. 4 are performed for each piece of video data 20.

[0057] <Generation of First Related Feature Sequence 50: S106> The first calculation unit 2040 generates the first related feature sequence 50 by calculating the first related feature 52 for each of the multiple video frames 22 (S106). For example, as described above, the first related feature sequence 50 is generated using the text feature 30 and the related feature sequence 40. That is, the first calculation unit 2040 calculates the text feature 30 from the query text data 10. The first calculation unit 2040 also calculates the related feature sequence 40 from the video data 20. Then, the first calculation unit 2040 generates the first related feature sequence 50 using the text feature 30 and the related feature sequence 40.

[0058] A specific example of a method for generating the first related feature sequence 50 will be described below.

[0059] <<Calculation of Text Feature 30>> The first calculation unit 2040 calculates the text feature 30, which is a feature of the query text data 10. The text feature 30 is calculated using a machine learning model such as a neural network. Hereinafter, the model that calculates the text feature 30 from the query text data 10 will be referred to as a text feature calculation model. The text feature calculation model may also be referred to as a text encoder or the like.

[0060] For example, the text feature calculation model is configured to output the feature of a text token in response to the input of the text token. The text token may be, for example, a word or a subword.

[0061] In this case, the first calculation unit 2040 divides the query text data 10 into multiple text tokens and inputs each text token to the text feature calculation model. As a result, the first calculation unit 2040 obtains features for each of the multiple text tokens that make up the query text data 10. In this way, a sequence of features is generated from the sequence of text tokens. For example, the first calculation unit 2040 uses the sequence of features as the text features 30.

[0062] The text feature calculation model may be configured to calculate the feature of each text token by taking into account one or more other text tokens before and after it (i.e., by taking into account the context). Such a text feature calculation model can utilize a machine learning model that handles time-series data (e.g., a recurrent neural network (RNN) or a Transformer encoder).

[0063] In this case, for example, the first calculation unit 2040 inputs a string of text tokens obtained from the query text data 10 to the text feature calculation model. As a result, the feature of each text token is output from the text feature calculation model. The first calculation unit 2040 uses the string of features output from the text feature calculation model as the text feature 30.

[0064] Alternatively, for example, the text feature calculation model may be configured to output one feature of text (e.g., a sentence) composed of a string of text tokens in response to the input of the string of text tokens. A machine learning model that handles time-series data can also be used for such a text feature calculation model.

[0065] In this case, for example, the first calculation unit 2040 inputs a string of text tokens obtained from the query text data 10 into the text feature calculation model. Then, the first calculation unit 2040 uses the feature output from the text feature calculation model as the text feature 30.

[0066] <<Generation of Relation Feature Sequence 40>> The first calculation unit 2040 generates the relation feature sequence 40 by calculating a relation feature 42 for each of the multiple video frames 22 included in the video data 20. The video frames 22 for which the relation feature 42 is calculated may be all of the video frames 22 included in the video data 20, or may be some of the video frames 22. In the latter case, for example, the acquisition unit 2020 extracts video frames 22 from the video data 20 at a predetermined interval (e.g., one every 10 frames), and calculates the relation feature 42 for each of the extracted video frames 22.

[0067] The process of calculating the relation feature 42 from the video frame 22 is performed using, for example, a machine learning model such as a neural network. For example, this machine learning model is trained in advance so that, in response to input of the video frame 22, it outputs data representing the inter-object relation included in the video frame 22. Hereinafter, this machine learning model is also referred to as a relation estimation model.

[0068] 5 is a diagram illustrating a relationship inference model. The relationship inference model 70 outputs output data 80 in response to input video frames 22. The output data 80 includes subject data 82, object data 84, and relationship data 86.

[0069] For example, the subject data 82, the object data 84, and the relational data 86 are each composed of a one-dimensional vector, and the output data 80 is a two-dimensional matrix composed of a set of these three one-dimensional vectors.

[0070] The output data 80 probabilistically indicates the relationships between objects included in the video frames 22. The subject data 82 indicates, for each of a plurality of object types, the probability that the subject of the relationships between objects included in the video frames 22 is an object of that type. The number of elements in the subject data 82 is the number of object types that can be detected by the relationship estimation model 70. Each element in the subject data 82 corresponds to a different object type. Then, for each object type, the subject data 82 indicates, in the element corresponding to that type, the probability that the object of that type is the subject.

[0071] For example, suppose the types of objects that can be detected by the relationship inference model 70 are four: "person, chair, bag, and desk." In this case, the first to fourth elements of the subject data 82 correspond to a person, a chair, a bag, and a desk, respectively. The subject data 82 output from the relationship inference model 70 indicates the probability that the subject is a person in the first element, the probability that the subject is a chair in the second element, the probability that the subject is a bag in the third element, and the probability that the subject is a desk in the fourth element.

[0072] The object data 84 indicates, for each of a plurality of object types, the probability that the object of the object relationship included in the video frame 22 is an object of that type. The structure of the object data 84 is similar to the structure of the subject data 82.

[0073] The relationship data 86 indicates, for each of a plurality of types of relationships, the probability that a subject-object relationship in the inter-object relationship included in the video frame 22 is of that type of relationship. The number of elements in the relationship data 86 is the number of types of relationships that can be detected by the relationship estimation model 70. Each element of the relationship data 86 corresponds to a different type of relationship. For each type of relationship, the relationship data 86 indicates, in the element corresponding to that type, the probability that the inter-object relationship included in the video frame 22 is of that type of relationship.

[0074] Here, when the video frame 22 contains multiple inter-object relationships, the relationship estimation model 70 outputs output data 80 for each inter-object relationship.

[0075] The first calculation unit 2040 calculates the relation feature 42 using the output data 80. For example, the first calculation unit 2040 uses the output data 80 as the relation feature 42 as is. When a plurality of pieces of output data 80 are output, for example, the first calculation unit 2040 uses data obtained by adding together the plurality of pieces of output data 80 element by element as the relation feature 42. Here, the "data obtained by adding together the output data 80 element by element" refers to a combination of a vector obtained by adding together the plurality of pieces of subject data 82 element by element, a vector obtained by adding together the plurality of pieces of object data 84 element by element, and a vector obtained by adding together the plurality of pieces of relation data 86 element by element.

[0076] The first calculation unit 2040 may identify inter-object relationships included in the video data 20 using the output data 80, and calculate data indicating the above-mentioned probability only for the identified inter-object relationships as the relationship feature 42. Specifically, the first calculation unit 2040 calculates a first vector, a second vector, and a third vector from the subject data 82, the object data 84, and the relationship data 86, respectively, and uses a combination of these vectors as the relationship feature 42.

[0077] The first vector indicates the maximum value of the element in the subject data 82 and zeros for all other elements. For example, if the subject data 82 is (0.9, 0.05, 0.03, 0.02), the first vector is (0.9, 0, 0, 0). Similarly, the second vector indicates the maximum value of the element in the object data 84 and zeros for all other elements. The third vector indicates the maximum value of the element in the relationship data 86 and zeros for all other elements.

[0078] The relation feature 42, which is formed by a combination of the first vector, the second vector, and the third vector, can be expressed using a three-dimensional matrix. Fig. 6 is a diagram illustrating an example of a three-dimensional matrix M representing the relation feature 42. In this example, the type of object that can be detected from the video frame 22 is type A. Also, the type of relationship that can be detected from the video frame 22 is type B. Therefore, the matrix M is a three-dimensional matrix of AxAxB.

[0079] Here, in the relation feature 42, the first vector indicates a value S1 in the i1-th element, the second vector indicates a value S2 in the j1-th element, and the third vector indicates a value S3 in the k1-th element. In this case, the matrix M indicates a score vector (S1, S2, S3) in the cell at position (i1, j1, k1), and indicates a zero vector (0, 0, 0) in all other cells. The position (i1, j1, k1) represents the inter-object relation detected from the video frame 22. Hereinafter, the above vector (S1, S2, S3) indicating the detected inter-object relation will be referred to as the score vector.

[0080] Note that multiple identical inter-object relationships may be detected from one video frame 22. For example, suppose two people are sitting on chairs in one video frame 22. In this case, two pieces of output data 80 representing the inter-object relationship "a person is sitting on a chair" are obtained from this video frame 22. The first calculation unit 2040 calculates a score vector for each of these two inter-object relationships, and sets the score vector with the larger magnitude to the position (i1, j1, k1).

[0081] As described above, a plurality of inter-object relationships can be detected from the video frame 22. For example, assume that two pieces of output data 80 are obtained by inputting the video frame 22 into the relationship estimation model 70. In this case, the first calculation unit 2040 calculates the values ​​of the cells of the matrix M for each of the two pieces of output data 80.

[0082] For example, suppose that a first vector indicating a probability S11 in the i1-th element, a second vector indicating a probability S12 in the j1-th element, and a third vector indicating a probability S13 in the k1-th element are obtained from the first output data 80. Also, suppose that a first vector indicating a probability S21 in the i2-th element, a second vector indicating a probability S22 in the j2-th element, and a third vector indicating a probability S23 in the k2-th element are obtained from the second output data 80. In this case, the first calculation unit 2040 sets score vectors (S11, S12, S13) in the cell at position (i1, j1, k1) of matrix M. Also, the first calculation unit 2040 sets score vectors (S21, S22, S23) in the cell at position (i2, j2, k2) of matrix M. A zero vector (0, 0, 0) is set in cells other than these two.

[0083] Note that the type of subject may be limited to a specific type (for example, a person) in the inter-object relationships handled by the search device 2000. In this case, the relationship feature amount 42 can be expressed as a two-dimensional matrix.

[0084] The first calculation unit 2040 may treat the matrix M as the relation feature 42 as it is, or may treat a one-dimensional vector obtained by flattening the matrix M as the relation feature 42.

[0085] <<<<Training of the Relationship Inference Model 70>>> Here, a method for training the relationship inference model 70 will be described. Hereinafter, a device that trains the model will be referred to as a training device. The models used by the search device 2000 may be trained using the same training device, or may be trained using different training devices.

[0086] 7 is a diagram illustrating a method for training the relationship estimation model 70. A training device trains the relationship estimation model 70 using training data 110. The training data 110 includes training images 112 and ground truth data 114. The training images 112 are images that include one or more object pairs, similar to the video frames 22. The ground truth data 114 represents the output data 80 that should be output from the relationship estimation model 70 in response to the input of the training images 112.

[0087] The training device calculates a loss based on output data 80 obtained by inputting training images 112 into the relationship estimation model 70 and based on ground truth data 114. The training device then updates trainable parameters (e.g., weights and biases of a neural network) of the relationship estimation model 70 based on the loss.

[0088] The training device trains the relationship estimation model 70 by repeatedly updating the parameters of the relationship estimation model 70 using a plurality of training data 110 .

[0089] <<Generation of First Related Feature Sequence 50>> The first calculation unit 2040 generates the first related feature sequence 50 based on the text feature 30 and the related feature sequence 40. The first related feature sequence 50 is generated using, for example, a machine learning model such as a neural network. Hereinafter, the model used to generate the first related feature sequence 50 is also referred to as a first related feature calculation model.

[0090] 8 is a diagram illustrating a method for generating the first related feature sequence 50. The first calculation unit 2040 inputs data (first linked data 100) obtained by linking the text feature 30 and the related feature sequence 40 to the first related feature calculation model 90. The first related feature calculation model 90 may be configured, for example, using the Video-text Fusion configuration disclosed in Non-Patent Document 1. However, the first related feature calculation model 90 may be configured using any machine learning model and is not limited to a model with the same configuration as Video-text Fusion.

[0091] There are various methods for concatenating the text feature 30 and the relation feature sequence 40 to generate the first concatenated data 100. FIG. 9 shows a first example of the configuration of the first concatenated data 100. In this example, the first concatenated data 100 is obtained by concatenating the text feature 30 with the entire relation feature sequence 40. Note that in this example, the text feature 30 is concatenated at the beginning of the relation feature sequence 40. However, the text feature 30 may also be concatenated at the end of the relation feature sequence 40.

[0092] 10 shows a second example of the configuration of the first linked data 100. In this example, the first linked data 100 is obtained by concatenating the text feature 30 with each relation feature 42 included in the relation feature sequence 40.

[0093] Here, the first calculation unit 2040 may add information indicating a ranking to the first related feature sequence 50. In this case, the ranking assigned to each related feature 42 is the ranking in the video data 20 of the video frame 22 used to generate that related feature 42. For example, assume that four video frames f1, f2, f3, and f4 are extracted from the video data 20 in this order, and a related feature 42 is calculated for each of these video frames. In this case, the related feature 42 calculated from f1 is assigned a ranking of 1. Furthermore, the related feature 42 calculated from f2 is assigned a ranking of 2.

[0094] For example, a position code can be used as information indicating the ranking of the relation feature 42.

[0095] Similarly, when the text feature 30 is a sequence of features, the first calculation unit 2040 may add information indicating the rank of each text token to the text feature 30. For example, a position code can also be used as the information indicating the rank of the text token.

[0096] <Detection of Interesting Section 60: S108> The detection unit 2060 detects the interesting section 60 using the first related feature sequence 50 (S114). Specifically, the detection unit 2060 detects a section from the video data 20 that has a high degree of relevance to the query text data 10 using the first related feature sequence 50. The detected section is then treated as the interesting section 60.

[0097] The detection of the section of interest 60 is performed using, for example, a machine learning model such as a neural network. This machine learning model is also called an interesting section detection model. For example, the configuration of Head disclosed in Non-Patent Document 1 can be adopted as the configuration of the interesting section detection model. However, the interesting section detection model can be configured using any machine learning model, and its configuration is not limited to the above-mentioned Head configuration.

[0098] The interesting section detection model outputs information that can identify an interesting section in response to input of the first related feature sequence 50. The information that can identify an interesting section is, for example, a combination of the start position and end position of the interesting section. The start position of the interesting section is represented, for example, by an identifier (e.g., frame number) of the video frame 22 located at the beginning of the interesting section. Similarly, the end position of the interesting section is represented, for example, by an identifier (e.g., frame number) of the video frame 22 located at the end of the interesting section.

[0099] Here, it is possible that the video data 20 does not include a scene highly related to the query text data 10 (in other words, does not include a scene matching the query text data 10). Therefore, it is possible that no section of interest exists in the video data 20. Therefore, it is preferable that the section of interest detection model be configured to be able to output data indicating that no section of interest exists. For example, if no section of interest exists in the video data 20, the section of interest detection model is configured to output predetermined values ​​(e.g., 0 or a negative value) as the start position and end position.

[0100] The method for dealing with a case where no interesting section exists in the video data 20 is not limited to the above-described method. For example, the detection unit 2060 may calculate a score (hereinafter referred to as a relevance score) representing the degree of relevance with the query text data 10 for each video frame 22. In this case, the detection unit 2060 calculates the statistical value of the above-described relevance score for the video frames 22 included in the section output by the interesting section detection model. If the calculated statistical value is equal to or greater than a threshold, the detection unit 2060 treats the section as an interesting section. On the other hand, if the calculated statistical value is not equal to or greater than the threshold, the detection unit 2060 does not treat the section as an interesting section.

[0101] The relevance score is calculated using, for example, a machine learning model such as a neural network. This machine learning model is called a relevance score calculation model. In response to input of the first relevance feature sequence 50, the relevance score calculation model outputs, for each video frame 22 used to generate the first relevance feature sequence 50, a relevance score that indicates the degree of relevance between the video frame 22 and the query text data 10.

[0102] The relevance score calculation model may be configured using, for example, the Head configuration disclosed in Non-Patent Document 1. However, the relevance score calculation model may be configured using any machine learning model, and its configuration is not limited to the Head configuration described above.

[0103] The number of interesting sections detected by the interesting section detection model is not limited to one, and the interesting section detection model may be configured to output a plurality of interesting sections.

[0104] <<Regarding Model Training>> The interesting section detection model is trained together with, for example, the above-described first related feature calculation model 90. Fig. 11 is a diagram illustrating an example of a training method for training the first related feature calculation model 90 and the interesting section detection model.

[0105] The first related feature calculation model 90 and the interesting section detection model 130 are trained using training data 120. The training data 120 includes a combination of a training query 122, a training image sequence 124, and a ground truth section 126.

[0106] The training query 122 and the training image sequence 124 are data corresponding to the query text data 10 and the video data 20, respectively. The ground truth interval 126 represents a ground truth interval of interest to be output from the interest interval detection model 130 for the combination of the training query 122 and the training image sequence 124. In other words, the ground truth interval 126 indicates a scene interval in the training image sequence 124 that matches the training query 122.

[0107] The training device uses a training query 122 and a training image sequence 124 to obtain first concatenated data 100. That is, the training device calculates text features 30 from the training query 122. The training device also generates a relation feature sequence 40 from the training image sequence 124. The training device then concatenates the text features 30 and the relation feature sequence 40 to generate first concatenated data 100.

[0108] The training device inputs the first linked data 100 into the first related feature calculation model 90 to obtain a first related feature sequence 50. The training device also inputs the first related feature sequence 50 into the interesting section detection model 130 to obtain an interesting section 60.

[0109] The training device calculates a loss using the section of interest 60 and the ground truth section 126. Then, the training device updates the trainable parameters of the first related feature calculation model 90 and the section of interest detection model 130 based on the calculated loss.

[0110] The training device repeatedly updates the parameters of the first related feature calculation model 90 and the interesting section detection model 130 using a plurality of training data 120. In this way, the training device trains the first related feature calculation model 90 and the interesting section detection model 130.

[0111] If the relevance score is used to detect the interesting section, the training device further trains the relevance score calculation model, which can be trained in the same manner as the interesting section detection model 130.

[0112] Figure 12 is a diagram illustrating a method for training a relevance score calculation model. Figure 12 differs from Figure 11 in the following two points. First, training data 120 includes ground truth scores 128 for each training image included in a training image sequence 124, rather than a ground truth section 126. Second, a first relevance feature sequence 50 output from a first relevance feature calculation model 90 is input to a relevance score calculation model 140, which outputs a relevance score 150.

[0113] The ground truth score 128 represents the ground truth relevance score to be output from the relevance score calculation model 140 for the combination of the training query 122 and the training image sequence 124. In other words, the ground truth score 128 represents the degree of relevance between each training image included in the training image sequence 124 and the training query 122.

[0114] The training device inputs first linked data 100 generated using a training query 122 and a training image sequence 124 into a first related feature calculation model 90 to obtain a first related feature sequence 50. The training device further inputs each first related feature 52 included in the first related feature sequence 50 into a related score calculation model 140 to obtain a related score 150 for each training image.

[0115] The training device calculates a loss using the obtained relevance score 150 for each training image and the ground truth score 128 for each training image, and then updates the trainable parameters of the first relevance feature calculation model 90 and the relevance score calculation model 140 based on the calculated loss.

[0116] The training device repeatedly updates the parameters of the first related feature calculation model 90 and the related score calculation model 140 using a plurality of training data 120. In this way, the training device trains the first related feature calculation model 90 and the related score calculation model 140.

[0117] As described above, the first related feature calculation model 90 is trained together with the interesting section detection model 130. Therefore, when training the relevance score calculation model 140, the first related feature calculation model 90 may not be trained (in other words, the parameters of the first related feature calculation model 90 may not be updated).

[0118] <Output by Search Device 2000> The search device 2000 outputs information representing the processing result (hereinafter, output information). The output information includes information about the detected section of interest 60. The information about the section of interest 60 indicates, for example, a pair of the start position and end position of the section of interest 60.

[0119] When multiple sections of interest 60 are detected, the output information may indicate information about all sections of interest 60 or may indicate information about only some of the sections of interest 60. In the latter case, for example, the search device 2000 calculates a score for each section of interest 60. The score of the section of interest 60 is determined, for example, by a statistical value of the relevance scores of each video frame 22 included in the section of interest 60. For example, the search device 2000 includes information about a predetermined number of sections of interest 60 in descending order of score in the output information.

[0120] When the detection of the section of interest 60 is performed for a plurality of video data 20, it is preferable that the output information further indicates an identifier (e.g., a file name) of the video data 20 in which the section of interest 60 is detected. In addition, the output information may indicate information about the camera that generated the video data 20 (such as the camera's identifier and installation location) together with or instead of the identifier of the video data 20.

[0121] By obtaining information about the camera that generated the video data 20 in which the interesting section 60 was detected, the user of the search device 2000 can easily understand when and where the scene matching the query text data 10 was captured. In other words, the user can easily understand the place and time when an event of interest occurred.

[0122] [Embodiment 2] <Overview> Fig. 13 is a second diagram illustrating an overview of the operation of the search device 2000. Note that Fig. 13 is a diagram for facilitating understanding of the overview of the search device 2000, and the operation of the search device 2000 is not limited to the operation shown in Fig. 13.

[0123] The search device 2000 of the second embodiment further generates an image feature sequence 160 from the video data 20. More specifically, the search device 2000 calculates an image feature 162 for each of a plurality of video frames 22 included in the video data 20, thereby generating the image feature sequence 160, which is a sequence of image feature values ​​162. The image feature values ​​162 represent the appearance features of objects included in the video frames 22. Note that the video frames 22 used to calculate the image feature values ​​162 are the same as the video frames 22 used to calculate the relation feature values ​​42.

[0124] Furthermore, the search device 2000 uses the text feature 30 and the image feature sequence 160 to calculate a second related feature 172 for each of the multiple video frames 22. As a result, a second related feature sequence 170, which is a sequence of the second related features 172, is obtained.

[0125] The search device 2000 detects an interesting section 60 from the video data 20 using the first related feature sequence 50 and the second related feature sequence 170 .

[0126] <Example of Effects> According to the search device 2000, in addition to the relation feature 42, the image feature 162 is calculated from the video frame 22. Therefore, the interesting section 60 is detected taking into consideration not only the features of the inter-object relationship included in the video frame 22 but also the appearance features of the objects included in the video frame 22. Therefore, according to the search device 2000, it is possible to detect with higher accuracy sections representing scenes that match the content of the text data from the video data.

[0127] The search device 2000 of this embodiment will be described in more detail below.

[0128] <Example of Functional Configuration> Fig. 14 is a second diagram illustrating the configuration of the search device 2000. In Fig. 14, the search device 2000 includes an acquisition unit 2020, a first calculation unit 2040, a detection unit 2060, and a second calculation unit 2080. The operations of the acquisition unit 2020 and the first calculation unit 2040 in the second embodiment are similar to those in the search device 2000 in the first embodiment.

[0129] The second calculation unit 2080 generates a second related feature sequence 170 using the query text data 10 and the video data 20. The detection unit 2060 of the second embodiment detects an interesting section 60 from the video data 20 using the first related feature sequence 50 and the second related feature sequence 170.

[0130] <Example of Hardware Configuration> The hardware configuration of the search device 2000 of the second embodiment can be represented in Fig. 3, similar to the hardware configuration of the search device 2000 of the first embodiment. However, the storage device 1080 of the second embodiment stores programs for realizing each function of the search device 2000 of the second embodiment.

[0131] <Processing Flow> Figure 15 is a second flowchart illustrating the flow of processing executed by the search device 2000. The acquisition unit 2020 acquires query text data 10 (S202). The acquisition unit 2020 acquires video data 20 (S204). The first calculation unit 2040 calculates a first related feature amount 52 for each of a plurality of video frames 22, thereby generating a first related feature sequence 50 (S206). Note that the processing performed from S202 to S206 is the same as the processing performed from S102 to S106, respectively.

[0132] The second calculation unit 2080 calculates the second related feature quantity 172 for each of the plurality of video frames 22 to generate the second related feature sequence 170 (S208). The detection unit 2060 detects an interesting section 60 from the video data 20 using the first related feature sequence 50 and the second related feature sequence 170 (S210).

[0133] <Calculation of second related feature sequence 170: S208> The second calculation unit 2080 generates the second related feature sequence 170 by calculating the second related feature 172 for each of the multiple video frames 22 (S208). As described above, the second related feature sequence 170 is generated using the text feature 30 and the image feature sequence 160. Therefore, the second calculation unit 2080 generates the image feature sequence 160 from the video data 20, and generates the second related feature sequence 170 using the text feature 30 and the image feature sequence 160.

[0134] A specific example of a method for generating the second related feature sequence 170 will be described below.

[0135] <<Generation of Image Feature Sequence 160>> The second calculation unit 2080 generates the image feature sequence 160 by calculating the image feature 162 for each of the multiple video frames 22 included in the video data 20. The process of calculating the image feature 162 from the video frames 22 is performed using, for example, a machine learning model such as a neural network. This machine learning model is also called an image feature calculation model. The image feature calculation model is also called an image encoder, etc.

[0136] The image feature calculation model is configured to output the feature of an image in response to an input image. The image feature calculation model may be configured to calculate the feature of each image by taking into account one or more other images before and after it (i.e., by taking into account the context). A machine learning model that handles time-series data can be used for such an image feature calculation model.

[0137] The second calculation unit 2080 inputs each of the multiple video frames 22 into the image feature calculation model to obtain a sequence of image feature amounts 162. The second calculation unit 2080 treats the sequence of image feature amounts 162 obtained in this manner as the image feature sequence 160.

[0138] Here, it is preferable that the text feature calculation model and the image feature calculation model are trained so as to share a feature space. In this case, the closer the distance between the features obtained from text using the text feature calculation model and the features obtained from an image using the image feature calculation model, the more similar the situations represented by the text and the image are to each other. For example, the text feature calculation model and the image feature calculation model can be a text encoder and an image encoder in a vision and language model, respectively.

[0139] <<Generation of Second Related Feature Sequence 170>> The second calculation unit 2080 generates the second related feature sequence 170 based on the text feature sequence 30 and the image feature sequence 160. As a method for generating the second related feature sequence 170 based on the text feature sequence 30 and the image feature sequence 160, a method similar to the method for generating the first related feature sequence 50 based on the text feature sequence 30 and the related feature sequence 40 can be used.

[0140] The second related feature sequence 170 is generated using, for example, a machine learning model such as a neural network. Hereinafter, the model used to generate the second related feature sequence 170 is also referred to as a second related feature amount calculation model.

[0141] 16 is a diagram illustrating a method for generating the second related feature sequence 170. The second calculation unit 2080 inputs data (second linked data 180) obtained by linking the text feature sequence 30 and the image feature sequence 160 to the second related feature calculation model 190. The second related feature calculation model 190 can have a configuration similar to that of the first related feature calculation model 90.

[0142] The method of linking the text feature 30 and the image feature sequence 160 to generate the second linked data 180 can be the same as the method of linking the text feature 30 and the relation feature sequence 40 to generate the first linked data 100. In other words, the configuration of the second linked data 180 can be the same as the configuration of the first linked data 100.

[0143] The second calculation unit 2080 may add information indicating ranking, such as a position code, to the first related feature sequence 50 and the text feature quantity 30 .

[0144] <Detection of Interesting Section 60: S210> The detection unit 2060 detects the interesting section 60 from the video data 20 using the first related feature sequence 50 and the second related feature sequence 170 (S210). To detect the interesting section 60, the interesting section detection model 130 is used, as in the first embodiment. However, both the first related feature sequence 50 and the second related feature sequence 170 are input to the interesting section detection model 130 in the second embodiment.

[0145] For example, the detection unit 2060 generates third linked data by linking the first related feature sequence 50 and the second related feature sequence 170, and inputs the third linked data to the interesting section detection model 130. In response to the input of the third linked data, the interesting section detection model 130 outputs information that can identify the interesting section 60.

[0146] The detecting unit 2060 generates the third linked data, for example, by adding together the first related feature amount 52 and the second related feature amount 172 that are ranked the same in the first related feature sequence 50 and the second related feature sequence 170. Alternatively, the detecting unit 2060 generates the third linked data by concatenating the first related feature amount 52 and the second related feature amount 172 that are ranked the same in the first related feature sequence 50 and the second related feature sequence 170. According to these methods, a data sequence having the same length as the first related feature sequence 50 and the second related feature sequence 170 is obtained as the third linked data.

[0147] The detection unit 2060 of embodiment 2 may further calculate an association score, similar to the detection unit 2060 of embodiment 1. For example, the third linked data is input to the association score calculation model 140 of embodiment 2, similar to the interesting section detection model 130 of embodiment 2. In response to the input of the third linked data, the association score calculation model 140 outputs an association score for each video frame 22 used to generate the first associated feature sequence 50 and the second associated feature sequence 170.

[0148] <<Regarding Model Training>> The interesting section detection model 130 is trained, for example, together with the first related feature calculation model 90 and the second related feature calculation model 190. FIG. 17 is a diagram illustrating an example of a training method for training the first related feature calculation model 90, the second related feature calculation model 190, and the interesting section detection model 130.

[0149] The first related feature calculation model 90, the second related feature calculation model 190, and the interesting section detection model 130 can be trained using the training data 120 described above.

[0150] First, the training device generates a first relevant feature sequence 50 using a training query 122 and a training image sequence 124 in the same manner as described in the first embodiment.

[0151] The training device further generates an image feature sequence 160 from the training image sequence 124, and concatenates the image feature sequence 160 with the text features 30 obtained from the training query 122 to obtain second concatenated data 180. The training device then inputs the second concatenated data 180 into a second related feature calculation model 190 to generate a second related feature sequence 170.

[0152] The training device concatenates the first related feature sequence 50 and the second related feature sequence 170 to generate third concatenated data 200. The training device then inputs the third concatenated data 200 into the interesting section detection model 130 to obtain the interesting section 60.

[0153] The training device calculates a loss using the section of interest 60 and the ground truth section 126. Then, the training device updates the trainable parameters of the first related feature calculation model 90, the second related feature calculation model 190, and the section of interest detection model 130 based on the calculated loss.

[0154] The training device repeatedly updates the parameters of the first related feature calculation model 90, the second related feature calculation model 190, and the interesting section detection model 130 using a plurality of training data 120. In this way, the training device trains the first related feature calculation model 90, the second related feature calculation model 190, and the interesting section detection model 130.

[0155] When the relevance score is used to detect the interesting section, the training device further trains the relevance score calculation model 140. Fig. 18 is a second diagram illustrating a method for training the relevance score calculation model 140. Here, the difference between Fig. 18 and Fig. 17 is the same as the difference between Fig. 12 and Fig. 11.

[0156] The training device obtains the third linked data 200 in the same manner as in training the interesting section detection model 130. Furthermore, the training device inputs each piece of data constituting the third linked data 200 (the concatenation of the first related feature 52 and the second related feature 172) into the related score calculation model 140, thereby obtaining the related score 150 for each training image.

[0157] The training device calculates a loss using the relevance score 150 obtained for each training image and the ground truth score 128 for each training image, and then updates the trainable parameters of the first relevance feature calculation model 90, the second relevance feature calculation model 190, and the relevance score calculation model 140 based on the calculated loss.

[0158] The training device repeatedly updates the parameters of the first related feature quantity calculation model 90, the second related feature quantity calculation model 190, and the related score calculation model 140 using a plurality of training data 120. In this way, the training device trains the first related feature quantity calculation model 90, the second related feature quantity calculation model 190, and the related score calculation model 140. However, when training the related score calculation model 140, the first related feature quantity calculation model 90 and the second related feature quantity calculation model 190 may not be trained (in other words, the parameters of the first related feature quantity calculation model 90 and the second related feature quantity calculation model 190 may not be updated).

[0159] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0160] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0161] Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes. (Supplementary Note 1) A search device comprising: an acquisition means for acquiring query text data and video data; a first calculation means for calculating, for each of a plurality of video frames included in the video data, a first related feature representing a characteristic of a relationship between a relationship between objects included in the video frame and a relationship between objects represented by the query text data; and a detection means for detecting, from the video data, an interesting section that matches content of the query text data, using the first related feature of each of the video frames. (Supplementary Note 2) The search device according to Supplementary Note 1, wherein the first calculation means calculates, for each of the plurality of video frames, a related feature representing a characteristic of a relationship between objects included in the video frame, and calculates the first related feature using a text feature obtained from the query text data and the related feature. (Supplementary Note 3) The search device according to Supplementary Note 2, wherein the relation feature indicates, for a pair of objects included in the video frames, a probability that the subject of the pair of objects is a specific type of object, a probability that the object of the pair of objects is a specific type of object, and a probability that the relationship between the pair of objects is a specific type of relationship, respectively. (Supplementary Note 4) The search device according to Supplementary Note 2, wherein the first calculation means calculates the first relation feature for each of the plurality of video frames by concatenating a string of the relation feature and the text feature obtained from the plurality of video frames and converting the first concatenated data into a string of features. (Supplementary Note 5) The search device according to any one of Supplements 1 to 4, further comprising: second calculation means for calculating, for each of the plurality of video frames, a second relation feature representing a feature of a relationship between an object included in the video frame and an object represented by query text data,(Supplementary Note 6) The search device according to Supplementary Note 5, wherein the second calculation means calculates, for each of the plurality of video frames, an image feature that is a feature of the appearance of an object included in the video frame, and calculates the second related feature using a text feature obtained from the query text data and the image feature. (Supplementary Note 7) The search device according to Supplementary Note 6, wherein the second calculation means generates second linked data by concatenating a string of the image feature and the text feature obtained from the plurality of video frames, and calculates the second related feature for each of the plurality of video frames by converting the second linked data into a string of features. (Supplementary Note 8) A search method executed by a computer, comprising: an acquisition step of acquiring query text data and video data; a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first related feature that represents a feature of a relationship between a relationship between objects included in the video frame and a relationship between the objects represented by the query text data; and a detection step of detecting an interesting section that matches content of the query text data from the video data using the first related feature of each of the video frames. (Supplementary Note 9) The search method according to Supplementary Note 8, wherein in the first calculation step, for each of the plurality of video frames, a relation feature representing a feature of a relation between objects included in the video frame is calculated, and the first relation feature is calculated using a text feature obtained from the query text data and the relation feature. (Supplementary Note 10) A program that causes a computer to execute the following steps: an acquisition step of acquiring query text data and video data; a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first relation feature representing a feature of a relation between the relationship between objects included in the video frame and the relationship between the objects represented by the query text data; and a detection step of detecting an interesting section that matches content of the query text data from the video data, using the first relation feature of each of the video frames.

[0162] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 3 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Note 8 in the same dependency relationship as Supplementary Notes 3 to 7. Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Note 10 in the same dependency relationship as Supplementary Notes 2 to 7. Some or all of the elements described in any Supplementary Note may be applicable to various hardware, software, recording means for recording software, systems, and methods.

[0163] This application claims priority based on Japanese Patent Application No. 2024-069843, filed April 23, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0164] 10 Query text data 20 Video data 22 Video frame 30 Text feature 40 Relation feature sequence 42 Relation feature 50 First relation feature sequence 52 First relation feature 60 Interest section 70 Relation estimation model 80 Output data 82 Subject data 84 Object data 86 Relation data 90 First relation feature calculation model 100 First linked data 110 Training data 112 Training image 114 Ground truth data 120 Training data 122 Training query 124 Training image sequence 126 Ground truth section 128 Ground truth score 130 Interest section detection model 140 Relation score calculation model 150 Relation score 160 Image feature sequence 162 Image feature 170 Second relation feature sequence 170 172 Second relation feature 172 180 Second linked data 190 Second related feature calculation model 200 Third linked data 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Search device 2020 Acquisition unit 2040 First calculation unit 2060 Detection unit 2080 Second calculation unit

Claims

1. A search device comprising: an acquisition means for acquiring query text data and video data; a first calculation means for calculating, for each of a plurality of video frames included in the video data, a first related feature representing a characteristic of the relationship between the relationship between objects included in the video frame and the relationship between objects represented by the query text data; and a detection means for detecting an interesting section from the video data that matches the content of the query text data using the first related feature of each of the video frames.

2. The search device described in claim 1, wherein the first calculation means calculates, for each of the plurality of video frames, a relation feature that represents a characteristic of the relationship between objects contained in that video frame, and calculates the first relation feature using the text feature obtained from the query text data and the relation feature.

3. The search device described in claim 2, wherein the relationship features indicate, for a pair of objects included in the video frame, the probability that the subject of the pair of objects is a specific type of object, the probability that the object of the pair of objects is a specific type of object, and the probability that the relationship between the pair of objects is a specific type of relationship.

4. The search device described in claim 2, wherein the first calculation means generates first linked data by linking the string of related features obtained from the plurality of video frames with the text features, and calculates the first related features for each of the plurality of video frames by converting the first linked data into a string of features.

5. A search device as described in any one of claims 1 to 4, further comprising a second calculation means for calculating, for each of the plurality of video frames, a second related feature representing a characteristic of the relationship between an object included in that video frame and an object represented by query text data, wherein the detection means detects the interesting section from the video data using the first related feature of each of the video frames and the second related feature of each of the video frames.

6. The search device described in claim 5, wherein the second calculation means calculates, for each of the plurality of video frames, image features which are features of the appearance of an object contained in that video frame, and calculates the second related features using text features obtained from the query text data and the image features.

7. The search device described in claim 6, wherein the second calculation means generates second linked data by linking the string of image features obtained from the plurality of video frames with the text features, and calculates the second related features for each of the plurality of video frames by converting the second linked data into a string of features.

8. A search method executed by a computer, comprising: an acquisition step of acquiring query text data and video data; a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first related feature representing a characteristic of the relationship between the relationship between objects included in the video frame and the relationship between the objects represented by the query text data; and a detection step of detecting an interesting section from the video data that matches the content of the query text data using the first related feature of each of the video frames.

9. The search method described in claim 8, wherein in the first calculation step, a relation feature representing a characteristic of a relationship between objects contained in each of the plurality of video frames is calculated, and the first relation feature is calculated using the text feature obtained from the query text data and the relation feature.

10. A program that causes a computer to execute the following steps: an acquisition step of acquiring query text data and video data; a first calculation step of calculating, for each of a plurality of video frames included in the video data, a first related feature that represents a characteristic of the relationship between the relationship between objects included in the video frame and the relationship between the objects represented by the query text data; and a detection step of detecting an interesting section from the video data that matches the content of the query text data using the first related feature of each of the video frames.

Images