Search device, search method, and program

Abstract

A search device according to the present disclosure: acquires query text data representing an event related to an object, and a plurality of pieces of video data; extracts, from the plurality of pieces of video data, object image sequences, which are time-series data of image regions that contain the same object; generates, for each object, a concatenated object image sequence obtained by concatenating the extracted object image sequences; for each object, uses feature quantities calculated from the concatenated object image sequence of the object and feature quantities calculated from the query text data to generate an associated feature sequence representing associated features in each image region included in the concatenated object image sequence and the query text data; and uses the associated feature sequence generated for each object to detect an object of interest, which is an object that matches the query text data. At least two of the plurality of pieces of video data are video data generated by mutually different cameras.

Description

Search device, search method, and program

[0001] The present disclosure relates to a search device, a search method, and a program.

[0002] Techniques have been developed to obtain information from video data using text data. For example, Non-Patent Document 1 discloses a technique for detecting scenes from video data that match events represented in text data.

[0003] Shen Yan and seven others, "UnLoc: A Unified Framework for Video Localization Tasks," [online], arXiv, August 21, 2023, [accessed January 25, 2024], Internet <URL: https: / / arxiv.org / pdf / 2308.11062.pdf>

[0004] Non-Patent Document 1 assumes that a single video data contains a scene that matches an event represented by text data. The present disclosure has been made in consideration of this problem, and one of its objectives is to provide a new technique for obtaining information from video data using text data.

[0005] A search device according to the present disclosure includes an acquisition means for acquiring query text data representing an event related to an object and a plurality of video data; a first generation means for extracting, from the plurality of video data, an object image sequence that is time-series data of image regions that include the same object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation means for generating, for each object, an associated feature sequence that represents a feature of a relationship between each of the image regions included in the concatenated object image sequence and the query text data, using a feature calculated from the concatenated object image sequence of the object and a feature calculated from the query text data; and a detection means for detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

[0006] A search method according to the present disclosure is executed by a computer and includes: an acquisition step of acquiring query text data representing an event related to an object and a plurality of video data; a first generation step of extracting, from the plurality of video data, object image sequences that are time-series data of image regions each including an identical object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a feature of a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature amount calculated from the concatenated object image sequence of the object and a feature amount calculated from the query text data; and a detection step of detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

[0007] A program according to the present disclosure causes a computer to execute the following steps: an acquisition step of acquiring query text data representing an event related to an object and a plurality of video data; a first generation step of extracting, from the plurality of video data, object image sequences that are time-series data of image regions each including an identical object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature amount calculated from the concatenated object image sequence of the object and a feature amount calculated from the query text data; and a detection step of detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object. At least two of the plurality of video data are video data generated by different cameras.

[0008] According to the present disclosure, new techniques are provided for utilizing text data to derive information from video data.

[0009] FIG. 1 is a diagram illustrating an overview of the operation of a search device. FIG. 2 is a diagram illustrating a connected object image sequence. FIG. 3 is a block diagram illustrating a functional configuration of a search device. FIG. 4 is a block diagram illustrating a hardware configuration of a computer that realizes the search device. FIG. 5 is a flowchart illustrating a flow of processing executed by the search device. FIG. 6 is a diagram illustrating a method for generating an associated feature sequence. FIG. 7 is a diagram illustrating a training method for training an associated feature calculation model and an interesting object identification model. FIG. 8 is a diagram illustrating a training method for training an associated feature calculation model and an associated score calculation model. FIG. 9 is a diagram illustrating an interesting section. FIG. 10 is a diagram illustrating a training method for training an associated feature calculation model and an interesting section identification model.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate explanations will be omitted as necessary for clarity. Furthermore, unless otherwise specified, predetermined values ​​such as predetermined values ​​and threshold values ​​are stored in advance in a storage device accessible from a device that uses the values. Furthermore, unless otherwise specified, the storage unit is composed of one or any number of storage devices.

[0011] <Overview> Fig. 1 is a diagram illustrating an example of an overview of the operation of the search device 2000. Here, Fig. 1 is a diagram for facilitating understanding of the overview of the search device 2000, and the operation of the search device 2000 is not limited to the operation shown in Fig. 1.

[0012] The search device 2000 is used to identify an object that matches a search query from a plurality of video data 20. An object that matches a search query refers to a subject, an object, or both of an event represented by the search query. Hereinafter, an object that matches a search query identified by the search device 2000 is also referred to as an object of interest. Note that at least two of the plurality of video data 20 are video data generated from different cameras 50.

[0013] The search device 2000 acquires query text data 10, which is text data representing a search query. The query text data 10 represents an event related to one or more objects. The event may be an event across multiple cameras 50.

[0014] An event spanning multiple cameras 50 is composed of multiple sub-events that occur in different locations. At least two or more sub-events are captured by different cameras 50. An example of an event composed of multiple sub-events is an event in which "a person comes to a store by bicycle and purchases a product at the store." This event is composed of a first sub-event that occurs outside the store, "a person comes to the store by bicycle," and a second sub-event that occurs inside the store, "the person purchases a product at the store." The first sub-event is captured by a camera 50 installed outside the store. Meanwhile, the second sub-event is captured by a camera 50 installed inside the store. Therefore, in order to identify an object of interest in the above event, it is necessary to use a combination of video data 20 generated by a camera 50 installed outside the store and video data 20 generated by a camera 50 installed inside the store.

[0015] Therefore, the search device 2000 acquires multiple video data 20 and detects one or more objects from the multiple video data 20. Furthermore, the search device 2000 extracts, for each detected object, time-series data of an image area containing the object. Hereinafter, an image area containing a specific object will also be referred to as an object image. Therefore, the time-series data extracted from the video data 20 can also be referred to as time-series data of object images. The time-series data of object images extracted from the video data 20 will also be referred to as an object image sequence 30.

[0016] Here, a plurality of object image sequences 30 for one object can be extracted from a plurality of video data 20. Also, a plurality of object image sequences 30 for one object can be extracted from a single video data 20.

[0017] Therefore, the search device 2000 generates a concatenated object image sequence 40 by concatenating a plurality of object image sequences 30 extracted for each of one or more objects. Note that when only one object image sequence 30 is extracted for a certain object, this object image sequence 30 is treated as the concatenated object image sequence 40.

[0018] 2 is a diagram illustrating a concatenated object image sequence 40. In the example of FIG. 2, an object image sequence 30-1, which is the object image sequence 30 of object X1, an object image sequence 30-2, which is the object image sequence 30 of object X2, and an object image sequence 30-3, which is the object image sequence 30 of object X3, are extracted from video data 20-1. Furthermore, an object image sequence 30-4, which is the object image sequence 30 of object X1, an object image sequence 30-5, which is the object image sequence 30 of object X3, and an object image sequence 30-6, which is the object image sequence 30 of object X4 are extracted from video data 20-2.

[0019] The search device 2000 uses a plurality of object image sequences 30 obtained from the video data 20-1 and the video data 20-2 to generate a connected object image sequence 40. Since there are four objects, objects X1 to X4, detected from the video data 20, a connected object image sequence 40 is generated for each of these four objects.

[0020] Specifically, the retrieval device 2000 generates a concatenated object image sequence 40-1 by concatenating the object image sequence 30-1 and the object image sequence 30-4, which are part of the object image sequence 30 of object X1. Here, the object image sequence 30-1 includes k1 object images, and the object image sequence 30-4 includes k4 object images. Therefore, in the concatenated object image sequence 40-1, the first object image to the k1th object image are composed of the object image sequence 30-1. On the other hand, in the concatenated object image sequence 40-1, the k1th object image to the k1+k4th object image are composed of the object image sequence 30-4.

[0021] Similarly, the search device 2000 generates a concatenated object image sequence 40-3 by concatenating the object image sequence 30-3 and the object image sequence 30-5, which are the object image sequence 30 of the object X3.

[0022] The object image sequence 30 for object X2 consists only of object image sequence 30-2. Therefore, the object image sequence 30-2 is used as is as the connected object image sequence 40-2. Similarly, the object image sequence 30 for object X4 consists only of object image sequence 30-6. Therefore, the object image sequence 30-6 is used as is as the connected object image sequence 40-4.

[0023] The search device 2000 generates a related feature sequence 60 for each connected object image sequence 40 by using a feature calculated from the query text data 10 and a feature calculated from the connected object image sequence 40. The related feature sequence 60 indicates a feature representing a relationship between each object image included in the connected object image sequence 40 and the query text data 10.

[0024] The search device 2000 uses the related feature sequence 60 to determine whether or not the object corresponding to the related feature sequence 60 is an object of interest.

[0025] <Example of Action and Effect> There are events that consist of multiple sub-events that are each captured by multiple different cameras, such as the aforementioned "a person comes to a store by bicycle and then purchases a product at the store." In order to identify the subject or object of such an event from video data, it is necessary to combine and analyze the multiple video data generated by the multiple cameras.

[0026] According to the search device 2000, an object image sequence 30, which is time-series data of an object's image region, is detected from each of the plurality of video data 20 generated by different cameras 50. Furthermore, a concatenated object image sequence 40 is generated for each object by concatenating the object image sequences 30 for that object from the plurality of object image sequences 30 detected from the plurality of video data 20. In this way, the image sequences of objects captured across the plurality of cameras 50 are aggregated into one for each object.

[0027] Furthermore, for each object, the search device 2000 generates a related feature sequence 60 that represents the features of the relationship between the query text data 10 and each object image of that object, based on the feature amount obtained from the query text data 10 and the feature amount obtained from the linked object image sequence 40 of that object. Then, the search device 2000 determines, for each object, whether or not the object is an object that matches the query text data 10 (object of interest), using the related feature sequence 60 of that object.

[0028] In this way, according to the search device 2000, for query text that describes an event spanning multiple cameras 50, the subject or object of the event is identified using multiple video data 20. Therefore, the search device 2000 provides a new technique for obtaining information from video data using text data. Furthermore, according to the search device 2000, the subject or object of an event spanning multiple cameras can be easily detected from multiple video data.

[0029] The search device 2000 of this embodiment will be described in more detail below.

[0030] <Example of Functional Configuration> FIG. 3 is a block diagram illustrating an example of the functional configuration of the search device 2000. For example, the search device 2000 includes an acquisition unit 2020, a first generation unit 2040, a second generation unit 2060, and a detection unit 2080. The acquisition unit 2020 acquires query text data 10 and multiple video data 20. The first generation unit 2040 extracts an object image sequence 30 for each of one or more objects from the multiple video data 20. The first generation unit 2040 further generates, for each object, a concatenated object image sequence 40 in which the object image sequences 30 for that object are concatenated. The second generation unit 2060 generates, for each object, an associated feature sequence 60 based on feature amounts calculated from the query text data 10 and the concatenated object image sequence 40 for that object. The detection unit 2080 detects an object of interest using the associated feature sequence 60 generated for each object.

[0031] <Example of Hardware Configuration> Each functional component of the search device 2000 may be realized by hardware that realizes the respective functional component (e.g., a hardwired electronic circuit, etc.), or may be realized by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it, etc.). Below, a case where each functional component of the search device 2000 is realized by a combination of hardware and software will be further described.

[0032] 4 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the search device 2000. The computer 1000 is any computer. For example, the computer 1000 is a stationary computer such as a PC (Personal Computer) or a server machine. Alternatively, the computer 1000 may be a portable computer such as a smartphone or a tablet terminal. The computer 1000 may be a dedicated computer designed to realize the search device 2000, or may be a general-purpose computer.

[0033] For example, by installing a predetermined application on the computer 1000, the computer 1000 realizes each function of the search device 2000. The application is configured with a program for realizing each functional component of the search device 2000. The method for acquiring the program is arbitrary. For example, the program can be acquired from a storage medium (such as a DVD (Digital Versatile Disc) or a USB (Universal Serial Bus) memory) on which the program is stored. Alternatively, the program can be acquired by downloading the program from a server device that manages the storage device on which the program is stored.

[0034] The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. However, the method of connecting the processor 1040 and the like to each other is not limited to bus connection.

[0035] The processor 1040 is one of various processors, such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device realized using a random access memory (RAM) or the like. The storage device 1080 is an auxiliary storage device realized using a hard disk, a solid state drive (SSD), a memory card, a read only memory (ROM), or the like.

[0036] The input / output interface 1100 is an interface for connecting the computer 1000 to an input / output device. For example, the input / output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

[0037] The network interface 1120 is an interface for connecting the computer 1000 to a network. This network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0038] The storage device 1080 stores a program (a program that realizes the above-mentioned application) that realizes each functional component of the search device 2000. The processor 1040 reads this program into the memory 1060 and executes it to realize each functional component of the search device 2000.

[0039] The search device 2000 may be realized by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same, and can be different from each other.

[0040] <Processing Flow> FIG. 5 is a flowchart illustrating the processing flow executed by the search device 2000. The acquisition unit 2020 acquires query text data 10 (S102). The acquisition unit 2020 acquires multiple video data 20 (S104). The first generation unit 2040 extracts an object image sequence 30 for one or more objects from the multiple video data 20 (S106). The first generation unit 2040 generates a connected object image sequence 40 for each object using the extracted object image sequence 30 (S108). The second generation unit 2060 generates an associated feature sequence 60 for each object using feature amounts calculated from the connected object image sequence 40 of that object and feature amounts calculated from the query text data 10 (S110). The detection unit 2080 detects an object of interest using the associated feature sequence 60 generated for each object (S112).

[0041] <Acquisition of Query Text Data 10: S102> The acquisition unit 2020 acquires the query text data 10 (S102). There are various methods for the acquisition unit 2020 to acquire the query text data 10. For example, the acquisition unit 2020 provides a screen for inputting the query text data 10 to the user of the search device 2000. The acquisition unit 2020 acquires the text input on this screen as the query text data 10. Alternatively, for example, the acquisition unit 2020 acquires the query text data 10 by receiving the query text data 10 transmitted from another device such as a terminal used by the user (for example, a PC or a smartphone).

[0042] Alternatively, for example, the query text data 10 may be stored in advance in a storage unit in a form that allows it to be acquired from the search device 2000. In this case, the acquisition unit 2020 acquires the query text data 10 by reading the query text data 10 from the storage unit.

[0043] Here, the acquiring unit 2020 may acquire query data that includes the query text data 10 and other search conditions. In this case, the query text data 10 is acquired as part of the query data. The method of acquiring the query data is the same as the method of acquiring the query text data 10 described above.

[0044] Various search conditions can be specified by the query data. For example, the search condition indicates a time range to be searched. When a time range is indicated as the search condition, the acquisition unit 2020 acquires each piece of video data generated by the camera 50 within the time range as the video data 20. For example, the acquisition unit 2020 acquires each piece of video data 20 by extracting only frames generated within the time range from each piece of video data generated by the camera 50.

[0045] Alternatively, for example, the search condition may indicate a spatial range (geographical range) to be searched. When a spatial range is indicated as a search condition, the acquisition unit 2020 acquires, as the video data 20, video data generated by each camera 50 located within the spatial range.

[0046] The spatial range is specified by information indicating a location, such as an address or GPS (Global Positioning System) coordinates. In this case, information indicating the installation location of each camera 50 (hereinafter referred to as installation location information) is prepared in advance. The installation location information is stored in advance in an arbitrary storage unit in a manner accessible from the search device 2000. The acquisition unit 2020 uses the installation location information to identify the camera 50 set at the location indicated in the search conditions.

[0047] The spatial range may be specified by the identifier of the camera 50. In other words, the search condition may be a condition that specifies the camera 50. In this case, the acquisition unit 2020 acquires, as the video data 20, video data generated by the camera 50 specified by the search condition.

[0048] Alternatively, for example, the search condition indicates the type of object to be treated as the object of interest. When the type of object is indicated as the search condition, the first generation unit 2040 detects each object belonging to that type from the video data 20. For example, when a person is specified as the type of object of interest, the first generation unit 2040 detects the person from the video data 20.

[0049] <Acquisition of Video Data 20: S104> The acquisition unit 2020 acquires the video data 20 (S104). There are various methods for the acquisition unit 2020 to acquire the video data 20. For example, the video data 20 is stored in advance in an arbitrary storage unit in a format that allows it to be acquired from the search device 2000. In this case, the acquisition unit 2020 acquires the video data 20 from the storage unit.

[0050] The acquisition unit 2020 may acquire all of the video data 20 stored in the storage unit, or may acquire a portion of the video data 20. In the latter case, for example, the acquisition unit 2020 acquires, as the video data 20, video data that matches the above-mentioned search conditions.

[0051] The acquisition unit 2020 may receive video data transmitted from another device such as a terminal used by a user (for example, a PC or a smartphone), and use the received video data as the video data 20 .

[0052] <Generation of object image sequence 30: S106> The first generation unit 2040 generates an object image sequence 30 from each of the plurality of video data 20 (S106). The object image sequence 30 of a certain object extracted from the video data 20 is time-series data of object images of the object extracted from one or more frames constituting the video data 20.

[0053] For example, the first generation unit 2040 performs object detection processing on each frame constituting the video data 20. Furthermore, the first generation unit 2040 identifies objects using a technique such as object tracking across multiple cameras. Object tracking across multiple cameras is performed using a technique such as face matching. In this way, by performing object tracking across multiple cameras, object images detected from the multiple video data 20 are classified by object.

[0054] As described above, an object image is an image region representing an object. For example, the object image is a circumscribing rectangle of the object. However, the object image may include not only an image region representing the object but also an image region representing the periphery of the object. For example, the first generation unit 2040 extracts the object image by adding a margin of a predetermined size or a predetermined ratio to the circumscribing rectangle of the object in each of the height and width directions.

[0055] The first generation unit 2040 may generate the object image sequence 30 using all object images of each object extracted from the video data 20, or may generate the object image sequence 30 using only some of the object images. In the latter case, for example, the acquisition unit 2020 extracts object images from all frames included in the video data 20, and then extracts the extracted object images at predetermined intervals (e.g., one frame every 30 frames) to generate the object image sequence 30. In other words, the first generation unit 2040 may generate the object image sequence 30 by thinning out the extracted object images to a certain extent.

[0056] The object image sequence 30 is composed of a series of object images in time series. It is possible that the same object appears multiple times at different times in the video data 20. In this case, multiple object image sequences 30 for one object may be generated from one piece of video data 20.

[0057] For example, suppose object X1 is detected from n1 frames, from the k1th to the k1+n1th frames, in the video data 20. Furthermore, suppose object X1 is also detected from n2 frames, from the k2th to the k2+n2th frames, in the video data 20 (where k2>k1+n1). In this case, the first generation unit 2040 generates an object image sequence 30 for object X1 using the object images of object X1 detected from each of the k1th to the k1+n1th frames. Furthermore, the first generation unit 2040 generates another object image sequence 30 for object X1 using the object images of object X1 detected from each of the k2th to the k2+n2th frames.

[0058] <Generation of Concatenated Object Image Sequence 40: S108> The first generation unit 2040 generates a concatenated object image sequence 40 for each object (S108). As illustrated in Fig. 2, the concatenated object image sequence 40 is generated by concatenating a plurality of object image sequences 30 for the same object. Furthermore, when only one object image sequence 30 is generated for a certain object, that object image sequence 30 is treated as the concatenated object image sequence 40 for that object.

[0059] Here, multiple connected object image sequences 40 may be generated for one object by dividing the time range into multiple segments. For example, assume that the time range is divided into segments of a predetermined time length P (e.g., P = 10 minutes). Assume also that the start point of the time search range is t1. In this case, the first generation unit 2040 generates the connected object image sequence 40 using each object image sequence 30 generated from frames included in the time range [t1, t1 + P). Here, the time range [t1, t1 + P) has a start point and an end point at t1 and t1 + P, respectively, but does not include t1 + P. The first generation unit 2040 further generates the connected object image sequence 40 using each object image sequence 30 generated from frames included in the time range [t1 + P, t1 + 2 * P). Similar processing is performed for each time range [t1 + i * P, t1 + (i + 1) * P) (i is an integer equal to or greater than 0).

[0060] <Generation of Related Feature Sequence 60: S110> The second generation unit 2060 generates, for each object, the related feature sequence 60 based on the feature amount calculated from the connected object image sequence 40 of that object and the feature amount calculated from the query text data 10 (S110). The related feature sequence 60 is generated using, for example, a machine learning model such as a neural network.

[0061] 6 is a diagram illustrating a method for generating a related feature sequence 60. The feature quantities of the query text data 10 (hereinafter referred to as text feature quantities 120) are calculated using a text encoder 110.

[0062] For example, the text encoder 110 is configured to output the feature quantity of a text token in response to the input of the text token. The text token may be, for example, a word or a subword.

[0063] For example, the second generation unit 2060 divides the query text data 10 into a plurality of text tokens and inputs each text token to the text encoder 110. As a result, the first generation unit 2040 obtains features for each of the plurality of text tokens constituting the query text data 10. As a result, a sequence of features corresponding to the sequence of text tokens obtained from the query text data 10 is generated. For example, the second generation unit 2060 uses the sequence of features as the text features 120.

[0064] The text encoder 110 may be configured to calculate the feature of each text token by taking into account one or more other text tokens before and after it (i.e., by taking into account the context). Such a text encoder 110 may utilize a machine learning model that handles time-series data (e.g., a recurrent neural network (RNN) or a Transformer encoder).

[0065] Alternatively, for example, the text encoder 110 may be configured to output one feature of text (e.g., a sentence) composed of a string of text tokens in response to the input of the string of text tokens. A machine learning model that handles time-series data can also be used for this text encoder 110.

[0066] In this case, for example, the second generation unit 2060 sequentially inputs each text token constituting the query text data 10 to the text encoder 110. Then, the second generation unit 2060 uses, as the text feature 120, data output from the text encoder 110 in response to the input of the last text token.

[0067] The feature quantities of the connected object image sequence 40 are, for example, a sequence of feature quantities of each image constituting the connected object image sequence 40. Hereinafter, feature quantities obtained from an image are also referred to as image feature quantities. Furthermore, feature quantities obtained from the connected object image sequence 40 are also referred to as an image feature sequence. In FIG. 5 , the feature quantities obtained from the connected object image sequence 40 are image feature sequence 140.

[0068] The image feature sequence 140 is generated using an image encoder 130. For example, the image encoder 130 is configured to output the feature quantities of an image in response to an input image. The image encoder 130 may be configured to calculate the feature quantities of each image by taking into account one or more other images before and after the image (i.e., by taking into account the context). Such an image encoder 130 can use a machine learning model that handles time-series data.

[0069] The second generation unit 2060 inputs each of the plurality of object images constituting the connected object image sequence 40 to the image encoder 130. As a result, the second generation unit 2060 obtains a sequence of image features of each object image constituting the connected object image sequence 40. The second generation unit 2060 uses the sequence of image features obtained in this manner as the image feature sequence 140.

[0070] The text encoder 110 and the image encoder 130 are pre-trained to share a feature space. This means that the closer the distance between features obtained from text using the text encoder 110 and features obtained from an image using the image encoder 130, the more similar the situations represented by the text and the image are to each other. For example, the text encoder 110 and the image encoder 130 can be a text encoder and an image encoder, respectively, in a vision and language model.

[0071] The second generation unit 2060 concatenates the text feature 120 and the image feature sequence 140 and inputs the concatenated text feature 120 and the image feature sequence 140 to the related feature calculation model 150. The related feature calculation model 150 is configured as a machine learning model such as a neural network. For example, the related feature calculation model 150 may be configured as the Video-text Fusion model disclosed in Non-Patent Document 1. However, the related feature calculation model 150 may be configured as any machine learning model and is not limited to a model with the same configuration as Video-text Fusion.

[0072] The related feature calculation model 150 is trained in advance along with other models described below. A specific method for training the related feature calculation model 150 will be described later.

[0073] Here, the second generating unit 2060 may add information indicating the order of each object image to the image feature sequence 140. For example, a position code can be used as the information indicating the order of the object images.

[0074] Similarly, when the text feature 120 is a sequence of features, the second generating unit 2060 may add information indicating the order of each text token to the text feature 120. For example, a position code can be used as the information indicating the order of the text tokens.

[0075] <Identifying an Object of Interest: S110> The detection unit 2080 determines, for each object, whether or not the object is an object of interest by using the associated feature sequence 60 generated for that object (S110). As described above, the subject, the object, or both of the event represented by the query text data 10 are objects of interest.

[0076] The determination of whether an object corresponding to the related feature sequence 60 is an object of interest is performed using, for example, a machine learning model such as a neural network. This machine learning model is called an object of interest identification model. For example, the configuration of Head disclosed in Non-Patent Document 1 can be adopted as the configuration of the object of interest identification model. However, the object of interest identification model can be configured using any machine learning model and is not limited to a model with the same configuration as Head.

[0077] The object of interest identification model is configured to output an identification result indicating whether or not an object corresponding to the related feature sequence 60 is an object of interest in response to input of the related feature sequence 60. For example, the object of interest identification model is configured to output 1 if the object corresponding to the related feature sequence 60 is an object of interest, and to output 0 if the object corresponding to the related feature sequence 60 is not an object of interest.

[0078] For example, assume that the related feature sequence 60 has been obtained for each of the objects X1, X2, and X3. In this case, the detection unit 2080 inputs the related feature sequence 60 of the object X1, the related feature sequence 60 of the object X2, and the related feature sequence 60 of the object X3 into the object-of-interest identification model, respectively.

[0079] Here, it is assumed that the object of interest identification model outputs 1 in response to input of the associated feature sequence 60 corresponding to object X1. Furthermore, it is assumed that the object of interest identification model outputs 0 in response to input of the associated feature sequence 60 corresponding to object X2. Furthermore, it is assumed that the object of interest identification model outputs 0 in response to input of the associated feature sequence 60 corresponding to object X3. In this case, it is determined that "the object of interest is object X1."

[0080] The object of interest identification model is trained together with the aforementioned related feature calculation model 150. Figure 7 is a diagram illustrating a training method for training the related feature calculation model 150 and the object of interest identification model. The related feature calculation model 150 and the object of interest identification model 160 are trained using training data 170. The training data 170 includes a combination of a training query 172, a training image sequence 174, and a ground truth flag 176.

[0081] The training query 172 is text data corresponding to the query text data 10. The training query 172 can also be expressed as the query text data 10 used for training.

[0082] The training image sequence 174 is an image sequence that corresponds to the connected object image sequence 40. The training image sequence 174 can also be expressed as the connected object image sequence 40 used for training.

[0083] Ground truth flag 176 is a flag indicating whether training image sequence 174 is a connected object image sequence for the object of interest. If the object corresponding to training image sequence 174 is an object of interest in the event represented by training query 172, ground truth flag 176 indicates 1. On the other hand, if the object corresponding to training image sequence 174 is not an object of interest in the event represented by training query 172, ground truth flag 176 indicates 0. Ground truth flag 176 indicates data that should be output from object of interest identification model 160 for the pair of training query 172 and training image sequence 174.

[0084] An apparatus (hereinafter referred to as a training apparatus) that trains the related feature calculation model 150 and the interested object identification model 160 calculates text features of a training query 172 using a text encoder 110. The training apparatus also calculates image features of a training image sequence 174 using an image encoder 130. The training apparatus concatenates the text features of the training query 172 and the image features of the training image sequence 174, and inputs the concatenated text features to the related feature calculation model 150. Furthermore, the search apparatus 2000 inputs the related feature sequence 60 output from the related feature calculation model 150 to the interested object identification model 160.

[0085] The training device calculates a loss using the data output from the object of interest identification model 160 and the ground truth flag 176, and updates the parameters of the related feature calculation model 150 and the object of interest identification model 160 based on the calculated loss. More specifically, the object of interest identification model 160 is configured to calculate the probability that an object corresponding to the related feature sequence 60 is an object of interest, and to identify whether or not the object corresponding to the related feature sequence 60 is an object of interest based on the probability. The training device inputs the related feature sequence 60 output from the related feature calculation model 150 to the object of interest identification model 160, and obtains from the object of interest identification model 160 the probability that the object corresponding to the related feature sequence 60 is an object of interest. The training device then updates the parameters of the related feature calculation model 150 and the object of interest identification model 160 based on the probability obtained from the object of interest identification model 160 and the ground truth flag 176.

[0086] The training device repeatedly updates the parameters of the related feature calculation model 150 and the interested object identification model 160 using a plurality of training data 170. In this way, the training device trains the related feature calculation model 150 and the interested object identification model 160.

[0087] <Other Processing by the Detecting Unit 2080> The detecting unit 2080 may be configured to perform the following processing, in addition to determining whether each object is an object of interest: (1) Calculating a relevance score; and (2) Identifying an interesting section.

[0088] The relevance score is a value representing the degree of relevance of an object image with the query text data 10. The interesting section is a scene section in the connected object image sequence 40 that has a high degree of relevance with the query text data 10.

[0089] The method for calculating the relevance score and the method for identifying the interesting section will be described below.

[0090] <<Method of Calculating Relevance Score>> The detection unit 2080 uses the associated feature sequence 60 to calculate the relevance score for each object image that constitutes the connected object image sequence 40 .

[0091] The relevance score is calculated using, for example, a machine learning model such as a neural network. This machine learning model is called a relevance score calculation model. For example, the configuration of Head disclosed in Non-Patent Document 1 can be adopted as the configuration of the relevance score calculation model. However, the relevance score calculation model can be configured using any machine learning model and is not limited to a model with the same configuration as Head.

[0092] The relevance score calculation model is configured to output, in response to an input of a related feature sequence 60, a related score for each object image constituting the connected object image sequence 40 corresponding to the related feature sequence 60. For example, if the connected object image sequence 40 is made up of k object images, the relevance score calculation model outputs a related score for each of the k object images.

[0093] The relevance score calculation model is trained together with the previously described relevance feature calculation model 150. Figure 8 illustrates an example of a training method for the relevance feature calculation model 150 and the relevance score calculation model. The relevance feature calculation model 150 and the relevance score calculation model 180 are trained using training data 190. The training data 190 includes a combination of a training query 192, a training image sequence 194, and a plurality of ground truth scores 196.

[0094] The training query 192 and the training image sequence 194 are data corresponding to the training query 172 and the training image sequence 174, respectively. The ground truth score 196 is a ground truth relevance score to be output from the relevance score calculation model 180. The ground truth score 196 is prepared for each of the multiple object images that make up the training image sequence 194.

[0095] In addition, the combination of training query 192 and training image sequence 194 used to train related score calculation model 180 may be the same as the combination of training query 172 and training image sequence 174 used to train object of interest identification model 160.

[0096] The training device obtains the relevant feature sequence 60 from the training query 192 and the training image sequence 194 in a manner similar to the manner in which the relevant feature sequence 60 is obtained from the training query 172 and the training image sequence 174. The training device then inputs the relevant feature sequence 60 into the relevance score calculation model 180 to obtain a relevance score for each object image in the training image sequence 194. The training device calculates a loss using a set of relevance scores obtained from the relevance score calculation model 180 and a set of ground truth scores 196 indicated in the training data 190. The loss can be calculated using, for example, a binary cross-entropy error for each index of the feature sequence. The training device then updates the parameters of the relevant feature calculation model 150 and the relevance score calculation model 180 based on the calculated loss.

[0097] The training device repeatedly updates the parameters of the related feature calculation model 150 and the related score calculation model 180 using a plurality of training data 190. In this way, the training device trains the related feature calculation model 150 and the related score calculation model 180.

[0098] <<Method for Identifying an Interesting Section>> As described above, an interest section is a period of a scene in the connected object image sequence 40 that has a high degree of relevance to the query text data 10. Here, the connected object image sequence 40 may include a plurality of object image sequences 30 each generated by a different camera 50. Therefore, by identifying the interest section, it is possible to identify not only the period of a scene that has a high degree of relevance to the query text data 10, but also the camera 50 that captured the scene that has a high degree of relevance to the query text data 10.

[0099] 9 is a diagram illustrating an example of an interest section. In the example of FIG. 9, video data 20-1, video data 20-2, and video data 20-3 generated by cameras 50-1, 50-2, and 50-3, respectively, are used. From the video data 20-1, video data 20-2, and video data 20-3, object image sequences 30-1, 30-2, and 30-3 are extracted for object X1, respectively. Then, a concatenated object image sequence 40 is generated by concatenating the object image sequences 30-1, 30-2, and 30-3.

[0100] In this example, two intervals of interest have been identified as a result of identifying the interval of interest for the concatenated object image sequence 40. Interest interval 1 is the interval from the f1th object image to the f2th object image in the object image sequence 30-1. The f1th object image and the f2th object image are object images extracted from frames generated at times t1 and t2, respectively. Therefore, interval 1 of interest, expressed using time, is from t1 to t2.

[0101] Section of interest 1 is a section of interest identified from the object image sequence 30-1. Furthermore, the object image sequence 30-1 is extracted from video data 20-1 generated by camera 50-1. Therefore, the camera 50 corresponding to section of interest 1 is camera 50-1, and its camera identifier (ID) is Cam1. Therefore, information about section of interest 1 can be expressed as a tuple of (camera ID, start time, end time) as (Cam1, t1, t2). Hereinafter, information about the section of interest generated by the detection unit 2080 will also be referred to as section of interest information.

[0102] Section of interest 2 is the section from the f3th object image to the f4th object image in the object image sequence 30-3. The f3th object image and the f4th object image are object images extracted from frames generated at times t3 and t4, respectively. Therefore, section of interest 2 expressed using time is from t3 to t4.

[0103] Section of interest 2 is a section of interest identified from the object image sequence 30-3. Furthermore, the object image sequence 30-3 is extracted from video data 20-3 generated by camera 50-3. Therefore, the camera 50 corresponding to section of interest 2 is camera 50-3. The camera ID of camera 50-3 is Cam3. Therefore, the section of interest information for section of interest 2 can be expressed as a tuple of (camera ID, start time, end time), which is (Cam3, t3, t4).

[0104] When the event represented by the query text data 10 is an event captured by multiple cameras 50, multiple sections of interest are identified as shown in FIG. 9 . For example, as described above, assume that the query text data 10 indicates "a person comes to a store by bicycle and purchases a product at the store." In this case, a section of interest corresponding to the scene "a person comes to a store by bicycle" and a section of interest corresponding to the scene "a person purchases a product at the store" are identified from one connected object image sequence 40. This identification makes it easy to understand which camera 50 captured each of the two scenes and when.

[0105] The section of interest is identified using a machine learning model such as a neural network. This machine learning model is called a section of interest identification model. For example, the configuration of Head disclosed in Non-Patent Document 1 can be adopted as the configuration of the section of interest identification model. However, the section of interest identification model can be configured using any machine learning model, and its configuration is not limited to the above-mentioned configuration of Head.

[0106] In response to an input of an associated feature sequence 60, the interesting section identification model identifies an interesting section in the connected object image sequence 40 corresponding to the associated feature sequence 60. For example, the interesting section identification model outputs the start position and end position of the interesting section in the connected object image sequence 40.

[0107] Here, the connected object image sequence 40 may not include a scene highly related to the query text data 10. Therefore, there may be no interesting section in the connected object image sequence 40. Therefore, the interesting section identification model is preferably configured to be able to output data indicating that no interesting section exists. For example, if there is no interesting section in the connected object image sequence 40, the interesting section identification model is configured to output predetermined values ​​(e.g., 0 or a negative value) as the start position and the end position.

[0108] The method for dealing with a case where no interesting section exists in the connected object image sequence 40 is not limited to the above-described method. For example, the search device 2000 may use the above-described relevance score to determine whether or not to treat a section output by the interesting section identification model as an interesting section. For example, the search device 2000 calculates the above-described statistical value of the interest score for the object images included in the section output by the interesting section identification model. If the calculated statistical value is equal to or greater than a threshold, the search device 2000 treats the section as an interesting section. On the other hand, if the calculated statistical value is not equal to or greater than the threshold, the search device 2000 does not treat the section as an interesting section.

[0109] The detection unit 2080 identifies the object image sequence 30 containing the section of interest and the start and end positions of the section of interest in the object image sequence 30, based on the connected object image sequence 40 and the section of interest output from the section of interest identification model. The detection unit 2080 also converts the start and end positions of the section of interest into times. Furthermore, the detection unit 2080 identifies the camera 50 that generated the object image sequence 30 containing the section of interest, based on the correspondence between the object image sequence 30 and the camera 50. As a result, time of interest information indicating a tuple of (camera ID, start time of the section of interest, end time of the section of interest), for example, as described above, is generated.

[0110] The number of interesting sections identified by the interesting section identification model is not limited to one, and the interesting section identification model may be configured to output a plurality of interesting sections.

[0111] The interesting section identification model is trained together with the aforementioned related feature calculation model 150. Fig. 10 is a diagram illustrating a training method for the related feature calculation model 150 and the interesting section identification model. The related feature calculation model 150 and the interesting section identification model 200 are trained using training data 210. The training data 210 includes a combination of a training query 212, a training image sequence 214, and a ground truth section 216.

[0112] The training query 212 and the training image sequence 214 are data corresponding to the training query 172 and the training image sequence 174, respectively. The ground truth section 216 indicates the ground truth section of interest to be output from the section of interest identification model 200 for the training image sequence 214. Note that the combination of the training query 212 and the training image sequence 214 used to train the section of interest identification model 200 may be the same as the combination of the training query 172 and the training image sequence 174 used to train the object of interest identification model 160, or the same combination as the combination of the training query 192 and the training image sequence 194 used to train the relevance score calculation model 180.

[0113] The training device obtains the related feature sequence 60 from the training query 212 and the training image sequence 214 in a manner similar to the manner in which the related feature sequence 60 is obtained from the training query 172 and the training image sequence 174. The training device then inputs the related feature sequence 60 into the interesting section identification model 200 to obtain an interesting section for the training image sequence 194. The training device calculates a loss using the interesting section obtained from the interesting section identification model 200 and the ground truth section 216. For example, the sum of squares of the error between the start time and the end time can be used as the loss. The training device then updates the parameters of the related feature calculation model 150 and the interesting section identification model 200 based on the calculated loss.

[0114] The training device repeatedly updates the parameters of the related feature calculation model 150 and the interesting section identification model 200 using a plurality of training data 210. In this way, the training device trains the related feature calculation model 150 and the interesting section identification model 200.

[0115] <Output by Search Device 2000> The search device 2000 outputs information representing the processing result (hereinafter, output information). The output information includes information related to the identified object of interest. The information related to the object of interest includes, for example, the connected object image sequence 40 of the object of interest. In addition, for example, the information related to the object of interest indicates information such as the identifier of the camera 50 that captured the image of the object of interest and the time when the image of the object of interest was captured (in other words, the time when the object of interest appeared).

[0116] The camera 50 that captured the object of interest is the camera 50 that generated the video data 20 from which the object image sequence 30 of the object of interest was extracted. For each object image sequence 30 included in the concatenated object image sequence 40 of the object of interest, the search device 2000 identifies the camera 50 corresponding to the video data 20 from which the object image sequence 30 was extracted. The output information indicates information on each identified camera 50. For example, the output information indicates each object image sequence 30 included in the concatenated object image sequence 40 of the object of interest in association with the camera 50.

[0117] Here, it is assumed that the object of interest is a person. In this case, the search device 2000 may identify information such as the name of the person who is the object of interest by performing processing such as face matching on the object image of the object of interest. In this case, the output information includes information such as the name of the object of interest.

[0118] The output information may further include information such as an association score, an interesting section, etc. For example, the search device 2000 includes, in the output information, an association score calculated for each object image included in the connected object image sequence 40 of the object of interest.

[0119] In addition, for example, the output information may include information associating an interesting section with a camera identifier. Here, when multiple interesting sections are identified, the output information may indicate all interesting sections or only some of the interesting sections.

[0120] In the latter case, for example, the search device 2000 calculates a relevance score for each section of interest. The relevance score for a section of interest is determined, for example, by a statistical value of the relevance scores for each object image included in the section of interest. For example, the search device 2000 includes in the output information a predetermined number of sections of interest with the highest relevance scores.

[0121] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0122] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0123] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes: (Supplementary Note 1) A search device comprising: an acquisition means for acquiring query text data representing an event related to an object and a plurality of video data; a first generation means for extracting, from the plurality of video data, an object image sequence that is time-series data of image regions including the same object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation means for generating, for each object, an associated feature sequence that represents a feature of a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature amount calculated from the concatenated object image sequence of the object and a feature amount calculated from the query text data; and a detection means for detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras. (Supplementary Note 2) The search device according to Supplementary Note 1, wherein the event represented by the query text data is made up of a plurality of sub-events that each occur in a different location, and at least two of the sub-events are captured by different cameras. (Supplementary Note 3) The search device according to Supplementary Note 1 or 2, wherein the concatenated object image sequence includes a plurality of object image sequences for the same object obtained from different video data. (Supplementary Note 4) The search device according to Supplementary Note 1 or 2, wherein the detection means has an object of interest identification model that determines, in response to input of the related feature sequence, whether an object corresponding to the related feature sequence is the object of interest, and determines, for each object, whether the object is the object of interest by inputting the related feature sequence of the object to the object of interest identification model. (Supplementary Note 5) The search device according to Supplementary Note 1 or 2, wherein the detection means uses the related feature sequence of each object to identify a period of interest that is a period representing a scene that matches the query text data.(Supplementary Note 6) The search device according to Supplementary Note 5, wherein the specifying the period of interest includes: using the related feature sequence of each object to specify one or more of the object image sequences including a sequence of the image regions representing a scene matching the query text data, specifying a camera that generated the video data from which the object image sequence was extracted, and generating period of interest information indicating an association between the camera and the period of interest. (Supplementary Note 7) The search device according to Supplementary Note 1 or 2, wherein the detection means uses the related feature sequence to calculate an association score indicating a degree of association between the image region and the query text data for each of the image regions included in the linked object image sequence used to generate the related feature sequence. (Supplementary Note 8) A computer-executed search method comprising: an acquisition step of acquiring query text data representing an event related to an object and a plurality of video data; a first generation step of extracting from the plurality of video data an object image sequence that is time-series data of image regions including the same object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a feature of a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature amount calculated from the concatenated object image sequence of the object and a feature amount calculated from the query text data; and a detection step of detecting an object of interest that is an object that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras. (Supplementary Note 9) The search method according to Supplementary Note 8, wherein the event represented by the query text data is composed of a plurality of sub-events that occurred at different locations, and at least two of the sub-events are captured by the different cameras.(Supplementary Note 10) A program causing a computer to execute the following steps: an acquisition step of acquiring query text data representing an event related to an object and a plurality of video data; a first generation step of extracting, from the plurality of video data, object image sequences that are time-series data of image regions each including an identical object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a feature of a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature amount calculated from the concatenated object image sequence of the object and a feature amount calculated from the query text data; and a detection step of detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

[0124] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 3 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Note 8 in the same dependency relationship as Supplementary Notes 3 to 7. Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 7 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Note 10 in the same dependency relationship as Supplementary Notes 2 to 7. Some or all of the elements described in any Supplementary Note may be applicable to various hardware, software, recording means for recording software, systems, and methods.

[0125] This application claims priority based on Japanese Patent Application No. 2024-012431, filed January 31, 2024, the disclosure of which is incorporated herein in its entirety by reference.

[0126] 10 Query text data 20 Video data 30 Object image sequence 40 Connected object image sequence 50 Camera 60 Relevant feature sequence 110 Text encoder 120 Text feature 130 Image encoder 140 Image feature sequence 150 Relevant feature calculation model 160 Interest object identification model 170 Training data 172 Training query 174 Training image sequence 176 Ground truth flag 180 Relevant score calculation model 190 Training data 192 Training query 194 Training image sequence 196 Ground truth score 200 Interest zone identification model 210 Training data 212 Training query 214 Training image sequence 216 Ground truth zone 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Search device 2020 Acquisition unit 2040 First generation unit 2060 Second generation unit 2080 Detection unit

Claims

1. A search device comprising: an acquisition means for acquiring query text data representing an event related to an object and a plurality of video data; a first generation means for extracting from the plurality of video data an object image sequence that is time-series data of image regions containing the same object, and generating a concatenated object image sequence for each object by concatenating the extracted object image sequences; a second generation means for generating, for each object, an associated feature sequence that represents a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature calculated from the concatenated object image sequence of the object and a feature calculated from the query text data; and a detection means for detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

2. The search device according to claim 1, wherein the event represented by the query text data is made up of multiple sub-events that occur in different locations, and at least two of the sub-events are captured by different cameras.

3. The search device according to claim 1 or 2, wherein the connected object image sequence includes a plurality of object image sequences of the same object obtained from different sets of video data.

4. The search device according to claim 1 or 2, wherein the detection means has an object of interest identification model that determines whether an object corresponding to the related feature sequence is the object of interest in response to the input of the related feature sequence, and determines for each object whether the object is the object of interest by inputting the related feature sequence of the object into the object of interest identification model.

5. The search device according to claim 1 or 2, wherein the detection means uses the associated feature sequence of each object to identify a period of interest that represents a scene that matches the query text data.

6. The search device described in claim 5, wherein the identification of the period of interest includes: using the related feature sequence of each object to identify one or more object image sequences that include a sequence of image regions that represent a scene that matches the query text data; identifying a camera that generated the video data from which the object image sequence was extracted; and generating period of interest information that represents an association between the camera and the period of interest.

7. The search device according to claim 1 or 2, wherein the detection means uses the related feature sequence to calculate a relevance score representing the degree of relevance between each image region included in the connected object image sequence used to generate the related feature sequence and the query text data.

8. A computer-executed search method comprising: an acquisition step of acquiring query text data representing an event related to an object and a plurality of video data; a first generation step of extracting from the plurality of video data an object image sequence that is time-series data of image regions containing the same object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature calculated from the concatenated object image sequence of the object and a feature calculated from the query text data; and a detection step of detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

9. The search method according to claim 8, wherein the event represented by the query text data is made up of multiple sub-events each occurring at a different location, and at least two of the sub-events are captured by different cameras.

10. A program that causes a computer to execute the following steps: an acquisition step of acquiring query text data that represents an event related to an object and a plurality of video data; a first generation step of extracting from the plurality of video data an object image sequence that is time-series data of image regions that include the same object, and generating, for each object, a concatenated object image sequence by concatenating the extracted object image sequences; a second generation step of generating, for each object, an associated feature sequence that represents a relation between each of the image regions included in the concatenated object image sequence and the query text data, using a feature calculated from the concatenated object image sequence of the object and a feature calculated from the query text data; and a detection step of detecting an object of interest that matches the query text data, using the associated feature sequence generated for each object, wherein at least two of the plurality of video data are video data generated by different cameras.

Images