Video metadata structuring device and program

Abstract

This invention provides a video metadata structuring device that reduces the data volume required to manage metadata and generates metadata-structured data that allows for quick access to the metadata. [Solution] The video metadata structuring device 1 includes an object recognition unit 10 that performs object recognition in the time direction of the video and generates a track ID, an identification candidate ID, and a plurality of recognition results for individual objects as metadata; a metadata encoding unit 21 that converts the values ​​of the plurality of recognition results for individual objects generated by the object recognition unit 10 into a data sequence of predetermined bit lengths; and a sparse tensor data embedding unit 22 that embeds the data sequence of the corresponding metadata into a sparse tensor data structure with the time-based track ID and identification candidate ID generated by the object recognition unit 10 as multidimensional axes.

Description

Technical Field

[0001] The present invention relates to a video metadata structuring device and its program.

Background Art

[0002] Conventionally, as a method for managing (registering and searching) video metadata, a method of table management using a database has been generally used. At this time, a method of a transposed index (inverted index) that manages various data related to temporal changes in a video using the temporal information of the video as key information is used. Information related to object information in a video is also mainly managed within the framework of this transposed index. For example, a method of associating and managing a plurality of metadata for an object in a video has been proposed (see Patent Document 1). Also, a mechanism for encapsulating metadata for an object in a video in the same file as the video and managing it has been proposed (see Patent Document 2).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the method of managing metadata in a database as in the prior art has a problem that it takes time to perform complex data operations between tables as the amount of video data increases. In addition, Patent Document 1 does not have a specific description of the data structure. Also, Patent Documents 1 and 2 do not have a description of the mechanism for accessing the data structure. Therefore, there was a need for a method to generate metadata using a data structure that could efficiently handle the metadata of video objects.

[0005] This invention has been made in view of the above problems and needs, and aims to provide a video metadata structuring device and program that reduces the data capacity for managing metadata and generates metadata structured data that allows quick access to metadata. [Means for solving the problem]

[0006] To solve the aforementioned problems, the video metadata structuring device according to the present invention is a video metadata structuring device that structures metadata for objects in video as sparse tensor data, and comprises an object recognition unit, a metadata encoding unit, and a sparse tensor data embedding unit.

[0007] In this configuration, the video metadata structuring device uses an object recognition unit to perform object recognition in the temporal direction of the video and generates a track ID, an identification candidate ID, and multiple recognition results for individual objects as metadata. The objects to be recognized are predetermined and include, for example, objects, people, etc.

[0008] The video metadata structuring device then uses a metadata encoding unit to convert the multiple recognition results of individual objects generated by the object recognition unit into a data sequence of predetermined bit lengths. For example, by making the data sequence 64 bits, it becomes possible to access the data with a single access across various platforms.

[0009] The video metadata structuring device then uses a sparse tensor data embedding unit to embed data sequences of the corresponding metadata into a sparse tensor data structure whose multidimensional axes are the time-based track IDs and candidate identification IDs generated by the object recognition unit. As a result, the video metadata structuring device constructs a sparse tensor-type data structure in which metadata represented by locations in a multidimensional space is arranged as a series of data scattered in parts of the space, thus allowing it to exclude data in spaces that do not correspond to the metadata. Furthermore, the video metadata structuring device can be operated using a program that enables a computer to function as a video metadata structuring device. [Effects of the Invention]

[0010] According to the present invention, metadata for objects in video can be structured using a sparse tensor data structure. Since this structured metadata data consists only of data with actually valid values, it reduces data size and allows for quick access. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the configuration of a video metadata structuring device according to the first embodiment of the present invention. [Figure 2] This is an explanatory diagram illustrating the processing overview of a video metadata structuring device according to the first embodiment of the present invention. [Figure 3] Figure 2 is a data structure diagram showing an example of the configuration of a data sequence in which metadata has been encoded by the metadata encoding unit. [Figure 4] Figure 2, which shows the sparse tensor data embedding section, is an explanatory diagram illustrating a method for embedding metadata data sequences into a sparse tensor data structure. [Figure 5] This is a flowchart illustrating the operation of the video metadata structuring device according to the first embodiment of the present invention. [Figure 6] This is a block diagram showing the configuration of a video metadata structuring device according to a second embodiment of the present invention. [Figure 7] Figure 6, which shows the sparse tensor data embedding section, is an explanatory diagram illustrating a method for embedding metadata data sequences into a sparse tensor data structure.

Embodiments for Carrying out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. <First Embodiment> [Configuration of Video Metadata Structuring Device] Referring to FIG. 1, the configuration of a video metadata structuring device 1 according to the first embodiment of the present invention will be described.

[0013] The video metadata structuring device 1 structures metadata for objects in a video as data in a sparse tensor format. That is, the video metadata structuring device 1 inputs a video, recognizes objects such as people in the video, structures metadata for identifying the objects as data in a sparse tensor format, and outputs it as metadata structured data.

[0014] Here, referring to FIG. 2, the outline of the processing of the video metadata structuring device 1 will be described. The video metadata structuring device 1 roughly performs recognition processing S1, encoding processing S2, and embedding processing S3.

[0015] The recognition processing S1 is processing for recognizing a video and generating metadata m of an object in the video as a recognition result. The metadata m has a plurality of pieces of information such as a track ID, an identification candidate ID, an identification score, an identification rank, and region information, and varies depending on the object to be recognized. Examples of the metadata m will be described later.

[0016] The encoding processing S2 is processing for arranging a plurality of pieces of metadata m for each object generated in the recognition processing S1 in a previously allocated area of a data sequence having a predetermined bit length. That is, the encoding processing S2 arranges a plurality of pieces of metadata m for each object as signed integer data in a data sequence ev in int64 format. This data sequence ev is generated for the number of recognized objects.

[0017] The embedding process S3 embeds the data sequence ev generated in the encoding process S2 into a sparse tensor data structure td. The sparse tensor data structure td is a data structure that holds non-zero values ​​(in this case, the data column ev) and their corresponding axis values ​​in a dataset that is mostly filled with zero values. Note that in Figure 2, the sparse tensor data structure td is shown as "0", but in reality, it would only contain the data column ev. In Figure 2, "0" is illustrated to illustrate the concept of the sparse tensor data structure. This allows the video metadata structuring device 1 to structure the video metadata in a sparse tensor data structure.

[0018] Returning to Figure 1, the configuration of the video metadata structuring device 1 will be explained in detail. As shown in Figure 1, the video metadata structuring device 1 comprises an object recognition unit 10 and a data structuring unit 20.

[0019] The object recognition unit 10 performs object recognition in the time direction of the video and generates a track ID, an identification candidate ID, and multiple recognition results for individual objects as metadata. The objects that the object recognition unit 10 recognizes are not limited to objects, people, etc.

[0020] Here, a person's face is described as an example of an object. That is, the object recognition unit 10 is described as a face recognition device. Note that the object recognition process can be done using existing methods, so the explanation is omitted here. For example, various metadata such as time information, track ID, and identification candidate ID can be generated by face recognition using the method described in Japanese Patent Application Publication No. 2022-112831. Here, we will describe an example of metadata generated by the object recognition unit 10.

[0021] "Time information (frid)" indicates the time taken for the recognition process on the video, and is represented as an integer value counted in ascending order from "0". Here, recognition is performed in frames at 1-second intervals. A "track ID" is an ID that identifies the same object over time. In this case, the track ID is an ID that identifies the facial region of the same person. The "recognition candidate ID (recogid)" indicates the ID assigned to the object as a result of the identification process. In this case, the recognition candidate ID is the ID assigned to candidates for people. A predetermined number of these recognition candidate IDs are generated, ordered from highest to lowest recognition score.

[0022] The "recognition score (recog_score)" indicates the confidence level of the candidate identification ID during the tracking period. The "Identification Rank (recog_rank)" indicates a value that ranks the identification scores in descending order. The "Current Recognition Score (cur_score)" indicates the confidence level of identifying the candidate person based on current time information (current frid).

[0023] "Face region information (bbox)" indicates information about the bounding box surrounding the face region. Here, bbox is assumed to be a square, and it represents the center position (objx, objy) and width (objsize) of the face region (bbox=[objx, objy, objsize]). The "Face Region Confidence Score (det_score)" indicates the confidence level of the face region. The object recognition unit 10 outputs the recognized metadata to the data structuring unit 20.

[0024] The data structuring unit 20 structures the metadata generated by the object recognition unit 10 as sparse tensor data. The data structuring unit 20 comprises a metadata encoding unit 21 and a sparse tensor data embedding unit 22.

[0025] The metadata encoding unit 21 converts the values ​​of multiple individual object recognition results generated by the object recognition unit 10 into a data sequence of predetermined bit lengths. The data sequence may be, for example, a 64-bit signed integer. Figure 3 illustrates an example of the data sequence ev generated by the metadata encoding unit 21. The example in Figure 3 shows an encoding example where the track ID (trackid) is 16 bits, the recognition score (recog_score) is 6 bits, the current recognition score (cur_score) is 6 bits, the identification rank (recog_rank) is 5 bits, the face region confidence score (det_score) is 6 bits, the center position (objx, objy), which is face region information, is 7 bits each, and the width (objsize) is 7 bits.

[0026] When the metadata structured data generated by this video metadata structuring device 1 is used as training data for object recognition, the metadata encoding unit 21 may encode bits for training data into the first two bits of the data sequence, as shown in Figure 3. Here, the most significant bit (1 bit) is the correct bit (istrue), and the second most significant bit (second most significant bit: 1 bit) is the bit (cur_top) that indicates whether the recognition result in the current time information (current frame) is the optimal candidate.

[0027] In this case, the metadata encoding unit 21 receives the correct values ​​for object-specific metadata from an external source on a frame-by-frame basis. The metadata encoding unit 21 then sets the correct bit to "1" if it matches the object-specific metadata generated by the object recognition unit 10, and sets the correct bit to "0" if it does not match. By setting the most significant bit to "1" in this way, the data sequence ev can be treated as a negative number. The benefits of data analysis using this negative number conversion will be described later.

[0028] Furthermore, the metadata encoding unit 21 sets the second most significant bit to "1" if the recognition rank (recog_rank), which is the recognition result in the current time information (current frame), is 1st, and sets the second most significant bit to "0" otherwise. This makes it possible to determine by bits whether or not the data sequence ev has an identification rank of 1st.

[0029] Returning to Figure 1, we will continue our explanation of the configuration of the video metadata structuring device 1. The metadata encoding unit 21 outputs the generated data sequence to the sparse tensor data embedding unit 22.

[0030] The sparse tensor data embedding unit 22 embeds a data sequence of metadata generated by the metadata encoding unit 21 into a sparse tensor data structure whose multidimensional axes are the track ID (trackid) and candidate identification ID (recogid) for each time period (time information [frid]) generated by the object recognition unit 10. Here, the data assigned to the multidimensional axes of the sparse tensor data structure are the track ID, time information, and candidate identification ID. The maximum dimensional length of each axis is set to a value that allows sufficient data output by the object recognition unit 10 to be assigned.

[0031] For example, the maximum dimensional length of a track ID is set to a number greater than or equal to the number of tracks appearing simultaneously, for example, "2000". The maximum dimensional length of a candidate identification ID is set to the total number of people pre-prepared for person recognition, for example, "20000". Furthermore, the maximum dimension length of the time information should be a number greater than or equal to the maximum duration of the video, for example, "3600". In this case, the duration of the video will be 1 hour (3600 seconds), but if you need to handle videos longer than 1 hour, you may add another axis.

[0032] Here, referring to Figure 4, we will schematically explain the method by which the sparse tensor data embedding unit 22 embeds metadata data sequences into the sparse tensor data structure td. Figure 4 shows the data structure with track ID, time information, and candidate identification ID as axes. The horizontal axis represents time information, the vertical axis represents recognition code ID, and the depth axis represents track ID. Furthermore, in addition to the time information for one hour, a time axis is provided for videos exceeding one hour in length.

[0033] Specifically, when the maximum dimension length of (track ID [trackid], time information [frid], candidate identification ID [recogid]) is (2000, 10*3600, 20000), the metadata structured data will be defined by a 4-dimensional sparse tensor data structure as shown in equation (1) below.

[0034]

number

[0035] Here, the time information is decomposed into 10 dimensions on the second axis and 3600 dimensions on the fourth axis. The definition formulas for the values ​​on the second axis (2_dim) and the fourth axis (4_dim) are given by equation (2) below.

[0036]

number

[0037] In equation (2), " / / " indicates rounding down division, and "%" indicates the remainder of the division. The sparse tensor data embedding unit 22 substitutes the data sequence ev (Figure 3), which has the remaining metadata encoded by the metadata encoding unit 21, into the index position of the sparse tensor data identified by (track ID [trackid], time information [frid], identification candidate ID [recogid]).

[0038] In other words, when the value of the data column ev is defined as enc_val, the sparse tensor data embedding unit 22 performs the process of setting the corresponding enc_val at the index position of the sparse tensor format of the metadata structured data defined by equation (1) for all recognition results of the object recognition unit 10, as shown in equation (3).

[0039]

number

[0040] The metadata structured data V_tensor, configured in this way, is generated as part of the data within the multidimensional space covered by the sparse tensor data.

[0041] Returning to Figure 1, we will continue our explanation of the configuration of the video metadata structuring device 1. The sparse tensor data embedding unit 22 embeds the data sequence generated by the metadata encoding unit 21 into the corresponding index positions on the multidimensional axes of the sparse tensor data, and outputs it externally as metadata structure data.

[0042] With the configuration described above, the video metadata structuring device 1 can structure metadata for objects in video using a sparse tensor data structure. Since the structured metadata data thus is composed only of data with actually valid values, it can reduce data size and be accessed quickly.

[0043] [Operation of the video metadata structuring device] Next, with reference to Figure 5 (and Figure 1 as appropriate for the configuration), the operation of the video metadata structuring device 1 according to the first embodiment of the present invention will be described.

[0044] In step S10, the object recognition unit 10 inputs video frames at predetermined time intervals (in this case, 1 second). In step S11, the object recognition unit 10 performs object recognition on the frames that are sequentially input in step S10. In step S11, the object recognition unit 10 performs object recognition in the time direction and generates metadata including the object's track ID and identification candidate ID, and multiple individual recognition results for each identification candidate ID.

[0045] In step S12, the metadata encoding unit 21 encodes the values ​​of multiple individual object recognition results from the metadata generated in step S11 into a data sequence of a predetermined bit length. Here, the metadata encoding unit 21 encodes them into 64-bit signed integer data (int64). In step S13, the sparse tensor data embedding unit 22 embeds the metadata data sequence generated by the metadata encoding unit 21 into a sparse tensor data structure whose multidimensional axes are the track ID (trackid) and candidate identification ID (recogid) for each time (time information [frid]) generated by the object recognition unit 10. In step S14, the sparse tensor data embedding unit 22 outputs metadata structured data with embedded data sequences to the outside.

[0046] In step S15, the object recognition unit 10 determines whether the recognition process for all frames of the video has been completed based on whether or not the next frame exists. If a next frame exists and processing of all video frames is not yet complete (No in step S15), the video metadata structuring device 1 returns to step S10 and continues its operation.

[0047] On the other hand, if there is no next frame and processing of all video frames is complete (Yes in step S15), the video metadata structuring device 1 terminates its operation. Through the above operations, the video metadata structuring device 1 can structure metadata for objects in the video using a sparse tensor data structure.

[0048] <Second Embodiment> [Video metadata structuring device configuration] Next, with reference to Figure 6, the configuration of the video metadata structuring device 1B according to the second embodiment of the present invention will be described.

[0049] The video metadata structuring device 1B structures metadata for objects in video as sparse tensor data. The video metadata structuring device 1B structures metadata for multiple video files.

[0050] As shown in Figure 6, the video metadata structuring device 1B comprises an object recognition unit 10, a video analysis unit 11, and a data structuring unit 20B. The object recognition unit 10 has the same configuration as the video metadata structuring device 1 (Figure 1), so its explanation is omitted.

[0051] The video analysis unit 11 analyzes the video and generates metadata for the entire video. While the object recognition unit 10 generates metadata for each frame, the video analysis unit 11 generates metadata for the entire video. Specifically, the video analysis unit 11 generates metadata such as the end time of each video when multiple videos are played consecutively, and cut times representing shot boundaries within the videos. Note that the metadata generation process for videos can be performed using existing methods, so a detailed explanation is omitted here. For example, the method described in Japanese Patent Publication No. 2022-112831, etc., can be used. The video analysis unit 11 outputs metadata for the entire video to the data structuring unit 20B.

[0052] The data structuring unit 20B structures the metadata generated by the object recognition unit 10 and the metadata generated by the video analysis unit 11 as sparse tensor data. The data structuring unit 20B comprises a metadata encoding unit 21 and a sparse tensor data embedding unit 22B. The metadata encoding unit 21 has the same configuration as the video metadata structuring device 1 (Figure 1) described in Figure 1, so its description is omitted.

[0053] The sparse tensor data embedding unit 22B embeds the metadata data sequence generated by the metadata encoding unit 21 into the sparse tensor data structure. The sparse tensor data structure handled by the sparse tensor data embedding unit 22B is basically the same as the sparse tensor data structure td described in Figure 4. However, since there are multiple input videos, it is necessary to extend it to identify the videos by vid, as shown in the sparse tensor data structure td2 in Figure 7. In this case, the definition of the sparse tensor data structure defined in equation (1) above can be extended to the definition in equation (4) below.

[0054]

number

[0055] N_vids is the total number of video clips, and the numerical value vid, which represents its dimension, is an identifier (ID) used to identify each video clip. Here, the sparse tensor data embedding unit 22B supports multiple images by expanding the number of dimensions in the definition of equation (1).

[0056] Specifically, the sparse tensor data embedding unit 22B adds the metadata of the entire video generated by the video analysis unit 11, along with the track ID [trackid], to the dimensions of the candidate identification ID [recogid] to generate metadata structured data.

[0057] The sparse tensor data embedding unit 22B takes the metadata input from the video analysis unit 11, namely the end time of the video and the cut time, and inserts "1" for the time the video ended and the cut number (cut No.) for the cut time into the dimensions of the time information [frid] where the respective events occurred. In other words, the sparse tensor data embedding unit 22B is defined by a four-dimensional sparse tensor data structure as shown in equation (5) below.

[0058]

number

[0059] In this way, the maximum dimension length of the candidate identification ID (recogid) is expanded by three dimensions, from "20000" to "20003". The following information is inserted into these three dimensions. The sparse tensor data embedding unit 22B inserts the end time and cut time into the extended dimensions of the recogid.

[0060] Furthermore, in this case, the sparse tensor data embedding unit 22B inserts the track ID control value into the extended recogid dimension in order to restore the track ID from the data sequence encoded by the metadata encoding unit 21 without decoding it. The track ID control value is the integer part obtained by dividing the true trackid (true_trackid) by the predetermined maximum dimension length of the track ID (here, true_trackid / / 2000). Then, the sparse tensor data embedding unit 22B substitutes the track ID control value (true_trackid / / 2000) for the corresponding track ID [true_trackid%2000] and time information [frid], as shown in equation (6) below.

[0061]

number

[0062] As shown in equation (6), the axis coordinates of the track ID are the remainder (dim_trackid) obtained by performing a modulo operation on the true track ID with respect to the predetermined maximum dimension length of the track ID. As a result, the true track ID (true_trackid) can be restored using the track ID control value of V_tensor by the following equation (7), without decoding the data sequence encoded by the metadata encoding unit 21.

[0063]

number

[0064] Furthermore, the mechanism for restoring this track ID from the data sequence encoded by the metadata encoding unit 21 without decoding can also be incorporated into the sparse tensor data embedding unit 22 of the video metadata structuring device 1 of the first embodiment. With the configuration described above, the video metadata structuring device 1B can structure metadata for objects in video using a sparse tensor data structure. Furthermore, the video metadata structuring device 1B can structure metadata using a sparse tensor data structure even in videos with multiple video segments or scenes with scene changes.

[0065] Note that the operation of the video metadata structuring device 1B differs only in that it performs video analysis in parallel with step S11 of the operation of the video metadata structuring device 1 described in Figure 5, and that it embeds data sequences into the expanded dimension in step S13; therefore, a diagrammatic explanation of its operation is omitted.

[0066] As explained above, by structuring metadata using a sparse tensor data structure, we were able to reduce the data size to approximately 1 / 10th of that of conventional tabular data handling, at an experimental level. This improves data retrieval speed.

[0067] Furthermore, structuring metadata using a sparse tensor data structure allows the use of fancy indexes such as NumPy (Numerical Python). This enables complex searches using lists or arrays of integers as indexes. Additionally, it eliminates the need to use complex syntax like SQL for data access. Furthermore, by structuring metadata in a sparse tensor data structure, dimensionality can be expanded simply by expanding the tensor dimension.

[0068] <Examples of data usage> The following provides a detailed explanation of how to use the sparse tensor-format metadata structured data generated by the video metadata structuring device 1,1B.

[0069] [Video Search] Metadata-structured data, using the fancy index mechanism, makes it easy to perform more complex searches than traditional database access searches. For example, this shows the procedure for obtaining time information when person A and person B are present in a video, but person C is not. Here, we will show an example using the Python language. It is assumed that the sparse tensor data structure is defined by equation (5) above. Also, the notation recogid_{*} represents the ID number of a specific person.

[0070] Furthermore, assuming that the V_tensor element extraction data, enc_val, has been decoded, the values ​​of each item after decoding will be represented using the "." notation. In other words, the recognition score will be represented as V_tensor[...].recog_score. In this case, the time information (AB_notC_timeinfo) in which person A and person B are present in the video but person C is not can be obtained by following the procedure shown in equation (8) below.

[0071]

number

[0072] Here, th_recog is set to a value that has a high probability of being present among the pre-configured thresholds. [Data Analysis] Next, we will explain an example of using metadata-structured data in sparse tensor format for data analysis. Here, we will explain an example of calculating the accuracy of recognition processing using the most significant bit (istrue) of the data sequence ev embedded in the sparse tensor explained in Figure 3. Since the correct answer bit is assigned to the most significant bit of the 64 bits, the correct answer data will be a negative number. Using this property, the accuracy rate (Acc_rate) can be calculated using the procedure shown in equation (9) below.

[0073]

number

[0074] Here, len(·) is a function that returns the number of data points. In this way, by using fancy indices from existing numerical libraries, it is possible to write programs that utilize sparse tensor data within the same programming language. Furthermore, since metadata, which is the result of the recognition process, can be embedded in sparse tensor data, metadata structured data in sparse tensor format can be implemented for various applications other than those mentioned above by programming using a similar mechanism.

[0075] Although embodiments of the present invention have been described in detail above, the present invention is not limited to the embodiments described above, and includes design changes and the like that that do not depart from the spirit of the present invention. Furthermore, although the above-described embodiments assumed that the video metadata structuring devices 1 and 1B are independent hardware, the present invention is not limited thereto. For example, the present invention can also be realized by a program that causes hardware resources such as the CPU, memory, and hard disk of a computer to function as the video metadata structuring devices 1 and 1B. This program may be distributed via a communication line, or it may be written to a recording medium such as a CD-ROM or flash memory and distributed. [Explanation of Symbols]

[0076] 1,1B Video metadata structuring device 10 Object recognition section 11. Video Analysis Department 20 Data Structuring Section 21 Metadata Encoding Section 22 Sparse Tensor Data Embedding Section

Claims

1. A video metadata structuring device that structures metadata for objects in video as sparse tensor data, An object recognition unit performs object recognition in the time direction of the video and generates a track ID, a candidate identification ID, and multiple recognition results for individual objects as metadata. A metadata encoding unit converts the values ​​of the multiple individual object recognition results generated by the object recognition unit into a data sequence of predetermined bit length, A sparse tensor data embedding unit embeds the data sequence of the corresponding metadata into the sparse tensor data structure, which has the time-based track IDs and identification candidate IDs generated by the object recognition unit as multidimensional axes, A video metadata structuring device characterized by comprising the following features.

2. The video metadata structuring device according to claim 1, characterized in that the object recognition unit recognizes a person's face as the object.

3. The video metadata structuring device according to claim 1, characterized in that the sparse tensor data embedding unit expands the axes of the candidate identification ID, inserts the integer part obtained by dividing the value of the true track ID by a predetermined maximum dimensional length of the track ID, and sets the coordinate value of the axis of the track ID to the remainder of the division.

4. The video metadata structuring device according to claim 1, characterized in that the data sequence is signed integer data.

5. Enter the correct value for the metadata of each object mentioned above, The video metadata structuring apparatus according to claim 4, characterized in that the metadata encoding unit sets the most significant bit of the data sequence to "1" to make it negative when it matches the object-specific metadata generated by the object recognition unit, and sets the most significant bit of the data sequence to "0" when it does not match.

6. The system further includes a video analysis unit that detects cut times representing shot boundaries in the aforementioned video. The video metadata structuring apparatus according to claim 1, characterized in that the sparse tensor data embedding unit associates the cut time and the end time of the video by extending the maximum dimensional length of the axis of the candidate identification ID.

7. The video metadata structuring apparatus according to claim 1, characterized in that the value of the track ID axis of the sparse tensor data structure is the quotient obtained by dividing the true track ID by a number greater than or equal to a predetermined number of simultaneously occurring tracks, and the remainder is associated with the maximum dimensional length of the axis of the candidate identification ID by extending it.

8. A program for causing a computer to function as a video metadata structuring device according to any one of claims 1 to 7.

Images