Program, video processing device, and video processing method

Abstract

To enable video processing as intended by a user in response to a request from the user.SOLUTION: A program causes a computer that receives text data describing scenes included in input first video data to function as an interpretation unit and a search unit. The interpretation unit interprets the text data and generates query information used to search for video elements in the first video data. The search unit uses the query information to search for video elements in the first video data, generates metadata for the first video data on the basis of search results, and stores the metadata in a storage unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] An embodiment of the present invention relates to a program, a video processing device, and a video processing method. [Background technology]

[0002] A technology is being used in which metadata describing the content of a video is added to video data and the metadata is used for editing the video. For example, a technology has been known in the past in which a video management device generates metadata by recognizing and converting video elements into text, edits the video corresponding to the metadata in response to a user request, and outputs the edited video. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7121378 Summary of the Invention [Problem to be solved by the invention]

[0004] However, with conventional technology, it has been difficult to enable video processing in accordance with a user's request as intended. [Means for solving the problem]

[0005] According to an embodiment, the program causes a computer that receives text data describing scenes included in input first video data to function as an interpretation unit and a search unit. The interpretation unit interprets the text data and generates query information used to search for video elements in the first video data. The search unit uses the query information to search for video elements in the first video data, generates metadata for the first video data based on search results, and stores the metadata in a storage unit. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a diagram showing an example of the hardware configuration of a video processing device according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of a video processing device according to a first embodiment. [Figure 3] FIG. 2 is a diagram showing an example of an operation screen of the video processing device according to the first embodiment. [Figure 4] FIG. 4 is a diagram showing an example of search result information by a search unit according to the first embodiment. [Figure 5] FIG. 2 is a diagram showing an example of the functional configuration of an interpretation unit according to the first embodiment. [Figure 6] FIG. 3 is a diagram showing an example of an image query table according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing Example 1 of a reply sentence from a reply unit of the first embodiment. [Figure 8] FIG. 3 is a diagram showing an example of a sound query table according to the first embodiment. [Figure 9] FIG. 10 is a diagram showing Example 2 of a reply sentence from the reply unit of the first embodiment. [Figure 10] FIG. 3 is a diagram showing an example of a closed caption query table according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing Example 3 of a reply sentence from the reply unit in the first embodiment. [Figure 12] FIG. 4 is a diagram showing an example of query information according to the first embodiment. [Figure 13] FIG. 10 is a diagram showing an example of query information after query importance adjustment according to the first embodiment. [Figure 14] FIG. 2 is a diagram showing an example of the functional configuration of a search unit according to the first embodiment. [Figure 15] 4 is a flowchart showing an example of a video processing method according to the first embodiment. [Figure 16] FIG. 10 is a diagram showing an example of the functional configuration of a video processing device according to a second embodiment. [Figure 17] FIG. 10 is a diagram showing an example of the functional configuration of an interpretation unit according to the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of a face query table according to the second embodiment. [Figure 19] FIG. 10 is a diagram showing an example of a character query table according to the second embodiment. [Figure 20]FIG. 10 is a diagram showing an example of video attribute data according to the second embodiment. [Figure 21] FIG. 11 is a diagram showing an example of a reply sentence from a reply unit according to the second embodiment. [Figure 22] FIG. 10 is a diagram showing an example of a voice query table according to the second embodiment. [Figure 23] FIG. 11 is a diagram showing an example of a reply sentence from a reply unit according to the second embodiment. [Figure 24] FIG. 10 is a diagram showing an example of the functional configuration of a search unit according to the second embodiment. [Figure 25] 4 is a flowchart showing an example of a video processing method according to the first embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, embodiments of a program, a video processing device, and a video processing method will be described in detail with reference to the accompanying drawings.

[0008] (First embodiment) First, an example of the hardware configuration of the video processing device 200 of the first embodiment will be described.

[0009] [Example of hardware configuration] Fig. 1 is a diagram showing an example of the hardware configuration of a video processing device 200 according to the first embodiment. The example in Fig. 1 shows a case where the video processing device 200 according to the first embodiment is implemented in a computer such as a PC (Personal Computer).

[0010] The video processing device 200 of the first embodiment includes a CPU (Central Processing Unit) 301, a RAM (Random Access Memory) 302, a ROM (Read Only Memory) 303, an operation input device 304, a display device 305, a storage device 306, and a communication device 307. The CPU 301, RAM 302, ROM 303, operation input device 304, display device 305, storage device 306, and communication device 307 are connected by a bus 308.

[0011] The CPU 301 is a processor that executes arithmetic processing, control processing, etc. in accordance with a program. The CPU 301 uses a predetermined area of ​​the RAM 302 as a work area and executes various processes in cooperation with programs stored in the ROM 303, the storage device 306, etc.

[0012] The RAM 302 is a memory such as an SDRAM (Synchronous Dynamic Random Access Memory), and functions as a work area for the CPU 301. The ROM 303 is a memory that stores programs and various types of information in a non-rewritable manner.

[0013] The operation input device 304 is an input device such as a touch screen, a mouse, a keyboard, etc. The operation input device 304 receives information input by a user as an instruction signal, and outputs the instruction signal to the CPU 301.

[0014] The display device 305 is a display device such as an LCD (Liquid Crystal Display), etc. The display device 305 displays various information based on a display signal from the CPU 301.

[0015] The storage device 306 is, for example, a semiconductor storage medium such as a flash memory. Alternatively, the storage device 306 is, for example, a device that writes and reads data to a magnetically or optically recordable storage medium. The storage device 306 writes and reads data to the storage medium under the control of the CPU 301.

[0016] The communication device 307 communicates with external devices via a network under the control of the CPU 301 .

[0017] [Example of functional configuration] 2 is a diagram showing an example of the functional configuration of the video processing device 200 according to the first embodiment. The video processing device 200 according to the first embodiment references text representing a desired scene input by a user, extracts the desired scene from long video data, and generates digest video data in which the desired scene is combined.

[0018] The video processing device 200 of the first embodiment includes a video input unit 201, an input video data storage unit 202, a text input unit 203, an interpretation unit 204, a query information storage unit 205, an editing unit 206, a search unit 207, a metadata storage unit 208, a division unit 209, a generation unit 210, and an edited video data storage unit 211.

[0019] Fig. 3 is a diagram showing an example of an operation screen of the video processing device 200 of the first embodiment. The example of Fig. 3 shows an example of an operation screen displayed on the display device 305 for operating the video processing device 200 of the first embodiment. The operation screen 100 of the first embodiment includes a scene input box 101, a query generation button 102, a query information editing box 103, a metadata generation button 104, a metadata display window 105, an original video display window 106, an original video operation button 107, a digest video display window 108, a digest video operation button 109, a digest video length input box 113, and a digest generation button 114.

[0020] An example of the operation of the video processing device 200 of the first embodiment will be described below with reference to FIGS.

[0021] The video input unit 201 receives input video data to be edited (an example of first video data) through a user's operation input. The input video data to be edited may be any input video data, and may be, for example, a soccer match video.

[0022] The input video data storage unit 202 stores input video data specified by the user. The input video data stored in the input video data storage unit 202 is displayed in the original video display window 106. The input video data stored in the input video data storage unit 202 can be viewed by operating the original video operation buttons 107 to perform operations such as play, stop, fast forward, and fast rewind.

[0023] The text input unit 203 acquires the text entered by the user in the scene input box 101 at the timing when the user presses the query generation button 102. In the example of Fig. 3, two pieces of text, "the scene where a goal is scored" and "the scene where the player is sent off with a red card," are acquired.

[0024] Interpretation unit 204 generates query information used to search for video elements from the text acquired by text input unit 203. Video elements represent search targets for video data stored in input video data storage unit 202. Video elements include images, sounds, and additional information. For example, the additional information is closed captions (an example of subtitle information) associated with time information of the video data.

[0025] For example, images and sounds may be used as video elements. Furthermore, for example, objects in images, human faces, and character strings may be added as video elements. Furthermore, for example, voices and music may be added as video elements. Furthermore, closed captions may be added as additional information included in video elements.

[0026] The query information includes a search query for searching each video element and its importance score (an example of query importance). The search query is information that expresses the target to be searched for from the video element using text, images, sound, etc. The importance score is a numerical value that indicates the degree of association between the text acquired by the text input unit 203 and the search query. For example, the larger the importance score, the greater the relevance (degree of association) between the text and the search query.

[0027] The query information storage unit 205 stores the query information output from the interpretation unit 204 .

[0028] The editing unit 206 displays the query information stored in the query information storage unit 205 in the query information editing box 103 and accepts editing of the search query and importance score from the user. The editing unit 206 updates the query information stored in the query information storage unit 205 based on the editing result accepted by operation input from the user.

[0029] In the example in Figure 3, images, sounds, and closed captions are considered as video elements, and for the input text "A scene where a goal is scored," the search query for the image video element is "The ball is in the goal," with an importance score of 1. The search query for the sound video element is "Applause," with an importance score of 0.5, and the search query for the closed caption video element is "Goal," with an importance score of 0.3.

[0030] The editing unit 206 updates the query information in the query information storage unit 205 by accepting edits from the user to the search query and importance score displayed in the query information editing box 103. For example, the editing unit 206 modifies "applause" to "applause and cheers" and modifies the importance score from 0.5 to 0.8 in response to the user's edits. In addition, for example, the editing unit 206 deletes each item of the query information in response to the user's edits (for example, deleting a search query "the ball is in the goal" with a video element of an image and an importance score of 1). In addition, for example, the editing unit 206 adds new query information in response to the user's edits (for example, adding a search query "whistle" with a video element of sound and an importance score of 0.2).

[0031] The search unit 207 searches the video data stored in the input video data storage unit 202 for each piece of query information stored in the query information storage unit 205, and outputs the search results. The search results include time information of the searched video section and the video importance score of the section. An example of the search result information is shown in FIG. 4.

[0032] 4 is a diagram showing an example of search result information by the search unit 207 in the first embodiment. The search result information in the first embodiment includes video elements, query importance, search query, start time, end time, search reliability, and video importance.

[0033] The video element indicates the searched video element. The query importance indicates the importance of the query. The search query indicates the search query used for the search. The start time indicates the start time of the video section including the searched video element. The end time indicates the end time of the video section including the searched video element. The search reliability indicates the reliability of each search result calculated for each video section including the searched video element. For example, the search reliability may be calculated using a statistical quantity such as the average, maximum, or median of the search reliability for the video section.

[0034] The video importance indicates the importance of the video containing the searched video element. In the example of FIG. 4, the video importance is the product of the query importance and the search confidence. Note that the video importance may be further multiplied by a coefficient proportional to the video section length to take the section length into consideration. This allows, for example, the longer an object is shown in the video, the higher the video importance can be.

[0035] 4, the time information of a video section is expressed by a start time and an end time, but the time information of a video section may be expressed by another value associated with time such as a video frame number. Also, for example, the time information of a video section may be expressed by the time of the center of the video section and the length of the video section (e.g., 30 seconds).

[0036] In addition, in the first embodiment, the search unit 207 has been described as an example of detecting video sections that match a search query, but it is also possible to divide all sections of video data stored in the input video data storage unit 202 into specified sections and calculate video importance for each section.

[0037] Returning to FIG. 2, the division unit 209 analyzes the video data stored in the input video data storage unit 202 and outputs scene boundary time information. A scene represents a section of video data that is continuous in terms of content. Scene division can be achieved in various ways. For example, one method is to detect the boundaries of cuts that represent the period from one camera switch that shoots the video to the next, combine consecutive cuts with similar content to form scene units, and detect the boundary times of these scene units.

[0038] Cut boundaries are detected, for example, by thresholding the similarity between image frames (for example, the difference between feature amounts). Specifically, if the similarity between image frames is smaller than a predetermined threshold, it is detected that they are different scenes, and if the similarity between image frames is equal to or greater than the predetermined threshold, it is detected that they are the same scene.

[0039] The feature quantity representing the content of the video in units of cuts is expressed by a vector obtained by averaging the feature vectors of the image or audio frames over the section of the cut, for example. Alternatively, the feature quantity may be a feature vector obtained by concatenating the image feature vector and the audio feature vector.

[0040] The dividing unit 209 may also adjust the length of a scene by referring to a preset parameter, such as an average scene length, a maximum scene length, or a threshold value for the similarity of feature amounts per cut.

[0041] The metadata storage unit 208 stores metadata including the search result information output from the search unit 207 and the scene boundary time information output from the division unit 209. For example, the metadata includes time information indicating the searched video section and the above-mentioned video importance indicating the importance of the video section. The search unit 207 calculates the video importance based on the reliability of the search result and the above-mentioned query importance.

[0042] Specifically, the search unit 207 and the division unit 209 perform processing when the user presses the metadata generation button 104, and the output of these processes is stored in the metadata storage unit 208 and reflected in the metadata display window 105.

[0043] 3, the horizontal axis of the metadata display window 105 represents the passage of time, with the left edge representing the start of the video data stored in the input video data storage unit 202 and the right edge representing the end of the video data. Thin lines 110 dividing the metadata display window 105 represent scene boundary times. Thick bars 111 represent search results, with the horizontal position representing the time information of the search results and the vertical length of the bar representing the video importance.

[0044] Next, when the user inputs a desired digest video length into the digest video length input box 113 and presses the digest generation button 114, the generation unit 210 generates a digest video. Specifically, the generation unit 210 refers to the metadata stored in the metadata storage unit 208, edits the video data stored in the input video data storage unit 202, and generates a digest video of the specified digest video length.

[0045] For example, the generation unit 210 generates a digest video (an example of second video data) that is shorter than the video data by editing the video data based on the above-mentioned video importance. First, the generation unit 210 calculates the scene importance for each scene by finding the sum of the video importance of the search results included in each scene. As the scene importance, various statistics such as the average or maximum value of the video importance may be used. Next, the generation unit 210 sorts each scene by scene importance and selects scenes to be used in the digest video in descending order. The generation unit 210 adds scenes to the digest video until the total length of the selected scenes exceeds the specified digest video length.

[0046] For example, the generation unit 210 generates a digest video by selecting and concatenating scenes including video sections with higher video importance in order of earliest boundary time based on a predetermined video length. Specifically, the generation unit 210 generates a digest video by concatenating scenes including video sections with higher video importance in order of earliest boundary time until the total exceeds a specified predetermined digest video length. The generation unit 210 may also add scenes to the digest video so that the total length of the selected scenes is the longest possible without exceeding the specified digest video length. Alternatively, the generation unit 210 may add scenes until the digest video length exceeds the digest video length, and then trim the last added scene to match the specified digest video length.

[0047] 3, scenes 112 painted in dark gray represent scenes used in the digest video among the scenes separated by lines 110 in the metadata display window 105. The generation unit 210 reads out the video of these scenes used in the digest video from the input video data storage unit 202 and generates digest video data by concatenating the video of these scenes.

[0048] The edited video data storage unit 211 stores the digest video data output from the generation unit 210. The stored digest video is displayed in the digest video display window 108, and is viewed by the user by operating the digest video operation buttons 109 (for example, operations such as play, stop, fast forward, and fast rewind).

[0049] Next, an example of the functional configuration of the interpretation unit 204 will be described. Fig. 5 is a diagram showing an example of the functional configuration of the interpretation unit 204 according to the first embodiment. The interpretation unit 204 according to the first embodiment includes an image query generation unit 400, an image query table storage unit 401, an audio query generation unit 402, an audio query table storage unit 403, a closed caption query generation unit 404, a closed caption query table storage unit 405, a response unit 406, and an adjustment unit 407.

[0050] The response unit 406 is a question-answering system configured to output a text answer to a question input in text. For example, the response unit 406 may be a question-answering system (generative AI (Artificial Intelligence)) that applies a large-scale language model, such as ChatGPT (registered trademark).

[0051] The image query table storage unit 401 stores an image query table that associates keywords that may be included in text input from the text input unit 203 with search queries corresponding to the keywords and query importance. For example, keywords that are expected to appear frequently in the input text are listed in advance by the user. Each listed keyword is stored in the image query table together with an appropriate search query for searching for video elements of images corresponding to the keyword and the query importance.

[0052] The image query generation unit 400 refers to the image query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0053] Fig. 6 is a diagram showing an example of an image query table according to the first embodiment. In the example of Fig. 6, for example, when the input text is "a scene in which a player is sent off with a red card," the keyword "red card" matches, and therefore the search query "a referee who raises a red card" and a query importance level of 1 are obtained from the image query table.

[0054] Furthermore, the image query generation unit 400 generates a search query using the response unit 406. The image query generation unit 400 generates a question using, as a template for creating a question to be input to the response unit 406, "What is shown on the screen in the broadcast video [input text]? Please list it in bullet points." For example, if the input text is "A scene where a player is sent off with a red card," the question will be "What is shown on the screen in the broadcast video scene where a player is sent off with a red card? Please list it in bullet points." The image query generation unit 400 inputs this question to the response unit 406 and obtains an answer from the response unit 406.

[0055] 7 is a diagram showing Example 1 of a response sentence by the response unit 406 of the first embodiment. For example, when the response sentence shown in FIG. 7 is obtained, the bulleted items "referee," "red card," and "fallen player" are extracted from the response sentence as search queries. The image query generation unit 400 determines the query importance corresponding to these search queries to be a predetermined constant (e.g., 0.5).

[0056] 5, the sound query table storage unit 403 stores a sound query table that associates keywords that may be included in the text input from the text input unit 203 with search queries corresponding to the keywords and query importance. In the sound query table, similar to the image query table described above, each listed keyword is stored together with an appropriate search query for searching for a sound video element corresponding to the keyword and its query importance.

[0057] The sound query generation unit 402 refers to the sound query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0058] Fig. 8 is a diagram showing an example of a sound query table according to the first embodiment. In the example of Fig. 8, for example, when the input text is "a scene in which a player is sent off with a red card," the keyword "red card" matches, and therefore the search query "whistle" and a query importance of 0.2 are acquired.

[0059] Furthermore, the sound query generation unit 402 generates a search query using the response unit 406. The sound query generation unit 402 generates a question using, as a template for creating a question to be input to the response unit 406, "What sounds can be heard in [input text] in the broadcast video? Please list them in bullet points." For example, if the input text is "A scene where a player is sent off with a red card in the broadcast video," the question will be "What sounds can be heard in the scene where a player is sent off with a red card in the broadcast video? Please list them in bullet points." The sound query generation unit 402 inputs this question to the response unit 406 and obtains an answer from the response unit 406.

[0060] 9 is a diagram showing a second example of a response sentence generated by the response unit 406 in the first embodiment. For example, when the response sentence shown in FIG. 9 is obtained, the itemized phrases "whistle" and "audience booing" are extracted from the response section as search queries. The sound query generation unit 402 determines the query importance corresponding to these search queries to be a predetermined constant (e.g., 0.5).

[0061] 5, the closed caption query table storage unit 405 stores a closed caption query table that associates keywords that may be included in the text input from the text input unit 203 with search queries corresponding to the keywords and query importance. As with the image query table and sound query table described above, the closed caption query table stores each listed keyword together with an appropriate search query for searching for a closed caption video element corresponding to the keyword and its query importance.

[0062] The closed caption query generation unit 404 refers to the closed caption query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0063] Fig. 10 is a diagram showing an example of a closed caption query table according to the first embodiment. In the example of Fig. 10, for example, when the input text is "a scene in which a character is sent off with a red card," the keyword "red card" matches, so the search query "red card" and a query importance of 1, and the search query "sent off" and a query importance of 0.5 are acquired.

[0064] Furthermore, the closed caption query generation unit 404 generates a search query using the response unit 406. The closed caption query generation unit 404 generates a question using a template for creating a question to be input to the response unit 406: "What words does the announcer say in [input text] in the broadcast video? Please list them in bullet points." For example, if the input text is "A scene where a player is sent off with a red card in the broadcast video," the question would be "What words does the announcer say in the scene where a player is sent off with a red card in the broadcast video? Please list them in bullet points." The closed caption query generation unit 404 inputs this question to the response unit 406 and obtains the answer.

[0065] 11 is a diagram showing Example 3 of a response sentence by the response unit 406 in the first embodiment. For example, when the response sentence shown in FIG. 11 is obtained, the itemized terms "red card," "sentence off," and "foul" are extracted from the response section as search queries. The closed caption query generation unit 404 determines the query importance corresponding to these search queries to be a predetermined constant (e.g., 0.5).

[0066] Returning to FIG. 5, the adjustment unit 407 receives query information from each of the image query generation unit 400, the sound query generation unit 402, and the closed caption query generation unit 404, and adjusts the query importance corresponding to each search query.

[0067] FIG. 12 is a diagram showing an example of query information according to the first embodiment. The query information according to the first embodiment includes a video element, a query importance level, and a search query. The video element indicates a video element to be searched. In the first embodiment, the video element is an image, sound, or closed caption.

[0068] For each video element, if there are multiple identical search queries, the query importance of the identical search queries is added and combined into one item. In the example of Fig. 12, the search query "whistle" for the sound video element exists in the sound query table and is also generated by response unit 406, so the query importance (0.2 and 0.5) is added and combined into one item with a query importance of 0.7.

[0069] Since the number of search queries and the value of query importance of each video element are different, adjustment unit 407 performs a normalization process of the query importance to adjust the query importance in order to equalize the contribution of each video element. Specifically, adjustment unit 407 performs normalization so that the sum of the query importance of each video element becomes 1.

[0070] 13 is a diagram showing an example of query information after the query importance level has been adjusted according to the first embodiment. The adjustment unit 407 outputs the query information after the query importance level has been adjusted.

[0071] 14 is a diagram showing an example of the functional configuration of the search unit 207 according to the first embodiment. The search unit 207 according to the first embodiment includes an image extraction unit 500, an image element search unit 501, a sound extraction unit 502, a sound element search unit 503, a closed caption extraction unit 504, and a closed caption element search unit 505.

[0072] The image extraction unit 500 extracts image data from the video data stored in the input video data storage unit 202 described above, and inputs the image data to the image element search unit 501 .

[0073] The image element search unit 501 searches the image data of each frame input from the image extraction unit 500 using each of the search queries corresponding to the image elements stored in the query information storage unit 205 described above.

[0074] For example, the search process of the image element search unit 501 is realized by a process of answering questions about images written in text (Visual Question Answering (VQA)).

[0075] The image element search unit 501 prepares a template for a question to be input to the VQA, "Does it contain [search query]?", and generates a question by embedding the search query. For example, if the search query is "a referee who holds up a red card," the question "Does it contain a referee who holds up a red card?" and image data are input to the VQA, and a Yes or No answer is output.

[0076] If the answer is Yes, the image element search unit 501 acquires a value representing the probability of Yes from the VQA. The image element search unit 501 sets the time of the first frame in the section where the answers "Yes" are consecutive as the start time, the time of the last frame as the end time, and calculates the average value of the probabilities of "Yes" as a search reliability score (an example of the reliability of the search results).

[0077] That is, the image element search unit 501 searches for image elements by VQA using a question sentence including a search query, and determines a larger search reliability score for a video segment that matches the search condition expressed by the question sentence.

[0078] The image element search unit 501 also calculates the product of the query importance score and the search confidence score corresponding to the search query as a video importance score. The image element search unit 501 outputs the start time, end time, and video importance score as search results for each search query.

[0079] The sound extraction unit 502 extracts sound data from the video data stored in the input video data storage unit 202 described above, and inputs the sound data to the sound element search unit 503 .

[0080] The sound element search unit 503 searches each frame, which is obtained by dividing the sound data output from the sound extraction unit 502 into frames of a predetermined length, using each search query corresponding to the sound element stored in the query information storage unit 205 described above.

[0081] For example, the search process of the sound element search unit 503 is realized by a process of answering questions about sounds described in text (Audio Question Answering (AQA)).

[0082] The sound element search unit 503 prepares a template for a question to be input into the AQA, "Can you hear [search query]?", and generates a question by embedding the search query. For example, if the search query is "whistle," the question "Can you hear the whistle?" and one frame of sound data are input into the AQA, and a Yes or No answer is output.

[0083] If the answer is Yes, the sound element search unit 503 obtains a value indicating the probability of Yes from the AQA. The sound element search unit 503 sets the start time of the first frame of the section in which Yes answers are consecutive as the start time of the detection result, sets the end time of the last frame as the end time of the detection result, and calculates the average value of the Yes probabilities as the search reliability score.

[0084] That is, the sound element search unit 503 searches for sound elements by AQA using a question sentence including a search query, and determines a larger search reliability score for a video section that matches the search conditions expressed by the question sentence.

[0085] The sound element search unit 503 also calculates a video importance score by multiplying the query importance score corresponding to the search query by the search confidence score. The sound element search unit 503 outputs the start time, end time, and video importance score as search results for each search query.

[0086] The closed caption extraction unit 504 extracts closed captions from the video data stored in the input video data storage unit 202, and inputs the closed captions to the closed caption element search unit 505. The closed captions include text to be displayed as subtitles and information about the time period during which the text is displayed.

[0087] The closed caption element search unit 505 searches the closed caption text input from the closed caption extraction unit 504 with each of the search queries corresponding to the closed caption elements stored in the query information storage unit 205 described above.

[0088] For example, the search process of the closed caption element search unit 505 is realized by string matching. The closed caption element search unit 505 sets the start time and end time of the time period in which subtitle text matching the search query is displayed as the start time and end time of the search result, sets the query importance score of the search query as the video importance score, and outputs the start time, end time, and video importance score.

[0089] [Example of video processing method] 15 is a flowchart showing an example of the video processing method of the first embodiment. First, the video input unit 201 accepts input of video data to be edited by a user's operation input (step S1). Next, the text input unit 203 accepts input of text data representing scenes included in the video data input in step S1 by a user's operation input (step S2).

[0090] Next, interpretation unit 204 generates query information to be used for searching for video elements from the text data input in step S2 (step S3). Next, search unit 207 uses the query information generated in step S3 to search for video elements (images, sounds, and subtitles in the first embodiment) of the video data input in step S1 (step S4), and generates metadata for the video data based on the search results (step S5).

[0091] Next, the generating unit 210 edits the video data input in step S1 with reference to the metadata generated in step S5, and generates a digest video with the specified digest video length (step S6).

[0092] As described above, the video processing device 200 of the first embodiment can perform video processing as intended by the user in response to a request from the user.

[0093] For example, the video processing device 200 of the first embodiment can generate appropriate search queries for searching different video elements, such as images, audio, and closed captions, based on text describing a scene desired by the user, and can search for different video elements using each search query. This makes it possible to generate metadata that appropriately reflects the user's requests.

[0094] Furthermore, for example, the video processing device 200 of the first embodiment generates a short digest video using the generated metadata, allowing the user to efficiently view the desired video.

[0095] For example, the video processing device 200 of the first embodiment generates multiple search queries for each video element, assigns a query importance score to each search query, and reflects the query importance score in the search results. This clarifies the ranking of scene importance, making it possible to generate an appropriate digest video by prioritizing more important scenes. Even if the generated search query does not match the user's intentions, the editing unit 206 can use an interface to accept user corrections, thereby generating the digest video desired by the user.

[0096] Furthermore, for example, when generating a digest video, the video processing device 200 of the first embodiment edits the video on a scene-by-scene basis, thereby making it possible to generate a digest video that does not have noticeable discontinuities at connecting portions and that does not create a sense of incongruity.

[0097] (Modification of the first embodiment) A modified example of the first embodiment will be described. In the image element search unit 501 of the first embodiment, a search is performed using VQA. However, VQA and object recognition may be used in combination. Object recognition is configured to recognize predetermined objects (e.g., cars, people, dogs, balls, cups, etc.) in an image by learning image features, and the recognition accuracy of objects included in the recognition target is generally higher than when VQA is used.

[0098] Therefore, as a modification of the first embodiment, the image element search unit 501 may be configured to obtain search results using object recognition if the object represented by the search query is included in the recognition targets of object recognition, and to obtain search results using VQA if not. Using dedicated object recognition for objects that frequently appear in user requests has the effect of improving recognition accuracy.

[0099] Similarly, in the first embodiment, the sound element search unit 503 has been described as using AQA for searches, but AQA and sound recognition may be used in combination. Sound recognition is configured to recognize a specific sound source in a sound (for example, applause, cheers, music, whistles, engine sounds, etc.) by learning acoustic features, and generally provides higher recognition accuracy for sound sources included in the recognition target than when AQA is used.

[0100] Therefore, as a modification of the first embodiment, the sound element search unit 503 may be configured to obtain search results using sound recognition if the object represented by the search query is included in the recognition targets of sound recognition, and to obtain search results using AQA if not. Using dedicated sound recognition for sound sources that frequently appear in user requests has the effect of improving recognition accuracy.

[0101] Furthermore, although the generation unit 210 of the first embodiment has been described as editing scenes as units, it may also be configured to edit without using scene boundaries. For example, the generation unit 210 may sort the search results stored in the metadata storage unit 208 by video importance, and cut out and connect video data of a predetermined length that includes the start and end times of the search results, starting from the top search results. By configuring in this way, it is possible to generate a digest video that allows desired scenes to be viewed in a shorter time.

[0102] (Second embodiment) Next, a second embodiment will be described. In the description of the second embodiment, the same description as in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0103] [Example of functional configuration] 16 is a diagram showing an example of the functional configuration of a video processing device 200-2 according to the second embodiment. The video processing device 200-2 according to the second embodiment includes a video input unit 201, an input video data storage unit 202, a text input unit 203, an interpretation unit 204-2, a query information storage unit 205, an editing unit 206, a search unit 207-2, a metadata storage unit 208, a division unit 209, a generation unit 210, an edited video data storage unit 211, and a video attribute input unit 212.

[0104] In the first embodiment, images, sounds, and closed captions are searched for as video elements, but in the second embodiment, faces in images, characters in images, and sounds in sounds are searched for as video elements. Therefore, in the configuration of a video processing device 200-2 in the second embodiment, an interpretation unit 204-2 and a search unit 207-2 are different from the interpretation unit 204 and the search unit 207 in the first embodiment. Furthermore, in the second embodiment, a video attribute input unit 212 is further added.

[0105] The video attribute input unit 212 acquires video attribute data by accepting input of video attribute data related to the video data input from the video input unit 201. For example, if the video data is a broadcast program, the video attribute data includes the program name, broadcast date and time, program genre, and performer names. This video attribute data may be acquired from the header of the video data or from the file name of the video data. Alternatively, an input interface for video attribute data may be provided and the video attribute data may be accepted by operation input from the user indicating the video attribute data.

[0106] 17 is a diagram showing an example of the functional configuration of the interpretation unit 204-2 according to the second embodiment. The interpretation unit 204-2 according to the second embodiment includes a response unit 406, an adjustment unit 407, a face query generation unit 600, a face query table storage unit 601, a text query generation unit 602, a text query table storage unit 603, a voice query generation unit 604, a voice query table storage unit 605, and an internet search unit 606.

[0107] The face query table storage unit 601 stores a face query table that associates keywords that may be included in text input from the text input unit 203 with search queries corresponding to the keywords and query importance. The search queries in the face query table are face image data of a person indicated by the keywords. For example, a user may list names of people that are expected to appear frequently in the input text as keywords in advance. Each listed keyword is stored in the image query table together with an appropriate search query for searching for video elements of images corresponding to the keyword and the query importance. The search queries are, for example, face image data of the person in question collected from the Internet.

[0108] The face query generation unit 600 refers to the face query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0109] Fig. 18 is a diagram showing an example of a face query table according to the second embodiment. In the example of Fig. 18, for example, when the input text is "the scene where Suzuki Jiro scored a goal," the keyword "Suzuki Jiro" matches, and therefore, corresponding face image data and a query importance level of 1 are obtained from the face query table as a search query.

[0110] Furthermore, the face query generation unit 600 generates a search query using the internet search unit 606. The face query generation unit 600 generates internet search text using "face image of [person's name]" as a template to be input to the internet search unit 606. The character string to be applied to [person's name] is obtained by extracting proper nouns from the input text. For example, if the input text is "The scene where Suzuki Jiro scored a goal," the face query generation unit 600 obtains "Suzuki Jiro" as [person's name] from the input text. In this case, the internet search text will be "face image of Suzuki Jiro."

[0111] The internet search unit 606 acquires face image data by connecting to the internet via the communication device 307 and inputting an internet search text into a search site on the internet. The internet search unit 606 acquires a predetermined number (for example, two) of image data items that are ranked high among the image data items in the search results, and inputs the predetermined number of image data items including the face images and the search rankings to the face query generation unit 600.

[0112] When the face query generation unit 600 acquires the face image and the search ranking from the internet search unit 606, it assigns a predetermined query importance according to the search ranking (for example, 0.5 for first place, 0.3 for second place, etc.).

[0113] The face query generation unit 600 inputs the search query acquired from the face query table and the search query acquired from the internet search unit 606 to the adjustment unit 407 .

[0114] The character query table storage unit 603 stores a character query table that associates keywords that may be included in text input from the text input unit 203, search queries corresponding to the keywords, and query importance. For example, keywords that are expected to appear frequently in the input text are listed in advance by the user. Each listed keyword is stored in the character query table together with an appropriate search query for a character video element in an image and its query importance.

[0115] The text query generation unit 602 refers to the text query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0116] Fig. 19 is a diagram showing an example of a character query table according to the second embodiment. In the example of Fig. 19, for example, when the input text is "the scene where Suzuki Jiro scored a goal," the keyword "score" matches, so the search query "goal" and query importance level 1, and the search query "GOAL" and query importance level 1 are acquired.

[0117] Furthermore, the text query generation unit 602 generates a search query using the video attributes input from the video attribute input unit 212 and the response unit 406. First, the text query generation unit 602 generates a prompt using, as a template for creating a prompt to be input to the response unit 406, "Please answer the following questions about the broadcast program whose program name is [program name], genre is [program genre], broadcast station is [broadcast station], broadcast date is [broadcast date], and cast members are [cast members]."

[0118] 20 is a diagram showing an example of video attribute data according to the second embodiment. For example, in the case of the video attribute data shown in FIG. 20, the following prompt is generated: "Please answer the following questions about the broadcast program whose title is J.League Soccer Yokohama Marinos vs. Kawasaki Frontale, whose genre is sports, whose broadcasting station is NHK, whose broadcasting date is January 26, 2024, and whose performers are Saburo Tanaka and Shiro Takahashi." The prompt is input to the response unit 406 before the question is input.

[0119] The text query generation unit 602 generates a question using a template for creating a question, "What characters are displayed on the screen in [input text]? Please list them in bullet points." For example, if the input text is "The scene where Sato Taro scores a goal," the question will be "What characters are displayed on the screen in the scene where Sato Taro scores a goal? Please list them in bullet points." The text query generation unit 602 inputs this question to the response unit 406 and obtains an answer from the response unit 406.

[0120] 21 is a diagram showing an example of a response sentence by the response unit 406 in the second embodiment. For example, when the response sentence shown in FIG. 20 is obtained, the itemized terms "GOAL," "Marinos," and "Frontale" are extracted from the response sentence as search queries. The text query generation unit 602 determines the query importance corresponding to these search queries to be a predetermined constant (e.g., 0.5).

[0121] 17, the voice query table storage unit 605 stores a voice query table that associates keywords that may be included in the text input from the text input unit 203, search queries corresponding to the keywords, and query importance. The voice query table stores keywords included in voices spoken by announcers, performers, etc., appropriate search queries for video elements of the voice, and their query importance.

[0122] The voice query generation unit 604 refers to the voice query table, searches for keywords included in the input text, and acquires the corresponding search query and query importance.

[0123] 22 is a diagram showing an example of a voice query table according to the second embodiment. For example, when the input text is "the scene where Suzuki Jiro scored a goal," the keyword "score" matches, so the search query "score" with a query importance level of 1, and the search query "goal" with a query importance level of 1 are obtained.

[0124] Furthermore, the voice query generation unit 604 generates a search query using the video attributes input from the video attribute input unit 212 and the response unit 406. First, a prompt is generated in the same manner as in the processing by the text query generation unit 602 described above.

[0125] For example, in the case of the video attribute data shown in Fig. 20 above, the prompt generated is "Please answer the following questions about the broadcast program whose title is J.League Soccer Yokohama Marinos vs. Kawasaki Frontale, whose genre is sports, whose broadcasting station is NHK, whose broadcasting date is January 26, 2024, and whose performers are Saburo Tanaka and Shiro Takahashi." The prompt is input to response unit 406 before the question is input.

[0126] The voice query generation unit 604 generates a question using, as a template for creating a question to be input to the response unit 406, "What words does the announcer say in [input text]? Please list them in bullet points." For example, if the input text is "The scene where Suzuki Jiro scores," the question will be "What words does the announcer say in the scene where Suzuki Jiro scores? Please list them in bullet points." The voice query generation unit 604 inputs this question to the response unit 406 and obtains the answer from the response unit 406.

[0127] 23 is a diagram showing an example of a response sentence by the response unit 406 of the second embodiment. For example, when the response sentence shown in FIG. 23 is obtained, the itemized words "goal," "Suzuki," and "decided" are extracted from the response sentence as search queries. The voice query generation unit 604 determines the query importance corresponding to these search queries to be a predetermined constant (e.g., 0.5).

[0128] 24 is a diagram showing an example of the functional configuration of the search unit 207-2 according to the second embodiment. The search unit 207-2 according to the second embodiment includes an image extraction unit 500, a sound extraction unit 502, a face element search unit 700, a character element search unit 701, and a voice element search unit 702. The description of the image extraction unit 500 and the sound extraction unit 502 is omitted as they are the same as those in the first embodiment.

[0129] The facial element search unit 700 searches the image data of each frame input from the image extraction unit 500 using each search query corresponding to the facial element stored in the query information storage unit 205 described above.

[0130] For example, searching for face elements is performed by detecting a face area from image data, calculating the image features of the face area, determining the similarity between the features of the face image obtained from the search query and the features of the image of the face area, and performing threshold processing.

[0131] For example, searches for facial elements are performed by learning and storing facial recognition models in advance for frequently appearing people, and when the name of a person in the stored facial recognition model matches the name of a person in the search query, the stored facial recognition model is used to perform detection.

[0132] The face element search unit 700 sets the time of the first frame in a section of consecutive frames in which a face is detected as the start time and the time of the last frame as the end time, and calculates the average probability that the face is the person in question as the search confidence score. The face element search unit 700 also calculates a video importance score as the product of the query importance score corresponding to the search query and the search confidence score. The face element search unit 700 outputs the start time, end time, and video importance score as the search results for each search query.

[0133] The character element search unit 701 searches the image data of each frame input from the image extraction unit 500 with each search query corresponding to the character element stored in the above-mentioned query information storage unit 205. The character element search unit 701 detects a character area from the image data, performs character recognition processing on the character area to generate text, and performs string matching between the text and the search query.

[0134] The text element search unit 701 sets the time of the first frame in a section of consecutive frames in which a search query is detected as the start time and the time of the last frame as the end time, and calculates the average value of the confidence levels of the recognition results of character strings that match the search query as the search confidence score. The text element search unit 701 also calculates a video importance score as the product of the query importance score and the search confidence score corresponding to the search query. The text element search unit 701 outputs the start time, end time, and video importance score as search results for each search query.

[0135] The audio element search unit 702 searches the audio data input from the audio extraction unit 502 with each search query corresponding to the audio element stored in the query information storage unit 205. The audio element search unit 702 performs audio recognition processing on the audio data to generate text, and performs string matching between the text and the search query.

[0136] The audio element search unit 702 sets the start time and end time of the speech section of the sound data that matches the search query, and calculates the average of the confidence levels of the recognition results of the character strings that match the search query as the search confidence score.The audio element search unit 702 also calculates a video importance score as the product of the query importance score and the search confidence score corresponding to the search query.The audio element search unit 702 outputs the start time, end time, and video importance score as the search results for each search query.

[0137] [Example of video processing method] 25 is a flowchart showing an example of a video processing method according to the second embodiment. Steps S11 and S12 are similar to steps S1 and S2 (see FIG. 15) in the first embodiment, and therefore a description thereof will be omitted.

[0138] Video attribute input unit 212 accepts input of video attribute data related to the video data input in step S11 (step S12-2). Next, interpretation unit 204 generates query information used to search for video elements from the text data input in step S12 and the video attribute data input in step S12-2 (step S13).

[0139] Next, the search unit 207 uses the query information generated in step S13 to search for the video elements (in the second embodiment, faces, text, and audio) of the video data input in step S1 (step S14), and generates metadata for the video data based on the search results (step S15).

[0140] Step S16 is similar to step S6 (see FIG. 15) in the first embodiment, and therefore a description thereof will be omitted.

[0141] As described above, the video processing device 200-2 of the second embodiment generates appropriate search queries for searching different video elements, such as faces, characters, and voices, based on text describing a scene desired by the user, and searches video data using each search query. This makes it possible to generate metadata that appropriately reflects the user's requests.

[0142] Furthermore, the video processing device 200-2 of the second embodiment uses the generated metadata to generate a short digest video, allowing the user to efficiently view the desired video.

[0143] Furthermore, the video processing device 200-2 of the second embodiment can generate an appropriate search query by referring to the video attributes when interpreting the input text and generating a search query. For example, if the program name of the video attributes includes "soccer," it becomes clear that the "goal" included in the input text is a soccer goal, enabling a more appropriate search.

[0144] (Modification of the second embodiment) A modified example of the second embodiment will be described. In the second embodiment, the face query table storage unit 601 is described as storing one piece of face image data per person, but multiple pieces of face image data may be stored. Increasing the number of pieces of face image data improves the detection accuracy.

[0145] Furthermore, in the face query generation unit 600 of the second embodiment, a search query is generated using the internet search unit 606. However, a face image may be generated by inputting "Please generate a face image of [person name]" to the response unit 406. Furthermore, a face image may be generated by inputting "Please generate a face image of [person name] appearing in [program name]" by referring to the video attributes from the video attribute input unit 212. This generates a query that is easy to match, including the background, and improves detection accuracy.

[0146] Although the video attribute data input from the video attribute input unit 212 has been described as the program title, genre, broadcasting station, broadcasting date, and cast, the video attribute data may also include information other than these. For example, the video attribute data may further include the broadcast time, an introduction to the program, and the like.

[0147] The functions of the video processing device 200 (200-2) according to the first and second embodiments described above may be realized by a program.

[0148] For example, the program executed by the video processing device 200 (200-2) is provided as a computer program product in the form of a file in an installable or executable format, recorded on a computer-readable storage medium such as a memory card, hard disk, CD-RW, CD-ROM, CD-R, DVD-RAM, or DVD-R.

[0149] Also, for example, the program executed by the video processing device 200 (200-2) may be stored on a computer connected to a network such as the Internet, and may be provided by being downloaded via the network.

[0150] Also, for example, the program executed by the video processing device 200 (200-2) may be provided via a network such as the Internet without being downloaded.

[0151] Furthermore, for example, the program for the video processing device 200 (200-2) may be provided by being pre-installed in the above-mentioned ROM 303 or the like.

[0152] Note that some or all of the functions of the video processing device 200 (200-2) according to the first and second embodiments may be realized by hardware such as an IC (Integrated Circuit) instead of by a program (software).

[0153] Furthermore, each function may be realized using a plurality of processors, in which case each processor may realize one of the plurality of functions, or may realize two or more functions.

[0154] Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as defined in the claims. [Explanation of symbols]

[0155] 200, 200-2 Image processing device 201 Video input section 202 Input video data storage unit 203 Text input section 204, 204-2 Interpretation Section 205 Query information storage unit 206 Editorial Department 207, 207-2 Search Section 208 Metadata storage unit 209 Split section 210 Generation part 211 Edited video data storage unit 212 Video attribute input unit 301 CPU 302 RAM 303 ROM 304 Operation input device 305 Display device 306 Storage device 307 Communication Equipment 308 Bus 400 Image query generation unit 401 Image query table storage unit 402 Sound Query Generation Unit 403 Sound query table storage unit 404 Closed Caption Query Generator 405 Closed Caption Query Table Storage 406 Response Section 407 Adjustment section 500 Image Extraction Unit 501 Image element search unit 502 Sound extraction section 503 Sound Element Search Unit 504 Closed Caption Extraction Unit 505 Closed Caption Element Search Unit 600 Face query generation unit 601 Face query table storage unit 602 Character Query Generation Unit 603 Character Query Table Storage 604 Voice Query Generation Unit 605 Voice query table storage unit 606 Internet Search Department 700 Face element search unit 701 Character element search unit 702 Audio element search unit

Claims

1. a computer that receives text data representing a scene included in input first video data, an interpretation unit that interprets the text data and generates query information used to search for video elements of the first video data; a search unit that searches for video elements of the first video data using the query information, generates metadata for the first video data based on a search result, and stores the metadata in a storage unit; A program to function as a

2. the query information includes a search query for searching the video element and a query importance indicating the importance of the search query; the interpretation unit increases the query importance as the relevance between the text data and the search query increases; The program according to claim 1.

3. the metadata includes time information indicating the searched video section and video importance indicating the importance of the video section; the search unit calculates the video importance level based on the reliability of the search result and the query importance level; The program according to claim 2.

4. the search unit searches for the video elements by VQA (Visual Question Answering) or AQA (Audio Question Answering) using a question sentence including the search query, and determines a higher reliability of the search result for a video section that matches the search condition expressed by the question sentence; The program according to claim 3.

5. The computer a generating unit that generates second video data shorter than the first video data by editing the first video data based on the video importance; The program according to claim 3, further functioning as:

6. The computer further functioning as a division unit that divides the first video data into a plurality of scenes by analyzing the first video data; the metadata further includes boundary times indicating boundaries between the plurality of scenes; the generation unit generates the second video data by selecting and linking scenes including video sections with higher video importance in order of earliest boundary time based on a predetermined video length. The program according to claim 5.

7. The computer an editing unit that edits the query information in response to an operation input from a user; 7. The program according to claim 1, further functioning as:

8. The video elements include at least one of an image, a sound, and a subtitle. The program according to any one of claims 1 to 6.

9. 1. A video processing device that accepts text data representing a scene included in input first video data, an interpretation unit that interprets the text data and generates query information used to search for video elements of the first video data; a search unit that searches for video elements of the first video data using the query information, generates metadata for the first video data based on a search result, and stores the metadata in a storage unit; A video processing device comprising:

10. 1. A video processing method for a video processing device that receives text data representing a scene included in input first video data, comprising: interpreting the text data by the video processing device to generate query information used to search for video elements in the first video data; the video processing device searches for video elements of the first video data using the query information, generates metadata for the first video data based on a search result, and stores the metadata in a storage unit; A video processing method including:

Images