Video subtitle matching methods, devices and computer-readable media
By extracting and fusing features from video frames and subtitle feature matrices, calculating similarity, and automatically matching subtitle text statements with the time information of video frames, the problem of low efficiency and insufficient accuracy in subtitle addition in existing technologies is solved, and efficient and accurate subtitle matching is achieved.
Patent Information
- Application Number
- CN202410604796.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-15
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-15
AI Technical Summary
Existing technologies are inefficient and inaccurate when adding subtitles to videos, and are easily affected by manual operation.
By extracting video frames from the target video and generating a frame feature matrix, combining it with a set of subtitle text to generate a subtitle feature matrix, performing feature fusion, calculating the similarity between video frames and subtitle text statements, and automatically matching the time information of subtitle text statements with video frames.
It improves the accuracy and efficiency of subtitle matching, and realizes automated and efficient subtitle addition.
Smart Images

Figure CN118378096B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of information technology, and in particular to a video subtitle matching method, device and computer-readable medium. Background Technology
[0002] With the development of information technology, watching video programs on computers using media player software has become a very common way of learning, leisure, and entertainment. During video playback, displaying subtitles related to the visuals helps viewers understand the program content; therefore, most users generally prefer videos to include subtitles. Currently, there is no highly efficient subtitle matching solution. Adding subtitles to videos requires preparing the corresponding subtitle text in advance, and then manually reviewing the entire video to determine the appropriate time and position for each subtitle text. This process is time-consuming, labor-intensive, and inefficient, and the accuracy of the subtitle addition is insufficient and easily affected by the operator. Summary of the Invention
[0003] One objective of this application is to provide a video subtitle matching method, device, and computer-readable medium to address the problems of low efficiency and insufficient accuracy in existing solutions when adding subtitles to videos.
[0004] To achieve the above objectives, embodiments of this application provide a video subtitle matching method, the method comprising:
[0005] Extract video frames from the target video and mark the time information of the video frames in the target video;
[0006] Generate a corresponding frame feature matrix based on the video frames;
[0007] Generate the corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set;
[0008] The frame feature matrix and the subtitle feature matrix are fused to obtain the frame feature fusion matrix and the subtitle feature fusion matrix;
[0009] Based on the frame feature fusion matrix and the subtitle feature fusion matrix, the similarity between each video frame and each subtitle text statement is calculated.
[0010] Based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statement and the video frame is determined, and the subtitle text statement is added to the target video according to the time information of the video frame in the target video.
[0011] Further, generating a corresponding frame feature matrix based on the video frames includes:
[0012] The video frames are input into the CLIP model image encoder, and the corresponding frame feature matrix is extracted using the CLIP model image encoder.
[0013] Furthermore, a corresponding subtitle feature matrix is generated based on the subtitle text statements in the subtitle text set, including:
[0014] The subtitle text statements in the subtitle text set are input into the text encoder of the CLIP model, and the corresponding subtitle feature matrix is extracted using the text encoder of the CLIP model.
[0015] Furthermore, the frame feature matrix has a shape of (A, X), and the subtitle feature matrix has a shape of (B, X), where A is the number of video frames, B is the number of subtitle text sentences contained in the subtitle text set, and X is the embedding dimension of the feature.
[0016] The frame feature matrix and the subtitle feature matrix are fused to obtain a frame feature fusion matrix and a subtitle feature fusion matrix, including:
[0017] The frame feature matrix and the subtitle feature matrix are concatenated to obtain a concatenated matrix with shape (A+B, X);
[0018] The concatenated matrix is input into a pre-trained feature fusion model, which outputs a feature fusion matrix with the same shape as the concatenated matrix.
[0019] The first A feature vectors are extracted from the feature fusion matrix to form the frame feature fusion matrix, and the last B feature vectors are extracted from the feature fusion matrix to form the subtitle feature fusion matrix. The frame feature fusion matrix has a shape of (A, X), and the subtitle feature fusion matrix has a shape of (B, X).
[0020] Further, based on the frame feature fusion matrix and the subtitle feature fusion matrix, the similarity between each video frame and each subtitle text statement is calculated, including:
[0021] The frame feature fusion matrix and the transposed subtitle feature fusion matrix are multiplied to obtain an intermediate result matrix of shape (A, B). Each element in the intermediate result matrix is then processed using a first activation function to obtain a similarity matrix of shape (A, B). The similarity matrix has the shape (A, B), and the element (a, b) in the similarity matrix represents the similarity between the a-th video frame and the b-th subtitle text statement.
[0022] Furthermore, based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statements and the video frames is determined, including:
[0023] For each video frame, the similarity between the video frame and each subtitle text statement is compared with a similarity threshold. Subtitle text statements with a similarity exceeding the similarity threshold are identified as candidate subtitle text statements.
[0024] If the candidate subtitle text statements include multiple statements, then the similarity between the video frame and the candidate subtitle text statements is sorted, and the subtitle text statement with the highest similarity in the sorting result is determined as the subtitle text statement that matches the video frame.
[0025] If there is only one candidate subtitle text statement, then the candidate subtitle text statement is determined as the subtitle text statement that matches the video frame;
[0026] If there are no candidate subtitle text statements, the video frame is marked as having no matching subtitle text statements.
[0027] Furthermore, before determining the matching relationship between subtitle text statements and video frames based on the similarity between each video frame and each subtitle text statement, the process includes:
[0028] Based on the subtitle feature fusion matrix, usable subtitle text statements are selected from the subtitle text statements.
[0029] Further, based on the subtitle feature fusion matrix, usable subtitle text statements are filtered from the subtitle text statements, including:
[0030] The subtitle feature fusion matrix is input into a pre-trained multilayer perceptron model for processing, and an intermediate result vector with B elements is output. The second activation function is used to calculate each element in the intermediate result vector to obtain a matching identifier vector with B elements. The b-th element of the matching identifier vector represents the availability score of the b-th subtitle text statement, and B is the number of subtitle text statements contained in the subtitle text set.
[0031] Based on the matching identifier vector, the usability score corresponding to each subtitle text statement is compared with a pre-set usability threshold;
[0032] If the availability score of a subtitle text statement is greater than or equal to the availability threshold, then the subtitle text statement is determined to be an available subtitle text statement.
[0033] Some embodiments of this application also provide a video caption matching device, wherein the device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned video caption matching method.
[0034] Other embodiments of this application also provide a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the video subtitle matching method.
[0035] Compared to existing technologies, the video subtitle matching scheme provided in this application first extracts video frames from the target video, marks the time information of the video frames in the target video, generates a corresponding frame feature matrix based on the video frames, and simultaneously generates a corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set. The frame feature matrix and the subtitle feature matrix are then fused to obtain a frame feature fusion matrix and a subtitle feature fusion matrix. Next, based on the frame feature fusion matrix and the subtitle feature fusion matrix, the similarity between each video frame and each subtitle text statement is calculated. Based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statement and the video frame is determined. Finally, based on the time information of the video frames in the target video, the subtitle text statement is added to the target video. In this scheme, by comparing the semantic features of the video image and the subtitle text, video frames with similar content expressions are automatically matched for the subtitle text statement. Simultaneously, information fusion is performed on the features of the video frames and the subtitle text statements during processing, so that the fused feature matrix can better reflect the similarity between the two, improving the accuracy of the matching results. Therefore, this provides a video subtitle matching scheme with good matching accuracy and high processing efficiency. Attached Figure Description
[0036] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0037] Figure 1 A flowchart illustrating a video subtitle matching method provided in this application embodiment;
[0038] Figure 2 This is a flowchart illustrating the process of adding subtitles to a video using the method described in this application.
[0039] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation
[0040] The present application will now be described in further detail with reference to the accompanying drawings.
[0041] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.
[0042] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0043] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program devices, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.
[0044] This application provides a video subtitle matching method. The method first extracts video frames from a target video, marks the time information of each video frame within the target video, generates a corresponding frame feature matrix based on the video frames, and simultaneously generates a corresponding subtitle feature matrix based on subtitle text statements in a subtitle text set. The frame feature matrix and the subtitle feature matrix are then fused to obtain a frame feature fusion matrix and a subtitle feature fusion matrix. Next, based on the frame feature fusion matrix and the subtitle feature fusion matrix, the similarity between each video frame and each subtitle text statement is calculated. Based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statement and the video frame is determined. Finally, based on the time information of each video frame within the target video, the subtitle text statement is added to the target video. In this scheme, by comparing the semantic features of video images and subtitle text, video frames with similar content expressions are automatically matched for subtitle text statements. Simultaneously, information fusion is performed on the features of video frames and subtitle text statements during processing, enabling the fused feature matrix to better reflect the similarity between the two, improving the accuracy of the matching results. Therefore, this provides a video subtitle matching scheme with good matching accuracy and high processing efficiency.
[0045] In practical scenarios, the execution subject of this method can be a user device, a network device, or a device composed of user devices and network devices integrated through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, and tablets; the network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets. Here, the cloud consists of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.
[0046] Figure 1 The present application illustrates a processing flow of a video subtitle matching method according to an embodiment of the present application. The method includes at least the following processing steps:
[0047] Step S101: Extract video frames from the target video and mark the time information of the video frames in the target video.
[0048] The target video is the video for which subtitles need to be matched during this processing. When extracting video frames, a pre-set frame rate can be used for frame splitting. For example, in this embodiment, the frame rate is 32 frames per second. If the target video is 120 seconds long, 3840 (120×32) video frames can be extracted from the target video.
[0049] For the extracted video frames, their time information within the target video can be marked. This time information can be in the form of start and end times. Taking the video frames obtained through frame splitting as an example, the first video frame has a start time of 0 seconds and an end time of 1 / 32 second, the second video frame has a start time of 1 / 32 second and an end time of 2 / 32 second. This allows marking the time information of all video frames within the target video for subsequent matching to determine the time position of the subtitles.
[0050] Step S102: Generate a corresponding frame feature matrix based on the video frames. The frame feature matrix represents the image semantic information contained in each video frame and reflects the potential features contained in the image content of each video frame. In some embodiments of this application, when generating the corresponding frame feature matrix based on the video frames, the image encoder of the CLIP (Contrastive Language-Image Pre-training) model can be used. The video frames are input into the CLIP model's image encoder, and the corresponding frame feature matrix is extracted using the CLIP model's image encoder.
[0051] The frame feature matrix has a shape of (A, X), where A is the number of video frames. Taking the aforementioned scenario as an example, if the number of video frames extracted from the target video is 3840, then A is 3840. X is the embedding dimension of the features, which can be set according to the needs of the actual scenario. For example, in this embodiment, it can be set to 256. Thus, when generating the corresponding frame feature matrix, the CLIP model's image encoder can extract image features from each video frame, obtaining 3840 feature vectors. Each feature vector has 256 elements, which can then be merged into a matrix of shape (3840, 256), i.e., the frame feature matrix.
[0052] Step S103: Generate the corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set.
[0053] The subtitle text set refers to the collection of all candidate subtitle text statements. For example, in a real-world scenario, the subtitle text statements in the subtitle text set could be narration subtitles that introduce the content of the video screen. When generating the corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set, the CLIP model's text encoder can be used. The subtitle text statements in the subtitle text set are input into the CLIP model's text encoder, and the corresponding subtitle feature matrix is extracted using the CLIP model's text encoder.
[0054] The shape of the subtitle feature matrix is (B, X), where B is the number of subtitle text statements contained in the subtitle text set. For example, in this embodiment, the subtitle text set contains 100 candidate subtitle text statements, so B is 100. X is the embedding dimension of the feature, which is the same as the embedding dimension set in the aforementioned frame feature matrix. Therefore, when generating the corresponding subtitle feature matrix, the CLIP model's text encoder can extract the text features of each subtitle text statement to obtain 100 feature vectors, and each feature vector has 256 elements. These can then be merged into a matrix of shape (100, 256), which is the subtitle feature matrix.
[0055] This scheme uses the same CLIP model to generate all feature matrices. Because the CLIP model learns to encode images and text into unified feature vectors through contrastive learning during training, it can understand the correspondence between images and text in both natural language and image vision. Therefore, the feature information obtained from the image encoder and text encoder of the same CLIP model can represent the association between text and images corresponding to the same action, accurately achieving similarity matching between textual semantics and image semantics.
[0056] Step S104: Perform feature fusion on the frame feature matrix and the subtitle feature matrix to obtain a frame feature fusion matrix and a subtitle feature fusion matrix. This allows the feature information contained in the frame feature matrix and the subtitle feature matrix to be fused according to the user's needs, so that the fused feature matrix can better reflect the similarity between the two and improve the accuracy of the matching results.
[0057] In some embodiments of this application, when performing feature fusion on the frame feature matrix and the subtitle feature matrix, the following method can be adopted: First, the frame feature matrix and the subtitle feature matrix are concatenated to obtain a new matrix with shape (A+B, X), i.e., the concatenated matrix. For example, if the shape of the frame feature matrix is (3840, 256) and the shape of the subtitle feature matrix is (100, 256), then the shape of the obtained concatenated matrix is (3840+100, 256).
[0058] Then, the concatenated matrix is input into a pre-trained feature fusion model, which outputs a feature fusion matrix with the same shape as the concatenated matrix. The feature fusion model can be pre-trained with necessary parameters according to the needs of the actual scenario, enabling each vector in the feature fusion matrix to effectively fuse the feature information contained in the frame feature matrix and the subtitle feature matrix according to the user's needs. In this embodiment, the feature fusion model can employ an LLM (Large Language Model), which allows for information fusion of text features and image features in the input concatenated matrix, outputting a feature fusion matrix. Since the shape of the feature fusion matrix is the same as the concatenated matrix (A+B, X), when the shape of the concatenated matrix is (3840+100, 256), the shape of the feature fusion matrix is also (3840+100, 256).
[0059] After obtaining the feature fusion matrix, the first A feature vectors can be extracted from the feature fusion matrix to determine the frame feature fusion matrix, with a shape of (A, X). Simultaneously, the last B feature vectors can be extracted from the feature fusion matrix to determine the subtitle feature fusion matrix, with a shape of (B, X). Taking the aforementioned scenario as an example, if the shape of the feature fusion matrix is (3840+100, 256), then the shape of the frame feature fusion matrix is (3840, 256), and the shape of the subtitle feature fusion matrix is (100, 256).
[0060] Step S105: Calculate the similarity between each video frame and each subtitle text statement based on the frame feature fusion matrix and the subtitle feature fusion matrix.
[0061] When calculating similarity, the frame feature fusion matrix and the transposed subtitle feature fusion matrix can be multiplied to obtain an intermediate result matrix of shape (A, B). Then, a first activation function is applied to each element of the intermediate result matrix to obtain a similarity matrix of shape (A, B). The similarity matrix, with shape (A, B), can represent the similarity between each video frame and each subtitle text statement. Specifically, the element (a, b) in the similarity matrix represents the similarity between the a-th video frame and the b-th subtitle text statement.
[0062] Taking the aforementioned scenario as an example, since the shape of the subtitle feature fusion matrix is (100, 256), its transpose yields a transposed matrix of shape (256, 100). Multiplying the frame feature fusion matrix of shape (3840, 256) with the transposed matrix of shape (256, 100) yields an intermediate result matrix of shape (3840, 100). For each element in the intermediate result matrix, in this embodiment, the softmax function (normalized exponential function) can be used as the activation function to map the value range of each element to [0, 1], thus obtaining a similarity matrix of shape (3840, 100). Each element in the similarity matrix represents the degree of similarity between each video frame and each subtitle text statement. Taking element (2, 52) as an example, it represents the similarity between the second video frame and the 52nd subtitle text statement. The larger the similarity value, the higher the degree of similarity between the two and the higher the probability of matching. Conversely, the smaller the similarity value, the lower the degree of similarity between the two and the lower the probability of matching.
[0063] Step S106: Determine the matching relationship between subtitle text statements and video frames based on the similarity between each video frame and each subtitle text statement, and add the subtitle text statements to the target video based on the time information of the video frames in the target video.
[0064] After obtaining the similarity between each video frame and each subtitle text statement, for each video frame, the similarity between that video frame and each subtitle text statement is compared with a similarity threshold. Subtitle text statements with similarity exceeding the threshold are identified as candidate subtitle text statements. For example, for the second video frame, frame2, its similarity with 100 subtitle text statements is s1 to s100. These 100 similarities are compared with the similarity threshold threshold, and the magnitude of the difference is determined. If the similarity is greater than the threshold, the corresponding subtitle text statement can be identified as a candidate subtitle text statement.
[0065] The similarity threshold `threshold_s` can be set according to the needs of the actual scenario. For example, in this embodiment, it can be set to 0.5. Therefore, if the similarity between a certain subtitle text statement and video frame 2 exceeds 0.5, the subtitle text statement can be identified as a candidate subtitle text statement. In actual scenarios, the candidate subtitle text statements corresponding to each video frame may have the following situations: 1. If the candidate subtitle text statements include multiple statements, for example, for video frame 2, there are 4 subtitle text statements with a similarity exceeding the similarity threshold `threshold_s`, namely subtitle text statements `sent4`, `sent22`, `sent52`, and `sent77`, with corresponding similarities of `s4`, `s22`, `s52`, and `s77`. By sorting the similarity between the video frame and the candidate subtitle text statements, the sorting result can be obtained as `s52` > `s4` > `s22` > `s77`. Therefore, the subtitle text statement with the highest similarity in the sorting result, i.e., `sent52`, can be identified as the subtitle text statement that matches the video frame. 2. If there is only one candidate subtitle text statement, it can be directly identified as the subtitle text statement that matches the video frame. For example, for video frame 59, there is only one subtitle text statement with a similarity exceeding the similarity threshold threshold_s, which is subtitle text statement send33. In this case, subtitle text statement send33 can be directly identified as the subtitle text statement that matches the video frame. 3. If there are no candidate subtitle text statements, it means that there is no matching subtitle text statement for the video frame, and the video frame can be marked as having no matching subtitle text statement. For example, for video frame 1, there is no subtitle text statement with a similarity exceeding the similarity threshold threshold_s. In this case, video frame 1 can be marked as having no matching subtitle text statement.
[0066] Therefore, through the above processing, the matching relationship between subtitle text statements and video frames is determined, thereby matching appropriate subtitle text statements for each video frame in the target video. Then, the time information of the video frames in the target video can be used to add the subtitle text statements to the target video. For example, for video frame 2, its corresponding time information is start time: 1 / 32 second, end time: 2 / 32 second, and the matched subtitle text statement is sent52. Thus, the subtitle text statement can be added to the target video at the position from 1 / 32 second to 2 / 32 second. After adding subtitles in the above manner for each video frame, the subtitle matching for the entire target video is completed.
[0067] This scheme automatically matches video frames with similar content to subtitle text by comparing the semantic features of video images and subtitle text. At the same time, the features of video frames and subtitle text are fused during the processing, so that the fused feature matrix can better reflect the similarity between the two and improve the accuracy of the matching results. Thus, a video subtitle matching scheme with good matching accuracy and high processing efficiency is provided.
[0068] In other embodiments of this application, before determining the matching relationship between subtitle text statements and video frames based on the similarity between each video frame and each subtitle text statement, usable subtitle text statements can be filtered from the subtitle text statements according to the subtitle feature fusion matrix. Since in real-world scenarios, various factors such as the language and culture of the video audience and the overall tone of the video may render some subtitle text statements unsuitable for addition to the target video, some unusable subtitle text statements can be excluded before matching video frames and subtitle text statements. This allows us to filter usable subtitle text statements from the subtitle text set, making the subtitles added to the target video more appropriate and improving the quality of subtitle addition.
[0069] Specifically, when selecting usable subtitle text statements from the subtitle text statements based on the subtitle feature fusion matrix, the subtitle feature fusion matrix can first be input into a pre-trained Multi-Layer Perceptron (MLP) model for processing, outputting an intermediate result vector with B elements. Each element in the intermediate result vector is then processed using a second activation function to obtain a matching identifier vector with B elements. The b-th element of the matching identifier vector represents the usability score of the b-th subtitle text statement, where B is the number of subtitle text statements in the subtitle text set. For example, for a subtitle feature fusion matrix of shape (100, 256), inputting it into an MLP model yields an intermediate result vector with 100 elements. If the second activation function uses the sigmoid function, a matching identifier vector with 100 elements can be obtained. Each element of this matching identifier vector takes values in the range (0, 1), with larger values indicating better usability of the corresponding subtitle text statement, and vice versa.
[0070] After obtaining the matching identifier vector, the usability score corresponding to each subtitle text statement can be compared with a pre-set usability threshold based on the matching identifier vector. If the usability score corresponding to a subtitle text statement is greater than or equal to the usability threshold, then the subtitle text statement is determined to be usable. The usability threshold `threshold_u` can also be set according to the needs of the actual scenario. For example, in this embodiment, it can be set to 0.5. Thus, if the usability score corresponding to a subtitle text statement exceeds 0.5, it means that the subtitle text statement is usable and can be used when performing the aforementioned step S106. Conversely, if the usability score corresponding to a subtitle text statement does not exceed 0.5, it means that the subtitle text statement is unusable and needs to be excluded when performing the aforementioned step S106. This ensures the effectiveness of subtitle matching and avoids situations where some subtitle text statements, although semantically consistent with the content of the video frame, are actually unsuitable.
[0071] Figure 2 The present invention illustrates a process for adding subtitles to a video using the solution provided in the embodiments of this application, including the following processing steps:
[0072] Step S201, video processing.
[0073] The target video is split into frames, with the frame rate set to 32 frames per second. Assuming the video is 120 seconds long, it will be split into 120 × 32 = 3840 video frames. When splitting the frames, the time information of the video frames in the target video is retained (in seconds). For example, the start time of the first video frame is 0 and the end time is 1 / 32. The start time of the second video frame is 1 / 32 and the end time is 2 / 32.
[0074] The image encoder of the CLIP model extracts the image features frame_feat of all video frames, resulting in 3840 feature vectors. Each vector has 256 elements, and these features are merged into a frame feature matrix with the shape (3840, 256), named A.
[0075] Step S202, subtitle processing.
[0076] Collect all the subtitle text statements that need to be added, assuming there are 100, and build them into a subtitle text set.
[0077] The CLIP model's text encoder extracts text features sent_feat from all subtitle text sentences, resulting in 100 feature vectors, each with 256 elements. These features are then combined into a subtitle feature matrix of shape (100, 256), named B.
[0078] Step S203, feature fusion.
[0079] Combine A and B to form a new concatenation matrix with the shape (3840+100, 256), and name it C.
[0080] LLM is used to fuse information between text and video frame images in C, and the output is a feature fusion matrix D with the same shape as C.
[0081] The first 3840 feature vectors of D are taken as the frame feature fusion matrix D_frame, and the last 100 feature vectors are taken as the subtitle feature fusion matrix D_subtitle.
[0082] Step S204: Video subtitle matching.
[0083] The frame feature fusion matrix D_frame and the transposed subtitle feature fusion matrix D_subtitle are multiplied by matrix multiplication and then calculated using the softmax function to obtain a similarity matrix with a shape of (3840, 100), which represents the degree of similarity between 3840 video frames and 100 subtitle text statements.
[0084] Simultaneously, the subtitle feature fusion matrix D_subtitle can be processed by an MLP model, and its output can be further processed by a sigmoid function to obtain a matching identifier vector. This matching identifier vector is a vector with 100 elements, representing the usability score of 100 subtitle text statements, with values ranging from 0 to 1. After comparing it with a usability threshold (set to 0.5), subtitle text statements corresponding to elements exceeding the usability threshold are considered usable, while those corresponding to elements below the threshold are considered unusable. For example, if the matching identifier vector is [0.1, 0.6, 0.7...], then the 2nd and 3rd subtitle text statements corresponding to 0.6 and 0.7 are considered usable, while the 1st subtitle text statement corresponding to 0.1 is considered unusable.
[0085] After selecting all available subtitle text statements using the aforementioned method, a query is performed in the similarity matrix based on these available subtitle text statements. If the similarity between an available subtitle text statement and a certain video frame exceeds a set similarity threshold (set to 0.5), it can be preliminarily determined that the subtitle text statement matches a certain video frame; otherwise, if the similarity threshold is not exceeded, it is determined to be a mismatch. When multiple subtitle text statements have a similarity value exceeding 0.5 with a certain video frame, the one with the highest similarity can be taken as the subtitle text statement for that video frame. After determining the matching relationship between 100 subtitle text statements and 3840 frames, the subtitle text statements can be placed in the corresponding positions according to the start-end time points of the video frames in the target video, thereby completing the addition of subtitles.
[0086] Based on another aspect of this application, embodiments of this application also provide a video subtitle matching device, which includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute the aforementioned video subtitle matching method.
[0087] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.
[0088] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0089] In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.
[0090] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0091] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0092] In another aspect, this application also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable medium carries one or more computer program instructions, which may be executed by a processor to implement the methods and / or technical solutions of the various embodiments of this application.
[0093] It should be noted that this application can be implemented in software and / or a combination of software and hardware, for example, using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, and similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that cooperates with a processor to perform the various steps or functions.
[0094] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within this application. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in the apparatus claims may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any particular order. Numbers corresponding to steps are used to label and distinguish different steps, and the magnitude of these numbers does not limit any particular order of execution.
Claims
1. A video subtitle matching method, characterized in that, The method includes: Extract video frames from the target video and mark the time information of the video frames in the target video; Generate a corresponding frame feature matrix based on the video frames; Generate the corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set; The frame feature matrix and the subtitle feature matrix are fused to obtain the frame feature fusion matrix and the subtitle feature fusion matrix; Based on the frame feature fusion matrix and the subtitle feature fusion matrix, calculate the similarity between each video frame and each subtitle text statement; Based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statement and the video frame is determined, and based on the time information of the video frame in the target video, the subtitle text statement is added to the target video; Wherein, the shape of the frame feature matrix is (A, X), the shape of the subtitle feature matrix is (B, X), A is the number of video frames, B is the number of subtitle text sentences contained in the subtitle text set, and X is the embedding dimension of the feature; The frame feature matrix and the subtitle feature matrix are fused to obtain a frame feature fusion matrix and a subtitle feature fusion matrix, including: The frame feature matrix and the subtitle feature matrix are concatenated to obtain a concatenated matrix with the shape (A+B, X); The concatenated matrix is input into a pre-trained feature fusion model, which outputs a feature fusion matrix with the same shape as the concatenated matrix. The first A feature vectors are extracted from the feature fusion matrix to form the frame feature fusion matrix, and the last B feature vectors are extracted from the feature fusion matrix to form the subtitle feature fusion matrix. The frame feature fusion matrix has a shape of (A, X), and the subtitle feature fusion matrix has a shape of (B, X).
2. The method according to claim 1, characterized in that, Generate a corresponding frame feature matrix based on the video frames, including: The video frames are input into the CLIP model image encoder, and the corresponding frame feature matrix is extracted using the CLIP model image encoder.
3. The method according to claim 1, characterized in that, Generate a corresponding subtitle feature matrix based on the subtitle text statements in the subtitle text set, including: The subtitle text statements in the subtitle text set are input into the text encoder of the CLIP model, and the corresponding subtitle feature matrix is extracted using the text encoder of the CLIP model.
4. The method according to claim 1, characterized in that, Based on the frame feature fusion matrix and the subtitle feature fusion matrix, the similarity between each video frame and each subtitle text statement is calculated, including: The frame feature fusion matrix and the transposed subtitle feature fusion matrix are multiplied to obtain an intermediate result matrix of shape (A, B). Each element in the intermediate result matrix is then processed using a first activation function to obtain a similarity matrix of shape (A, B). The similarity matrix has the shape (A, B), and the element (a, b) in the similarity matrix represents the similarity between the a-th video frame and the b-th subtitle text statement.
5. The method according to claim 1, characterized in that, Based on the similarity between each video frame and each subtitle text statement, the matching relationship between the subtitle text statement and the video frame is determined, including: For each video frame, the similarity between the video frame and each subtitle text statement is compared with a similarity threshold. Subtitle text statements with a similarity exceeding the similarity threshold are identified as candidate subtitle text statements. If the candidate subtitle text statements include multiple statements, then the similarity between the video frame and the candidate subtitle text statements is sorted, and the subtitle text statement with the highest similarity in the sorting result is determined as the subtitle text statement that matches the video frame. If there is only one candidate subtitle text statement, then the candidate subtitle text statement is determined as the subtitle text statement that matches the video frame; If there are no candidate subtitle text statements, the video frame is marked as having no matching subtitle text statements.
6. The method according to claim 1, characterized in that, Before determining the matching relationship between subtitle text statements and video frames based on the similarity between each video frame and each subtitle text statement, the process includes: Based on the subtitle feature fusion matrix, usable subtitle text statements are selected from the subtitle text statements.
7. The method according to claim 6, characterized in that, Based on the subtitle feature fusion matrix, usable subtitle text statements are selected from the subtitle text statements, including: The subtitle feature fusion matrix is input into a pre-trained multilayer perceptron model for processing, and an intermediate result vector with B elements is output. The second activation function is used to calculate each element in the intermediate result vector to obtain a matching identifier vector with B elements. The b-th element of the matching identifier vector represents the availability score of the b-th subtitle text statement, and B is the number of subtitle text statements contained in the subtitle text set. Based on the matching identifier vector, the usability score corresponding to each subtitle text statement is compared with a pre-set usability threshold; If the availability score of a subtitle text statement is greater than or equal to the availability threshold, then the subtitle text statement is determined to be an available subtitle text statement.
8. A video subtitle matching device, wherein, The device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to perform the method of any one of claims 1 to 7.
9. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 7.
Citation Information
Patent Citations
Method for performing multi-mode video question answering by using frame-subtitle self-supervision
CN112860945A
Video bullet screen matching method and device, computer equipment and storage medium
CN114339362A
