Video processing method, apparatus, device, and storage medium

Abstract

The present invention provides a video processing method, an apparatus, a device, and a storage medium. The method comprises: firstly, acquiring a target video clip, and extracting a video frame feature sequence of the target video clip; then, determining, from the video frame feature sequence, a key frame feature sequence of the target video clip; and on the basis of the video frame feature sequence and the key frame feature sequence, determining a text fragment corresponding to the target video clip. According to embodiments of the present invention, the key frame feature sequence of the target video clip is determined, and the text fragment corresponding to the target video clip is determined in light of the video frame feature sequence and the key frame feature sequence. Because the key frame feature sequence can better reflect a livestreaming viewing assistance action in the target video clip, the text fragment corresponding to the target video clip can be determined more effectively on the basis of the method.

Description

Video processing method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese invention patent application with application number 202311121214.X, entitled “A video processing method, device, equipment and storage medium” and filing date August 31, 2023, and incorporates the entirety of that application into this document by reference. Technical Field

[0003] The present disclosure relates to the field of data processing, and in particular to a video processing method, apparatus, device, and storage medium. Background Art

[0004] With the continuous development of online live broadcast technology, video live broadcast has gradually become one of the main ways for people to obtain information and social entertainment. Therefore, the number of live broadcast rooms of various types is increasing.

[0005] For the live broadcast room type of sign language live broadcast, how to effectively determine the text content corresponding to the sign language live broadcast video is a technical problem that needs to be solved urgently.

[0006] Summary of the Invention

[0007] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for processing live video.

[0008] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0009] Acquire a target video clip, and extract a video frame feature sequence of the target video clip;

[0010] Determining a key frame feature sequence of the target video segment from the video frame feature sequence;

[0011] Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined.

[0012] In an optional implementation, the target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment.

[0013] In an optional embodiment, the video frame feature sequence includes video frame features having a sequential relationship, and determining the key frame feature sequence of the target video segment from the video frame feature sequence includes:

[0014] Determining N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters respectively; wherein N is a preset natural number;

[0015] A clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters.

[0016] In an optional embodiment, the determining of the key frame feature sequence of the target video segment using a clustering algorithm based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters includes:

[0017] Determining the Euclidean distance between a first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters, and assigning the first video frame feature to the cluster corresponding to the minimum Euclidean distance;

[0018] Determining, based on video frame features of a first cluster among the N clusters, a centroid vector of the first cluster as a target centroid vector, and determining whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold;

[0019] If it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is not less than the preset threshold, updating the target centroid vector to the current centroid vector of the first cluster, and continuing to perform the step of determining the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold;

[0020] Based on the current centroid vector of the first cluster, a key frame feature sequence of the target video segment is determined.

[0021] In an optional implementation, determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence includes:

[0022] The target model is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence; wherein the target model is obtained by pre-training the model using video segment samples and text segment samples with corresponding relationships.

[0023] In an optional embodiment, before determining the text segment corresponding to the target video segment using the target model based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0024] Obtaining video segment samples and text segment samples with corresponding relationships;

[0025] Extracting a video frame feature sequence of the video clip sample using a feature extraction module, and determining a key frame feature sequence of the video clip sample from the video frame feature sequence;

[0026] Using an encoder to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, and inputting the fused feature sequence into a decoder;

[0027] Using the decoder to generate a predicted text segment based on the fused feature sequence, and determining a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update model parameters of the feature extraction module, the encoder, and the decoder through backpropagation;

[0028] After determining the model parameters that minimize the loss value, a trained target model is obtained; wherein the target model includes the encoder and the decoder.

[0029] In an optional implementation, the method is applied to a live broadcast server, and obtaining the target video segment includes:

[0030] Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment;

[0031] Accordingly, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0032] The text segment corresponding to the target video segment is sent to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0033] In an optional implementation, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0034] The text segment corresponding to the target video segment is converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room; wherein the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

[0035] In an optional implementation, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0036] Determine a live broadcast special effect corresponding to the text segment; wherein the live broadcast special effect is used to be displayed on the live broadcast room page of the target live broadcast room.

[0037] In a second aspect, the present disclosure provides a video processing device, the device comprising:

[0038] Video acquisition module, used to acquire target video clips;

[0039] A first feature extraction module is used to extract a video frame feature sequence of the target video clip;

[0040] A first determining module is configured to determine a key frame feature sequence of the target video segment from the video frame feature sequence;

[0041] The second determining module is configured to determine a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0042] In a third aspect, the present disclosure provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device implements the above method.

[0043] In a fourth aspect, the present disclosure provides a live video processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned method when executing the computer program.

[0044] In a fifth aspect, the present disclosure provides a computer program product, which includes a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0046] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] FIG1 is a flow chart of a video processing method provided by an embodiment of the present disclosure;

[0048] FIG2 is a schematic diagram of training a target model provided by an embodiment of the present disclosure;

[0049] FIG3 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure;

[0050] FIG4 is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0051] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0052] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0053] With the continuous development of live streaming technology, video streaming has gradually become one of the primary ways people obtain information and engage in social entertainment. As a result, the number of various types of live streaming rooms is increasing. For example, live streaming rooms where anchors use sign language to broadcast live. For these live streaming rooms, how to effectively determine the text content corresponding to the sign language live video is a technical problem that needs to be solved.

[0054] To this end, an embodiment of the present disclosure provides a video processing method. First, a target video clip is obtained and a video frame feature sequence of the target video clip is extracted. Then, from the video frame feature sequence, a key frame feature sequence of the target video clip is determined. Based on the video frame feature sequence and the key frame feature sequence, a text clip corresponding to the target video clip is determined. The embodiment of the present disclosure determines the text clip corresponding to the target video clip by determining the key frame feature sequence of the target video clip and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language movements in the target video clip, the above method can more effectively determine the text clip corresponding to the sign language video.

[0055] Based on this, an embodiment of the present disclosure provides a video processing method. Referring to FIG1 , which is a flow chart of a video processing method provided by an embodiment of the present disclosure, the method specifically includes:

[0056] S101: Acquire a target video segment and extract a video frame feature sequence of the target video segment.

[0057] The video processing method provided by the embodiments of the present disclosure can be applied to a server, where the server may include a cloud server, a local server, etc.

[0058] The target video segment in the embodiment of the present disclosure includes auxiliary actions, and may be a video segment containing auxiliary action content. The auxiliary action may be, for example, a sign language action. The target video segment is also referred to as a sign language video segment hereinafter.

[0059] In one optional embodiment, the target video segment may be a video segment in a live video stream. Specifically, the target video segment may be a sign language video segment in a sign language live video stream. That is, the host in the live broadcast room uses sign language gestures to broadcast live, and a sign language live video stream can be obtained. The host in the live broadcast room uses sign language gestures to broadcast live, and the host's sign language can help viewers watching the live broadcast understand the host's live content.

[0060] In the embodiment of the present disclosure, after acquiring the target video segment, the image features of each video frame in the target video segment are extracted to form a video frame feature sequence of the target video segment. The video frame feature sequence includes the image features of the video frames in a sequential relationship.

[0061] Assume that the target video segment is a sign language video segment V, and its video frame feature sequence f = feature1, feature2, ..., feature n , where n is the total number of frames of the sign language video segment V, feature i is the image feature of the i-th frame of the sign language video segment V.

[0062] In an optional embodiment, a trained feature extraction model can be used to extract a video frame feature sequence of the target video clip. The feature extraction model can be implemented based on a 3D-ResNet three-dimensional convolutional residual neural network model.

[0063] S102: Determine a key frame feature sequence of the target video segment from the video frame feature sequence.

[0064] In order to more effectively determine the text segment corresponding to the sign language video segment, the embodiment of the present disclosure can not only extract the video frame feature sequence of the target video segment, but also extract the key frame feature sequence of the target video segment to determine the text segment corresponding to the sign language video segment.

[0065] The key frame feature sequence may include image features of key frame images in the target video clip. For example, the key frame images may be video images containing sign language actions in the sign language video clip.

[0066] In one optional embodiment, after extracting the video frame feature sequence of the target video clip, a key frame feature sequence is obtained from the video frame feature sequence. Specifically, a clustering algorithm can be used to determine the key frame feature sequence from the video frame feature sequence. The specific implementation of the clustering algorithm is not limited and can be, for example, a K-means algorithm.

[0067] In an optional embodiment, after extracting the video frame feature sequence of the target video clip, N video frame features are first determined from the video frame feature sequence as the initial centroid vectors corresponding to the N clusters, also known as the current centroid vectors. N is a natural number set based on demand. Optionally, N can be determined based on the total number of frames in the video frame feature sequence. N is proportional to the total number of frames. The larger the total number of frames, the larger the N setting is. Conversely, the smaller the total number of frames, the smaller the N setting is. Assume that N is set to 32.

[0068] After determining the current centroid vectors corresponding to the N clusters, a clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters.

[0069] In an optional embodiment, for any video frame feature in the video frame feature sequence (assuming it is the first video frame feature), the Euclidean distance between the first video frame feature and the current centroid vectors corresponding to the N clusters is calculated, and the first video frame feature is classified into the cluster corresponding to the smallest Euclidean distance among the Euclidean distances (assuming it is the jth cluster).

[0070] After the classification is completed for each video frame feature in the video frame feature sequence, for any one of the N clusters (assuming it is the first cluster), based on the various video frame features classified in the first cluster, the centroid vector of the first cluster is recalculated as the target centroid vector, and it is determined whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than a preset threshold. If not, the target centroid vector is determined as the current centroid vector of the first cluster. Determining whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than the preset threshold can be specifically determining whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than the preset threshold.

[0071] Using the above method, after updating the current centroid vector for each of the N clusters, the cluster calculation is performed again, that is, for each video frame feature in the video frame feature sequence, the cluster corresponding to the current centroid vector with the smallest Euclidean distance is recalculated until it is determined that the Euclidean distance between the recalculated target centroid vector and the current centroid vector is less than a preset threshold, then the update of the current centroid vector of the cluster is stopped.

[0072] After the current centroid vector corresponding to each of the N clusters stops updating, a key frame feature sequence of the target video segment is determined based on the current centroid vectors of each cluster. Specifically, based on the order relationship between the clusters, the current centroid vectors of each cluster constitute the key frame feature sequence of the target video segment.

[0073] For ease of understanding, assume that the current centroid vectors corresponding to the N clusters are u1, u2, ..., u n , video frame feature sequence f = feature1, feature2, ..., feature m , the Euclidean distance d between the i-th video frame feature and the current centroid vector of the j-th cluster i =||feature i -u j ||, after calculating the Euclidean distance between the i-th video frame feature and the current centroid vector of each cluster, the i-th video frame feature is classified into the cluster corresponding to the smallest Euclidean distance.

[0074] For each cluster, the centroid vector is recalculated based on the features of each video frame. The specific method for calculating the centroid vector is not limited. For clusters whose recalculated centroid vector is the same as the current centroid vector, the current centroid vector of the cluster is not updated.

[0075] Assume that the current centroid vectors corresponding to the N clusters finally determined are u1', u2', ..., u n ', then u1', u2', ..., u n 'Determine the key frame feature sequence of the target video segment.

[0076] S103: Determine a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0077] The text segment is used to express the meaning of the sign language action in the target video segment.

[0078] In the embodiment of the present disclosure, after obtaining the video frame feature sequence and key frame feature sequence of the target video clip, the text segment corresponding to the target video clip is determined based on the video frame feature sequence and the key frame feature sequence, that is, the text segment corresponding to the sign language action included in the target video clip is determined.

[0079] In an optional implementation, a pre-trained target model may be used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0080] Specifically, the target model may be a sign language recognition model. As shown in FIG2 , it is a schematic diagram of a target model training provided by an embodiment of the present disclosure.

[0081] First, a large number of corresponding video clip samples and text clip samples are obtained. Then, a feature extraction module is used to extract the video frame feature sequence of the video clip sample, and the key frame feature sequence of the video clip sample is determined from the video frame feature sequence. Then, an encoder is used to perform feature fusion on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, wherein the fused feature sequence can highlight the key frame feature sequence through the video frame feature sequence, so that the fused feature sequence can better reflect the characteristics of the sign language movements in the sign language video clip.

[0082] After obtaining the fused feature vector, the encoder inputs it into the decoder, which generates a predicted text segment based on the fused feature sequence and determines the loss value by comparing the predicted text segment with the real text segment sample. The loss value is used to update the model parameters of the feature extraction module, encoder, and decoder through back propagation.

[0083] After training a large number of video clip samples and text clip samples with corresponding relationships, when determining the model parameters that minimize the loss value, a trained target model is obtained; specifically, the target model may include a trained encoder and decoder.

[0084] In the disclosed embodiment, the fused feature sequence is input into a decoder. The decoder uses a long short-term memory (LSTM) model to decode the feature vectors in the fused feature sequence to generate a text sequence describing the image, i.e., a predicted text segment. After obtaining the predicted text segment, the decoder compares it with the corresponding text segment sample to calculate a loss function and obtain a loss value. Backpropagation is then used based on the loss function to update the model parameters, such as weight parameters, specifically including the weight parameters for both the encoder and decoder.

[0085] During the model prediction (also known as model inference) phase, a trained feature extraction module is used to extract the target video clip's video frame feature sequence and, from this video frame feature sequence, determine the target video clip's key frame feature sequence. A trained encoder then fuses the video frame feature sequence and key frame feature sequence to generate a fused feature sequence, which is then fed into a trained decoder. The decoder then generates a text segment based on the fused feature sequence, which serves as the text segment corresponding to the target video segment.

[0086] In the video processing method provided by the embodiment of the present disclosure, first, a target video segment is obtained, and a video frame feature sequence of the target video segment is extracted. Then, from the video frame feature sequence, a key frame feature sequence of the target video segment is determined. Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined; wherein, the text segment is used to express the meaning of the sign language action in the target video segment. The embodiment of the present disclosure determines the text segment corresponding to the target video segment by determining the key frame feature sequence of the target video segment and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language action in the target video segment, the text segment corresponding to the sign language video can be more effectively determined based on the above method.

[0087] Based on the above embodiments, the present disclosure provides a specific application scenario of a video processing method. Specifically, the video processing method can be applied to a live broadcast scenario of a target live broadcast room, where the target live broadcast room refers to a live broadcast room type where the anchor broadcasts live via sign language.

[0088] The target video segment may be a live video segment in a live video stream of a target live broadcast room.

[0089] In actual applications, during the live broadcast of the target live broadcast room, the live broadcast server obtains a live video segment from the live video stream of the target live broadcast room as the target video segment. Through the video processing method of the embodiment of the present invention, the text segment corresponding to each live video segment in the live video stream of the target live broadcast room is determined in turn, and the text segment is returned to the client corresponding to the target live broadcast room in chronological order.

[0090] In an optional implementation, after determining the text segment corresponding to the target video segment, the text segment is sent to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0091] In another optional implementation, the text segment corresponding to the target video segment can be converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room and played synchronously with the target video segment in the target live broadcast room.

[0092] In another optional implementation, after determining the text segment corresponding to the target video segment, the live broadcast special effect corresponding to the text segment can be determined, and the live broadcast special effect can be displayed on the live broadcast room page of the target live broadcast room.

[0093] By displaying subtitles, synchronously playing voice, and displaying live special effects on the live broadcast room page of the target live broadcast room, the display effect of the live broadcast room of the anchor through sign language can be improved, thereby improving the experience of the users watching the broadcast.

[0094] Based on the above method embodiment, the present disclosure further provides a video processing device. Referring to FIG3 , which is a schematic structural diagram of a video processing device provided by an embodiment of the present disclosure, the device includes:

[0095] The video acquisition module 301 is used to acquire a target video clip;

[0096] A first feature extraction module 302 is configured to extract a video frame feature sequence of the target video segment;

[0097] A first determining module 303 is configured to determine a key frame feature sequence of the target video segment from the video frame feature sequence;

[0098] The second determining module 304 is configured to determine a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0099] In an optional embodiment, the target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment. In other words, the text segment can be used to explain the auxiliary action, making it easier for the user to understand the meaning of the auxiliary action.

[0100] In an optional implementation, the video frame feature sequence includes video frame features having a sequential relationship, and the first determining module includes:

[0101] A first determining submodule is configured to determine N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters, wherein N is a preset natural number;

[0102] The second determining submodule is configured to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters using a clustering algorithm.

[0103] In an optional implementation, the first determining module includes:

[0104] A third determining submodule is configured to determine the Euclidean distance between a first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters, and classify the first video frame feature into the cluster corresponding to the minimum Euclidean distance;

[0105] a fourth determination submodule, configured to determine, based on video frame features of a first cluster classified into the N clusters, a centroid vector of the first cluster as a target centroid vector, and determine whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold;

[0106] an updating submodule, configured to update the target centroid vector to the current centroid vector of the first cluster if it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is not less than the preset threshold, and continue to perform the step of calculating the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold;

[0107] The fifth determining submodule is configured to determine a key frame feature sequence of the target video segment based on the current centroid vector of the first cluster.

[0108] In an optional implementation manner, the second determining module is specifically configured to:

[0109] The target model is used to determine a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0110] In an optional embodiment, the device further includes:

[0111] A sample acquisition module is used to acquire video segment samples and text segment samples that have corresponding relationships;

[0112] a second feature extraction module, configured to extract a video frame feature sequence of the video clip sample using a feature extraction module, and determine a key frame feature sequence of the video clip sample from the video frame feature sequence;

[0113] A feature fusion module is used to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence using an encoder to obtain a fused feature sequence, and input the fused feature sequence into a decoder;

[0114] a loss value calculation module, configured to generate a predicted text segment based on the fused feature sequence using the decoder, and determine a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update the model parameters of the feature extraction module, the encoder, and the decoder through backpropagation;

[0115] A model acquisition module is used to obtain a trained target model after determining the model parameters that minimize the loss value; wherein the target includes the encoder and the decoder.

[0116] In an optional implementation manner, the device is applied to a live broadcast server, and the video acquisition module is specifically used to:

[0117] Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment;

[0118] Accordingly, the device further includes:

[0119] The first sending module is used to send the text segment corresponding to the target video segment to the client corresponding to the target live broadcast room; wherein, the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0120] In an optional embodiment, the device further includes:

[0121] A voice conversion module is used to convert the text segment corresponding to the target video segment into a voice segment, and send the voice segment to the client corresponding to the target live broadcast room; wherein, the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

[0122] In an optional embodiment, the device further includes:

[0123] The special effects determination module is used to determine the live broadcast special effects corresponding to the text segment; wherein the live broadcast special effects are used to be displayed on the live broadcast room page of the target live broadcast room.

[0124] In the video processing device provided by the embodiment of the present disclosure, first, a target video segment is obtained and a video frame feature sequence of the target video segment is extracted. Then, from the video frame feature sequence, a key frame feature sequence of the target video segment is determined. Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined. The embodiment of the present disclosure determines the text segment corresponding to the target video segment by determining the key frame feature sequence of the target video segment and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language movements in the target video segment, the above-mentioned method can more effectively determine the text segment corresponding to the sign language video.

[0125] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure also provide a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device implements the live video processing method described in the embodiments of the present disclosure.

[0126] The embodiments of the present disclosure further provide a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the live video processing method described in the embodiments of the present disclosure is implemented.

[0127] In addition, the embodiment of the present disclosure further provides a video processing device, as shown in FIG4 , which may include:

[0128] Processor 401, memory 402, input device 403, and output device 404. The number of processors 401 in the video processing device can be one or more, and Figure 4 uses one processor as an example. In some embodiments of the present disclosure, the processor 401, memory 402, input device 403, and output device 404 can be connected via a bus or other means, wherein Figure 4 uses the bus connection as an example.

[0129] Memory 402 can be used to store software programs and modules. Processor 401 executes various functional applications and data processing of the video processing device by running the software programs and modules stored in memory 402. Memory 402 may primarily include a program storage area and a data storage area. The program storage area may store an operating system, at least one application required for a function, and the like. Furthermore, memory 402 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Input device 403 may be used to receive input digital or character information and generate signal input related to user settings and function control of the video processing device.

[0130] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, thereby realizing the various functions of the above-mentioned video processing device.

[0131] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0132] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, comprising: Acquire a target video clip, and extract a video frame feature sequence of the target video clip; Determining a key frame feature sequence of the target video clip from the video frame feature sequence; Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined.

2. The method according to claim 1, wherein: The target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment.

3. The method according to claim 1, wherein: The video frame feature sequence includes video frame features having a sequential relationship, and determining the key frame feature sequence of the target video segment from the video frame feature sequence includes: Determine N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters respectively; wherein N is a preset natural number; A clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters.

4. The method according to claim 3, wherein: The method of using a clustering algorithm to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors respectively corresponding to the N clusters includes: Determine the Euclidean distance between a first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, and classify the first video frame feature into the cluster corresponding to the minimum Euclidean distance; Based on the video frame features of a first cluster classified into the N clusters, determining a centroid vector of the first cluster as a target centroid vector, and determining whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold; If the Euclidean distance between the target centroid vector and the current centroid vector is determined is not less than the preset threshold, updating the target centroid vector to the current centroid vector of the first cluster, and continuing to perform the step of determining the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold; Based on the current centroid vector of the first cluster, a key frame feature sequence of the target video segment is determined.

5. The method according to claim 1, wherein: The determining, based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment comprises: The target model is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

6. The method according to claim 5, wherein: Before determining the text segment corresponding to the target video segment by using the target model based on the video frame feature sequence and the key frame feature sequence, the method further includes: Obtaining video segment samples and text segment samples having corresponding relationships; Extracting a video frame feature sequence of the video clip sample using a feature extraction module, and determining a key frame feature sequence of the video clip sample from the video frame feature sequence; Using an encoder to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, and inputting the fused feature sequence into a decoder; Using the decoder to generate a predicted text segment based on the fused feature sequence, and determining a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update the model parameters of the feature extraction module, the encoder and the decoder through back propagation; After determining the model parameters that minimize the loss value, obtain the trained target Model; wherein the target model includes the encoder and the decoder.

7. The method according to claim 1, wherein: The method is applied to a live broadcast server, and obtaining a target video segment includes: Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment; Correspondingly, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: The text segment corresponding to the target video segment is sent to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

8. The method according to claim 7, wherein: After determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: The text segment corresponding to the target video segment is converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room; wherein the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

9. The method according to claim 7, wherein: After determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: Determine a live broadcast special effect corresponding to the text segment; wherein the live broadcast special effect is used to be displayed on a live broadcast room page of the target live broadcast room.

10. A video processing device, comprising: A video acquisition module is used to acquire a target video clip; A first feature extraction module, used to extract a video frame feature sequence of the target video clip; A first determining module, configured to determine a key frame feature sequence of the target video segment from the video frame feature sequence; The second determination module is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

11. A computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium. When the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 9.

12. A video processing device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images