Video processing method and device, equipment and storage medium

Abstract

The invention provides a video processing method and device, equipment and a storage medium, and the method comprises the steps: firstly, obtaining a target video clip, extracting a video frame feature sequence of the target video clip, then determining a key frame feature sequence of the target video clip from the video frame feature sequence, and obtaining a key frame feature sequence of the target video clip; and determining a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence. According to the embodiment of the invention, the key frame feature sequence of the target video clip is determined, the video frame feature sequence and the key frame feature sequence are combined, the text clip corresponding to the target video clip is determined, and the key frame feature sequence can better reflect the watching and playing auxiliary action in the target video clip, so that the watching and playing efficiency is improved. Based on the above mode, the text segment corresponding to the target video segment can be determined more effectively.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing, and in particular to a video processing method, device, equipment and storage medium. Background Art

[0002] With the continuous development of online live broadcast technology, video live broadcast has gradually become one of the main ways for people to obtain information and social entertainment. Therefore, the number of live broadcast rooms of various types is increasing.

[0003] For the live broadcast room type of sign language live broadcast, how to effectively determine the text content corresponding to the sign language live broadcast video is a technical problem that urgently needs to be solved. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present disclosure provide a method for processing live video.

[0005] In a first aspect, the present disclosure provides a video processing method, the method comprising:

[0006] Acquire a target video clip, and extract a video frame feature sequence of the target video clip;

[0007] Determining a key frame feature sequence of the target video clip from the video frame feature sequence;

[0008] Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined.

[0009] In an optional implementation, the target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment.

[0010] In an optional implementation, the video frame feature sequence includes video frame features having a sequential relationship, and determining the key frame feature sequence of the target video segment from the video frame feature sequence includes:

[0011] Determine N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters respectively; wherein N is a preset natural number;

[0012] A clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters.

[0013] In an optional implementation, the determining of the key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters using a clustering algorithm includes:

[0014] Determine the Euclidean distance between a first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, and classify the first video frame feature into the cluster corresponding to the minimum Euclidean distance;

[0015] Based on the video frame features of a first cluster classified into the N clusters, determining a centroid vector of the first cluster as a target centroid vector, and determining whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold;

[0016] If it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is not less than the preset threshold, the target centroid vector is updated to the current centroid vector of the first cluster, and the step of determining the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters is continued until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold;

[0017] Based on the current centroid vector of the first cluster, a key frame feature sequence of the target video segment is determined.

[0018] In an optional implementation, determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence includes:

[0019] The target model is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence; wherein the target model is obtained by pre-training the model using video segment samples and text segment samples with corresponding relationships.

[0020] In an optional implementation manner, before determining the text segment corresponding to the target video segment using the target model based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0021] Obtaining video segment samples and text segment samples having corresponding relationships;

[0022] Extracting a video frame feature sequence of the video clip sample using a feature extraction module, and determining a key frame feature sequence of the video clip sample from the video frame feature sequence;

[0023] Using an encoder to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, and inputting the fused feature sequence into a decoder;

[0024] Using the decoder to generate a predicted text segment based on the fused feature sequence, and determining a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update the model parameters of the feature extraction module, the encoder and the decoder through back propagation;

[0025] After determining the model parameters that minimize the loss value, a trained target model is obtained; wherein the target model includes the encoder and the decoder.

[0026] In an optional implementation manner, the method is applied to a live broadcast server, and obtaining the target video segment includes:

[0027] Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment;

[0028] Correspondingly, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0029] The text segment corresponding to the target video segment is sent to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0030] In an optional implementation manner, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0031] The text segment corresponding to the target video segment is converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room; wherein the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

[0032] In an optional implementation manner, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes:

[0033] Determine a live broadcast special effect corresponding to the text segment; wherein the live broadcast special effect is used to be displayed on a live broadcast room page of the target live broadcast room.

[0034] In a second aspect, the present disclosure provides a video processing device, the device comprising:

[0035] A video acquisition module is used to acquire a target video clip;

[0036] A first feature extraction module, used to extract a video frame feature sequence of the target video clip;

[0037] A first determining module, configured to determine a key frame feature sequence of the target video segment from the video frame feature sequence;

[0038] The second determination module is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0039] In a third aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the above method.

[0040] In a fourth aspect, the present disclosure provides a live video processing device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above method when executing the computer program.

[0041] In a fifth aspect, the present disclosure provides a computer program product, wherein the computer program product comprises a computer program / instructions, and the computer program / instructions implement the above method when executed by a processor.

[0042] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has at least the following advantages:

[0043] The disclosed embodiment provides a video processing method, first, obtaining a target video segment and extracting a video frame feature sequence of the target video segment, then determining a key frame feature sequence of the target video segment from the video frame feature sequence, and determining a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence. The disclosed embodiment determines the text segment corresponding to the target video segment by determining the key frame feature sequence of the target video segment and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language movements in the target video segment, the text segment corresponding to the sign language video can be more effectively determined based on the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0046] Figure 1 A flowchart of a video processing method provided by an embodiment of the present disclosure;

[0047] Figure 2 A training diagram of a target model provided in an embodiment of the present disclosure;

[0048] Figure 3 A schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure;

[0049] Figure 4 A schematic diagram of the structure of a video processing device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0050] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0052] With the continuous development of online live broadcast technology, video live broadcast has gradually become one of the main ways for people to obtain information and social entertainment. As a result, the number of various types of live broadcast rooms is increasing, for example, the type of live broadcast room where the anchor uses sign language to broadcast live. For the type of live broadcast room where the anchor uses sign language to broadcast live, how to effectively determine the text content corresponding to the sign language live broadcast video is a technical problem that needs to be solved urgently.

[0053] To this end, the disclosed embodiment provides a video processing method, first, obtaining a target video segment and extracting a video frame feature sequence of the target video segment, then determining a key frame feature sequence of the target video segment from the video frame feature sequence, and determining a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence. The disclosed embodiment determines the text segment corresponding to the target video segment by determining the key frame feature sequence of the target video segment and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language movements in the target video segment, the text segment corresponding to the sign language video can be more effectively determined based on the above method.

[0054] Based on this, the present disclosure provides a video processing method, referring to Figure 1 , is a flow chart of a video processing method provided by an embodiment of the present disclosure, the method specifically comprising:

[0055] S101: Acquire a target video segment, and extract a video frame feature sequence of the target video segment.

[0056] The video processing method provided by the embodiments of the present disclosure may be applied to a server, wherein the server may include a cloud server, a local server, etc.

[0057] The target video segment in the embodiment of the present disclosure includes auxiliary actions, and may be a video segment containing auxiliary action content. The auxiliary action may be, for example, a sign language action. The target video segment is also referred to as a sign language video segment hereinafter.

[0058] In an optional implementation, the target video segment may be a video segment in a live video stream, and specifically, the target video segment may be a sign language video segment in a sign language live video stream, that is, the host in the live broadcast room broadcasts live with sign language movements, and a sign language live video stream may be obtained. The host in the live broadcast room broadcasts live with sign language movements, and the host's sign language may assist viewers watching the live broadcast to understand the host's live content.

[0059] In the disclosed embodiment, after acquiring the target video segment, the image features of each video frame in the target video segment are extracted to form a video frame feature sequence of the target video segment, wherein the video frame feature sequence includes image features of video frames having a sequential relationship.

[0060] Assume that the target video segment is a sign language video segment V, and its video frame feature sequence f = feature1, feature2, ..., feature n , where n is the total number of frames of the sign language video segment V, feature i is the image feature of the i-th frame of the sign language video segment V.

[0061] In an optional implementation, a trained feature extraction model may be used to extract a video frame feature sequence of a target video clip. The feature extraction model may be implemented based on a 3D-ResNet three-dimensional convolutional residual neural network model.

[0062] S102: Determine a key frame feature sequence of the target video clip from the video frame feature sequence.

[0063] In order to more effectively determine the text segment corresponding to the sign language video segment, the disclosed embodiment can not only extract the video frame feature sequence of the target video segment, but also extract the key frame feature sequence of the target video segment to determine the text segment corresponding to the sign language video segment.

[0064] The key frame feature sequence may include image features of key frame images in the target video clip. For example, the key frame images may be video images containing sign language movements in the sign language video clip.

[0065] In an optional implementation, after extracting the video frame feature sequence of the target video clip, a key frame feature sequence is obtained from the video frame feature sequence. Specifically, a clustering algorithm can be used to determine the key frame feature sequence from the video frame feature sequence. There is no limitation on the specific implementation of the clustering algorithm, for example, it can be a K-means algorithm.

[0066] In an optional implementation, after extracting the video frame feature sequence of the target video clip, N video frame features are first determined from the video frame feature sequence as the initial centroid vectors corresponding to the N clusters, also called the current centroid vector. Wherein, N is a natural number set based on demand. Optionally, N can be determined based on the total number of frames in the video frame feature sequence. N is proportional to the total number of frames. The larger the total number of frames, the larger the N setting is. Conversely, the smaller the total number of frames, the smaller the N setting is. Assume that N is set to 32.

[0067] After determining the current centroid vectors corresponding to the N clusters respectively, a clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively.

[0068] In an optional implementation, for any video frame feature in a video frame feature sequence (assumed to be the first video frame feature), the Euclidean distance between the first video frame feature and the current centroid vectors corresponding to N clusters is calculated, and the first video frame feature is classified into the cluster corresponding to the smallest Euclidean distance among the Euclidean distances (assumed to be the jth cluster).

[0069] After the classification is completed for each video frame feature in the video frame feature sequence, for any cluster among the N clusters (assuming it is the first cluster), based on the various video frame features classified in the first cluster, the centroid vector of the first cluster is recalculated as the target centroid vector, and it is determined whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than a preset threshold, and if not, the target centroid vector is determined as the current centroid vector of the first cluster. Determining whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than the preset threshold can be specifically determining whether the Euclidean distance between the target centroid vector and the current centroid vector of the first cluster is less than the preset threshold.

[0070] Using the above method, after updating the current centroid vector for each of the N clusters, the cluster calculation is performed again, that is, for each video frame feature in the video frame feature sequence, the cluster corresponding to the current centroid vector with the smallest Euclidean distance is recalculated, until it is determined that the Euclidean distance between the recalculated target centroid vector and the current centroid vector is less than a preset threshold, then the update of the current centroid vector of the cluster is stopped.

[0071] After the current centroid vectors corresponding to each cluster in the N clusters stop updating, the key frame feature sequence of the target video segment is determined based on the current centroid vectors of each cluster. Specifically, based on the order relationship between the clusters, the current centroid vectors of each cluster constitute the key frame feature sequence of the target video segment.

[0072] For ease of understanding, assume that the current centroid vectors corresponding to the N clusters are u1, u2, ..., u n , video frame feature sequence f = feature1, feature2, ..., feature m , the Euclidean distance d between the i-th video frame feature and the current centroid vector of the j-th cluster i =||feature i -u j ||, after calculating the Euclidean distance between the i-th video frame feature and the current centroid vector of each cluster, the i-th video frame feature is classified into the cluster corresponding to the smallest Euclidean distance.

[0073] For each cluster, the centroid vector is recalculated based on the features of each video frame, and the specific method of calculating the centroid vector is not limited. For a cluster whose recalculated centroid vector is the same as the current centroid vector, the current centroid vector of the cluster is no longer updated.

[0074] Assume that the current centroid vectors corresponding to the final N clusters are """

[0075] u1,u2,...,u n , then u1, u2, ..., u n The key frame feature sequence of the target video segment is determined.

[0076] S103: Determine a text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0077] The text segment is used to express the meaning of the sign language action in the target video segment.

[0078] In the disclosed embodiment, after obtaining the video frame feature sequence and key frame feature sequence of the target video segment, the text segment corresponding to the target video segment is determined based on the video frame feature sequence and the key frame feature sequence, that is, the text segment corresponding to the sign language action included in the target video segment is determined.

[0079] In an optional implementation, a pre-trained target model may be used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0080] Specifically, the target model may be a sign language recognition model. Figure 2 As shown, it is a training diagram of a target model provided in an embodiment of the present disclosure.

[0081] Firstly, a large number of corresponding video clip samples and text clip samples are obtained, and then a feature extraction module is used to extract a video frame feature sequence of the video clip samples, and a key frame feature sequence of the video clip samples is determined from the video frame feature sequence. Then, an encoder is used to perform feature fusion on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, wherein the fused feature sequence can highlight the key frame feature sequence through the video frame feature sequence, so that the fused feature sequence can better reflect the characteristics of the sign language movements in the sign language video clip.

[0082] After obtaining the fused feature vector, the encoder inputs it into the decoder, which generates a predicted text segment based on the fused feature sequence and determines the loss value by comparing the predicted text segment with the real text segment sample. The loss value is used to update the model parameters of the feature extraction module, encoder and decoder through back propagation.

[0083] After training with a large number of corresponding video clip samples and text clip samples, when determining the model parameters that minimize the loss value, the trained target model is obtained; specifically, the target model may include a trained encoder and decoder.

[0084] In the disclosed embodiment, the fused feature sequence is input into the decoder. The function of the decoder is to decode the feature vector in the fused feature sequence using the long short-term memory (LSTM) model to generate a text sequence describing the image, i.e., a predicted text segment. After obtaining the predicted text segment, the decoder compares it with the corresponding text segment sample to calculate the loss function to obtain the loss value. Then, based on the loss function, the parameters of the model, such as weight parameters, are updated by back propagation, specifically including the weight parameters of the encoder and decoder, respectively.

[0085] In the model prediction (also called model inference) stage, the trained feature extraction module is used to extract the video frame feature sequence of the target video segment, and the key frame feature sequence of the target video segment is determined from the video frame feature sequence. Then, the trained encoder performs feature fusion on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, which is input into the trained decoder. The decoder generates a text segment based on the fused feature sequence as the text segment corresponding to the target video segment.

[0086] In the video processing method provided by the embodiment of the present disclosure, first, a target video segment is obtained, and a video frame feature sequence of the target video segment is extracted, then a key frame feature sequence of the target video segment is determined from the video frame feature sequence, and a text segment corresponding to the target video segment is determined based on the video frame feature sequence and the key frame feature sequence; wherein the text segment is used to express the meaning of the sign language action in the target video segment. The embodiment of the present disclosure determines the text segment corresponding to the target video segment by determining the key frame feature sequence of the target video segment, and combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language action in the target video segment, the text segment corresponding to the sign language video can be more effectively determined based on the above method.

[0087] Based on the above embodiments, the embodiments of the present disclosure provide a specific application scenario of a video processing method. Specifically, the video processing method can be applied to a live broadcast scenario of a target live broadcast room, where the target live broadcast room refers to a type of live broadcast room where the anchor broadcasts live through sign language.

[0088] The target video segment may be a live video segment in a live video stream of a target live broadcast room.

[0089] In actual applications, during the live broadcast of the target live broadcast room, the live broadcast server obtains a live video segment from the live video stream of the target live broadcast room as the target video segment, and through the video processing method of the embodiment of the present invention, determines the text segment corresponding to each live video segment in the live video stream of the target live broadcast room in turn, and returns the text segment to the client corresponding to the target live broadcast room in chronological order.

[0090] In an optional implementation, after determining the text segment corresponding to the target video segment, the text segment is sent to a client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0091] In another optional implementation, the text segment corresponding to the target video segment can also be converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room, and played synchronously with the target video segment in the target live broadcast room.

[0092] In another optional implementation, after determining the text segment corresponding to the target video segment, the live broadcast special effect corresponding to the text segment can be determined, and the live broadcast special effect can be displayed on the live broadcast room page of the target live broadcast room.

[0093] By displaying subtitles, synchronously playing voice, and displaying live special effects on the live broadcast room page of the target live broadcast room, the display effect of the live broadcast room where the anchor broadcasts live via sign language can be improved, thereby improving the experience of users watching the broadcast.

[0094] Based on the above method embodiment, the present disclosure also provides a video processing device, referring to Figure 3 , is a schematic diagram of the structure of a video processing device provided by an embodiment of the present disclosure, the device comprising:

[0095] The video acquisition module 301 is used to acquire a target video segment;

[0096] A first feature extraction module 302, configured to extract a video frame feature sequence of the target video segment;

[0097] A first determination module 303 is used to determine a key frame feature sequence of the target video segment from the video frame feature sequence;

[0098] The second determination module 304 is configured to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0099] In an optional implementation, the target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment. That is, the text segment can be used to explain the auxiliary action to facilitate the user to understand the meaning of the auxiliary action.

[0100] In an optional implementation, the video frame feature sequence includes video frame features having a sequential relationship, and the first determining module includes:

[0101] A first determination submodule is used to determine N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters respectively; wherein N is a preset natural number;

[0102] The second determination submodule is used to determine the key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters by using a clustering algorithm.

[0103] In an optional implementation manner, the first determining module includes:

[0104] A third determination submodule is used to determine the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, and classify the first video frame feature into the cluster corresponding to the minimum Euclidean distance;

[0105] a fourth determination submodule, configured to determine, based on video frame features of a first cluster classified into the N clusters, a centroid vector of the first cluster as a target centroid vector, and determine whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold;

[0106] An updating submodule, for updating the target centroid vector to the current centroid vector of the first cluster if it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is not less than the preset threshold, and continuing to perform the step of calculating the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold;

[0107] The fifth determination submodule is configured to determine a key frame feature sequence of the target video segment based on the current centroid vector of the first cluster.

[0108] In an optional implementation manner, the second determining module is specifically configured to:

[0109] The target model is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

[0110] In an optional implementation, the device further includes:

[0111] A sample acquisition module is used to acquire video segment samples and text segment samples having corresponding relationships;

[0112] A second feature extraction module is used to extract a video frame feature sequence of the video clip sample using a feature extraction module, and determine a key frame feature sequence of the video clip sample from the video frame feature sequence;

[0113] A feature fusion module is used to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence using an encoder to obtain a fused feature sequence, and input the fused feature sequence into a decoder;

[0114] A loss value calculation module, used to generate a predicted text segment based on the fused feature sequence using the decoder, and determine a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update the model parameters of the feature extraction module, the encoder and the decoder through back propagation;

[0115] A model acquisition module is used to acquire a trained target model after determining the model parameters that minimize the loss value; wherein the target includes the encoder and the decoder.

[0116] In an optional implementation manner, the device is applied to a live broadcast server, and the video acquisition module is specifically used to:

[0117] Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment;

[0118] Accordingly, the device further comprises:

[0119] The first sending module is used to send the text segment corresponding to the target video segment to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

[0120] In an optional implementation, the device further includes:

[0121] A voice conversion module is used to convert the text segment corresponding to the target video segment into a voice segment, and send the voice segment to the client corresponding to the target live broadcast room; wherein the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

[0122] In an optional implementation, the device further includes:

[0123] A special effect determination module is used to determine the live broadcast special effect corresponding to the text segment; wherein the live broadcast special effect is used to be displayed on the live broadcast room page of the target live broadcast room.

[0124] In the video processing device provided by the embodiment of the present disclosure, first, a target video segment is obtained, and a video frame feature sequence of the target video segment is extracted. Then, from the video frame feature sequence, a key frame feature sequence of the target video segment is determined. Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined. The embodiment of the present disclosure determines the key frame feature sequence of the target video segment, and determines the text segment corresponding to the target video segment by combining the video frame feature sequence and the key frame feature sequence. Since the key frame feature sequence can better reflect the sign language movements in the target video segment, the text segment corresponding to the sign language video can be more effectively determined based on the above method.

[0125] In addition to the above-mentioned method and apparatus, the embodiments of the present disclosure further provide a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device implements the live video processing method described in the embodiments of the present disclosure.

[0126] The embodiments of the present disclosure also provide a computer program product, which includes a computer program / instructions. When the computer program / instructions are executed by a processor, the live video processing method described in the embodiments of the present disclosure is implemented.

[0127] In addition, the present disclosure also provides a video processing device, see Figure 4 As shown, it may include:

[0128] Processor 401, memory 402, input device 403 and output device 404. The number of processors 401 in the video processing device can be one or more. Figure 4 In some embodiments of the present disclosure, the processor 401, the memory 402, the input device 403 and the output device 404 may be connected via a bus or other means, wherein: Figure 4 The example of connecting through bus is taken in the following.

[0129] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing of the video processing device by running the software programs and modules stored in the memory 402. The memory 402 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc. In addition, the memory 402 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. The input device 403 can be used to receive input digital or character information, and generate signal input related to user settings and function control of the video processing device.

[0130] Specifically in this embodiment, the processor 401 will load the executable files corresponding to the processes of one or more applications into the memory 402 according to the following instructions, and the processor 401 will run the applications stored in the memory 402, thereby realizing various functions of the above-mentioned video processing device.

[0131] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0132] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire a target video clip, and extract a video frame feature sequence of the target video clip; Determining a key frame feature sequence of the target video clip from the video frame feature sequence; Based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment is determined.

2. The method according to claim 1, characterized in that The target video segment includes an auxiliary action, and the text segment is used to express the meaning of the auxiliary action in the target video segment.

3. The method according to claim 1, characterized in that The video frame feature sequence includes video frame features having a sequential relationship, and determining the key frame feature sequence of the target video segment from the video frame feature sequence includes: Determine N video frame features from the video frame feature sequence as current centroid vectors corresponding to the N clusters respectively; wherein N is a preset natural number; A clustering algorithm is used to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors corresponding to the N clusters.

4. The method according to claim 3, characterized in that The method of using a clustering algorithm to determine a key frame feature sequence of the target video segment based on the video frame feature sequence and the current centroid vectors respectively corresponding to the N clusters includes: Determine the Euclidean distance between a first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters respectively, and classify the first video frame feature into the cluster corresponding to the minimum Euclidean distance; Based on the video frame features of a first cluster classified into the N clusters, determining a centroid vector of the first cluster as a target centroid vector, and determining whether a Euclidean distance between the target centroid vector and the current centroid vector is less than a preset threshold; If it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is not less than the preset threshold, the target centroid vector is updated to the current centroid vector of the first cluster, and the step of determining the Euclidean distance between the first video frame feature in the video frame feature sequence and the current centroid vectors corresponding to the N clusters is continued until it is determined that the Euclidean distance between the target centroid vector and the current centroid vector is less than the preset threshold; Based on the current centroid vector of the first cluster, a key frame feature sequence of the target video segment is determined.

5. The method according to claim 1, characterized in that The determining, based on the video frame feature sequence and the key frame feature sequence, a text segment corresponding to the target video segment comprises: The target model is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

6. The method according to claim 5, characterized in that Before determining the text segment corresponding to the target video segment by using the target model based on the video frame feature sequence and the key frame feature sequence, the method further includes: Obtaining video segment samples and text segment samples having corresponding relationships; Extracting a video frame feature sequence of the video clip sample using a feature extraction module, and determining a key frame feature sequence of the video clip sample from the video frame feature sequence; Using an encoder to perform feature fusion processing on the video frame feature sequence and the key frame feature sequence to obtain a fused feature sequence, and inputting the fused feature sequence into a decoder; Using the decoder to generate a predicted text segment based on the fused feature sequence, and determining a loss value by comparing the predicted text segment with the text segment sample; the loss value is used to update the model parameters of the feature extraction module, the encoder and the decoder through back propagation; After determining the model parameters that minimize the loss value, a trained target model is obtained; wherein the target model includes the encoder and the decoder.

7. The method according to claim 1, characterized in that The method is applied to a live broadcast server, and obtaining a target video segment includes: Obtain a live video segment from the live video stream of the target live broadcast room as the target video segment; Correspondingly, after determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: The text segment corresponding to the target video segment is sent to the client corresponding to the target live broadcast room; wherein the text segment is used to be displayed as a subtitle on the live broadcast room page of the target live broadcast room.

8. The method according to claim 7, characterized in that After determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: The text segment corresponding to the target video segment is converted into a voice segment, and the voice segment is sent to the client corresponding to the target live broadcast room; wherein the voice segment is used to be played synchronously with the target video segment in the target live broadcast room.

9. The method according to claim 7, characterized in that: After determining the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence, the method further includes: Determine a live broadcast special effect corresponding to the text segment; wherein the live broadcast special effect is used to be displayed on a live broadcast room page of the target live broadcast room.

10. A video processing device, characterized in that: The device comprises: A video acquisition module is used to acquire a target video clip; A first feature extraction module, used to extract a video frame feature sequence of the target video clip; A first determining module, configured to determine a key frame feature sequence of the target video segment from the video frame feature sequence; The second determination module is used to determine the text segment corresponding to the target video segment based on the video frame feature sequence and the key frame feature sequence.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device implements the method according to any one of claims 1 to 9.

12. A video processing device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method according to any one of claims 1 to 9 is implemented.

Images