Method and apparatus for processing video data, electronic device and storage medium

Abstract

The application provides a method and device for processing video data, an electronic device and a storage medium, wherein the method comprises: obtaining video data to be processed; inputting a video frame and / or initial text into a target model to obtain target visual features and target text features; and determining a processing result of the video data to be processed according to the target visual features and the target text features. Through the application, the problem that the processing operation speed is slow in the related art, and the text information associated with the input video and capable of accurately representing the video cannot be obtained is solved.

Description

Technical Field

[0001] This application relates to the field of big data, and in particular to a method and apparatus for video data processing, an electronic device, and a storage medium. Background Technology

[0002] Currently, when determining whether there is a certain correlation between each video frame in video information and the corresponding audio text, a pre-trained model can be used to obtain the final trained model, and the final model can be used to output the correlation between the video and the current text.

[0003] Most existing video and text pre-trained models are based on the comparative training of video clips and text. Some have added audio for comparison. However, the operation of video clips involves the processing of three-dimensional data, so the operation speed is relatively slow and consumes too much GPU performance and graphics card resources. Moreover, since the video and text in related technologies are compared and learned, the final model cannot obtain text information that is highly correlated with the input video and can accurately represent the video. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for video data processing, to at least solve the problems in related technologies where the processing speed is slow and the text information that is highly relevant to and can accurately represent the input video cannot be obtained well.

[0005] According to one aspect of the embodiments of this application, a method for video data processing is provided, the method comprising:

[0006] Acquire video data to be processed, wherein the video data to be processed contains multiple video frames and audio information corresponding to the video frames;

[0007] The video frame and / or initial text are input into the target model to obtain target visual features and target text features. The initial text is information obtained after speech recognition of the audio information, and the target visual features are video feature information without audio. The target model is used to obtain the target visual features and the target text features with a correlation degree greater than the correlation degree threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0008] Based on the target visual features and the target text features, the processing result of the video data to be processed is determined.

[0009] According to another aspect of the embodiments of this application, a video data processing apparatus is also provided, the apparatus comprising:

[0010] The first acquisition unit is used to acquire video data to be processed, wherein the video data to be processed includes multiple video frames and audio information corresponding to the video frames;

[0011] The first input unit is used to input the video frame and / or initial text into the target model to obtain target visual features and target text features. The initial text is information obtained after speech recognition of the audio information, and the target visual features are video feature information without audio. The target model is used to obtain the target visual features and the target text features with a correlation degree greater than the correlation degree threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0012] The determining unit is used to determine the processing result of the video data to be processed based on the target visual features and the target text features.

[0013] Optionally, the device further includes:

[0014] The second acquisition unit is used to acquire initial video data, wherein the initial video data is a sample set for training the initial model;

[0015] A partitioning unit is used to partition the initial video frames in the initial video data into equal-frequency frame number groups, wherein each video frame group contains the same number of the initial video frames.

[0016] The second input unit is used to input the initial video frame in the video frame group into the encoder to obtain visual features, wherein the number of visual features is at least one, and the visual features are used to characterize visualization information.

[0017] The conversion unit is used to convert the initial audio into characters using a text preprocessing model to generate a first text feature, wherein the initial audio is the audio information contained in the initial video data that corresponds to the initial video frame;

[0018] The unit is configured to obtain reference visual features and reference text features based on the visual features, the first text features, and the self-attention model.

[0019] The adjustment unit is used to adjust the model parameters based on the reference visual features, the reference text features, and positive and negative sample pairs in the training sample set to obtain the target model.

[0020] Optionally, the obtained unit includes:

[0021] The processing module is used to input the first text features into the translation model and perform dimensionality reduction processing through a downsampling algorithm to obtain the second text features, wherein the number of the first text features is at least one, and the second text features correspond to the first text features;

[0022] The generation module is used to generate the reference visual features and the reference text features based on the visual features, the second text features, and the self-attention model.

[0023] Optionally, the generation module includes:

[0024] The splicing subunit is used to splice the visual features and the second text features dimensionally to obtain the spliced ​​first sequence features;

[0025] The first obtaining subunit is used to obtain aggregated visual features based on the visual features, wherein the aggregated visual features are feature vectors obtained by aggregating multiple visual features;

[0026] The second subunit is used to obtain aggregated text features based on the second text features, wherein the aggregated text features are feature vectors obtained by aggregating multiple second text features;

[0027] The third subunit is used to concatenate the first sequence features with the aggregated visual features and the aggregated text features to obtain the concatenated second sequence features.

[0028] The fourth sub-unit is used to input the second sequence features into the self-attention model to obtain the reference visual features and the reference text features.

[0029] Optionally, the adjustment unit includes:

[0030] The matching module is used to match the reference visual features and the reference text features with the positive sample pairs and the negative samples in the training sample set, respectively, to obtain matching values;

[0031] An adjustment module is used to adjust the vector distance between the reference visual features and the reference text features and the model parameters using the matching value and the matching threshold to obtain the target model.

[0032] Optionally, the adjustment module includes:

[0033] The reduction subunit is used to reduce the vector distance between the reference visual feature and the reference text feature when the matching value between the reference visual feature and the reference text feature and the positive sample pair is greater than the matching threshold;

[0034] An additional subunit is added to increase the vector distance between the reference visual feature and the reference text feature when the matching value between the reference visual feature and the reference text feature and the negative sample pair is greater than the matching threshold.

[0035] Optionally, the device further includes:

[0036] The arrangement unit is used to arrange the second text features sequentially according to the time order in which the first text features were input into the translation model after the second text features were obtained through dimensionality reduction processing by the downsampling algorithm;

[0037] An additional unit is added to the second text feature to add a location identifier vector, wherein the location identifier vector is used to locate the current time when the second text feature was generated.

[0038] According to another aspect of the embodiments of this application, an electronic device is also provided, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; wherein the memory is used to store a computer program; and the processor is used to execute the method steps of any of the above embodiments by running the computer program stored in the memory.

[0039] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to execute the method steps of any of the above embodiments when running.

[0040] This application embodiment is applied to computer vision in the field of deep learning technology. In this application embodiment, video data to be processed is acquired, wherein the video data to be processed contains multiple video frames and audio information corresponding to the video frames; the video frames and / or initial text are input into a target model to obtain target visual features and target text features, wherein the initial text is information obtained after speech recognition of the audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation greater than a correlation threshold. The target model is obtained by adjusting the model parameters of the initial model; the processing result of the video data to be processed is determined based on the target visual features and target text features. Since the embodiments of this application are obtained by continuously adjusting the model parameters of the initial model, and the correlation between the target visual features and target text features obtained by the target model is greater than the correlation threshold, the target visual features and target text features can be obtained by inputting the video data to be processed into the target model. This improves the processing speed, and the mutual representation ability between the target visual features and target text features is also strong. This solves the problem in related technologies that the processing speed is slow and that it is not possible to obtain text information that is highly correlated with the input video and can accurately represent the video. Attached Figure Description

[0041] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0042] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the hardware environment for an optional video data processing method according to an embodiment of the present invention;

[0044] Figure 2 This is a flowchart illustrating an optional video data processing method according to an embodiment of this application;

[0045] Figure 3 This is a schematic diagram of an optional video data processing model framework according to an embodiment of this application;

[0046] Figure 4 This is a structural block diagram of an optional video data processing apparatus according to an embodiment of this application;

[0047] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application. Detailed Implementation

[0048] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] According to one aspect of the embodiments of this application, a method for video data processing is provided. Optionally, in this embodiment, the above-described video data processing method can be applied to, for example... Figure 1 In the hardware environment shown. For example... Figure 1 As shown, terminal 102 may include memory 104, processor 106, and display 108 (optional component). Terminal 102 can communicate with server 112 via network 110. Server 112 can provide services (such as application services) to the terminal or clients installed on the terminal. Database 114 can be set up on or independently of server 112 to provide data storage services to server 112. In addition, server 112 may run a processing engine 116, which can be used to execute the steps performed by server 112.

[0051] Optionally, terminal 102 may be, but is not limited to, a terminal capable of computing data, such as a mobile terminal (e.g., mobile phone, tablet computer), laptop computer, PC (Personal Computer), etc. The aforementioned network may include, but is not limited to, a wireless network or a wired network. The wireless network includes Bluetooth, Wi-Fi (Wireless Fidelity), and other networks that enable wireless communication. The aforementioned wired network may include, but is not limited to, a wide area network (WAN), a metropolitan area network (MAN), and a local area network (LAN). The aforementioned server 112 may include, but is not limited to, any hardware device capable of computing.

[0052] Furthermore, in this embodiment, the video data processing method described above can also be applied to, but is not limited to, a powerful independent processing device without requiring data interaction. For example, the processing device can be, but is not limited to, a powerful terminal device; that is, the various operations in the video data processing method described above can be integrated into a single independent processing device. The above is merely an example, and no limitation is made in this embodiment.

[0053] Optionally, in this embodiment, the video data processing method described above can be executed by server 112, by terminal 102, or by both server 112 and terminal 102. The video data processing method of this embodiment can also be executed by a client installed on terminal 102.

[0054] Taking running on a server as an example, Figure 2 This is a flowchart illustrating an optional video data processing method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0055] Step S201: Obtain the video data to be processed, wherein the video data to be processed contains multiple video frames and audio information corresponding to the video frames;

[0056] Step S202: Input the video frame and / or initial text into the target model to obtain target visual features and target text features. The initial text is the information obtained after speech recognition of the audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation greater than the correlation threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0057] Step S203: Determine the processing result of the video data to be processed based on the target visual features and target text features.

[0058] Optionally, in this embodiment, a trained target model is used to process the video data. Furthermore, the server first obtains the video data to be processed, which contains multiple video frames, each with corresponding audio information. ASR speech recognition can be performed on this audio information to generate corresponding initial text. Then, the video frames and initial text are input into the target model to output target visual features and target text features. The target model is obtained by adjusting the model parameters of the initial model, which is the trained model.

[0059] It should be noted that in the embodiments of this application, the target visual features do not contain audio information, but only some visual image information. The target text features are obtained by recognizing the audio information. At this time, the correlation between the target visual features and the target text features is greater than the correlation threshold. That is, the target visual features and the target text features have good mutual representation ability. That is, the text corresponding to the target visual features is the target text features, and the image corresponding to the target text features is the target visual features.

[0060] In this way, the target visual features and target text features obtained from the video data to be processed in the target model are the final processing results, and the target visual features and target text features have a strong correlation and good mutual representation ability.

[0061] In this embodiment, video data to be processed is acquired, which includes multiple video frames and corresponding audio information. The video frames and / or initial text are input into a target model to obtain target visual features and target text features. The initial text is obtained through speech recognition of the audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation greater than a correlation threshold. The target model is obtained by adjusting the model parameters of the initial model. Based on the target visual features and target text features, the processing result of the video data to be processed is determined. Since this embodiment obtains the target model by continuously adjusting the model parameters of the initial model, and the correlation between the target visual features and target text features obtained by this target model is greater than the correlation threshold, the processing speed is improved because the target visual features and target text features can be obtained simply by inputting the video data to be processed into the target model. Furthermore, the mutual representation ability between the target visual features and target text features is strong, thus solving the problem in related technologies where the processing speed is slow and the text information that is highly correlated with and accurately represents the video cannot be obtained well from the input video.

[0062] As an optional embodiment, before inputting video frames and / or initial text into the target model to obtain target visual features and target text features, the method further includes:

[0063] Obtain initial video data, which is the sample set for training the initial model;

[0064] The initial video frames in the initial video data are divided into equal-frequency frame groups, with each video frame group containing the same number of initial video frames.

[0065] The initial video frames within the video frame group are input into the encoder to obtain visual features, wherein the number of visual features is at least one, and the visual features are used to characterize visualization information.

[0066] The initial audio is converted into characters using a text preprocessing model to generate the first text feature, where the initial audio is the audio information contained in the initial video data that corresponds to the initial video frame;

[0067] Based on visual features, first text features, and a self-attention model, reference visual features and reference text features are obtained.

[0068] The model parameters are adjusted based on reference visual features, reference text features, and positive and negative sample pairs in the training sample set to obtain the target model.

[0069] Optionally, in the embodiments of this application, the process of obtaining the target model from the initial training model is described: first, the sample set for training the initial model is obtained, that is, the initial video data is obtained.

[0070] Then, the initial video is uniformly sampled at the same frequency to obtain multiple video frame groups. For example, the number of frames extracted (or divided) is N, that is, each video frame group contains N initial video frames. The data of each initial video frame is obtained as N*C*H*W, where N is the number of frames, which is a hyperparameter that can be adjusted, C is the number of channels of the input image, and W and H are the length and width of the image in the input initial video frame.

[0071] like Figure 3 The initial video frames are input into the encoder to obtain visual features. This encoder can be a model pre-trained on large-scale image data, such as Faster R-CNN trained on the COCO dataset. Figure 3 The video features are N*D dimensional, where D is 512.

[0072] For the initial audio information corresponding to the initial video frame contained in the initial video data, perform ASR speech recognition, select the first M characters, padding with 0s if necessary, and perform character conversion (the BERT text preprocessing model can be used for conversion), converting it into M*D dimensional first text features, see [link to documentation]. Figure 3 .

[0073] like Figure 3 The first text features are sequentially input into the translation model (i.e., the Transformer model). This Transformer model is designed to process sequential text features. The output M*D dimensional features are downsampled and reduced in dimensionality using a Pooling module to obtain K*D dimensional second text features. These second text features are output sequentially according to the order in which the first text features were input into the translation model. To clarify the temporal order of the second text features' generation, a 1*D dimensional positional vector is added to each second text feature to pinpoint the current time of its generation.

[0074] Then, based on the visual features, the second text features, and the self-attention model, the visual features and the second text features are input into the self-attention model, and reference visual features and reference text features are output. Among them, the number of first text features is at least one, and the second text features correspond to the first text features.

[0075] Finally, the model parameters are adjusted based on the reference visual features, reference text features, and positive and negative sample pairs in the training sample set to obtain the target model.

[0076] In this embodiment, the length of video data is controlled by extracting video frames at equal frequencies, and the length of text in the video data is controlled by passing a transformer model and a downsampling module in between. This allows visual features and text features to be input together into the self-attention module for cross-modal learning, thus solving the problem of inconsistent lengths between visual features and text features.

[0077] As an optional embodiment, generating reference visual features and reference text features based on visual features, second text features, and a self-attention model includes:

[0078] The visual features and the second text features are concatenated dimensionally to obtain the first sequence of features after concatenation.

[0079] Based on the visual features, aggregated visual features are obtained, where the aggregated visual features are feature vectors obtained by aggregating multiple visual features.

[0080] Based on the second text features, aggregated text features are obtained, where the aggregated text features are feature vectors obtained by aggregating multiple second text features;

[0081] The first sequence features are concatenated with the aggregated visual features and aggregated text features to obtain the concatenated second sequence features.

[0082] The second sequence features are input into the self-attention model to obtain reference visual features and reference text features.

[0083] Optionally, the N*D dimensional visual features and the K*D dimensional second text features are concatenated to obtain a (K+N)*D first sequence feature. Multiple visual features are then aggregated to obtain a higher-level feature vector: aggregated visual features (i.e.,...) Figure 3 In the cls token1, multiple second text features are aggregated to obtain the higher-level feature vector: aggregated text features (i.e. Figure 3 The first sequence feature is concatenated with the aggregated visual features and aggregated text features to obtain the second sequence feature with dimensions of (K+N+2)*D. Finally, the second sequence feature is input into the self-attention model, and the output cls token3 and cls token4 are the final reference visual features and reference text features after cross-modal learning.

[0084] In this embodiment of the application, cross-modal learning is further achieved by setting aggregated visual features and aggregated text features.

[0085] As an optional implementation, the model parameters are adjusted based on reference visual features, reference text features, and positive and negative sample pairs in the training sample set to obtain the target model, including:

[0086] The reference visual features and reference text features are matched with positive and negative sample pairs in the training sample set, respectively, to obtain matching values;

[0087] By using the matching value and the matching threshold, the vector distance between the reference visual features and the reference text features, as well as the model parameters, are adjusted to obtain the target model.

[0088] Optionally, after obtaining the reference visual features and reference text features, the reference visual features and reference text features are matched with positive sample pairs in the training sample set to obtain a matching value. The reference visual features and reference text features are then matched with negative sample pairs in the training sample set to obtain another matching value. Simultaneously, a preset matching threshold is set, and the obtained matching values ​​are compared with this preset matching threshold. The matching threshold is also a custom value and can be arbitrarily set based on historical experience.

[0089] If the matching value between the reference visual features and reference text features and positive sample pairs is greater than the matching threshold, it indicates that the reference visual features and reference text features are close to the positive sample pairs. In this case, the vector distance between the reference visual features and reference text features should be decreased. Conversely, if the matching value between the reference visual features and reference text features and negative sample pairs is greater than the matching threshold, it indicates that the reference visual features and reference text features are close to the negative sample pairs. In this case, the vector distance between the reference visual features and reference text features should be increased. Simultaneously, loss is calculated using a loss function, and backpropagation is performed based on the obtained probability results to adjust the model parameters of the initial model, control the vector distance, and finally obtain the trained target model.

[0090] As an optional implementation, if a downstream task wants to obtain visual features from video data, it can extract video frames at equal intervals and then input them into the target model to obtain visual features. If it wants to obtain ASR text features, it can input the ASR text into the target model to obtain text features, and then apply the visual features and text features to the downstream task.

[0091] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0093] According to another aspect of the embodiments of this application, an apparatus for video data processing for implementing the above-described video data processing method is also provided. Figure 4 This is a structural block diagram of an optional video data processing apparatus according to an embodiment of this application, such as... Figure 4 As shown, the device may include:

[0094] The first acquisition unit 401 is used to acquire video data to be processed, wherein the video data to be processed contains multiple video frames and audio information corresponding to the video frames.

[0095] The first input unit 402 is connected to the first acquisition unit 401 and is used to input video frames and / or initial text into the target model to obtain target visual features and target text features. The initial text is information obtained after speech recognition of audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation degree greater than the correlation degree threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0096] The determining unit 403, connected to the first input unit 402, is used to determine the processing result of the video data to be processed based on the target visual features and target text features.

[0097] It should be noted that the first acquisition unit 401 in this embodiment can be used to execute the above step S201, the first input unit 402 in this embodiment can be used to execute the above step S202, and the determination unit 403 in this embodiment can be used to execute the above step S203.

[0098] Through the above modules, the target model is obtained by continuously adjusting the model parameters of the initial model. At the same time, the correlation between the target visual features and target text features obtained by the target model is greater than the correlation threshold. Since the target visual features and target text features can be obtained by inputting the video data to be processed into the target model, the processing speed is improved. At the same time, the mutual representation ability between the target visual features and target text features is also strong. This solves the problem in related technologies that the processing speed is slow and that it is not possible to obtain text information that is highly correlated with the input video and can accurately represent the video.

[0099] As an optional embodiment, the device further includes:

[0100] The second acquisition unit is used to acquire initial video data, wherein the initial video data is the sample set for training the initial model;

[0101] The partitioning unit is used to divide the initial video frames in the initial video data into equal-frequency frame number partitions to obtain multiple video frame groups, wherein each video frame group contains the same number of initial video frames.

[0102] The second input unit is used to input the initial video frame in the video frame group into the encoder to obtain visual features, wherein the number of visual features is at least one, and the visual features are used to characterize visualization information.

[0103] The conversion unit is used to convert the initial audio into characters using a text preprocessing model to generate the first text feature, wherein the initial audio is the audio information contained in the initial video data that corresponds to the initial video frame;

[0104] The unit is used to obtain reference visual features and reference text features based on visual features, first text features, and self-attention model;

[0105] The adjustment unit is used to adjust the model parameters based on reference visual features, reference text features, and positive and negative sample pairs in the training sample set to obtain the target model.

[0106] As an optional embodiment, the obtained unit includes:

[0107] The processing module is used to input the first text features into the translation model and perform dimensionality reduction processing through a downsampling algorithm to obtain the second text features. The number of the first text features is at least one, and the second text features correspond to the first text features.

[0108] The generation module is used to generate reference visual features and reference text features based on visual features, second text features, and a self-attention model.

[0109] As an optional embodiment, the generation module includes:

[0110] The splicing subunit is used to splice the visual features and the second text features dimensionally to obtain the spliced ​​first sequence features.

[0111] The first sub-unit is used to obtain aggregated visual features based on visual features, wherein the aggregated visual features are feature vectors obtained by aggregating multiple visual features;

[0112] The second subunit is used to obtain aggregated text features based on the second text features, wherein the aggregated text features are feature vectors obtained by aggregating multiple second text features;

[0113] The third sub-unit is used to concatenate the first sequence features with the aggregated visual features and aggregated text features to obtain the concatenated second sequence features.

[0114] The fourth sub-unit is used to input the second sequence features into the self-attention model to obtain reference visual features and reference text features.

[0115] As an optional embodiment, the adjustment unit includes:

[0116] The matching module is used to match the reference visual features and reference text features with the positive and negative sample pairs in the training sample set, respectively, to obtain the matching value.

[0117] The adjustment module is used to adjust the vector distance and model parameters between the reference visual features and the reference text features using the matching value and the matching threshold, so as to obtain the target model.

[0118] As an optional embodiment, the adjustment module includes:

[0119] The reduction subunit is used to reduce the vector distance between the reference visual features and the reference text features when the matching value between the reference visual features and the reference text features and the positive sample pair is greater than the matching threshold.

[0120] A sub-unit is added to increase the vector distance between the reference visual features and the reference text features when the matching value between the reference visual features and the reference text features and the negative sample pair is greater than the matching threshold.

[0121] As an optional embodiment, the device further includes:

[0122] The arrangement unit is used to arrange the second text features sequentially according to the time order in which the first text features were input into the translation model after the dimensionality reduction process is performed by the downsampling algorithm to obtain the second text features;

[0123] An additional unit is added to add a location identifier vector to the second text feature, wherein the location identifier vector is used to locate the current time when the second text feature was generated.

[0124] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented through software or hardware, and the hardware environment includes the network environment.

[0125] According to another aspect of the embodiments of this application, an electronic device for implementing the above-described video data processing method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0126] Figure 5 This is a structural block diagram of an optional electronic device according to an embodiment of this application, such as... Figure 5 As shown, it includes a processor 501, a communication interface 502, a memory 503, and a communication bus 504. The processor 501, communication interface 502, and memory 503 communicate with each other via the communication bus 504.

[0127] Memory 503 is used to store computer programs;

[0128] When processor 501 executes a computer program stored in memory 503, it performs the following steps:

[0129] Acquire the video data to be processed, which contains multiple video frames and corresponding audio information for each video frame;

[0130] Video frames and / or initial text are input into the target model to obtain target visual features and target text features. The initial text is the information obtained after speech recognition of audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation greater than the correlation threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0131] Based on the target's visual features and target text features, determine the processing result of the video data to be processed.

[0132] Optionally, in this embodiment, the communication bus can be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0133] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0134] The memory may include RAM, or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0135] As an example, such as Figure 5 As shown, the memory 503 may include, but is not limited to, the first acquisition unit 401, the first input unit 402, and the determination unit 403 in the video data processing apparatus. Furthermore, it may include, but is not limited to, other module units in the video data processing apparatus, which will not be elaborated upon in this example.

[0136] The processors mentioned above can be general-purpose processors, including but not limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; they can also be DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0137] In addition, the aforementioned electronic device also includes a display for showing the results of video data processing.

[0138] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0139] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. The device that implements the above video data processing method can be a terminal device, such as a smartphone (e.g., Android phone, iOS phone), tablet computer, PDA, mobile Internet device (MID), PAD, etc. Figure 5 This does not limit the structure of the aforementioned electronic devices. For example, the terminal device may also include components that are more advanced than those described above. Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0140] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, ROM, RAM, disk or optical disk, etc.

[0141] According to another aspect of the embodiments of this application, a storage medium is also provided. Optionally, in this embodiment, the storage medium may be used to execute program code for a method of video data processing.

[0142] Optionally, in this embodiment, the storage medium may be located on at least one of the multiple network devices in the network shown in the above embodiment.

[0143] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps:

[0144] Acquire the video data to be processed, which contains multiple video frames and corresponding audio information for each video frame;

[0145] Video frames and / or initial text are input into the target model to obtain target visual features and target text features. The initial text is the information obtained after speech recognition of audio information, and the target visual features are video feature information without audio. The target model is used to obtain target visual features and target text features with a correlation greater than the correlation threshold. The target model is obtained by adjusting the model parameters of the initial model.

[0146] Based on the target's visual features and target text features, determine the processing result of the video data to be processed.

[0147] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated in this embodiment.

[0148] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, ROMs, RAMs, portable hard drives, magnetic disks, or optical disks.

[0149] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, causing the computer device to perform the video data processing method steps of any of the above embodiments.

[0150] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] If the integrated units in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the video data processing methods of the various embodiments of this application.

[0152] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0153] In the several embodiments provided in this application, it should be understood that the disclosed client can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0154] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the solution provided in this embodiment, depending on actual needs.

[0155] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0156] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method of video data processing, the method comprising: The method comprises: acquiring video data to be processed, wherein the video data to be processed contains a plurality of video frames and audio information corresponding to the video frames; inputting the video frames and / or initial text into a target model to obtain target visual features and target text features, wherein the initial text is information obtained by performing speech recognition on the audio information, the target visual features are video feature information without audio, the target model is used to obtain the target visual features and the target text features with a correlation degree greater than a correlation degree threshold, the target model is obtained by matching reference visual features and reference text features with positive sample pairs and negative samples in a training sample set respectively to obtain matching values, and adjusting vector distances between the reference visual features and the reference text features and model parameters of an initial model by using the matching values and a matching threshold, wherein the reference visual features and the reference text features are obtained according to visual features, first text features and a self-attention model of initial video data used to train the initial model; determining a processing result of the video data to be processed according to the target visual features and the target text features.

2. The method of claim 1, wherein, Before the inputting the video frames and / or the initial text into the target model to obtain the target visual features and the target text features, the method further comprises: acquiring initial video data, wherein the initial video data is a sample set used to train the initial model; performing equal-frequency-frame-number division on initial video frames in the initial video data to obtain a plurality of video frame groups, wherein the number of the initial video frames contained in each video frame group is the same; inputting the initial video frames in the video frame groups into an encoder to obtain visual features, wherein the number of the visual features is at least one, and the visual features are used to represent visual information; performing character conversion on initial audio by using a text preprocessing model to generate first text features, wherein the initial audio is audio information corresponding to the initial video frames and contained in the initial video data; obtaining reference visual features and reference text features according to the visual features, the first text features and a self-attention model; adjusting the model parameters according to the reference visual features, the reference text features and positive sample pairs and negative sample pairs in a training sample set to obtain the target model.

3. The method of claim 2, wherein, The obtaining the reference visual features and the reference text features according to the visual features, the first text features and the self-attention model comprises: inputting the first text features into a translation model and performing dimension reduction processing on the first text features by using a down-sampling algorithm to obtain second text features, wherein the number of the first text features is at least one, and the second text features correspond to the first text features; generating the reference visual features and the reference text features according to the visual features, the second text features and the self-attention model.

4. The method of claim 3, wherein, The generating the reference visual features and the reference text features according to the visual features, the second text features and the self-attention model comprises: dimensionally splice the visual features and the second text features to obtain spliced first sequence features; obtain aggregated visual features according to the visual features, wherein the aggregated visual features are feature vectors aggregated from the visual features; obtain aggregated text features according to the second text features, wherein the aggregated text features are feature vectors aggregated from the second text features; dimensionally splice the first sequence features, the aggregated visual features, and the aggregated text features to obtain spliced second sequence features; input the second sequence features into the self-attention model to obtain the reference visual features and the reference text features.

5. The method of claim 1, wherein, adjusting the vector distance between the reference visual features and the reference text features by using the matching value and a matching threshold value includes: decreasing the vector distance between the reference visual features and the reference text features when the matching value between the reference visual features and the reference text features and the positive sample pair is greater than the matching threshold value; increasing the vector distance between the reference visual features and the reference text features when the matching value between the reference visual features and the reference text features and the negative sample pair is greater than the matching threshold value.

6. The method of claim 3, wherein, after the second text features are obtained by the dimension reduction processing of the down-sampling algorithm, the method further includes: arranging the second text features in sequence according to the time sequence in which the first text features are input into the translation model; adding a position identification vector to the second text features, wherein the position identification vector is used to locate the current time at which the second text features are generated.

7. An apparatus for video data processing, the apparatus comprising: The device includes: a first acquisition unit configured to acquire video data to be processed, wherein the video data to be processed includes a plurality of video frames and audio information corresponding to the video frames; a first input unit configured to input the video frames and / or initial text into a target model to obtain target visual features and target text features, wherein the initial text is information obtained by performing speech recognition on the audio information, the target visual features are video feature information without audio, the target model is configured to obtain the target visual features and the target text features with a correlation degree greater than a correlation degree threshold, the target model is obtained by adjusting vector distance between reference visual features and reference text features and model parameters of an initial model by using matching values and a matching threshold value, the matching values being obtained by matching the reference visual features and the reference text features with positive sample pairs and negative samples in a training sample set, respectively, the reference visual features and the reference text features being obtained according to visual features, first text features, and a self-attention model of initial video data used to train the initial model; a determination unit configured to determine a processing result of the video data to be processed according to the target visual features and the target text features.

8. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein, The processor, the communication interface, and the memory complete communication with each other through the communication bus, and the device is characterized in that The memory is configured to store a computer program. The processor is configured to execute the method steps of any one of claims 1 to 6 by running the computer program stored on the memory.

9. A computer readable storage medium, characterized in that, The storage medium has stored therein a computer program, wherein the computer program, when executed by a processor, implements the method steps of any one of claims 1 to 6.

Images