Video processing method and device, equipment, medium and program product
By extracting the timing information of video data and combining the cross attention mechanism, the problems of low accuracy and high training cost in video processing tasks are solved, and the efficiency and accuracy of multimodal video processing tasks are achieved.
Patent Information
- Application Number
- CN202310582633.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-22
AI Technical Summary
The existing video language pre-training methods have low accuracy and high training costs in video processing tasks. The performance of image language pre-training methods is poor when transferred to video processing, mainly due to the differences in image data and video data and the heavy processing task. Differently, high-quality video text data cannot be fully utilized.
By extracting time-series image features with timing information from video data, the timing information is used to enhance the representation ability of image features, combining image encoder, text encoder and video adapter, multimodal feature fusion is adopted to improve video processing by using cross-attention mechanism. The accuracy and effectiveness of the task.
It improves the accuracy and effectiveness of multimodal video processing tasks, enhances the alignment of video and language features, reduces training costs, and is suitable for various video processing tasks such as video subtitles and video Q&A.
Smart Images

Figure CN119007060A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, and in particular to a video processing method, device, equipment, medium and program product. Background Art
[0002] Video language pre-training (VLP) technology is often used to improve the performance of various multimodal video processing tasks. Existing video language pre-training methods usually require high-quality video data and text data for training. However, the amount of high-quality video data and text data is limited and cannot be fully trained, resulting in low accuracy and high training costs for the models used for video processing tasks. Although the image language pre-training (ILP) method can be pre-trained based on a sufficient number of high-quality images, due to the differences between image data and video data, as well as the different emphases of their respective processing tasks, the model obtained by the image language training method cannot improve accuracy and other performance when used for video processing tasks. Summary of the invention
[0003] The present disclosure proposes a video processing method, device, equipment, storage medium and program product to solve the technical problem of low accuracy of video processing to a certain extent.
[0004] In a first aspect, the present disclosure provides a video processing method, comprising:
[0005] Obtaining video data to be processed;
[0006] Obtaining a time-series image feature having time-series information based on the video data;
[0007] Determine, based on the time-series image feature, a target text feature in a text feature set that matches the time-series image feature;
[0008] Target text data corresponding to the video data is obtained based on the target text features.
[0009] In a second aspect of the present disclosure, a video processing device is provided, comprising:
[0010] An acquisition module, used for acquiring video data to be processed;
[0011] A model module is used to obtain a time-series image feature with time-series information based on the video data; determine a target text feature in a text feature set that matches the time-series image feature based on the time-series image feature; and obtain target text data corresponding to the video data based on the target text feature.
[0012] According to a third aspect of the present disclosure, an electronic device is provided, characterized in that it includes one or more processors, a memory; and one or more programs, wherein the one or more programs are stored in the memory and executed by the one or more processors, and the program includes instructions for executing the method described in the first aspect.
[0013] According to a fourth aspect of the present disclosure, a non-volatile computer-readable storage medium containing a computer program is provided. When the computer program is executed by one or more processors, the processors execute the method described in the first aspect.
[0014] In a seventh aspect of the present disclosure, a computer program product is provided, comprising computer program instructions, which, when executed on a computer, cause the computer to execute the method described in the first aspect.
[0015] From the above, it can be seen that the video processing method, device, equipment, medium and program product provided by the present disclosure obtain time-series image features with timing information by extracting the time information of video data, and use the timing information to enhance the characterization ability of image features, thereby improving the accuracy and effectiveness of multimodal video processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the present disclosure or related technologies, the drawings required for use in the embodiments or related technical descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0017] Figure 1 A schematic diagram of a video processing architecture according to an embodiment of the present disclosure.
[0018] Figure 2 The figure is a schematic diagram of the hardware structure of an exemplary electronic device according to an embodiment of the present disclosure.
[0019] Figure 3 A schematic diagram of the model architecture of the video processing model of the embodiment of the present disclosure
[0020] Figure 4 is a schematic structural diagram of a video adapter according to an embodiment of the present disclosure.
[0021] Figure 5 A schematic diagram of the stacked cross attention structure of an embodiment of the present invention.
[0022] Figure 6 A schematic diagram of the parallel cross attention structure of the embodiment of the present disclosure
[0023] Figure 7 The figure is a schematic flow chart of a video processing method according to an embodiment of the present disclosure.
[0024] Figure 8 Schematic diagram of a video processing device according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0025] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.
[0026] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the embodiments of the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0027] Figure 1 FIG. 1 is a schematic diagram showing a video processing architecture of an embodiment of the present disclosure. Figure 1 The video processing architecture 100 may include a server 110, a terminal 120, and a network 130 that provides a communication link. The server 110 and the terminal 120 may be connected via a wired or wireless network 130. The server 110 may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, security services, and CDN.
[0028] The terminal 120 may be implemented in hardware or software. For example, when the terminal 120 is implemented in hardware, it may be various electronic devices having a display screen and supporting page display, including but not limited to smart phones, tablet computers, e-book readers, laptop portable computers, and desktop computers, etc. When the terminal 120 device is implemented in software, it may be installed in the electronic devices listed above; it may be implemented as multiple software or software modules (such as software or software modules used to provide distributed services), or it may be implemented as a single software or software module, which is not specifically limited here.
[0029] It should be noted that the video processing method provided in the embodiment of the present application can be executed by the terminal 120 or by the server 110. It should be understood that Figure 1 The number of terminals, networks and servers in the embodiment is only for illustration and is not intended to limit the number of terminals, networks and servers. Any number of terminals, networks and servers may be provided as required.
[0030] Figure 2 FIG. 2 shows a schematic diagram of the hardware structure of an exemplary electronic device 200 provided in an embodiment of the present disclosure. Figure 2 As shown, the electronic device 200 may include: a processor 202, a memory 204, a network module 206, a peripheral interface 208 and a bus 210. The processor 202, the memory 204, the network module 206 and the peripheral interface 208 are connected to each other in communication within the electronic device 200 through the bus 210.
[0031] Processor 202 may be a central processing unit (CPU), an image processor, a neural network processor (NPU), a microcontroller (MCU), a programmable logic device, a digital signal processor (DSP), an application specific integrated circuit (ASIC), or one or more integrated circuits. Processor 202 may be used to perform functions related to the technology described in this disclosure. In some embodiments, processor 202 may also include multiple processors integrated into a single logical component. For example, Figure 2 As shown, processor 202 may include a plurality of processors 202a, 202b, and 202c.
[0032] The memory 204 may be configured to store data (eg, instructions, computer code, etc.). Figure 2As shown, the data stored in the memory 204 may include program instructions (e.g., program instructions for implementing the video processing method of the embodiment of the present disclosure) and data to be processed (e.g., the memory may store configuration files of other modules, etc.). The processor 202 may also access the program instructions and data stored in the memory 204, and execute the program instructions to operate on the data to be processed. The memory 204 may include a volatile storage device or a non-volatile storage device. In some embodiments, the memory 204 may include a random access memory (RAM), a read-only memory (ROM), an optical disk, a magnetic disk, a hard disk, a solid-state drive (SSD), a flash memory, a memory stick, etc.
[0033] The network module 206 can be configured to provide communication with other external devices to the electronic device 200 via a network. The network can be any wired or wireless network capable of transmitting and receiving data. For example, the network can be a wired network, a local wireless network (e.g., Bluetooth, WiFi, near field communication (NFC) etc.), a cellular network, the Internet or a combination thereof. It is understood that the type of network is not limited to the above specific examples. In some embodiments, the network module 306 can include any number of network interface controllers (NICs), radio frequency modules, transceivers, modems, routers, gateways, adapters, cellular network chips, etc., in any combination.
[0034] The peripheral interface 208 can be configured to connect the electronic device 200 to one or more peripheral devices to achieve information input and output. For example, the peripheral devices can include input devices such as a keyboard, a mouse, a touch pad, a touch screen, a microphone, and various sensors, and output devices such as a display, a speaker, a vibrator, and an indicator light.
[0035] The bus 210 can be configured to transmit information between various components of the electronic device 200 (e.g., the processor 202, the memory 204, the network module 206, and the peripheral interface 208), such as an internal bus (e.g., a processor-memory bus), an external bus (USB port, PCI-E bus), etc.
[0036] It should be noted that, although the architecture of the electronic device 200 only shows the processor 202, the memory 204, the network module 206, the peripheral interface 208 and the bus 210, in the specific implementation process, the architecture of the electronic device 200 may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the architecture of the electronic device 200 may also only include the components necessary for implementing the embodiments of the present disclosure, and does not necessarily include all the components shown in the figure.
[0037] Existing video language pre-training (VLP) techniques have been used to improve the performance of various multimodal video processing tasks, such as text video retrieval, video captioning, and video question answering (VQA). Although image language pre-training (ILP) methods, such as CLIP (Contrastive Language-Image Pre-training), have achieved remarkable success in learning high-quality visual multimodal representations. However, there are the following problems in transferring image-text models such as those obtained based on CLIP to video processing models: processing domain, due to the difference between image data and video data, the image-text model performs poorly when used to process video data; different emphasis on processing tasks, CLIP models mainly process contrast tasks, while VLP should also process generation tasks, such as video captioning and video question answering; and differences in pre-training data, due to curation rules and the availability of open source data, the pre-training data of these models are significantly different. VLP usually requires high-quality video text data for training, but the amount of such high-quality video text data is relatively limited, resulting in insufficient training data and relatively high training costs. In addition, the CLIP model has defects in processing temporal information, so it may not achieve significant performance improvements when processing video data. Moreover, CLIP models are generally targeted at specific tasks (such as text and video retrieval), rather than handling all tasks in a unified model. Therefore, how to improve the accuracy and effectiveness of video processing tasks has become a technical problem that needs to be solved urgently.
[0038] In view of this, the embodiments of the present disclosure provide a video processing method, apparatus, device, medium, and program product, which obtains time-series image features with timing information by extracting the time information of video data, and uses the timing information to enhance the characterization capability of image features, thereby improving the accuracy and effectiveness of multimodal video processing tasks.
[0039] See also Figure 3 , Figure 3 A schematic diagram of a video processing model architecture according to an embodiment of the present disclosure is shown. Figure 3 In the embodiment, the video processing model 300 may include an image encoder 310, a text encoder 320, a video adapter 330, and a multimodal encoder 340. The input of the image encoder 310 may be a video frame image, and the output may be a corresponding image feature. The input of the text encoder 320 may be text information, and the output may be a text feature corresponding to the input text information. The input of the video adapter 330 may be a video frame sequence, and the output may be an image feature of specific time sequence information. The input of the multimodal encoder 340 includes the image features output by the video encoder 310, the text features output by the text encoder 320, and the image features of specific time sequence information output by the video adapter 330, and the output is a multimodal fusion feature fused based on the cross-attention mechanism.
[0040] In some embodiments, the video adapter may include a temporal network and a dynamic convolutional network. The video adapter may be used to enhance the temporal modeling capability of the model, thereby improving the alignment capability of video and language features. For a video frame in the video data, it may be divided into multiple image blocks, and the video adapter may aggregate temporal information and enhance the representation of each image block. Figure 4 , Figure 4 FIG. 2 shows a schematic structural diagram of a video adapter according to an embodiment of the present disclosure. Figure 4 As shown, the video adapter 330 may include a timing network 331 and a dynamic convolutional network 332. Video data is input into the video adapter 330, and after passing through the multi-layer perceptron MLP (which may be a fully connected layer FC), it is input into the timing network 331 and the dynamic convolutional network 332 respectively. The timing network 331 outputs timing features, including:
[0041]
[0042] Among them, the timing network 331 can be a timing Transformer, v [CLS] Can be a category label feature, It can be an updated category label feature that encodes the visual temporal information context, which can be used as a temporal feature.
[0043] For each image patch, the representation of the image patch features can be enhanced by temporal features, and the image patch is encoded with temporal and spatial information to obtain convolutional features, including:
[0044]
[0045] Among them, v patch The image block feature, DyConv is a convolution operation, which applies the kernel of the class label feature from the time sequence to the feature of the spatial image block to obtain the encoded video time sequence feature of the image block to improve its representation ability. The time sequence feature and the convolution feature can be connected to obtain the second image feature
[0046] In some embodiments, the video processing model needs to pre-train the initial model based on the training samples, specifically including:
[0047] Acquire the training samples, where the training samples include image samples, video samples and text samples;
[0048] Performing feature extraction based on the image sample to obtain an image feature sample, performing feature extraction based on the text sample to obtain a text feature sample, and performing feature extraction based on the video sample to obtain a video feature sample with time sequence information;
[0049] Obtaining a multimodal fusion feature based on the image feature sample, the text feature sample and the video feature sample;
[0050] The initial model is pre-trained based on the training samples and the multimodal fusion features.
[0051] The initial model may be pre-trained based on the training sample to obtain the pre-trained model. In some embodiments, the training sample may include at least one combination, the combination including an image sample, a video sample, and a text sample. The image sample may be a video frame of the video sample. For example, the training sample<I,V,T> It includes an image sample I, a video sample V and a corresponding text sample T. The text sample T can be used to describe the content of the video sample V, and the image sample I can refer to a video frame in the video sample V. A video pre-annotated or matched with a corresponding text can be used as a video-text pair.
[0052] In some embodiments, fusing the image feature sample, the text feature sample, and the video feature sample to obtain a multimodal fusion feature includes:
[0053] Performing self-attention calculation based on the text feature sample to obtain text self-attention features;
[0054] Performing cross attention calculation based on the text self-attention feature and the image feature sample to obtain a text-image attention feature;
[0055] The multimodal fusion feature is obtained by performing cross-attention calculation based on the text-image attention feature and the video feature sample.
[0056] Among them, the cross-attention layers can be stacked to process the output features of the image encoder and the video adapter respectively. For example, one or more stacked structures can be used, each stacked structure including a self-attention layer (Self-Attention), two cross-attention layers (Cross-Attention) and a feedback layer (Feed-Forward). See Figure 5 , Figure 5 FIG. 2 shows a schematic diagram of a stacked cross attention structure according to an embodiment of the present disclosure. Figure 5 As shown, the output of the stacked cross attention structure can include:
[0057]
[0058] in, is the output feature of the cross attention layer in the lth image block, is the output feature of the self-attention layer in the lth image block, CA vis the output characteristic of the video adapter, CA i is the output feature of the image encoder, I is the image feature, and v is the video feature.
[0059] In some embodiments, fusing the image feature sample, the text feature sample, and the video feature sample to obtain a multimodal fusion feature includes:
[0060] Performing self-attention calculation based on the text feature sample to obtain text self-attention features;
[0061] Performing cross-attention calculation based on the text self-attention feature and the image feature sample to obtain a text-image attention feature; and performing cross-attention calculation based on the text self-attention feature and the video feature sample to obtain a text-video attention feature;
[0062] The multimodal fusion feature is obtained by weighting the text-image attention feature and the text-video attention feature.
[0063] Among them, a parallel cross-attention layer can be used to process the output features of the image encoder and the video adapter respectively. Figure 6 , Figure 6 FIG. 2 shows a schematic diagram of a parallel cross-attention structure according to an embodiment of the present disclosure. Figure 6 As shown, the output of the parallel cross-attention structure can include:
[0064]
[0065] Among them, α and β are weight parameters and can be adjusted adaptively.
[0066] In some embodiments, pre-training the initial model based on the training sample and the multimodal fusion feature includes:
[0067] Performing a first pre-training based on the training samples and the multimodal fusion features, adjusting adapter parameters of the video adapter and keeping non-adapter parameters of the initial model unchanged, so that the loss function best meets the training requirements;
[0068] A second pre-training is performed based on the training samples and the multimodal fusion features, and all parameters of the initial model are adjusted so that the loss function meets the training requirements.
[0069] Specifically, when pre-training the initial model, the training process can be divided into two stages, adaptive transfer and integrated tuning. In the adaptive transfer stage, the first pre-training can be performed based on the training sample, the parameters of the video adapter can be adjusted, and other parameters of the initial model can be fixed. In the integrated tuning stage, the second pre-training can be performed based on the same training sample, all parameters of the video adapter can be adjusted, and the performance of the pre-trained model obtained by pre-training can be further improved.
[0070] Figure 3 In the example, the image sample 311 can be input into the image encoder 310 for feature extraction to obtain the image feature sample Fi; the text sample 321 can be input into the text encoder 320 for feature extraction to obtain the text feature sample Ft; the video sample 331 can be input into the video adapter 330 for feature extraction to obtain the video feature sample Fv with timing information. The image feature sample Fi, the text feature sample Ft and the video feature sample Fv are input into the multimodal encoder 340 for feature fusion to obtain the multimodal fusion feature.
[0071] In some embodiments, pre-training the initial model based on the training sample and the multimodal fusion feature includes:
[0072] The image sample and the corresponding text sample are used as a positive sample pair, and the image sample and a non-corresponding text sample in the training sample are used as a negative sample pair;
[0073] Calculating a first similarity of the positive sample pair and a second similarity of the negative sample pair;
[0074] The first model parameters of the initial model are adjusted so that the first similarity is minimized and the second similarity is maximized.
[0075] Among them, the contrast loss function of image-text contrastive learning (ITC) can be used to align the two visual representations of image features and video features, and then feed the visual representation into the multimodal encoder. Specifically, an image sample and its corresponding text sample can be regarded as a positive sample pair, while all other samples in the same batch are regarded as negative sample pairs. Then, the loss is calculated by calculating the cosine similarity distance between them. Comparing the distance between positive and negative pairs can make the distance between positive sample pairs closer and the distance between negative sample pairs farther. In this way, a better semantic structure can be established in the representation space, thereby improving visual-text matching and retrieval.
[0076] In some embodiments, pre-training the initial model based on the training sample and the multimodal fusion feature includes:
[0077] Generating a text mask sample based on the text sample;
[0078] Performing self-attention calculation based on the text mask sample to obtain a first text mask self-attention feature;
[0079] Performing a cross-attention calculation based on the image feature sample and the first text mask self-attention feature to obtain a first image-text mask cross-attention feature; and performing a cross-attention calculation based on the video feature sample and the first text mask self-attention feature to obtain a first video-text mask cross-attention feature;
[0080] Obtaining a first loss function based on the first image-text mask cross-attention feature, the first video-text mask cross-attention feature, and the multimodal fusion feature;
[0081] The model parameters of the initial model are adjusted based on the first loss function to minimize the first loss function.
[0082] Specifically, Masked Language Modeling (MLM) can be adopted to encourage the model to generate captions given a visual representation. Since the multimodal fusion feature is based on the cross-attention Transformer, it enables the model to query the information in the visual representation to generate text tags. Therefore, this training task can effectively promote the fusion of visual and textual information.
[0083] In some embodiments, a text mask sample can be obtained based on the text sample and a preset text mask strategy. Further, obtaining the text mask sample based on the text sample and a preset text mask strategy includes: randomly selecting a preset proportion of words in the text sample for masking to generate the text mask sample. For example, Figure 3 In the example, the text sample is "a person is putting food into a microwave.", which contains 8 words, of which the preset ratio (e.g., 30%) is about 2. Then, the 2 preset type words in the text sample can be randomly masked to obtain the text "a[mask]is[mask]food into amicrowave." as the text mask sample. It should be understood that the number of words in the preset ratio may not be an integer, and an integer (e.g., rounded) can be taken as needed for masking, which is not limited here.
[0084] In some embodiments, pre-training the initial model based on the training sample and the multimodal fusion feature includes:
[0085] Generating a text mask sample based on the text sample;
[0086] Performing causal self-attention calculation based on the text mask sample to obtain a second text mask self-attention feature;
[0087] Performing a cross-attention calculation based on the image feature sample and the second text mask self-attention feature to obtain a second image-text mask cross-attention feature; and performing a cross-attention calculation based on the video feature sample and the second text mask self-attention feature to obtain a second video-text mask cross-attention feature;
[0088] Obtaining a second loss function based on the second image-text mask cross-attention feature, the second video-text mask cross-attention feature, and the multimodal fusion feature;
[0089] The model parameters of the initial model are adjusted based on the second loss function to minimize the second loss function.
[0090] Specifically, to enhance the integration of visual and textual information, Uni-ML can be used as an additional training task. In addition to the MLM task, both use the same multimodal encoder. The difference between the two tasks lies in the self-attention mechanism of the Transformer. Uni-ML uses causal self-attention masking to regulate the interaction between text inputs. This encourages the generated text tokens to rely heavily on visual and textual inputs, thereby promoting a more effective fusion of the two modalities.
[0091] After the above pre-training stage, a pre-training model is obtained. On this basis, the pre-training model can be trained specifically for different downstream tasks to obtain a multimodal data model for different downstream tasks.
[0092] In some embodiments, the method may further include: training the pre-trained model based on task training samples to obtain the video processing model.
[0093] Furthermore, in some embodiments, training the pre-trained model based on the task training sample to obtain the video processing model may further include:
[0094] Acquire the task training sample, wherein the second training sample includes at least one video-text training pair, and each video-text training pair includes a video training sample and a corresponding text training sample;
[0095] The pre-trained model is trained based on the task training samples until the target training requirements are met to obtain the video processing model.
[0096] Among them, for different video processing tasks, the content contained in the task training samples may be different. For example, for a video information generation task, the video processing model can generate text information (such as a summary, title, introduction, etc.) associated with the video data based on the video data. Then, the task training sample corresponding to the video information generation task may include at least one video-text information pair, each video-text information pair including a video training sample and corresponding text information such as a summary, title, introduction, etc. It can be seen that the pre-trained model obtained by the video processing method according to the embodiment of the present disclosure can ensure high efficiency and high performance.
[0097] See also Figure 7 , Figure 7 FIG. 2 shows a schematic flow chart of a video processing method according to an embodiment of the present disclosure. Figure 7 As shown, the video processing method 700 can be based on Figure 3 The video processing model implementation shown further includes the following steps.
[0098] Step S710: obtaining video data to be processed.
[0099] Step S720: obtaining a temporal image feature having temporal information based on the video data.
[0100] In some embodiments, obtaining a temporal image feature having temporal information based on the video data includes:
[0101] Perform feature extraction based on the video frame of the video data to obtain a first image feature and a second image feature with time sequence information;
[0102] The first image feature and the second image feature are fused to obtain the temporal image feature.
[0103] Specifically, the first image feature can be obtained by extracting features from the video frame based on the image encoder 310, and the second image feature with time sequence information can be obtained by extracting features from the video data based on the video adapter 330. The first image feature and the second image feature are fused via the multimodal encoder 340 to obtain a time sequence image feature.
[0104] In some embodiments, extracting features based on the video frames of the video data to obtain second image features with time sequence information includes:
[0105] Extracting image block features of image blocks in video frames and temporal features of the video frames based on the video data;
[0106] The second image feature is obtained based on the fusion of the temporal feature and the image block feature.
[0107] Specifically, the timing features of the video data can be extracted based on the timing network 331 in the video adapter 330, and the image block features of the video frame can be fused with the timing features based on the dynamic convolution network 332 to obtain a second image feature with timing information.
[0108] Step S730: determining a target text feature in a text feature set that matches the time-series image feature based on the time-series image feature.
[0109] Step S740, obtaining target text data corresponding to the video data based on the target text features.
[0110] Specifically, for the video data to be processed, the user hopes to generate corresponding summary information for the video data. The video processing model can perform feature extraction on the video data to obtain a time-series image feature with time-series information, and the time-series image feature can be a feature vector. The video processing model searches and matches in a text feature set based on the time-series image feature, and the text feature set can be a set of text features obtained after feature extraction of a preset text. After searching and matching, one or more target text features that match the time-series image feature can be obtained, and based on the target preset text corresponding to the one or more target text features, a target text about the video data can be formed as summary information. According to the video processing method of the embodiment of the present disclosure, the accuracy of the text information can be improved by using a video processing model to generate relevant text information based on the video.
[0111] It should be noted that the method of the embodiment of the present disclosure can be performed by a single device, such as a computer or a server. The method of the present embodiment can also be applied in a distributed scenario and completed by multiple devices cooperating with each other. In the case of such a distributed scenario, one of the multiple devices can only perform one or more steps in the method of the embodiment of the present disclosure, and the multiple devices will interact with each other to complete the described method.
[0112] It should be noted that the above describes some embodiments of the present disclosure. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the above embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0113] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present disclosure also provides a video processing device, see Figure 8 , the video processing device comprises:
[0114] An acquisition module, used for acquiring video data to be processed;
[0115] A model module is used to obtain a time-series image feature with time-series information based on the video data; determine a target text feature in a text feature set that matches the time-series image feature based on the time-series image feature; and obtain target text data corresponding to the video data based on the target text feature.
[0116] For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the present disclosure, the functions of each module can be implemented in the same or multiple software and / or hardware.
[0117] The device of the above embodiment is used to implement the corresponding video processing method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.
[0118] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present disclosure also provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the video processing method described in any of the above embodiments.
[0119] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.
[0120] The computer instructions stored in the storage medium of the above embodiments are used to enable the computer to execute the video processing method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.
[0121] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Based on the concept of the present disclosure, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present disclosure as described above, which are not provided in detail for the sake of simplicity.
[0122] In addition, to simplify the description and discussion, and in order not to make the embodiments of the present disclosure difficult to understand, the known power / ground connections to the integrated circuit (IC) chips and other components may or may not be shown in the provided figures. In addition, the device can be shown in the form of a block diagram to avoid making the embodiments of the present disclosure difficult to understand, and this also takes into account the fact that the details of the implementation of these block diagram devices are highly dependent on the platform on which the embodiments of the present disclosure will be implemented (that is, these details should be fully within the scope of understanding of those skilled in the art). Where specific details (e.g., circuits) are set forth to describe exemplary embodiments of the present disclosure, it is apparent to those skilled in the art that the embodiments of the present disclosure can be implemented without these specific details or with changes in these specific details. Therefore, these descriptions should be considered illustrative rather than restrictive.
[0123] Although the present disclosure has been described in conjunction with specific embodiments of the present disclosure, many replacements, modifications and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
[0124] The embodiments of the present disclosure are intended to cover all such substitutions, modifications and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present disclosure should be included in the scope of protection of the present disclosure.
Claims
1. A video processing method, characterized in that: include: Obtaining video data to be processed; Obtaining a time-series image feature having time-series information based on the video data; Determine, based on the time-series image feature, a target text feature in a text feature set that matches the time-series image feature; Target text data corresponding to the video data is obtained based on the target text features.
2. The method according to claim 1, characterized in that: Obtaining a time-series image feature having time-series information based on the video data includes: Perform feature extraction based on the video frame of the video data to obtain a first image feature and a second image feature with time sequence information; The first image feature and the second image feature are fused to obtain the temporal image feature.
3. The method according to claim 2, characterized in that Extracting features based on the video frames of the video data to obtain second image features with time sequence information includes: Extracting image block features of image blocks in video frames and temporal features of the video frames based on the video data; The second image feature is obtained based on the fusion of the temporal feature and the image block feature.
4. The method according to claim 3, characterized in that Obtaining the second image feature based on the fusion of the temporal feature and the image block feature includes: Obtaining a convolution feature based on a convolution operation of the temporal feature and the image block feature; The temporal feature and the convolution feature are connected to obtain the second image feature.
5. The method according to claim 1, characterized in that Obtaining a time-series image feature having time-series information based on the video data includes: The time-series image features are obtained by extracting features from the video data based on a video processing model; wherein the video processing model needs to pre-train an initial model based on a training sample, specifically including: Acquire the training samples, where the training samples include image samples, video samples and text samples; Performing feature extraction based on the image sample to obtain an image feature sample, performing feature extraction based on the text sample to obtain a text feature sample, and performing feature extraction based on the video sample to obtain a video feature sample with time sequence information; Obtaining a multimodal fusion feature based on the image feature sample, the text feature sample and the video feature sample; The initial model is pre-trained based on the training samples and the multimodal fusion features.
6. The method according to claim 5, characterized in that The multimodal fusion feature is obtained by fusing the image feature sample, the text feature sample and the video feature sample, including: Performing self-attention calculation based on the text feature sample to obtain text self-attention features; Performing cross attention calculation based on the text self-attention feature and the image feature sample to obtain a text-image attention feature; The multimodal fusion feature is obtained by performing cross-attention calculation based on the text-image attention feature and the video feature sample.
7. The method according to claim 5, characterized in that The multimodal fusion feature is obtained by fusing the image feature sample, the text feature sample and the video feature sample, including: Performing self-attention calculation based on the text feature sample to obtain text self-attention features; Performing cross-attention calculation based on the text self-attention feature and the image feature sample to obtain a text-image attention feature; and performing cross-attention calculation based on the text self-attention feature and the video feature sample to obtain a text-video attention feature; The multimodal fusion feature is obtained by weighting the text-image attention feature and the text-video attention feature.
8. The method according to claim 5, characterized in that The initial model includes a video adapter for extracting image features with time sequence information, and pre-training the initial model based on the training sample and the multimodal fusion feature includes: Performing a first pre-training based on the training samples and the multimodal fusion features, adjusting adapter parameters of the video adapter and keeping non-adapter parameters of the initial model unchanged, so that the loss function best meets the training requirements; A second pre-training is performed based on the training samples and the multimodal fusion features, and all parameters of the initial model are adjusted so that the loss function meets the training requirements.
9. A video processing device, characterized in that: include: An acquisition module, used for acquiring video data to be processed; A model module, used for obtaining time-series image features with time-series information based on the video data; Based on the time-series image feature, a target text feature in a text feature set that matches the time-series image feature is determined; and based on the target text feature, target text data corresponding to the video data is obtained.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 8 when executing the program.
11. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 8.
12. A computer program product, comprising computer program instructions, which, when executed on a computer, enable the computer to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Video dynamic thumbnail generation method and model training method and device
CN109885723A
Method and device for generating video from text, computer equipment and storage medium
CN115935010A
Method for producing carbon fiber composition
KR1020230014143A
Video image processing method and apparatus
WO2022142702A1
