Video feature fusion method and device, storage medium and processor

Abstract

This invention discloses a method, apparatus, storage medium, and processor for fusing video features. The method includes: acquiring a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; processing the first image feature matrix based on the first text vector to obtain a second image feature matrix; obtaining a third image feature matrix based on the first and second image feature matrices; and processing the third image feature matrix based on the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information. This invention solves the technical problem of inaccurate information description in videos caused by insufficient fusion of multiple modal information.

Description

Technical Field

[0001] This invention relates to the field of video processing, and more specifically, to a method, apparatus, storage medium, and processor for fusing video features. Background Technology

[0002] With the rise of live-streaming e-commerce and short social media videos, video technology is increasingly widely used in modern society. As a new interactive medium, video not only contains rich image information but also multimodal information such as text and audio. To more accurately describe the information in videos, it is necessary to fuse the features of multiple modalities present. However, current technologies mostly employ a post-fusion approach for fusing multimodal features in videos. This involves extracting features from multiple submodals such as images, text, or audio, and then performing weighted fusion at the end of the network structure. This fusion method has limited effectiveness in information interaction, failing to leverage the complementary effects of modal information across different layers of network training. Some techniques that perform cross-modal training only consider simple feature concatenation, failing to locate effective image regions and resulting in poor fusion performance. Fully utilizing the multimodal information in videos can achieve performance superior to single image features, but how to effectively fuse multimodal feature information remains a challenging problem.

[0003] There is currently no effective solution to the problem of inaccurate information description in videos caused by insufficient fusion of multiple modal information. Summary of the Invention

[0004] This invention provides a method, apparatus, storage medium, and processor for fusing video features, to at least solve the technical problem of inaccurate information description in videos caused by insufficient fusion of multiple modal information.

[0005] According to one aspect of the present invention, a method for fusing video features is provided, comprising: acquiring a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; processing the first image feature matrix according to the first text vector to obtain a second image feature matrix; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; and processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0006] Further, processing the first image feature matrix based on the first text vector to obtain the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain a fourth image feature matrix; processing the fourth image feature matrix based on the first text vector to obtain a fifth image feature matrix; and processing the fifth image feature matrix according to a preset function to obtain the second image feature matrix.

[0007] Further, processing the fourth image feature matrix based on the first text vector to obtain the fifth image feature matrix includes: multiplying the first text vector with the fourth image feature matrix to obtain the sixth image feature matrix; and multiplying the sixth image feature matrix with the first image feature matrix to obtain the fifth image feature matrix.

[0008] Furthermore, before obtaining the first image feature matrix and the first text vector, the method includes: obtaining audio information from the video; processing the audio information to obtain corresponding text information; and processing the text information through a recurrent neural network to obtain text feature information.

[0009] Further, processing the third image feature matrix based on the first image feature matrix to obtain the target image feature matrix includes: performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first image feature matrix and the seventh image feature matrix to obtain the target image feature matrix.

[0010] Furthermore, the target image feature matrix has the same size and structure as the first image feature matrix, and the method is applied to the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

[0011] Further, obtaining the third image feature matrix based on the first image feature matrix and the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and performing matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain the third image feature matrix.

[0012] According to one aspect of the present invention, a video processing method is provided, comprising: acquiring a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information; training the target image feature matrix to obtain a target video analysis model; and processing the video based on the target video analysis model to obtain a processed target video.

[0013] Furthermore, the video includes live video. The video is processed based on the target video analysis model to obtain the processed target video, which includes: classifying the live video based on the target video analysis model to obtain the classified live video and the classification tag corresponding to the live video.

[0014] According to one aspect of the present invention, a video processing method is provided, comprising: receiving a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; responding to the service call request, processing a first image feature matrix in a server according to a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and outputting the target image feature matrix.

[0015] Furthermore, the target image feature matrix is ​​used in the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

[0016] According to another aspect of the present invention, a video feature fusion apparatus is also provided, comprising: a first acquisition unit, configured to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; a first processing unit, configured to process the first image feature matrix according to the first text vector to obtain a second image feature matrix; a second acquisition unit, configured to obtain a third image feature matrix according to the first image feature matrix and the second image feature matrix; and a second processing unit, configured to process the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0017] Further, the first processing unit includes: a first processing module, used to perform convolution processing on the first image feature matrix to obtain a fourth image feature matrix; a second processing module, used to process the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; and a third processing module, used to process the fifth image feature matrix according to a preset function to obtain a second image feature matrix.

[0018] According to one aspect of the present invention, a video processing apparatus is provided, the apparatus comprising: a fourth acquisition unit, configured to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; a fifth acquisition unit, configured to obtain a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information; a training unit, configured to train the target image feature matrix to obtain a target video analysis model; and a fifth processing unit, configured to process the video based on the target video analysis model to obtain a processed target video.

[0019] According to one aspect of the present invention, a video processing apparatus is provided, comprising: a receiving unit for receiving a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; a responding unit for responding to the service call request, processing a first image feature matrix in a server according to a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and an output unit for outputting the target image feature matrix.

[0020] To achieve the above objectives, according to another aspect of this application, a storage medium is provided, the storage medium including a stored program, wherein, when the program is executed, the device on which the storage medium is located is controlled to perform the video feature fusion method described in any one of the above claims.

[0021] To achieve the above objectives, according to another aspect of this application, a processor is provided for running a program, wherein the program, when running, executes the video feature fusion method described in any one of the preceding claims.

[0022] In this embodiment of the invention, a multimodal location coding module is used to fuse multiple modal data across multiple levels. This is achieved by obtaining a first image feature matrix and a first text vector, where the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video. The first image feature matrix is ​​then processed based on the first text vector to obtain a second image feature matrix. A third image feature matrix is ​​obtained based on the first and second image feature matrices. Finally, the third image feature matrix is ​​processed based on the first image feature matrix to obtain a target image feature matrix, which includes fused image feature information and text feature information. This process fully integrates multiple modal features from the video, thereby improving the robustness of video features and solving the problem of inaccurate information description in the video caused by insufficient fusion of multiple modal information. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0024] Figure 1 This is a hardware structure block diagram of a computer terminal according to an embodiment of the present invention;

[0025] Figure 2 This is a flowchart of the video feature fusion method provided in Embodiment 1 of the present invention. Figure 1 ;

[0026] Figure 3 This is a flowchart of the video feature fusion method provided in Embodiment 1 of the present invention. Figure 2 ;

[0027] Figure 4 This is a schematic diagram illustrating an application scenario of the video feature fusion method provided in Embodiment 1 of the present invention;

[0028] Figure 5 This is a flowchart of a video processing method provided according to Embodiment 2 of the present invention;

[0029] Figure 6 This is a flowchart of a video processing method provided according to Embodiment 3 of the present invention;

[0030] Figure 7 This is a schematic diagram of a video feature fusion apparatus provided according to Embodiment 4 of the present invention;

[0031] Figure 8 This is a flowchart of a video processing method provided in Embodiment 5 of the present invention;

[0032] Figure 9 This is a flowchart of a video processing method provided according to Embodiment Six of the present invention; and

[0033] Figure 10 This is a structural block diagram of an optional computer terminal according to an embodiment of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0037] Detection: Locate objects of interest (such as products being explained by the host) in video frames and output rectangular coordinate boxes.

[0038] CNN: Convolutional Neural Network, commonly used for image feature extraction.

[0039] RNN: Recurrent Neural Network, commonly used for feature extraction of sequence types such as text or speech.

[0040] Word2Vector: A word vector, which represents a text word in vector form.

[0041] MPE: Multimodal Location Encoding Feature Fusion Module.

[0042] Example 1

[0043] According to an embodiment of the present invention, a method for fusing video features is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0044] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a video feature fusion method is shown. Figure 1As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0045] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0046] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video feature fusion method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the video feature fusion method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0047] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0048] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0049] Under the aforementioned operating environment, this application provides the following: Figure 2 The video feature fusion method shown. Figure 2 This is a flowchart of the video feature fusion method according to Embodiment 1 of the present invention. Figure 1 .

[0050] Step S101: Obtain a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video.

[0051] This embodiment utilizes a Video Feature Fusion (MPE) module to fully fuse image modal feature information with text / audio modal feature information from a video. The input to the MPE module consists of two parts: an input feature map from the image source and a feature vector from the text / audio modality. Before fusing the image and text modal feature information, the image and text modal feature information from the video are first acquired. The image modal feature information can be represented by a first image feature matrix, and the text modal feature information can be represented by a first text vector. For example, acquiring the image feature information from the video yields a first image feature matrix TxHxWxC; acquiring the text feature information from the video yields a first text vector 1x1x1xC′.

[0052] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, before obtaining the first image feature matrix and the first text vector, the method includes: obtaining audio information in the video; processing the audio information to obtain corresponding text information; and processing the text information through a recurrent neural network to obtain text feature information.

[0053] Before feature fusion, the audio information contained in the video needs to be processed into an input format that meets the requirements of the MPE module in this embodiment, namely, a text feature vector. The specific method includes: first, converting the acquired audio information into text information; then, using a gate-based recurrent neural network (RNN) to obtain the text feature vector, that is, representing the converted audio text information in vector form. This allows the audio modal information in the video to be converted into the input format required by the MPE module, and then the converted text feature vector is fused with the video feature matrix.

[0054] Step S102: Process the first image feature matrix according to the first text vector to obtain the second image feature matrix.

[0055] The first text vector includes feature information of the text modality in the video, which is auxiliary modality information. In this embodiment, the feature information of the image modality is processed by the auxiliary modality information, and the feature information of the text modality is used to locate the key regions appearing in the video frame. Only the features of specific key positions are obtained, thereby improving the robustness of the video features and realizing a more accurate description of the information in the video.

[0056] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, processing the first image feature matrix based on the first text vector to obtain the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain a fourth image feature matrix; processing the fourth image feature matrix based on the first text vector to obtain a fifth image feature matrix; and processing the fifth image feature matrix according to a preset function to obtain the second image feature matrix.

[0057] To achieve deep fusion of image and text features in a video, interactive processing of image and text modal features is performed at various levels. Specifically, processing the first image feature matrix based on the first text vector to obtain the second image feature matrix includes: convolving the first image feature matrix to obtain the fourth image feature matrix; then processing the fourth image feature matrix based on the first text vector to obtain the fifth image feature matrix; and finally processing the fifth image feature matrix based on a preset function to obtain the second image feature matrix. Figure 3 This is a flowchart of the video feature fusion method provided in Embodiment 1 of the present invention. Figure 2 ,like Figure 3As shown, in the process of processing the first image feature vector through the first text vector, the size of the first image feature vector is TxHxWxC, and the size of the first text vector is 1x1x1xC′. First, the first image feature vector is convolved, and then the result of the convolution between the first text vector and the first image feature vector is processed to obtain the fifth image feature matrix. Then, the fifth image feature matrix is ​​processed by the preset SoftMax function to obtain the second image feature matrix THWxTHW.

[0058] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, processing the fourth image feature matrix based on the first text vector to obtain the fifth image feature matrix includes: performing matrix multiplication of the first text vector and the fourth image feature matrix to obtain the sixth image feature matrix; and multiplying the sixth image feature matrix with the sixth image feature matrix to obtain the fifth image feature matrix.

[0059] The fourth image feature matrix is ​​the result of convolving the first image feature matrix, such as... Figure 3 As shown, the process of processing the fourth image feature matrix based on the first text vector is as follows: multiply the first text vector by the fourth image feature matrix to obtain the sixth image feature matrix, and then multiply the sixth image feature matrix by itself to obtain the fifth image feature matrix TxHxWxC′.

[0060] Step S103: Obtain the third image feature matrix based on the first image feature matrix and the second image feature matrix.

[0061] After processing the first image feature matrix based on the first text vector to obtain the second image feature matrix, the text modal feature information and image modal information are further fused, including processing the second image features based on the first image feature matrix to obtain the third image feature matrix.

[0062] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, obtaining a third image feature matrix based on the first image feature matrix and the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and performing matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain the third image feature matrix.

[0063] like Figure 3As shown, the process of processing the second image feature matrix using the first image feature matrix includes: performing convolution on the first image feature matrix to obtain the eighth image feature matrix; and then performing matrix multiplication between the second image feature matrix (THWxTHW) and the eighth image feature matrix to obtain TxHxWxC. * The third image feature matrix. It should be noted that the eighth image feature matrix and the fourth image feature matrix are essentially the same, both being the result of convolution processing of the first image feature matrix.

[0064] Step S104: Process the third image feature matrix according to the first image feature matrix to obtain the target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0065] After processing the first image feature matrix according to the first text vector to obtain the second image feature matrix, and obtaining the third image feature matrix according to the first image feature matrix and the second image feature matrix, the text modal feature information and image modal information are further fused, including processing the third image feature matrix according to the first image feature matrix to obtain the target image feature matrix, wherein the target image feature matrix includes the fused image feature information and text feature information.

[0066] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, processing the third image feature matrix based on the first image feature matrix to obtain the target image feature matrix includes: performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first image feature matrix and the seventh image feature matrix to obtain the target image feature matrix.

[0067] To achieve deep fusion of image and text features in a video, interactive processing of image and text modal features is performed at various levels. After processing the first image feature matrix based on the first text vector to obtain the second image feature matrix, and then processing the first and second image feature matrices to obtain the third image feature matrix, further fusion processing of the text and image modal features is performed, including processing the third image feature matrix based on the first image feature matrix to obtain the target image feature matrix. For example... Figure 3As shown, processing the third image feature matrix based on the first image feature matrix to obtain the target image feature matrix includes: performing convolution processing on the third image feature matrix, adding the result of the convolution processing of the first image feature matrix and the third image feature matrix to obtain the target image feature matrix. The target image feature matrix is ​​still TxHxWxC, and has the same size and structure as the first image feature matrix.

[0068] Optionally, in the video feature fusion method provided according to Embodiment 1 of this application, the target image feature matrix has the same size and structure as the first image feature matrix, and the method is applied to the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

[0069] The video feature fusion method in this embodiment is applied to the training of a multimodal signal model in a video. This multimodal signal model is used to fuse features from multiple modal signals in the video. By inserting an MPE module in the middle of the training network, key locations in the video can be encoded, fully utilizing cross-modal information to assist image information in obtaining more robust video features. To achieve plug-and-play functionality for the MPE module, this embodiment requires that the output image feature matrix has the same size and structure as the input image feature matrix. Figure 3 As shown, the above processing method also achieves multi-level interactive fusion of the first image feature matrix TxHxWxC and the first text vector 1x1x1xC′, and the resulting target feature matrix is ​​still TxHxWxC, which is the same as the first image feature matrix in size and structure.

[0070] Figure 4 This is a schematic diagram illustrating an application scenario of the video feature fusion method provided in Embodiment 1 of the present invention. It shows an example of a multimodal feature training network using an MPE module, such as... Figure 4 As shown, the video is sampled at equal intervals to obtain a sequence of video frames with a temporal order. Then, a CNN network is used to obtain the temporal fusion features of multiple frames. At the same time, the audio information in the video is preprocessed to convert it into text feature vectors. Then, the MPE module is used to fuse the cross-modal features in the training network. The MPE module is a general modality fusion module that can be plugged and played and can be easily fused into any CNN network.

[0071] The Multimodal Position Encoding (MPE) module operates during the training of multimodal feature information. Unlike post-fusion methods, it fully integrates cross-modal information from different network layers and utilizes this information to assist in locating key information in the video, resulting in more robust video representations. Simultaneously, by learning a position encoding matrix, MPE assigns greater weight to key regions in the video, achieving more efficient video fusion. For example, in live video, audio text can effectively describe the product being discussed, revealing salient regions in the image that are difficult to distinguish. It can also assess the validity of the current video frame, eliminating frames with invalid content. Furthermore, descriptions of information that is difficult to differentiate in the image, such as product style, length, and size, can assist in feature extraction from the image chain. Through the MPE module, this difficult-to-describe information is transformed into a position encoding matrix for the original input feature map, used to eliminate interference from ineffective regions.

[0072] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0073] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0074] Example 2

[0075] In the operating environment provided in Embodiment 1 above, this application provides the following: Figure 5 The video processing method shown. Figure 5 This is a flowchart of a video processing method according to Embodiment 2 of the present invention.

[0076] Step S501: Obtain a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video.

[0077] This embodiment utilizes a Video Feature Fusion (MPE) module to fully fuse image modal feature information with text / audio modal feature information from a video. The input to the MPE module consists of two parts: an input feature map from the image source and a feature vector from the text / audio modality. Before fusing the image and text modal feature information, the image and text modal feature information from the video are first acquired. The image modal feature information can be represented by a first image feature matrix, and the text modal feature information can be represented by a first text vector. For example, acquiring the image feature information from the video yields a first image feature matrix TxHxWxC; acquiring the text feature information from the video yields a first text vector 1x1x1xC′.

[0078] Step S502: Based on the first image feature matrix and the first text vector, a target image feature matrix is ​​obtained, wherein the target image feature matrix includes fused image feature information and text information.

[0079] When analyzing video, image modal feature information and speech / text modal feature vectors can complement each other. The MPE structure learns the weighted matrix of the image modality in the spatiotemporal domain, i.e., the first image feature matrix, through the feature vector of the auxiliary modality, i.e., the first text vector. This achieves full fusion of image modal feature information and text modal feature vectors at multiple levels to obtain the target image feature matrix. By fusing the first text vector with the first image feature information, auxiliary information in the text / audio can be used to locate key regions in the image corresponding to the image feature matrix. For example, the weight of the product or key region being highlighted in the video frame will be greater, as will the weight of the key frames in the video. Furthermore, the structure and size of the target image feature matrix after fusion are exactly the same as the first image feature matrix before fusion.

[0080] Step S503: Train the target image feature matrix to obtain the target video analysis model.

[0081] Since the structure and size of the target image feature matrix after fusion are exactly the same as those of the first image feature matrix before fusion, the process of obtaining the target image feature matrix based on the first image feature matrix and the first text vector can be easily integrated into any CNN network. That is, the target image feature matrix after fusing multiple modal feature information can be trained, replacing the method of training a single image modal feature. The model obtained by training the target image feature matrix after fusing multiple modal information can achieve better video analysis results.

[0082] Step S504: Process the video based on the target video analysis model to obtain the processed target video.

[0083] Through steps S501-S504, the video can be processed using the target video analysis model to achieve better video analysis results.

[0084] Optionally, in the video processing method provided according to Embodiment 2 of this application, the video includes live video. The video is processed based on the target video analysis model to obtain the processed target video, which includes: classifying the live video based on the target video analysis model to obtain the classified live video and the classification label corresponding to the live video.

[0085] The MPE module is used to fuse features from multiple modalities in a video. It can be applied plug-and-play to video training networks, making it applicable to various video analysis scenarios and improving the performance of trained models. For example, in live video analysis, different live videos may contain different types of product images. Simultaneously, the host provides audio descriptions of the products. By fusing the image modal information from the video with the text vector converted from the host's audio, the resulting fused target image feature matrix increases the weight of product feature information in the video. Based on this, the target video analysis model trained can more accurately identify product types in the video. Using this target video analysis model to classify live videos improves classification accuracy, resulting in classified live videos and their corresponding classification labels.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0087] Example 3

[0088] In the operating environment provided in Embodiment 1 above, this application provides the following: Figure 6 The video processing method shown. Figure 6 This is a flowchart of a video processing method according to Embodiment 3 of the present invention.

[0089] Step S601: Receive a service call request sent by the client, wherein the service call request carries a request to fuse image feature information and text feature information in the video.

[0090] Step S602: In response to the service call request, the server processes the first image feature matrix based on the first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; based on the first and second image feature matrices, a third image feature matrix is ​​obtained; the third image feature matrix is ​​processed based on the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and

[0091] Step S603: Output the target image feature matrix.

[0092] By calling the server to process the video, processing efficiency can be improved, and textual and image feature information can be quickly fused. This achieves the goal of fully integrating multiple modal features in the video, thereby enhancing the robustness of video features.

[0093] Optionally, in the video processing method provided according to Embodiment 2 of this application, the target image feature matrix is ​​used in the training of a multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0094] By using the feature matrix of the target image in the training of the multimodal signal model, the accuracy of the trained multimodal signal model is improved, so that the fusion of features of multiple modal signals will be more effective in the future.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention.

[0096] Example 4

[0097] According to embodiments of the present invention, an apparatus for implementing the above-described video feature fusion method is also provided, such as... Figure 7As shown, the device includes: a first acquisition unit 701, a first processing unit 702, a second acquisition unit 703, and a second processing unit 704.

[0098] Specifically, the first acquisition unit 701 is used to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video;

[0099] The first processing unit 702 is used to process the first image feature matrix according to the first text vector to obtain the second image feature matrix;

[0100] The second acquisition unit 703 is used to obtain the third image feature matrix based on the first image feature matrix and the second image feature matrix;

[0101] The second processing unit 704 is used to process the third image feature matrix according to the first image feature matrix to obtain the target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0102] In summary, the apparatus for video feature fusion provided in this application embodiment employs a multimodal location coding module to fuse multiple modal data across multiple levels. This is achieved by acquiring a first image feature matrix and a first text vector, where the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video. The first image feature matrix is ​​then processed based on the first text vector to obtain a second image feature matrix. A third image feature matrix is ​​obtained based on the first and second image feature matrices. Finally, the third image feature matrix is ​​processed based on the first image feature matrix to obtain a target image feature matrix, which includes fused image feature information and text feature information. This process effectively fuses multiple modal features in the video, thereby improving the robustness of video features and solving the technical problem of inaccurate information description in the video caused by insufficient fusion of multiple modal information.

[0103] Optionally, in the video feature fusion apparatus provided in this application embodiment, the first processing unit includes: a first processing module, used to perform convolution processing on the first image feature matrix to obtain a fourth image feature matrix; a second processing module, used to process the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; and a third processing module, used to process the fifth image feature matrix according to a preset function to obtain a second image feature matrix.

[0104] Optionally, in the video feature fusion apparatus provided in this application embodiment, the second processing module includes: a first processing submodule, used to perform matrix multiplication of the first text vector and the fourth image feature matrix to obtain a sixth image feature matrix; and a second processing submodule, used to multiply the sixth image feature matrix with the sixth image feature matrix to obtain a fifth image feature matrix.

[0105] Optionally, in the video feature fusion apparatus provided in this application embodiment, the apparatus includes: a third acquisition unit, configured to acquire audio information in the video before acquiring the first image feature matrix and the first text vector; a third processing unit, configured to process the audio information to obtain corresponding text information; and a fourth processing unit, configured to process the text information through a recurrent neural network to obtain text feature information.

[0106] Optionally, in the video feature fusion apparatus provided in this application embodiment, the second processing unit includes: a first processing module, used to perform convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and a second processing module, used to sum the first image feature matrix and the seventh image feature matrix to obtain a target image feature matrix.

[0107] Optionally, in the video feature fusion apparatus provided in this application embodiment, the target image feature matrix has the same size and structure as the first image feature matrix. The apparatus is applied to the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

[0108] Optionally, in the video feature fusion apparatus provided in this application embodiment, the second acquisition unit includes: a first processing module, used to perform convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and a second processing module, used to perform matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain a third image feature matrix.

[0109] It should be noted that the first acquisition unit 701, the first processing unit 702, the second acquisition unit 703, and the second processing unit 704 mentioned above correspond to steps S101 to S104 in Embodiment 1. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0110] Example 5

[0111] According to embodiments of the present invention, an apparatus for implementing the video processing method of Embodiment 2 described above is also provided, such as... Figure 8As shown, the device includes: a fourth acquisition unit 801, a fifth acquisition unit 802, a training unit 803, and a fifth processing unit 804.

[0112] Specifically, the fourth acquisition unit 801 is used to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video.

[0113] This embodiment utilizes a Video Feature Fusion (MPE) module to fully fuse image modal feature information with text / audio modal feature information from a video. The input to the MPE module consists of two parts: an input feature map from the image source and a feature vector from the text / audio modality. Before fusing the image and text modal feature information, the image and text modal feature information from the video are first acquired. The image modal feature information can be represented by a first image feature matrix, and the text modal feature information can be represented by a first text vector. For example, acquiring the image feature information from the video yields a first image feature matrix TxHxWxC; acquiring the text feature information from the video yields a first text vector 1x1x1xC′.

[0114] The fifth acquisition unit 802 is used to obtain a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information.

[0115] When analyzing video, image modal feature information and speech / text modal feature vectors can complement each other. The MPE structure learns the weighted matrix of the image modality in the spatiotemporal domain, i.e., the first image feature matrix, through the feature vector of the auxiliary modality, i.e., the first text vector. This achieves full fusion of image modal feature information and text modal feature vectors at multiple levels to obtain the target image feature matrix. By fusing the first text vector with the first image feature information, auxiliary information in the text / audio can be used to locate key regions in the image corresponding to the image feature matrix. For example, the weight of the product or key region being highlighted in the video frame will be greater, as will the weight of the key frames in the video. Furthermore, the structure and size of the target image feature matrix after fusion are exactly the same as the first image feature matrix before fusion.

[0116] Training unit 803 is used to train the feature matrix of the target image to obtain the target video analysis model.

[0117] Since the structure and size of the target image feature matrix after fusion are exactly the same as those of the first image feature matrix before fusion, the process of obtaining the target image feature matrix based on the first image feature matrix and the first text vector can be easily integrated into any CNN network. That is, the target image feature matrix after fusing multiple modal feature information can be trained, replacing the method of training a single image modal feature. The model obtained by training the target image feature matrix after fusing multiple modal information can achieve better video analysis results.

[0118] The fifth processing unit 804 is used to process the video based on the target video analysis model to obtain the processed target video.

[0119] Optionally, in the video processing apparatus provided in Embodiment 5 of this application, the fifth processing unit 804 includes: a third processing module, used to classify the live video based on the target video analysis model to obtain the classified live video and the classification tag corresponding to the live video.

[0120] The MPE module is used to fuse features from multiple modalities in a video. It can be applied plug-and-play to video training networks, making it applicable to various video analysis scenarios and improving the performance of trained models. For example, in live video analysis, different live videos may contain different types of product images. Simultaneously, the host provides audio descriptions of the products. By fusing the image modal information from the video with the text vector converted from the host's audio, the resulting fused target image feature matrix increases the weight of product feature information in the video. Based on this, the target video analysis model trained can more accurately identify product types in the video. Using this target video analysis model to classify live videos improves classification accuracy, resulting in classified live videos and their corresponding classification labels.

[0121] It should be noted that the aforementioned units and modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 2.

[0122] Example 6

[0123] According to embodiments of the present invention, an apparatus for implementing the video processing method of Embodiment 2 described above is also provided, such as... Figure 9 As shown, the device includes a receiving unit 901, a response unit 902, and an output unit 903.

[0124] Specifically, the receiving unit 901 is used to receive a service call request sent by the client, wherein the service call request carries a request to fuse image feature information and text feature information in the video;

[0125] Response unit 902 is used to respond to service call requests. In the server, it processes a first image feature matrix based on a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; it then obtains a third image feature matrix based on the first and second image feature matrices; and finally processes the third image feature matrix based on the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and

[0126] Output unit 903 is used to output the feature matrix of the target image.

[0127] By calling the server to process the video, processing efficiency can be improved, and textual and image feature information can be quickly fused. This achieves the goal of fully integrating multiple modal features in the video, thereby enhancing the robustness of video features.

[0128] Optionally, in the video processing apparatus provided in Embodiment Six of this application, the target image feature matrix is ​​used in the training of a multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0129] By using the feature matrix of the target image in the training of the multimodal signal model, the accuracy of the trained multimodal signal model is improved, so that the fusion of features of multiple modal signals will be more effective in the future.

[0130] It should be noted that the aforementioned units and modules, as part of the device, can run on the computer terminal 10 provided in Embodiment 3.

[0131] Example 7

[0132] Embodiments of the present invention can provide a computer terminal, which can be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the computer terminal can also be replaced by a mobile terminal or other terminal device.

[0133] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0134] In this embodiment, the computer terminal described above can execute the following steps of the video feature fusion method of the application: obtaining a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; processing the first image feature matrix according to the first text vector to obtain a second image feature matrix; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0135] The program code for the following steps in the video feature fusion method of the above-mentioned computer terminal that can execute the application: processing the first image feature matrix according to the first text vector to obtain the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain the fourth image feature matrix; processing the fourth image feature matrix according to the first text vector to obtain the fifth image feature matrix; and processing the fifth image feature matrix according to a preset function to obtain the second image feature matrix.

[0136] The program code for the following steps in the video feature fusion method of the above-mentioned computer terminal that can execute the application: processing the fourth image feature matrix according to the first text vector to obtain the fifth image feature matrix includes: performing matrix multiplication of the first text vector and the fourth image feature matrix to obtain the sixth image feature matrix; multiplying the sixth image feature matrix with the sixth image feature matrix to obtain the fifth image feature matrix.

[0137] The above-mentioned computer terminal can execute the program code for the following steps in the video feature fusion method of the application: before obtaining the first image feature matrix and the first text vector, the method includes: obtaining audio information in the video; processing the audio information to obtain corresponding text information; and processing the text information through a recurrent neural network to obtain text feature information.

[0138] The program code for the following steps in the video feature fusion method of the above-mentioned computer terminal can execute the application: processing the third image feature matrix according to the first image feature matrix to obtain the target image feature matrix includes: performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first image feature matrix and the seventh image feature matrix to obtain the target image feature matrix.

[0139] The computer terminal described above can execute the program code for the following steps in the video feature fusion method of the application: the target image feature matrix has the same size and structure as the first image feature matrix, the method is applied to the training of a multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0140] The program code for the following steps in the video feature fusion method of the above-mentioned computer terminal that can execute the application: obtaining a third image feature matrix based on the first image feature matrix and the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and performing matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain the third image feature matrix.

[0141] The computer terminal described above can execute the following steps in the video feature fusion method of the application: obtaining a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information; training the target image feature matrix to obtain a target video analysis model; and processing the video based on the target video analysis model to obtain a processed target video.

[0142] The above-mentioned computer terminal can execute the program code for the following steps in the video feature fusion method of the application: the video includes live video, and the video is processed based on the target video analysis model to obtain the processed target video, including: classifying the live video based on the target video analysis model to obtain the classified live video and the classification label corresponding to the live video.

[0143] The computer terminal described above can execute the following steps in the video feature fusion method of the application: receiving a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; responding to the service call request, processing a first image feature matrix in the server according to a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and outputting the target image feature matrix.

[0144] The computer terminal described above can execute the program code for the following steps in the video feature fusion method of the application: the target image feature matrix is ​​used in the training of the multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0145] Optionally, Figure 10 This is a structural block diagram of a computer terminal according to an embodiment of the present invention. Figure 10 As shown, the computer terminal may include: one or more ( Figure 10 (Only one is shown in the image) processor and memory.

[0146] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video feature fusion method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned video feature fusion method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0147] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: acquiring a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; processing the first image feature matrix according to the first text vector to obtain a second image feature matrix; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0148] Optionally, the processor may also execute program code for the following steps: processing the first image feature matrix according to the first text vector to obtain a second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain a fourth image feature matrix; processing the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; and processing the fifth image feature matrix according to a preset function to obtain a second image feature matrix.

[0149] Optionally, the processor may also execute program code for the following steps: processing the fourth image feature matrix based on the first text vector to obtain the fifth image feature matrix includes: performing matrix multiplication of the first text vector and the fourth image feature matrix to obtain the sixth image feature matrix; and multiplying the sixth image feature matrix with the first image feature matrix to obtain the fifth image feature matrix.

[0150] Optionally, the processor may also execute program code for the following steps: before acquiring the first image feature matrix and the first text vector, the method includes: acquiring audio information from the video; processing the audio information to obtain corresponding text information; and processing the text information through a recurrent neural network to obtain text feature information.

[0151] Optionally, the processor may also execute program code for the following steps: processing the third image feature matrix according to the first image feature matrix to obtain the target image feature matrix includes: performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first image feature matrix and the seventh image feature matrix to obtain the target image feature matrix.

[0152] Optionally, the processor may also execute program code with the following steps: the target image feature matrix has the same size and structure as the first image feature matrix, the method is applied to the training of a multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0153] Optionally, the processor may also execute program code for the following steps: obtaining a third image feature matrix based on the first image feature matrix and the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and performing matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain the third image feature matrix.

[0154] Optionally, the processor may also execute program code for the following steps: obtaining a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; obtaining a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information; training the target image feature matrix to obtain a target video analysis model; and processing the video based on the target video analysis model to obtain a processed target video.

[0155] Optionally, the processor may also execute program code for the following steps: the video includes a live video, and the video is processed based on the target video analysis model to obtain a processed target video, including: classifying the live video based on the target video analysis model to obtain a classified live video and a classification label corresponding to the live video.

[0156] Optionally, the processor may also execute program code that performs the following steps: receiving a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; responding to the service call request, processing a first image feature matrix in the server according to a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; obtaining a third image feature matrix based on the first image feature matrix and the second image feature matrix; processing the third image feature matrix based on the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and outputting the target image feature matrix.

[0157] Optionally, the processor may also execute program code for the following steps: the target image feature matrix is ​​used in the training of a multimodal signal model, and the multimodal signal model is used to fuse the features of multiple modal signals.

[0158] This invention provides a method for fusing video features. By acquiring a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; processing the first image feature matrix based on the first text vector yields a second image feature matrix; obtaining a third image feature matrix based on the first and second image feature matrices; and processing the third image feature matrix based on the first image feature matrix yields a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information. This achieves the goal of fully fusing multiple modal features in the video, thereby solving the technical problem of inaccurate information description in the video caused by insufficient fusion of multiple modal information.

[0159] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a mobile internet device (MID), a PAD, and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include components that are more advanced than those described above. Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0160] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0161] Example 8

[0162] Embodiments of the present invention also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the video feature fusion method provided in Embodiment 1.

[0163] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0164] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; processing the first image feature matrix according to the first text vector to obtain a second image feature matrix; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

[0165] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: processing the first image feature matrix according to the first text vector to obtain a second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain a fourth image feature matrix; processing the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; and processing the fifth image feature matrix according to a preset function to obtain a second image feature matrix.

[0166] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: processing the fourth image feature matrix according to the first text vector to obtain the fifth image feature matrix includes: performing matrix multiplication of the first text vector and the fourth image feature matrix to obtain the sixth image feature matrix; and multiplying the sixth image feature matrix with the sixth image feature matrix to obtain the fifth image feature matrix.

[0167] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: before obtaining the first image feature matrix and the first text vector, the method includes: obtaining audio information in the video; processing the audio information to obtain corresponding text information; and processing the text information through a recurrent neural network to obtain text feature information.

[0168] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix includes: performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first image feature matrix and the seventh image feature matrix to obtain the target image feature matrix.

[0169] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the target image feature matrix has the same size and structure as the first image feature matrix, the method is applied to the training of a multimodal signal model, and the multimodal signal model is used to fuse features of multiple modal signals.

[0170] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a third image feature matrix based on the first image feature matrix and the second image feature matrix includes: performing convolution processing on the first image feature matrix to obtain an eighth image feature matrix; and performing matrix multiplication between the eighth image feature matrix and the second image feature matrix to obtain the third image feature matrix.

[0171] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; obtaining a target image feature matrix based on the first image feature matrix and the first text vector, wherein the target image feature matrix includes fused image feature information and text information; training the target image feature matrix to obtain a target video analysis model; and processing the video based on the target video analysis model to obtain a processed target video.

[0172] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the video includes a live video, and the video is processed based on the target video analysis model to obtain a processed target video, including: classifying the live video based on the target video analysis model to obtain a classified live video and a classification label corresponding to the live video.

[0173] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; responding to the service call request, processing a first image feature matrix in the server according to a first text vector to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; obtaining a third image feature matrix according to the first image feature matrix and the second image feature matrix; processing the third image feature matrix according to the first image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and outputting the target image feature matrix.

[0174] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: the target image feature matrix is ​​used in the training of a multimodal signal model, the multimodal signal model being used to fuse features of multiple modal signals.

[0175] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0176] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0177] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0178] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0179] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0180] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0181] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for fusing video features, characterized in that, The method includes: Obtain a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; The first image feature matrix is ​​convolved to obtain the fourth image feature matrix; the fourth image feature matrix is ​​processed according to the first text vector to obtain the fifth image feature matrix; the fifth image feature matrix is ​​processed according to a preset normalization exponential function to obtain the second image feature matrix. Based on the first image feature matrix and the second image feature matrix, a third image feature matrix is ​​obtained; The third image feature matrix is ​​convolved to obtain the seventh image feature matrix; the first image feature matrix and the seventh image feature matrix are summed to obtain the target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

2. The method according to claim 1, characterized in that, The fifth image feature matrix is ​​obtained by processing the fourth image feature matrix based on the first text vector, including: Multiply the first text vector by the fourth image feature matrix to obtain the sixth image feature matrix; Multiply the sixth image feature matrix by the sixth image feature matrix to obtain the fifth image feature matrix.

3. The method according to claim 1, characterized in that, Before obtaining the first image feature matrix and the first text vector, the method includes: Extract audio information from a video; The audio information is processed to obtain the corresponding text information; The text information is processed by a recurrent neural network to obtain text feature information.

4. The method according to claim 1, characterized in that, The target image feature matrix has the same size and structure as the first image feature matrix. The method is applied to the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

5. The method according to claim 1, characterized in that, The third image feature matrix is ​​obtained based on the first image feature matrix and the second image feature matrix, including: The first image feature matrix is ​​convolved to obtain the eighth image feature matrix. The eighth image feature matrix is ​​multiplied by the second image feature matrix to obtain the third image feature matrix.

6. A video processing method, characterized in that, The method includes: Obtain a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; The first image feature matrix is ​​convolved to obtain a fourth image feature matrix; the fourth image feature matrix is ​​processed according to the first text vector to obtain a fifth image feature matrix; the fifth image feature matrix is ​​processed according to a preset normalization exponential function to obtain a second image feature matrix; the first image feature matrix and the second image feature matrix are combined to obtain a third image feature matrix; the third image feature matrix is ​​convolved to obtain a seventh image feature matrix; the first image feature matrix and the seventh image feature matrix are summed to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text information; The target image feature matrix is ​​trained to obtain a target video analysis model; The video is processed based on the target video analysis model to obtain the processed target video.

7. The method according to claim 6, characterized in that, The video includes live video. The video is processed based on the target video analysis model to obtain the processed target video, which includes: The live video is classified based on the target video analysis model to obtain the classified live video and the corresponding classification tags.

8. A video processing method, characterized in that, include: Receive a service call request sent by a client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; In response to the service call request, the server performs convolution processing on the first image feature matrix to obtain a fourth image feature matrix; processes the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; processes the fifth image feature matrix according to a preset normalization exponential function to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; a third image feature matrix is ​​obtained based on the first and second image feature matrices; the third image feature matrix is ​​convolved to obtain a seventh image feature matrix; and the first and seventh image feature matrices are summed to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and Output the feature matrix of the target image.

9. The method according to claim 8, characterized in that, The target image feature matrix is ​​used in the training of a multimodal signal model, which is used to fuse the features of multiple modal signals.

10. A video feature fusion apparatus, characterized in that, The device includes: The first acquisition unit is used to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; The first processing unit is configured to perform convolution processing on the first image feature matrix to obtain a fourth image feature matrix; process the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; and process the fifth image feature matrix according to a preset normalization exponential function to obtain a second image feature matrix. The second acquisition unit is used to obtain a third image feature matrix based on the first image feature matrix and the second image feature matrix; The second processing unit is used to perform convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and to sum the first image feature matrix and the seventh image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information.

11. A video processing apparatus, characterized in that, The device includes: The fourth acquisition unit is used to acquire a first image feature matrix and a first text vector, wherein the first image feature matrix includes image feature information in the video, and the first text vector includes text feature information in the video; The fifth acquisition unit is configured to perform convolution processing on the first image feature matrix to obtain a fourth image feature matrix; process the fourth image feature matrix according to the first text vector to obtain a fifth image feature matrix; process the fifth image feature matrix according to a preset normalization exponential function to obtain a second image feature matrix; obtain a third image feature matrix according to the first image feature matrix and the second image feature matrix; perform convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and sum the first image feature matrix and the seventh image feature matrix to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text information. The training unit is used to train the feature matrix of the target image to obtain the target video analysis model; The fifth processing unit is used to process the video based on the target video analysis model to obtain the processed target video.

12. A video processing apparatus, characterized in that, include: The receiving unit is used to receive a service call request sent by the client, wherein the service call request carries a request to fuse image feature information and text feature information in the video; A response unit is configured to respond to the service call request by performing convolution processing on a first image feature matrix in the server to obtain a fourth image feature matrix; processing the fourth image feature matrix according to a first text vector to obtain a fifth image feature matrix; processing the fifth image feature matrix according to a preset normalization exponential function to obtain a second image feature matrix, wherein the first image feature matrix includes image feature information from the video, and the first text vector includes text feature information from the video; obtaining a third image feature matrix based on the first and second image feature matrices; performing convolution processing on the third image feature matrix to obtain a seventh image feature matrix; and summing the first and seventh image feature matrices to obtain a target image feature matrix, wherein the target image feature matrix includes fused image feature information and text feature information; and The output unit is used to output the feature matrix of the target image.

13. A storage medium, characterized in that, The storage medium includes a stored program, wherein when the program is executed, it controls the device where the storage medium is located to perform the method according to any one of claims 1 to 5, or the method according to any one of claims 6 to 7, or the method according to any one of claims 8 to 9.

14. A processor, characterized in that, The processor is used to run a program, wherein the program, when running, executes the method according to any one of claims 1 to 5, or the method according to any one of claims 6 to 7, or the method according to any one of claims 8 to 9.

Images