Video processing method, device, equipment and storage medium based on artificial intelligence

Abstract

The present application provides a video processing method, device, electronic device and computer-readable storage medium based on artificial intelligence; it relates to artificial intelligence technology; the method includes: performing feature extraction processing on multiple sub-video sequences of a video respectively to obtain a feature sequence of each sub-video sequence; performing encoding processing based on multiple channels on the feature sequence of each sub-video sequence to obtain channel features corresponding to multiple channels of each sub-video sequence; performing feature aggregation processing based on multiple channels on the channel features corresponding to the multiple sub-video sequences respectively to obtain a fixed-length feature whose dimension is associated with the number of multiple channels; performing mapping processing on the fixed-length feature whose dimension is associated with the number of multiple channels to obtain an embedded vector of the video; performing video application processing based on the embedded vector of the video to obtain an application result of the video. Through the present application, the application effect of the video can be improved.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an artificial intelligence-based video processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive technology in computer science. By studying the design principles and implementation methods of various intelligent machines, it enables machines to have the functions of perception, reasoning and decision-making. Artificial Intelligence technology is a comprehensive discipline that covers a wide range of fields, such as natural language processing technology and machine learning / deep learning. With the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0003] The related art lacks an effective solution for video applications based on videos of variable length, and mainly relies on video cutting to obtain multiple video clips of fixed length, and extracting embedded features of the video clips for video applications such as video classification and video recall. However, this solution has the problem of variable length input and the timing problem of embedded features, which in turn affects the application effect of video applications. Summary of the invention

[0004] The embodiments of the present application provide a video processing method, device, electronic device and computer-readable storage medium based on artificial intelligence, which can improve the application effect of the video.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The embodiment of the present application provides a video processing method based on artificial intelligence, including:

[0007] Performing feature extraction processing on multiple sub-video sequences of the video respectively to obtain a feature sequence of each sub-video sequence;

[0008] Performing encoding processing based on multiple channels on the feature sequence of each of the sub-video sequences to obtain channel features corresponding to the multiple channels of each of the sub-video sequences;

[0009] Performing feature aggregation processing based on the multiple channels on the channel features respectively corresponding to the multiple sub-video sequences to obtain fixed-length features whose dimensions are associated with the number of the multiple channels;

[0010] Mapping the fixed-length features whose dimensions are associated with the number of the multiple channels to obtain an embedding vector of the video;

[0011] Video application processing is performed based on the embedded vector of the video to obtain an application result of the video.

[0012] The embodiment of the present application provides a video processing device based on artificial intelligence, including:

[0013] A feature extraction module is used to perform feature extraction processing on multiple sub-video sequences of a video respectively to obtain a feature sequence of each sub-video sequence;

[0014] An encoding module, used for performing encoding processing on the feature sequence of each sub-video sequence based on multiple channels to obtain channel features of each sub-video sequence corresponding to the multiple channels;

[0015] an aggregation module, configured to perform feature aggregation processing based on the multiple channels on the channel features respectively corresponding to the multiple sub-video sequences, so as to obtain fixed-length features whose dimensions are associated with the number of the multiple channels;

[0016] A mapping module, configured to perform mapping processing on the fixed-length features whose dimensions are associated with the number of the plurality of channels to obtain an embedding vector of the video;

[0017] The application module is used to perform video application processing based on the embedding vector of the video to obtain the application result of the video.

[0018] In the above technical solution, the encoding process based on multiple channels is implemented by an encoding model, and the encoding model includes multiple cascaded convolutional layers; the encoding module is also used to perform convolution processing based on multiple channels on the feature sequence of each sub-video sequence through the first convolutional layer in the multiple cascaded convolutional layers;

[0019] Outputting the convolution result of the first convolution layer to a subsequent cascaded convolution layer, and continuing to perform convolution processing and output convolution results through the subsequent cascaded convolution layer;

[0020] The convolution result output by the last convolution layer is used as the channel feature of each of the sub-video sequences corresponding to the multiple channels.

[0021] In the above technical solution, the encoding module is further used to perform convolution processing on the convolution results output by the j-1th convolution layer through the multiple channels in the jth convolution layer of the multiple cascaded convolution layers, respectively, to obtain the convolution results corresponding to the multiple channels respectively;

[0022] Concatenate the convolution results corresponding to the multiple channels to obtain the convolution result of the j-th convolution layer;

[0023] Outputting the convolution result of the j-th convolutional layer through the j-th convolutional layer;

[0024] Among them, j is an increasing natural number and its value range is 1 < j ≤ N, where N is the number of convolutional layers.

[0025] In the above technical solution, the aggregation module is further configured to perform the following processing on any one of the multiple channels:

[0026] From the channel features of each sub-video sequence corresponding to the multiple channels, obtain the sub-channel features of each sub-video sequence corresponding to the channel;

[0027] Perform statistical pooling processing on the sub-channel features of the multiple sub-video sequences corresponding to the channel respectively to obtain the statistical features of the channel;

[0028] Perform splicing processing on the statistical features corresponding to the multiple channels respectively to obtain a fixed-length feature whose dimension is associated with the number of the multiple channels.

[0029] In the above technical solution, the aggregation module is further configured to perform mean processing based on the sub-channel features of the multiple sub-video sequences corresponding to the channel respectively to obtain the mean features of the multiple sub-video sequences corresponding to the channel;

[0030] Perform variance processing based on the sub-channel features of the multiple sub-video sequences corresponding to the channel respectively to obtain the variance features of the multiple sub-video sequences corresponding to the channel;

[0031] Perform splicing processing on the mean features of the multiple sub-video sequences corresponding to the channel and the variance features of the multiple sub-video sequences corresponding to the channel to obtain the statistical features of the channel.

[0032] In the above technical solution, the mapping processing is implemented by a mapping model, and the mapping model includes multiple cascaded fully connected layers; the mapping module is further configured to perform mapping processing on the fixed-length feature whose dimension is associated with the number of the multiple channels through the first fully connected layer in the multiple cascaded fully connected layers;

[0033] Output the mapping result of the first fully connected layer to the subsequent cascaded fully connected layers, and continue to perform mapping processing and output of the mapping result through the subsequent cascaded fully connected layers;

[0034] Take the mapping result output by the last fully connected layer as the embedding vector of the video.

[0035] In the above technical solution, the feature extraction processing is implemented by a feature extraction model, and the feature extraction model includes a pooling layer and multiple cascaded convolutional layers. Each sub-video sequence includes multiple image frames; the feature extraction module is further configured to perform the following processing on any one of the image frames in each sub-video sequence:

[0036] Performing cascade convolution processing on the image frame through the multiple cascaded convolution layers to obtain convolution features of the image frame;

[0037] Performing pooling processing on the convolution features of the image frame through the pooling layer to obtain image features of the image frame;

[0038] The image features respectively corresponding to the multiple image frames are spliced ​​to obtain a feature sequence of each sub-video sequence.

[0039] In the above technical solution, the application module is further used to perform similarity recall processing on candidate similar videos based on the embedding vector of the video to obtain similar videos of the video; or,

[0040] Video classification processing is performed based on the embedding vector of the video to obtain the video type to which the video belongs.

[0041] In the above technical solution, the video processing method is implemented by a video processing model; the device also includes:

[0042] A training module, used for mapping the fixed-length features of the video sample through the initialized video processing model to obtain an embedding vector of the video sample;

[0043] Constructing a target loss function of the video processing model based on the embedding vector of the video sample;

[0044] The parameters of the video processing model are updated based on the target loss function, and the updated parameters of the video processing model are used as the parameters of the trained video processing model.

[0045] In the above technical solution, the training module is also used to perform video classification processing based on the embedding vector of the video sample to obtain the predicted type of the video sample;

[0046] Constructing a classification loss function of the video processing model based on the predicted type of the video sample and the type label of the video sample;

[0047] Constructing an embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vectors of similar video samples of the video sample, and the embedding vectors of dissimilar video samples of the video sample;

[0048] A weighted summation process is performed on the classification loss function of the video processing model and the embedding loss function of the video processing model to obtain a target loss function of the video processing model.

[0049] In the above technical solution, the training module is further used to perform similarity processing on the embedding vector of the video sample and the embedding vector of a similar video sample of the video sample to obtain a first similarity between the video sample and the similar video sample;

[0050] Performing similarity processing on the embedding vector of the video sample and the embedding vector of a video sample that is not similar to the video sample to obtain a second similarity between the video sample and the dissimilar video sample;

[0051] Based on the first similarity and the second similarity, an embedding loss function of the video processing model is constructed.

[0052] In the above technical solution, the training module is further used to extract at least one candidate video sample from a plurality of candidate similar video sample pairs, wherein the plurality of candidate similar video sample pairs do not include the video sample;

[0053] Based on the similarity between the candidate video samples and the video samples, sorting the candidate video samples in descending order;

[0054] The candidate video samples that are sorted later in the descending sorting result are sorted in descending order as dissimilar video samples of the video samples.

[0055] An embodiment of the present application provides an electronic device for video processing, the electronic device comprising:

[0056] A memory for storing executable instructions;

[0057] The processor is used to implement the artificial intelligence-based video processing method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0058] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the artificial intelligence-based video processing method provided in the embodiment of the present application.

[0059] The embodiments of the present application have the following beneficial effects:

[0060] By encoding the feature sequence of each sub-video sequence of the video based on multiple channels, the channel features of each sub-video sequence are obtained, and the channel features are aggregated based on multiple channels to obtain the fixed-length features of the video, so that variable-length video input can be effectively processed and videos of any length can be input to improve the flexibility of the input video, thereby improving the utilization rate of the computing resources consumed in the video processing process; the fixed-length features are mapped to obtain time-free embedded vectors, so there is no need to consider the time sequence problem of the embedded features, which improves the accuracy of the embedded vectors and further improves the application effect of the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 is a schematic diagram of an application scenario of a video processing system provided in an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0063] Figure 3-Figure 5 It is a flowchart of a video processing method based on artificial intelligence provided by an embodiment of the present application;

[0064] Figure 6 is a structural diagram of a feature extraction model provided in an embodiment of the present application;

[0065] Figure 7 It is a structural diagram of the coding model provided in the embodiment of the present application;

[0066] Figure 8 It is a model learning framework provided in the embodiments of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0068] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] It should be pointed out that the collection and processing of relevant data in this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained, and subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0071] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0072] 1) Metric Learning: Given some similar samples and some dissimilar samples, by learning the similarity (distance) metric, the similarity between the originally similar samples increases (or the distance decreases), while the similarity between the originally dissimilar samples decreases (or the distance increases). In the embodiment of the present application, the video processing model can be trained by metric learning.

[0073] 2) Image recognition: refers to the use of computers to process, analyze and understand images to identify targets and objects of various different patterns. It is a practical application of deep learning algorithms. Image recognition is a category-level recognition that does not consider the specific instance of the object. It only recognizes the category of the object (such as people, dogs, cats, birds, etc.) and gives the category to which the object belongs. For example, the recognition task in the large-scale general object recognition open source dataset (imagenet) identifies which of the 1,000 categories an object belongs to.

[0074] 3) Fragmented video: A editing technique that divides a long video into many short segments and recombines them into a new video in a different order from the original. For example, the original video is composed of segment A-segment B-segment C-segment D, and segment A1-segment C-segment A2-segment D-segment B is a fragmented video, in which segment A1 and segment A2 are two smaller segments of segment A.

[0075] 4) Blockchain: It includes a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0076] The embodiments of the present application provide a video processing method, device, electronic device and computer-readable storage medium based on artificial intelligence, which can improve the application effect of the video.

[0077] The artificial intelligence-based video processing method provided in the embodiments of the present application can be implemented by the terminal / server alone; it can also be implemented by the terminal and the server in collaboration. For example, the terminal alone undertakes the artificial intelligence-based video processing method described below, or the terminal sends an application request for the video to the server, and the server executes the artificial intelligence-based video processing method based on the received application request for the video, maps the fixed-length features, and obtains a time-free embedded vector, thereby improving the application effect of the video based on the time-free embedded vector.

[0078] The electronic device for video processing provided in the embodiments of the present application may be various types of terminals or servers, wherein the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms; the terminal may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, smart TV, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the present application.

[0079] Taking the server as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access one or more artificial intelligence services provided by the AIaaS platform through an application programming interface.

[0080] For example, one of the artificial intelligence cloud services may be a video processing service, that is, a server in the cloud is encapsulated with a video processing program provided by an embodiment of the present application. The user calls the video processing service in the cloud service through a terminal (running a client, such as a video client), so that the server deployed in the cloud calls the encapsulated video processing program, encodes each sub-video sequence in the video based on multiple channels, obtains channel features corresponding to multiple channels of each sub-video sequence, and performs feature aggregation based on multiple channels on all channel features to obtain fixed-length features of the video, maps the fixed-length features, and obtains a time-free embedding vector of the video, thereby improving the application effect of the video based on the time-free embedding vector, such as improving the accuracy of video recall based on similarity, improving the accuracy of video classification, speeding up the efficiency of video recommendation, and improving the effect of video recommendation based on user behavior in the later stage.

[0081] See also Figure 1 , Figure 1 It is a schematic diagram of an application scenario of the video processing system 10 provided in an embodiment of the present application. The terminal 200 is connected to the server 100 via a network 300. The network 300 may be a wide area network or a local area network, or a combination of the two.

[0082] A terminal (running a client, such as a video client) can be used to obtain an application request for a video. For example, when a user opens a video client running on the terminal and inputs a video that needs to be processed, the terminal automatically obtains an application request for the video (including a video for video processing).

[0083] In some embodiments, a video processing plug-in may be implanted in the client running in the terminal to implement a video processing method based on artificial intelligence locally on the client. For example, the terminal 200 calls the video processing plug-in to implement a video processing method based on artificial intelligence, encodes each sub-video sequence in the video based on multiple channels, obtains channel features corresponding to multiple channels of each sub-video sequence, and performs feature aggregation based on multiple channels on all channel features to obtain fixed-length features of the video, performs mapping processing on the fixed-length features, and obtains a time-sequentially non-embedded vector of the video, thereby improving the application effect of the video based on the time-sequentially non-embedded vector.

[0084] As an application example, for video classification applications, since the input video may be of variable length, in order to implement the video classification application, the related technology cuts or pads the variable-length video to obtain a fixed-length video, and then performs video classification based on the fixed-length video. This solution makes the video content inaccurate, thereby affecting the accuracy of video classification.

[0085] In order to solve the problem of inaccurate video classification, the terminal calls the video processing plug-in to encode each sub-video sequence in the video (i.e., video of variable length) based on multiple channels, obtain channel features corresponding to multiple channels of each sub-video sequence, and perform feature aggregation based on multiple channels on all channel features to obtain fixed-length features of the video, thereby solving the variable-length problem of the input video, and mapping the fixed-length features to obtain a time-free embedding vector of the video, thereby not considering the time sequence problem of the sub-video sequence, so as to improve the accuracy of video classification.

[0086] In some embodiments, after the terminal obtains an application request for a video, it calls the video processing interface of the server 100 (which can be provided in the form of a cloud service, i.e., a video processing service). Based on the application request for the video, the server 100 encodes each sub-video sequence in the video based on multiple channels, obtains channel features corresponding to multiple channels of each sub-video sequence, and performs feature aggregation based on multiple channels on all channel features to obtain fixed-length features of the video, maps the fixed-length features to obtain a time-free embedding vector of the video, thereby improving the application effect of the video based on the time-free embedding vector.

[0087] As an application example, for video retrieval applications, since the input video may be of variable length, in order to perform video similarity retrieval, the related technology cuts the variable-length video to obtain video segments, and then performs similarity comparison based on the time sequence of the video segments to determine whether two videos are similar. This similarity judgment based on time sequence is inaccurate, which affects the accuracy of video retrieval.

[0088] In order to solve the problem of inaccurate video retrieval, after the terminal obtains the retrieval request for the video, it calls the video processing interface of the server. Based on the retrieval request for the video, the server encodes each sub-video sequence in the video (i.e., a video of variable length) based on multiple channels to obtain channel features corresponding to multiple channels for each sub-video sequence, and performs feature aggregation based on multiple channels on all channel features to obtain fixed-length features of the video. The fixed-length features are mapped to obtain a time-free embedding vector of the video, thereby solving the video timing problem, and performing video similarity comparison based on the time-free embedding vector, thereby accurately retrieving similar videos that are similar to the video, so as to improve the accuracy of video retrieval.

[0089] In some embodiments, the terminal or server can implement the video processing method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0090] In some embodiments, multiple servers may form a blockchain, and server 100 is a node on the blockchain. Information connections may exist between each node in the blockchain, and information may be transmitted between nodes through the above information connections. Among them, data related to the artificial intelligence-based video processing method provided in the embodiment of the present application (such as the logic of video processing, the application results of the video) may be stored on the blockchain.

[0091] The structure of the electronic device provided by the embodiment of the present application is described below. Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, taking the electronic device 500 as a server as an example. Figure 2 The electronic device 500 shown includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Figure 2 Various buses are labeled as bus system 540 .

[0092] The processor 510 may be an integrated circuit chip having signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., wherein the general-purpose processor may be a microprocessor or any conventional processor, etc.

[0093] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiment of the present application is intended to include any suitable type of memory. The memory 550 may optionally include one or more storage devices that are physically far away from the processor 510.

[0094] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0095] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0096] A network communication module 553, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0097] In some embodiments, the artificial intelligence-based video processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 The video processing device 555 based on artificial intelligence stored in the memory 550 is shown, which can be software in the form of programs and plug-ins, etc., including the following software modules: feature extraction module 5551, encoding module 5552, aggregation module 5553, mapping module 5554, application module 5555 and training module 5556. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0098] As mentioned above, the artificial intelligence-based video processing method provided in the embodiments of the present application can be implemented by various types of electronic devices. Figure 3 , Figure 3 is a flow chart of a video processing method based on artificial intelligence provided by an embodiment of the present application, combined with Figure 3 The steps shown are explained.

[0099] In step 101, feature extraction processing is performed on multiple sub-video sequences of a video to obtain a feature sequence of each sub-video sequence.

[0100] As an example of obtaining a video, when a user opens a video client running on a terminal and inputs a video that needs to be processed, the terminal automatically obtains an application request for the video (including a video for video processing) and sends the application request for the video to a server. After receiving the application request for the video, the server parses the application request for the video and obtains the video for video processing.

[0101] Since the video may be relatively long and the video length input to the video processing model is limited, it is necessary to extract the video to obtain multiple sub-video sequences, where the sub-video sequence includes multiple image frames. For example, for video Vi, video frames are extracted every Fl lengths to obtain multiple sub-video sequences (i.e., sub-video frame sequence 1, sub-video frame sequence 2, ..., sub-video frame sequence K), where each sub-video sequence includes Fl frames of image. The method of extracting sub-video sequences may be overlapping extraction between adjacent sub-video sequences or non-overlapping extraction, that is, there may or may not be repeated frames between sub-video frame sequence 1 and sub-video frame sequence 2.

[0102] For example, after performing video extraction to obtain multiple sub-video sequences of the video, it is first necessary to perform feature extraction processing on each sub-video sequence through the feature extraction model in the video processing model to obtain a feature sequence of each sub-video sequence, wherein the feature sequence is used to characterize the image features (image embedding) in the sub-video sequence, so as to facilitate subsequent feature processing based on the feature sequence of the sub-video sequence, so as to convert the variable-length features of the video into fixed-length features, effectively solving the input problem of indefinite-length videos.

[0103] In some embodiments, feature extraction processing is implemented through a feature extraction model, which includes a pooling layer and multiple cascaded convolution layers, and each sub-video sequence includes multiple image frames; feature extraction processing is performed on multiple sub-video sequences of the video respectively to obtain a feature sequence of each sub-video sequence, including: performing the following processing on any image frame in each sub-video sequence: performing cascaded convolution processing on the image frame through multiple cascaded convolution layers to obtain convolution features of the image frame; pooling the convolution features of the image frame through a pooling layer to obtain image features of the image frame; splicing the image features corresponding to multiple image frames to obtain a feature sequence of each sub-video sequence.

[0104] like Figure 6As shown, for any sub-video sequence, each image frame included in the sub-video sequence is input to the first convolution layer in the feature extraction model, the image frame is convolved by the first convolution layer to obtain the first feature map of the image frame, the first feature map of the image frame is output to the second convolution layer, the first feature map of the image frame is convolved by the second convolution layer to obtain the second feature map of the image frame, the above convolution processing is performed in sequence until the Mth convolution layer performs convolution processing on the M-1th feature map of the image frame to obtain the convolution feature of the image frame, wherein M is a positive integer greater than 2, and the convolution feature of the image frame is output to the pooling layer in the feature extraction model, the convolution feature of the image frame is averaged or max pooled by the pooling layer to obtain the image feature (image embedding) of the image frame, the image features of all the image frames of any sub-video sequence are spliced ​​to obtain the feature sequence of any sub-video sequence. Through the above feature extraction processing, the feature sequence of the sub-video sequence can be preliminarily extracted to save the computational amount of subsequent processing based on the feature sequence.

[0105] Among them, the feature extraction model can be a convolutional neural network (such as a ResNet101 model) or a deep neural network.

[0106] In step 102, a coding process based on multiple channels is performed on the feature sequence of each sub-video sequence to obtain channel features corresponding to multiple channels of each sub-video sequence.

[0107] For example, since the feature sequences corresponding to multiple sub-video sequences are of indefinite length, in order to be able to convert indefinite-length feature sequences into fixed-length features, the feature sequence of each sub-video sequence can be first encoded based on multiple channels to obtain channel features corresponding to multiple channels of each sub-video sequence, and the channel features are associated with the channels for subsequent feature aggregation based on the channel features. The image frame level features are converted into fixed-length video features to solve the input problem of indefinite-length videos.

[0108] It should be noted that the encoding process based on multiple channels is implemented through an encoding model, which includes multiple cascaded convolutional layers. Figure 4 , Figure 4 is a flow chart of a video processing method based on artificial intelligence provided in an embodiment of the present application, Figure 4 Show Figure 3Step 102 can be implemented through steps 1021 - 1023: In step 1021, the feature sequence of each sub-video sequence is subjected to convolution processing based on multiple channels through the first convolutional layer in multiple cascaded convolutional layers; in step 1022, the convolution result of the first convolutional layer is output to the subsequent cascaded convolutional layers, and convolution processing and convolution result output continue through the subsequent cascaded convolutional layers; in step 1023, the convolution result output by the last convolutional layer is used as the channel features corresponding to multiple channels of each sub-video sequence.

[0109] As Figure 7 shown, the first convolutional layer performs encoding convolution based on multiple channels on the feature sequence of the sub-video sequence to obtain the convolution result of the first convolutional layer, and the convolution result of the first convolutional layer is output to the second convolutional layer. The second convolutional layer continues convolution encoding and convolution result output until it is output to the Nth convolutional layer. The Nth convolutional layer performs convolution encoding on the convolution result output by the (N - 1)th convolutional layer to obtain the channel features corresponding to multiple channels of the sub-video sequence, where N is the total number of multiple cascaded convolutional layers, and N is a positive integer greater than 2. Among them, through convolution processing by cascaded convolutional layers, high-order features of the sub-video sequence can be gradually extracted, the receptive field can be expanded to extract more video information.

[0110] For example, the feature sequence is subjected to convolution processing through multiple channels in the first convolutional layer among multiple cascaded convolutional layers to obtain the convolution results corresponding to multiple channels in the first convolutional layer respectively. The convolution results corresponding to multiple channels in the first convolutional layer are subjected to splicing processing to obtain the convolution result of the first convolutional layer, and the convolution result of the first convolutional layer is output to the second convolutional layer; the convolution results output by the (j - 1)th convolutional layer are subjected to convolution processing through multiple channels in the jth convolutional layer among multiple cascaded convolutional layers to obtain the convolution results corresponding to multiple channels in the jth convolutional layer respectively; the convolution results corresponding to multiple channels in the jth convolutional layer are subjected to splicing processing to obtain the convolution result of the jth convolutional layer; the convolution result of the jth convolutional layer is output through the jth convolutional layer; where j is an increasing natural number and the value range is 1 < j ≤ N, and N is the number of convolutional layers.

[0111] In step 103, feature aggregation processing based on multiple channels is performed on the channel features corresponding to multiple sub-video sequences respectively to obtain fixed-length features whose dimensions are associated with the number of multiple channels.

[0112] For example, after obtaining the channel features of multiple channels corresponding to each sub-video sequence, since the channel features are associated with the channels, the channel features corresponding to the sub-video sequences are aggregated based on multiple channels to obtain fixed-length features of the video, where the dimension of the fixed-length features is the same as the number of channels, thereby converting the image frame level features into fixed-length video features, solving the input problem of variable-length videos, and solving the timing problem of multiple sub-video sequences.

[0113] See also Figure 5 , Figure 5 is a flow chart of a video processing method based on artificial intelligence provided in an embodiment of the present application, Figure 5 Show Figure 3 Step 103 can be implemented through steps 1031-1033: in step 1031, the following processing is performed on any channel of the multiple channels: from the channel features of the multiple channels corresponding to each sub-video sequence, the sub-channel features of the channel corresponding to each sub-video sequence are obtained; in step 1032, the sub-channel features of the channels corresponding to the multiple sub-video sequences are respectively subjected to statistics-based pooling processing to obtain the statistical features of the channels; in step 1033, the statistical features corresponding to the multiple channels are respectively subjected to splicing processing to obtain fixed-length features whose dimensions are associated with the number of multiple channels.

[0114] It should be noted that video processing is implemented through a video processing model, which includes a statistical pooling layer. The statistical pooling layer is different from a general maximum pooling layer or an average pooling layer, and performs pooling processing based on a statistical method.

[0115] For example, the statistics-based pooling process is as follows: performing mean processing on sub-channel features of channels corresponding to multiple sub-video sequences respectively to obtain mean features of channels corresponding to multiple sub-video sequences; performing variance processing on sub-channel features of channels corresponding to multiple sub-video sequences respectively to obtain variance features of channels corresponding to multiple sub-video sequences; performing splicing processing on mean features of channels corresponding to multiple sub-video sequences and variance features of channels corresponding to multiple sub-video sequences to obtain statistical features of the channels.

[0116] As an example, the above cascaded convolutional layer outputs 1*512 channel features of Kvi sub-video sequences, that is, the 512 channels that need to be learned in the video processing model have Kvi feature values. In statistical pooling, for each channel, the mean (i.e., mean feature) and variance (i.e., variance feature) of the Kvi feature values ​​are obtained, and finally 512 means mean_vi and 512 variances std_vi are obtained, which represent the mean features and variance features of the video under all 512 channels; then, all mean features and variance features [mean_vi,std_vi] are spliced ​​to obtain 1*1024-dimensional fixed-length features.

[0117] In step 104, a mapping process is performed on the fixed-length features whose dimensions are associated with the number of channels to obtain an embedding vector of the video.

[0118] For example, after obtaining the fixed-length features of a video, it is necessary to convert the fixed-length features into an embedding vector for video applications, that is, to map the fixed-length features through the fully connected layer in the video processing model to obtain the embedding vector of the video. The embedding vector has the characteristics of being time-free, thereby removing the timing influence of multiple sub-video sequences and improving the effect of subsequent video applications.

[0119] It should be noted that the video processing model includes a mapping model, and the mapping processing is implemented through the mapping model, which includes multiple cascaded fully connected layers; the mapping processing process is as follows: through the first fully connected layer in the multiple cascaded fully connected layers, the fixed-length features whose dimensions are associated with the number of multiple channels are mapped; the mapping result of the first fully connected layer is output to the subsequent cascaded fully connected layers, and the mapping processing and mapping result output are continued through the subsequent cascaded fully connected layers; the mapping result output by the last fully connected layer is used as the embedding vector of the video.

[0120] For example, the first fully connected layer performs nonlinear mapping processing on the fixed-length features to obtain the mapping result of the first fully connected layer, and outputs the mapping result of the first fully connected layer to the second fully connected layer. The second fully connected layer continues to perform nonlinear mapping processing and output the mapping result until it is output to the Kth fully connected layer. The Kth fully connected layer performs nonlinear mapping on the mapping result output by the K-1th fully connected layer to obtain the embedding vector of the video, where K is the total number of multiple cascaded fully connected layers, and K is a positive integer greater than 2. Among them, by performing nonlinear mapping processing through cascaded fully connected layers, the embedding vector can be gradually refined to perform video applications based on accurate embedding vectors.

[0121] In step 105, video application processing is performed based on the embedded vector of the video to obtain an application result of the video.

[0122] It should be noted that after obtaining an accurate embedding vector, video application processing is performed based on the embedding vector to obtain the application result of the video. For example, based on the embedding vector of the video, similarity recall processing is performed on the candidate similar videos to obtain similar videos of the video. The similarity recall processing process is as follows: the embedding vector of the candidate similar video in the video library is calculated, and the similarity between the embedding vector of the candidate similar video and the embedding vector of the video is obtained. When the similarity is greater than the similarity threshold, the candidate similar video is used as the similar video of the video; based on the embedding vector of the video, video classification processing is performed to obtain the video type to which the video belongs, that is, the embedding vector of the video is subjected to video classification processing through the fully connected layer to obtain the predicted video type.

[0123] It should be noted that the video processing method is implemented through a video processing model; the training process of the video processing model includes: mapping the fixed-length features of the video sample through the initialized video processing model to obtain the embedding vector of the video sample; constructing the target loss function of the video processing model based on the embedding vector of the video sample; updating the parameters of the video processing model based on the target loss function, and using the updated parameters of the video processing model as the parameters of the trained video processing model.

[0124] Continuing with the above example, the video processing model includes a feature extraction model, an encoding model, a pooling layer, and a mapping model. The feature extraction model is used to perform feature extraction processing on multiple sub-video sequences of the video sample to obtain a feature sequence of each sub-video sequence, and the feature sequence of each sub-video sequence is encoded based on multiple channels to obtain channel features corresponding to multiple channels of each sub-video sequence. The channel features corresponding to the multiple sub-video sequences are subjected to feature aggregation processing based on multiple channels to obtain fixed-length features whose dimensions are associated with the number of multiple channels, and the fixed-length features whose dimensions are associated with the number of multiple channels are mapped to obtain an embedding vector of the video sample. After determining the value of the target loss function of the video processing model based on the embedding vector of the video sample, it can be determined whether the value of the target loss function exceeds a preset threshold. When the value of the target loss function exceeds the preset threshold, the error signal of the video processing model is determined based on the target loss function, and the error information is back-propagated in the video processing model, and the model parameters of each layer are updated during the propagation process.

[0125] Here, the back propagation is explained. The training sample data is input into the input layer of the neural network model, passes through the hidden layer, and finally reaches the output layer and outputs the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the output result and the actual value is calculated, and the error is back-propagated from the output layer to the hidden layer until it propagates to the input layer. In the back propagation process, the value of the model parameter is adjusted according to the error, that is, the loss function is constructed according to the error between the output result and the actual value, and the partial derivative of the loss function to the model parameter is calculated layer by layer to generate the gradient of the loss function to the model parameters of each layer. Since the direction of the gradient indicates the direction of error expansion, the gradient of the model parameter is inverted and summed with the original parameters of each layer model. The summed result is used as the updated model parameters of each layer, thereby reducing the error caused by the model parameters; the above process is iterated continuously until convergence. Among them, the video processing model is a neural network model.

[0126] In some embodiments, based on the embedding vector of the video sample, a target loss function of the video processing model is constructed, including: performing video classification processing based on the embedding vector of the video sample to obtain a predicted type of the video sample; constructing a classification loss function of the video processing model based on the predicted type of the video sample and the type label of the video sample; constructing an embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vector of similar video samples of the video sample, and the embedding vector of dissimilar video samples of the video sample; performing weighted summation processing on the classification loss function of the video processing model and the embedding loss function of the video processing model to obtain the target loss function of the video processing model.

[0127] For example, when the video application is similarity recall, the classification loss function and the embedding loss function are combined for joint training to improve the accuracy of the embedding vector and achieve accurate representation of the video, so as to improve the accuracy of the similarity comparison; when the video application is video classification, the classification loss function and the embedding loss function are combined for joint training to improve the accuracy of the embedding vector and achieve accurate representation of the video, so as to improve the accuracy of video classification.

[0128] It should be noted that the process of constructing the embedding loss function of the video processing model is as follows: the embedding vector of the video sample and the embedding vector of similar video samples to the video sample are processed for similarity to obtain a first similarity between the video sample and the similar video samples; the embedding vector of the video sample and the embedding vector of dissimilar video samples to the video sample are processed for similarity to obtain a second similarity between the video sample and the dissimilar video samples; based on the first similarity and the second similarity, the embedding loss function of the video processing model is constructed.

[0129] For example, the embedding loss function is calculated as ,in, Indicates the margin, represents the L2 distance between the embedding vector of video sample a and the embedding of similar video sample p, Represents the L2 distance between the embedding of video sample a and the embedding of dissimilar video sample n. By embedding the loss function, the similarity between originally similar samples increases (or the distance decreases), while the similarity between originally dissimilar samples decreases (or the distance increases).

[0130] In some embodiments, before constructing an embedding loss function of a video processing model based on an embedding vector of a video sample, an embedding vector of similar video samples of the video sample, and an embedding vector of dissimilar video samples of the video sample, at least one candidate video sample is extracted from multiple candidate similar video sample pairs, wherein the multiple candidate similar video sample pairs do not include the video sample; based on the similarity between the candidate video sample and the video sample, the candidate video samples are sorted in descending order; and some of the candidate video samples that are sorted at the end of the descending sorting result are sorted in descending order as dissimilar video samples of the video sample.

[0131] For example, for a video sample x in a similar video sample pair, a candidate video sample i is randomly selected from each candidate similar video sample pair, and the distance between the candidate video sample i and the video sample x is calculated. The distances are sorted from small to large, and the first 10 candidate video samples are taken as negative samples (i.e., samples that are not similar to video sample x), and are respectively formed into triplets with the positive samples of video sample x (i.e., the other video sample in the similar video sample pair).

[0132] The following is an exemplary application of an embodiment of the present application in an actual video recall scenario.

[0133] Video similarity judgment is achieved by using image embedding vectors (embedding) to judge whether two videos are similar based on temporal logic. For example, for video a composed of ABCD segments, video b composed of BCD is a duplicate of video a, where BCD is a sub-video with the same segment order in ABCD. The method based on image embedding temporal judgment does not work for fragmented videos, that is, videos are generated by using smaller video segments and disrupting the original video order, thereby bypassing the temporal constraints in the review, causing such videos to cause losses to video websites by leaking the plot in advance. How to effectively identify such fragmented videos is a difficulty in video similarity recognition.

[0134] In the related art, video representation methods include 2D feature joint temporal logic method and R(2+1)D method. Among them, the feature joint temporal logic method uses a 2D image representation model to extract feature sequences from each frame of the video, and uses temporal judgment logic to determine whether two videos are repeated. For example, for N frames of video a and M frames of video b, when min(N,M) / 2 consecutive frames are similar, video a and video b are repeated; the R(2+1)D method is to cut the video into segments of equal length and extract the embedding of each video segment. When this method is applied to video similarity judgment, it is also necessary to perform temporal logic judgment on the embeddings of multiple video segments of the two videos to be compared, so as to determine whether the two videos are repeated.

[0135] However, the above video representation methods have the following problems: 1) They all require temporal logic to assist in judgment. Considering that the fragmented videos may be cut in different ways, such as extremely short and extremely long, it is difficult for temporal logic to cover various fragmented video similarity judgment scenarios; 2) The R(2+1)D method requires a fixed number of frames of video clips during training and cannot handle feature training of videos of unequal lengths. Therefore, after the model outputs the video clip embedding, temporal logic is still required for the final video similarity judgment.

[0136] In order to solve the above problems, the embodiment of the present application provides a video processing method based on artificial intelligence, which realizes the generation of video embedding from variable-length sequences through a neural network that supports variable-length sequence input, and enhances the representation of frame-level features to video features. By solving the problem that neural networks are difficult to process videos of different lengths and the representation difficulty caused by the lack of time sequence of videos, the representation features of any fragmented videos can be extracted, thereby realizing the retrieval and recall of fragmented videos.

[0137] like Figure 8 As shown, the embodiment of the present application combines the variable-length feature learning module (i.e., encoding model) and the fixed-length feature learning module (i.e., mapping model) in the deep neural network (DNN), and connects the two modules with the help of statistical pooling method, and finally outputs a fixed-length video embedding under the fixed-length feature learning module, that is, the variable-length feature learning module DNN1 supports feature sequence inputs at the video frame level of different lengths. After extracting the frame sequence features (Frame feat) of the specified time window, multiple frame sequence features are statistically pooled to generate fixed-length features (feat), and then pass through the fixed-length feature learning module DNN2 to finally obtain the embedding that represents the complete video. The statistical-based video embedding characterization method under variable-length input provided by the embodiment of the present application will be specifically described below:

[0138] First, the image feature extraction of the embodiment of the present application adopts the feature extraction module of ResNet101 pre-trained by ImageNet and the output of the pooling layer as the image embedding representation (image embedding). The network structure of ResNet101 is shown in Table 1.

[0139] Table 1 Structure of ResNet101

[0140]

[0141] Secondly, the neural network of the embodiment of the present application is shown in Table 2, including a variable-length feature learning module (i.e., conv1-conv4) and a fixed-length feature learning module (i.e., Fc1-Fc2). In order to enhance the video embedding learning effect, a classification layer as shown in Table 3 is also provided in the training for classifying the video. The video embedding learned by Fc2 is input into the classification layer Fc3 in Table 3 to learn the video classification task, so that the video embedding has a stronger ability to represent the video and is sufficient to support the classification task.

[0142] Table 2 Embedding extraction network

[0143]

[0144] Table 3 Embedding classification layer (Nc is the number of video categories)

[0145]

[0146] like Figure 8 As shown, the relevant model calculation process is described in detail below:

[0147] 1) Model variable length input

[0148] Different from general video convolution learning, since the embodiment of the present application needs to process variable-length video input, it is necessary to first perform the following processing on the variable-length video:

[0149] For a video Vi to be learned, firstly, video frames are extracted every Fl length to obtain a sub-video frame sequence (the method of extracting sub-videos can be overlapping one frame, non-overlapping, or overlapping multiple frames between adjacent sub-videos). Assuming that the video Vi is divided into Kvi sub-video sequences in total, the image feature sequences corresponding to these sub-video sequences are input into the model, that is, the fixed-length feature learning module DNN2.

[0150] 2) Model processing of input

[0151] As shown in Table 2, the input of the first module conv1 of the model is Fl consecutive image frame embeddings (i.e., sub-video sequences), where Fl is the frame length (frame length) processed by the preset DNN1 network each time. For example, if 3 frames are processed each time, the input is 3*2048 (1*2048 is the embedding length output by ResNet101). After learning 512 (the number of channels) convolution kernels of conv1, conv1 finally learns the 1*512 vector representation of 3 consecutive image frames. For Kvi sub-video image feature sequences, each sub-video image feature sequence generates a 1*512 vector feature after conv1, i.e., a total of Kvi 1*512 vector features are generated.

[0152] 3) Model Statistics Pooling

[0153] The pooling layer of the embodiment of the present application is different from the general mean or maximum pooling. The embodiment of the present application uses a statistical method to perform feature aggregation.

[0154] First, conv4 outputs 1*512 feature vectors (i.e., channel features) of Kvi sub-video sequences, that is, there are Kvi feature values ​​in each of the 512 channels that the model needs to learn. In statistical pooling, for each channel, the mean and variance of the Kvi feature values ​​are obtained, and finally 512 means mean_vi and 512 variances std_vi are obtained, which represent the mean features and variance features of the video Vi in all 512 channels; then, all mean features and variance features [mean_vi,std_vi] are concatenated to obtain 1*1024-dimensional fixed-length features.

[0155] At this point, the Kvi variable-length sub-video sequences are output through the conv4 and pooling layers to generate a fixed-length 1*1024-dimensional video overall element description vector (i.e., fixed-length feature). This fixed-length feature depends only on the number of conv4 channels (512) and has nothing to do with the length of the input video.

[0156] 4) Fixed-length feature learning

[0157] For the 1*1024 dimensional fixed-length features generated above, after learning by a fixed-length feature learning module composed of two fully connected layers, an embedding representation of the video Vi is finally generated, which is subsequently optimized with the help of metric learning loss.

[0158] The following is a detailed description of the learning process of the relevant model:

[0159] 1) Parameter initialization: The parameters in Table 1 and Table 2 are initialized using a Gaussian distribution with a variance of 0.01 and a mean of 0.

[0160] 2) Data preparation

[0161] (a) Prepare video similarity samples: video pairs (i.e., sample pairs that mark whether two videos are the same video or similar videos).

[0162] (b) Prepare video category labels

[0163] For the above video pairs, randomly select one video to mark the video classification label (such as educational video, artistic creation video, calligraphy, knowledge animation, sports competition, news, selfie talent, product promotion, natural environment, etc.). If there is no specially specified category, the category of the youtube-8M large-scale open source classification video dataset can be used as the classification category.

[0164] (c) Data sampling during learning

[0165] Similarity triplet mining: The embodiment of the present application adopts triplet learning, takes similar sample pairs as input, and performs the following mining on the sample pairs in each batch (the number of sample pairs in each batch is bs) to obtain triplets: for a certain sample x, randomly select a sample i from the remaining bs-1 sample pairs, and calculate the distance between sample i and sample x, sort them from small to large in order of distance, take the first 10 samples as negative samples, and form triplets with the positive sample of sample x respectively, so each sample x generates 10 triplets, that is, the entire batch obtains 10*bs triplets.

[0166] The above triplet data is used for both metric learning (i.e., triplet learning) and classification learning.

[0167] 3) Overall training process: For all sample pairs (H sample pairs in total), each bs sample pairs is a batch, and there are H / bs batches in total. For each batch, the model is forward calculated and the parameters are updated. After completing H / bs updates, a training cycle (epoch) is completed, and a total of Z epochs are trained (or when the average loss of 10 consecutive epochs does not decrease).

[0168] 4) Model forward: All model parameters are set to the state that needs to be learned. During training, the neural network performs forward calculations on the input video samples to obtain dual-branch prediction results (i.e., embedding vector prediction results em and classification prediction results fc). The classification prediction results (output of Fc3) are compared with the real category labels of the video to calculate the classification loss of the model. The triplet loss (triplet loss) is calculated based on the embedding representation of the video samples. The above two losses are combined to obtain the total loss.

[0169] The following describes the calculation process of the loss function:

[0170] a) Classification loss

[0171] The cross entropy loss function of sampling multi-classification is shown in formula (1):

[0172] (1)

[0173] in, represents the predicted probability that sample i belongs to category c, Indicates whether the label of sample i is c. If it is c, then =1, otherwise =0.

[0174] b) Triplet loss

[0175] After finding the triplet (a, p, n) in the batch samples, the embedding representation of these triplet samples is used to calculate the triplet loss. The calculation of triplet loss is shown in formula (2):

[0176] (2)

[0177] in, Indicates margin, set to 0.6, Represents the L2 distance between the embedding of sample a and the embedding of sample p, It represents the L2 distance between the embedding of sample a and the embedding of sample n. The purpose of triplet loss is to make the distance between the anchor sample and the negative sample greater than the distance between the anchor sample and the positive sample.

[0178] c) Total loss

[0179] Total loss The calculation formula is shown in formula (3):

[0180] (3)

[0181] Among them, w is the weight, which can be 1 or adjusted according to experience.

[0182] 5) Model parameter update process: Use the total loss to perform gradient backward calculation to obtain the updated values ​​of all model parameters and update the network. In order to prevent the Fc3 classification effect from affecting the basic features too quickly, a learning rate of 0.05 is used for Fc3 and a learning rate of 0.005 (10 times smaller than Fc3) is used for other network features. After the gradient is obtained by the total loss in each round of learning, the network weights are updated according to different learning rates.

[0183] In summary, the embodiments of the present application have the following beneficial effects:

[0184] 1) Generate the overall element description vector of the video through statistical analysis of variable-length video subsequences, and innovatively implement a deep learning solution for variable-length video learning embedding;

[0185] 2) Jointly learn video embedding representation through classification and metric learning to achieve video embedding representation without time sequence;

[0186] 3) Based on video embedding, time-sequential and non-time-sequential video similarity retrieval can be performed, such as supporting non-time-sequential similarity retrieval of fragmented videos.

[0187] So far, the video processing method based on artificial intelligence provided by the embodiment of the present application has been described in combination with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. The embodiment of the present application also provides a video processing device based on artificial intelligence. In practical applications, each functional module in the video processing device based on artificial intelligence can be implemented by hardware resources of electronic devices (such as terminals, servers or server clusters), such as computing resources such as processors, communication resources (such as for supporting various communication methods such as optical cables and cellular), and memory. Figure 2 An artificial intelligence-based video processing device 555 stored in a memory 550 is shown, which may be software in the form of programs and plug-ins, for example, software modules designed in programming languages ​​such as C / C++ and Java, application software designed in programming languages ​​such as C / C++ and Java, or dedicated software modules in large software systems, application programming interfaces, plug-ins, cloud services, and other implementation methods. Different implementation methods are illustrated below.

[0188] The video processing device 555 based on artificial intelligence includes a series of modules, including a feature extraction module 5551, an encoding module 5552, an aggregation module 5553, a mapping module 5554, and an application module 5555. The following further describes how the various modules in the video processing device 555 based on artificial intelligence provided in the embodiment of the present application cooperate to implement the video processing solution.

[0189] A feature extraction module 5551 is configured to perform feature extraction processing on multiple sub-video sequences of a video respectively to obtain a feature sequence of each of the sub-video sequences; an encoding module 5552 is configured to perform encoding processing based on multiple channels on the feature sequence of each of the sub-video sequences to obtain channel features corresponding to the multiple channels of each of the sub-video sequences; an aggregation module 5553 is configured to perform feature aggregation processing based on the multiple channels on the channel features respectively corresponding to the multiple sub-video sequences to obtain a fixed-length feature whose dimension is associated with the number of the multiple channels; a mapping module 5554 is configured to perform mapping processing on the fixed-length feature whose dimension is associated with the number of the multiple channels to obtain an embedding vector of the video; an application module 5555 is configured to perform video application processing based on the embedding vector of the video to obtain an application result of the video.

[0190] In some embodiments, the encoding processing based on multiple channels is implemented by an encoding model, and the encoding model includes a plurality of cascaded convolutional layers; the encoding module 5552 is further configured to perform convolutional processing based on multiple channels on the feature sequence of each of the sub-video sequences through a first convolutional layer among the plurality of cascaded convolutional layers; output the convolutional result of the first convolutional layer to subsequent cascaded convolutional layers, and continue to perform convolutional processing and output of convolutional results through the subsequent cascaded convolutional layers; and use the convolutional result output by the last convolutional layer as the channel features corresponding to the multiple channels of each of the sub-video sequences.

[0191] In some embodiments, the encoding module 5552 is further configured to perform convolutional processing on the convolutional result output by the (j - 1)-th convolutional layer respectively through the multiple channels in the j-th convolutional layer among the plurality of cascaded convolutional layers to obtain convolutional results respectively corresponding to the multiple channels; perform splicing processing on the convolutional results respectively corresponding to the multiple channels to obtain the convolutional result of the j-th convolutional layer; and output the convolutional result of the j-th convolutional layer through the j-th convolutional layer; where j is an increasing natural number and the value range is 1 < j ≤ N, and N is the number of convolutional layers.

[0192] In some embodiments, the aggregation module 5553 is further configured to perform the following processing for any one of the multiple channels: obtain sub-channel features corresponding to the channel of each of the sub-video sequences from the channel features corresponding to the multiple channels of each of the sub-video sequences; perform pooling processing based on statistics on the sub-channel features respectively corresponding to the multiple sub-video sequences for the channel to obtain statistical features of the channel; and perform splicing processing on the statistical features respectively corresponding to the multiple channels to obtain a fixed-length feature whose dimension is associated with the number of the multiple channels.

[0193] In some embodiments, the aggregation module 5553 is also used to perform mean processing based on the sub-channel features of the channels corresponding to the multiple sub-video sequences respectively, to obtain mean features of the channels corresponding to the multiple sub-video sequences; perform variance processing based on the sub-channel features of the channels corresponding to the multiple sub-video sequences respectively, to obtain variance features of the channels corresponding to the multiple sub-video sequences; and perform splicing processing on the mean features of the channels corresponding to the multiple sub-video sequences and the variance features of the channels corresponding to the multiple sub-video sequences to obtain statistical features of the channels.

[0194] In some embodiments, the mapping processing 5554 is implemented through a mapping model, which includes multiple cascaded fully connected layers; the mapping module is also used to perform mapping processing on fixed-length features whose dimensions are associated with the number of multiple channels through the first fully connected layer among the multiple cascaded fully connected layers; output the mapping result of the first fully connected layer to the subsequent cascaded fully connected layers, and continue the mapping processing and mapping result output through the subsequent cascaded fully connected layers; use the mapping result output by the last fully connected layer as the embedding vector of the video.

[0195] In some embodiments, the feature extraction processing is implemented through a feature extraction model, which includes a pooling layer and multiple cascaded convolution layers, and each of the sub-video sequences includes multiple image frames; the feature extraction module 5551 is also used to perform the following processing on any of the image frames in each of the sub-video sequences: performing cascaded convolution processing on the image frames through the multiple cascaded convolution layers to obtain the convolution features of the image frames; performing pooling processing on the convolution features of the image frames through the pooling layer to obtain the image features of the image frames; and performing splicing processing on the image features corresponding to the multiple image frames to obtain a feature sequence of each sub-video sequence.

[0196] In some embodiments, the application module 5555 is also used to perform similarity recall processing on candidate similar videos based on the embedded vector of the video to obtain similar videos of the video; or, perform video classification processing based on the embedded vector of the video to obtain the video type to which the video belongs.

[0197] In some embodiments, the video processing method is implemented through a video processing model; the device also includes: a training module 5556, which is used to map the fixed-length features of the video sample through the initialized video processing model to obtain the embedding vector of the video sample; based on the embedding vector of the video sample, construct a target loss function of the video processing model; based on the target loss function, update the parameters of the video processing model, and use the updated parameters of the video processing model as the parameters of the trained video processing model.

[0198] In some embodiments, the training module 5556 is also used to perform video classification processing based on the embedding vector of the video sample to obtain the predicted type of the video sample; construct a classification loss function of the video processing model based on the predicted type of the video sample and the type label of the video sample; construct an embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vector of similar video samples to the video sample, and the embedding vector of dissimilar video samples to the video sample; perform weighted summation processing on the classification loss function of the video processing model and the embedding loss function of the video processing model to obtain the target loss function of the video processing model.

[0199] In some embodiments, the training module 5556 is also used to perform similarity processing on the embedding vector of the video sample and the embedding vector of similar video samples to the video sample to obtain a first similarity between the video sample and the similar video samples; perform similarity processing on the embedding vector of the video sample and the embedding vector of dissimilar video samples to the video sample to obtain a second similarity between the video sample and the dissimilar video samples; and construct an embedding loss function of the video processing model based on the first similarity and the second similarity.

[0200] In some embodiments, the training module 5556 is also used to extract at least one candidate video sample from multiple candidate similar video sample pairs, wherein the multiple candidate similar video sample pairs do not include the video sample; based on the similarity between the candidate video sample and the video sample, sort the candidate video samples in descending order; and sort the part of the candidate video samples that are sorted at the end of the descending sorting result in descending order as dissimilar video samples of the video sample.

[0201] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs the above-mentioned artificial intelligence-based video processing method of the embodiment of the present application.

[0202] The present application embodiment provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the artificial intelligence-based video processing method provided by the present application embodiment, for example, Figure 3-Figure 5 The video processing method based on artificial intelligence is shown.

[0203] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0204] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0205] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file storing other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0206] As an example, the executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0207] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A video processing method based on artificial intelligence, characterized in that: The method includes: Performing feature extraction processing on multiple sub-video sequences of the video respectively to obtain the feature sequence of each sub-video sequence; Performing encoding processing based on multiple channels on the feature sequence of each sub-video sequence to obtain the channel features corresponding to the multiple channels of each sub-video sequence; Performing the following processing on any one of the multiple channels: obtaining the sub-channel features corresponding to the channel of each sub-video sequence from the channel features corresponding to the multiple channels of each sub-video sequence; performing statistical pooling processing on the sub-channel features corresponding to the channel of the multiple sub-video sequences respectively to obtain the statistical features of the channel; Performing splicing processing on the statistical features corresponding to the multiple channels respectively to obtain a fixed-length feature whose dimension is associated with the number of the multiple channels; Performing mapping processing on the fixed-length feature whose dimension is associated with the number of the multiple channels to obtain the embedding vector of the video; Performing video application processing based on the embedding vector of the video to obtain the application result of the video.

2. The method according to claim 1, wherein The encoding processing based on multiple channels is implemented by an encoding model, and the encoding model includes multiple cascaded convolutional layers; The performing encoding processing based on multiple channels on the feature sequence of each sub-video sequence to obtain the channel features corresponding to the multiple channels of each sub-video sequence includes: Performing convolutional processing based on multiple channels on the feature sequence of each sub-video sequence through the first convolutional layer in the multiple cascaded convolutional layers; Outputting the convolutional result of the first convolutional layer to the subsequent cascaded convolutional layers, and continuing to perform convolutional processing and output of the convolutional result through the subsequent cascaded convolutional layers; Taking the convolutional result output by the last convolutional layer as the channel features corresponding to the multiple channels of each sub-video sequence.

3. The method according to claim 2, characterized in that The continuing to perform convolutional processing and output of the convolutional result through the subsequent cascaded convolutional layers includes: Performing convolutional processing on the convolutional result output by the (j - 1)-th convolutional layer respectively through the multiple channels in the j-th convolutional layer in the multiple cascaded convolutional layers to obtain the convolutional results corresponding to the multiple channels respectively; Performing splicing processing on the convolutional results corresponding to the multiple channels respectively to obtain the convolutional result of the j-th convolutional layer; Outputting the convolutional result of the j-th convolutional layer through the j-th convolutional layer; where j is an increasing natural number and the value range is 1 < j ≤ N, and N is the number of convolutional layers.

4. The method according to claim 1, characterized in that: The performing statistical pooling processing on the sub-channel features corresponding to the channel of the multiple sub-video sequences respectively to obtain the statistical features of the channel includes: Performing mean processing based on the sub-channel features corresponding to the channel of the multiple sub-video sequences respectively to obtain the mean features corresponding to the channel of the multiple sub-video sequences; Performing variance processing based on the sub-channel features corresponding to the channel of the multiple sub-video sequences respectively to obtain the variance features corresponding to the channel of the multiple sub-video sequences; The mean features of the channels corresponding to the multiple sub-video sequences and the variance features of the channels corresponding to the multiple sub-video sequences are concatenated to obtain statistical features of the channels.

5. The method according to claim 1, characterized in that The mapping process is implemented by a mapping model, and the mapping model includes a plurality of cascaded fully connected layers; The mapping process is performed on the fixed-length features whose dimensions are associated with the number of the multiple channels to obtain the embedding vector of the video, including: Mapping the fixed-length features whose dimensions are associated with the number of the multiple channels through a first fully-connected layer of the multiple cascaded fully-connected layers; Outputting the mapping result of the first fully connected layer to the subsequent cascaded fully connected layer, and continuing the mapping process and outputting the mapping result through the subsequent cascaded fully connected layer; The mapping result output by the last fully connected layer is used as the embedding vector of the video.

6. The method according to claim 1, characterized in that The feature extraction process is implemented by a feature extraction model, the feature extraction model includes a pooling layer and a plurality of cascaded convolutional layers, and each of the sub-video sequences includes a plurality of image frames; The step of performing feature extraction processing on a plurality of sub-video sequences of a video to obtain a feature sequence of each of the sub-video sequences comprises: The following processing is performed for any image frame in each of the sub-video sequences: Performing cascade convolution processing on the image frame through the multiple cascaded convolution layers to obtain convolution features of the image frame; Performing pooling processing on the convolution features of the image frame through the pooling layer to obtain image features of the image frame; The image features respectively corresponding to the multiple image frames are spliced ​​to obtain a feature sequence of each sub-video sequence.

7. The method according to claim 1, characterized in that The performing video application processing based on the embedding vector of the video to obtain the application result of the video includes: Performing similarity recall processing on candidate similar videos based on the embedding vector of the video to obtain similar videos of the video; or, Video classification processing is performed based on the embedding vector of the video to obtain the video type to which the video belongs.

8. The method according to any one of claims 1 to 7, characterized in that: The video processing method is implemented by a video processing model; the training process of the video processing model includes: Mapping the fixed-length features of the video sample using the initialized video processing model to obtain an embedding vector of the video sample; Constructing a target loss function of the video processing model based on the embedding vector of the video sample; The parameters of the video processing model are updated based on the target loss function, and the updated parameters of the video processing model are used as the parameters of the trained video processing model.

9. The method according to claim 8, characterized in that The constructing the target loss function of the video processing model based on the embedding vector of the video sample includes: Performing video classification processing based on the embedding vector of the video sample to obtain a predicted type of the video sample; Constructing a classification loss function of the video processing model based on the predicted type of the video sample and the type label of the video sample; Constructing an embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vectors of similar video samples of the video sample, and the embedding vectors of dissimilar video samples of the video sample; A weighted summation process is performed on the classification loss function of the video processing model and the embedding loss function of the video processing model to obtain a target loss function of the video processing model.

10. The method according to claim 9, characterized in that The constructing the embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vector of similar video samples of the video sample, and the embedding vector of dissimilar video samples of the video sample comprises: Performing similarity processing on the embedding vector of the video sample and the embedding vector of a video sample similar to the video sample to obtain a first similarity between the video sample and the similar video sample; Performing similarity processing on the embedding vector of the video sample and the embedding vector of a video sample that is not similar to the video sample to obtain a second similarity between the video sample and the dissimilar video sample; Based on the first similarity and the second similarity, an embedding loss function of the video processing model is constructed.

11. The method according to claim 9, characterized in that Before constructing the embedding loss function of the video processing model based on the embedding vector of the video sample, the embedding vector of similar video samples of the video sample, and the embedding vector of dissimilar video samples of the video sample, the method further includes: Extracting at least one candidate video sample from a plurality of candidate similar video sample pairs, wherein the plurality of candidate similar video sample pairs do not include the video sample; Based on the similarity between the candidate video samples and the video samples, sorting the candidate video samples in descending order; The candidate video samples that are sorted later in the descending sorting result are sorted in descending order as dissimilar video samples of the video samples.

12. A video processing device based on artificial intelligence, characterized in that: The device comprises: A feature extraction module is used to perform feature extraction processing on multiple sub-video sequences of a video respectively to obtain a feature sequence of each sub-video sequence; An encoding module, used for performing encoding processing on the feature sequence of each sub-video sequence based on multiple channels to obtain channel features of each sub-video sequence corresponding to the multiple channels; an aggregation module, configured to perform the following processing on any of the multiple channels: obtaining a sub-channel feature of each sub-video sequence corresponding to the channel from the channel features of each sub-video sequence corresponding to the multiple channels; performing statistics-based pooling processing on the sub-channel features of the multiple sub-video sequences corresponding to the channels, respectively, to obtain statistical features of the channel; The aggregation module is further used to perform splicing processing on the statistical features corresponding to the multiple channels respectively to obtain a fixed-length feature whose dimension is associated with the number of the multiple channels; A mapping module, configured to perform mapping processing on the fixed-length features whose dimensions are associated with the number of the plurality of channels to obtain an embedding vector of the video; The application module is used to perform video application processing based on the embedding vector of the video to obtain the application result of the video.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing executable instructions; A processor, used to implement the artificial intelligence-based video processing method described in any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the artificial intelligence-based video processing method described in any one of claims 1 to 11 when executed by a processor.

15. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the artificial intelligence-based video processing method according to any one of claims 1 to 11 is implemented.

Images