Video processing methods, devices, equipment, and storage media

Abstract

This application provides a video processing method, apparatus, device, storage medium, and computer program product, which can be applied to various scenarios such as cloud technology, artificial intelligence, intelligent transportation, and assisted driving. The method includes: dividing each image frame in the video to be processed into a first number of first image blocks; for a target image frame in the video to be processed, obtaining the distance between a reference image frame and the target image frame, and dividing the reference image frame into a second number of second image blocks based on the distance, the second number being negatively correlated with the distance; for each first image block in the target image frame, using the target image frame and the second image blocks included in the reference image frame as references, extracting features from the first image blocks to obtain image block features of the first image blocks; and aggregating the image block features of the first image blocks included in each image frame to obtain video features of the video. This application can improve the efficiency of video feature extraction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a video processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] Video processing is also an important application area of ​​artificial intelligence. In related technologies, video feature extraction is usually achieved using 3D convolution, but 3D convolution often involves a large amount of computation, resulting in low efficiency in video feature extraction. Summary of the Invention

[0004] This application provides a video processing method, apparatus, device, storage medium, and computer program product that can ensure the accuracy of video feature extraction and improve the efficiency of video feature extraction.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a video processing method, including:

[0007] Each image frame in the video to be processed is divided into a first number of first image blocks;

[0008] For a target image frame in the video to be processed, the distance between a reference image frame and the target image frame is obtained, and based on the distance, the reference image frame is divided into a second number of second image blocks;

[0009] The reference image frame refers to the image frames in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance.

[0010] For each first image block in the target image frame, with reference to the target image frame and the second image blocks included in the reference image frame, feature extraction is performed on the first image block to obtain the image block features of the first image block;

[0011] Feature aggregation is performed on the image block features of the first image block included in each of the image frames to obtain the video features of the video to be processed.

[0012] This application also provides a video processing apparatus, including:

[0013] The first partitioning module is used to divide each image frame in the video to be processed into a first number of first image blocks;

[0014] The second segmentation module is used to obtain the distance between the reference image frame and the target image frame in the video to be processed, and to divide the reference image frame into a second number of second image blocks based on the distance.

[0015] The reference image frame refers to the image frames in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance.

[0016] The feature extraction module is used to extract features from each first image block in the target image frame, taking the target image frame and the second image blocks included in the reference image frame as references, to obtain the image block features of the first image block.

[0017] The feature aggregation module is used to perform feature aggregation on the image block features of the first image block included in each of the image frames to obtain the video features of the video to be processed.

[0018] In the above scheme, the second division module is further used to obtain the number of image frames between the reference image frame and the target image frame, and use the number of frames as the distance between the reference image frame and the target image frame.

[0019] In the above scheme, the second division module is further used to obtain the playback time point of the reference image frame and the time interval between the playback time point of the target image frame, and use the time interval as the distance between the reference image frame and the target image frame.

[0020] In the above scheme, the second division module is further used to obtain the mapping relationship between distance and division number, wherein the division number is used for the division of the reference image frame; based on the distance and the mapping relationship, the target division number corresponding to the distance is determined, and the target division number is used as the second number.

[0021] In the above scheme, the second division module is further configured to obtain multiple distance intervals corresponding to the reference image frame, each distance interval being associated with a corresponding division number, the division number being used for dividing the reference image frame; from the multiple distance intervals, to determine the target distance interval in which the distance is located, and to use the division number corresponding to the target distance interval as the second number.

[0022] In the above scheme, the feature extraction module is further configured to obtain the block vector of the first image block included in the target image frame, and obtain the block vector of the second image block included in the reference image frame; determine the vector similarity between each block vector and the block vector of the first image block respectively; perform a first aggregation process on the multiple block vectors based on the vector similarity corresponding to each block vector to obtain the aggregated feature of the first image block; and perform a second aggregation process on the block vector of the first image block and the aggregated feature to obtain the image block feature of the first image block.

[0023] In the above scheme, the feature extraction module is further used to normalize the vector similarity corresponding to each block vector and use the result as the vector weight of the corresponding block vector; based on the vector weight of each block vector, the multiple block vectors are weighted and summed to obtain the aggregated features of the first image block.

[0024] In the above scheme, the block vector includes a query vector, a key vector, and a value vector; the feature extraction module is further configured to determine the similarity between the key vector of each image block and the query vector of the first image block, and use the similarity as the vector similarity; the feature extraction module is further configured to perform a first aggregation process on multiple value vectors based on the vector similarity corresponding to each block vector to obtain the aggregated features of the first image block.

[0025] In the above scheme, the block vector includes a query vector, a key vector, and a value vector; the feature extraction module is further configured to perform vector transformation on the second image block included in the reference image frame to obtain the image block vector of the second image block; perform linear projection processing on the image block vector to obtain a first intermediate query vector, a first intermediate key vector, and a first intermediate value vector of the second image block; perform a first pooling process on the first intermediate query vector to obtain the query vector; perform a second pooling process on the first intermediate key vector, and process the result obtained by the second pooling process based on the distance to obtain the key vector; perform a third pooling process on the first intermediate value vector, and process the result obtained by the third pooling process based on the distance to obtain the value vector.

[0026] In the above scheme, the feature extraction module is further configured to perform a first feature extraction process on the result obtained by the second pooling process to obtain the key vector when the distance reaches a distance threshold; and to perform a second feature extraction process on the result obtained by the second pooling process to obtain the key vector when the distance does not reach the distance threshold; wherein the density of the features obtained by the first feature extraction process is lower than the density of the features obtained by the second feature extraction process.

[0027] In the above scheme, the feature extraction module is further configured to perform feature extraction on the first image block at least twice, with reference to the target image frame and the second image block included in the reference image frame, to obtain intermediate image block features obtained from each feature extraction; and to concatenate the intermediate image block features obtained from each feature extraction to obtain the image block features of the first image block.

[0028] In the above scheme, the feature aggregation module is further configured to, for each image frame, stitch together the image block features of at least two adjacent first image blocks in the image frame to obtain multiple intermediate image block features corresponding to the image frame; for each intermediate image block feature, with reference to the multiple intermediate image block features and the features of the second image block included in the reference image frame, perform feature extraction on the intermediate image block feature to obtain the stitched image block features corresponding to the image frame; and perform feature aggregation on the stitched image block features corresponding to each image frame to obtain the video features of the video to be processed.

[0029] This application also provides an electronic device, including:

[0030] Memory, used to store executable instructions;

[0031] The processor, when executing executable instructions stored in the memory, implements the video processing method provided in the embodiments of this application.

[0032] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the video processing method provided in this application.

[0033] This application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement the video processing method provided in this application.

[0034] The embodiments of this application have the following beneficial effects:

[0035] Each image frame in the video to be processed is divided into a first number of first image blocks. For the target image frame in the video to be processed, the distance between the reference image frame and the target image frame is obtained, and the reference image frame is divided into a second number of second image blocks based on the distance. 1) In this way, for each first image block in the target image frame, the first image block can be extracted with reference to the target image frame and the second image blocks included in the reference image frame to obtain the image block features of the first image block. The image block features of the first image blocks included in each image frame are then aggregated to obtain the video features of the video to be processed, ensuring the accuracy of video feature extraction. 2) The reference image frame is the image frame in the video to be processed other than the target image frame. The second number is negatively correlated with the distance between the reference image frame and the target image frame. That is, as the distance between the reference image frame and the target image frame increases, the number of image blocks included in the reference image frame will decrease. In this way, when the image blocks included in the reference image frame are used as a reference for feature extraction of the image blocks of the target image frame, the amount of computation in the feature extraction process is reduced, and the efficiency of video feature extraction is improved. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the architecture of the video processing system 100 provided in an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the structure of an electronic device 500 that implements the video processing method provided in this application embodiment;

[0038] Figure 3 This is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0039] Figure 4 This is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0040] Figure 5 This is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0041] Figure 6 This is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0042] Figure 7 This is a schematic flowchart of the video processing method provided in the embodiments of this application;

[0043] Figure 8 This is a schematic diagram of the feature extraction model provided in the embodiments of this application;

[0044] Figure 9 This is a schematic diagram of the structure of the second feature extraction layer provided in an embodiment of this application;

[0045] Figure 10 This is a schematic diagram of the experimental data provided in the embodiments of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0047] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0048] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0049] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0050] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0051] 1) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0052] 2) Pooling is an important concept in convolutional neural networks (CNNs), and it is essentially a form of downsampling. There are various non-linear pooling functions, with max pooling being the most common. It divides the input image into several rectangular regions and outputs the maximum value for each sub-region. Pooling layers continuously reduce the spatial size of the data, thus decreasing the number of parameters and computational cost, which to some extent controls overfitting. Typically, pooling layers are periodically inserted between convolutional layers in a CNN.

[0053] 3) Convolutional layer: Composed of several convolutional units, the parameters of each convolutional unit are optimized through backpropagation. The purpose of convolution is to extract different features from the input. The first convolutional layer may only extract some low-level features such as edges, lines, and corners, while more layers of the network can iteratively extract more complex features from low-level features.

[0054] 4) The normalization exponential function, or softmax function, is a generalization of the logistic function. It can "compress" a K-dimensional vector z containing arbitrary real numbers into another K-dimensional real vector σ(z), such that each element is in the range (0, 1), and the sum of all elements is 1. This function is frequently used in multi-class classification problems.

[0055] This application provides a video processing method, apparatus, device, storage medium, and computer program product that can ensure the accuracy of video feature extraction and improve the efficiency of video feature extraction.

[0056] The following describes the implementation scenarios of the video processing method provided in the embodiments of this application. See also... Figure 1 , Figure 1 This is a schematic diagram of the architecture of the video processing system 100 provided in the embodiments of this application. In order to support an exemplary application, the terminal 400 connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0057] Terminal 400 is used to send a feature extraction request for the video to server 200 in response to a feature extraction instruction for the video.

[0058] Server 200 is configured to receive feature extraction requests for videos; in response to the feature extraction requests, divide each image frame in the video to be processed into a first number of first image blocks; for a target image frame in the video to be processed, obtain the distance between a reference image frame and the target image frame, and divide the reference image frame into a second number of second image blocks based on the distance; for each first image block in the target image frame, perform feature extraction on the first image block with reference to the target image frame and the second image blocks included in the reference image frame to obtain image block features of the first image block; perform feature aggregation on the image block features of the first image blocks included in each image frame to obtain video features of the video to be processed; wherein, the reference image frame is an image frame in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance;

[0059] In some embodiments, the extracted video features can be applied to video tasks, such as video classification tasks and video action recognition tasks.

[0060] In some embodiments, the video processing method provided in this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. For instance, the terminal executes the video processing method provided in this application independently, or the terminal sends a feature extraction request for the video to the server, and the server executes the video processing method provided in this application based on the received feature extraction request. This application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0061] In some embodiments, the electronic device for processing video provided in this application can be various types of terminal devices or servers. The server (e.g., server 200) can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal (e.g., terminal 400) can be a smartphone, tablet computer, laptop computer, desktop computer, smart voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, in-vehicle terminal, etc., but is not limited thereto. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0062] In some embodiments, the video processing method provided in this application can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies hardware, software, network, and other resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form resource pools, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technology network system require a large amount of computing and storage resources. As an example, a server (e.g., server 200) can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0063] In some embodiments, the terminal or server can implement the video processing method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run; it can also be a mini-program, that is, a program that only needs to be downloaded to the browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module or plugin.

[0064] In some embodiments, multiple servers can form a blockchain, with each server being a node on the blockchain. Information connections can exist between each node in the blockchain, allowing for information transmission between them. Furthermore, data related to the video processing method provided in this application (e.g., video footage, extracted video features) can be stored on the blockchain.

[0065] The following describes an electronic device that implements a video processing method according to embodiments of this application. See also: Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 implementing the video processing method provided in this application embodiment. The electronic device 500 is used as an example. Figure 1 Taking the server shown as an example, the electronic device 500 implementing the video processing method provided in this application embodiment includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0066] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0067] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may optionally include one or more storage devices physically located remote from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0068] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0069] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0070] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0071] In some embodiments, the video processing apparatus provided in this application can be implemented in software. Figure 2 A video processing device 553 stored in memory 550 is shown. It may be software in the form of programs and plug-ins, including the following software modules: a first partitioning module 5531, a second partitioning module 5532, a feature extraction module 5533, and a feature aggregation module 5534. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0072] The following describes the video processing method provided in the embodiments of this application. In some embodiments, the video processing method provided in the embodiments of this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. Taking server implementation as an example, see [link to documentation]. Figure 3 , Figure 3 This is a flowchart illustrating a video processing method provided in an embodiment of this application. The video processing method provided in this embodiment includes:

[0073] Step 101: The server divides each image frame in the video to be processed into a first number of first image blocks.

[0074] In step 101, the server first acquires the video to be processed, which includes multiple image frames (or video frames). Then, for each image frame, the following processing is performed: the image frame is divided to obtain the first image blocks comprising that image frame. In actual implementation, a first number of first image blocks obtained after dividing the image frame can be preset, and then the image frame is divided according to the preset first number to divide it into a first number of first image blocks, where the first number is greater than or equal to 1. In this way, the purpose of dividing each image frame in the video to be processed into a first number of first image blocks is achieved.

[0075] Step 102: For the target image frame in the video to be processed, obtain the distance between the reference image frame and the target image frame, and based on the distance, divide the reference image frame into a second number of second image blocks.

[0076] The reference image frame refers to the image frames in the video to be processed, excluding the target image frame, and the second quantity is negatively correlated with the distance.

[0077] When performing feature extraction on the video to be processed, features can be extracted from each image frame in the video separately. In step 102, the target image frame is the image frame currently undergoing feature extraction processing. For the target image frame, a reference image frame corresponding to the target image frame is first determined. In practice, the reference image frame is any image frame in the video to be processed other than the target image frame, and there is at least one reference image frame. Then, for each reference image frame, the distance between the reference image frame and the target image frame is obtained. Based on this distance, the number of segments to be divided for the reference image frame is determined as a second number, and the reference image frame is then divided into the second number of second image blocks. In practical applications, this second number is negatively correlated with the distance; that is, as the distance increases, the second number generally decreases. It should be noted that the second number and the distance can be linearly negatively correlated or non-linearly negatively correlated.

[0078] In some embodiments, the server may obtain the distance between the reference image frame and the target image frame by: obtaining the number of image frames between the reference image frame and the target image frame, and using the number of frames as the distance between the reference image frame and the target image frame; or, obtaining the playback time point of the reference image frame and the time interval between the playback time point of the target image frame, and using the time interval as the distance between the reference image frame and the target image frame.

[0079] In practical applications, the distance between the reference image frame and the target image frame can be represented by the number of image frames between them. Therefore, when obtaining the distance between the reference image frame and the target image frame, the number of image frames between them can be obtained, and this number can be used as the distance between the reference image frame and the target image frame.

[0080] In practical applications, the distance between the reference image frame and the target image frame can also be characterized by the playback time of the reference image frame and the time interval between the playback time of the target image frame. Therefore, when obtaining the distance between the reference image frame and the target image frame, the playback time of the reference image frame and the time interval between the playback time of the target image frame can be obtained, and the time interval can be used as the distance between the reference image frame and the target image frame.

[0081] In some embodiments, the server may determine a second number of partitions for a reference image frame by: obtaining a mapping relationship between distance and partition number, wherein the partition number is used for partitioning the reference image frame; determining a target partition number corresponding to the distance based on the distance and the mapping relationship, and using the target partition number as the second number.

[0082] In practical applications, a mapping relationship between distance and number of divisions can be pre-set. This mapping relationship can be represented by a mapping formula, such as number of divisions = k * distance + initial number (k < 0). Then, based on the distance between the reference image frame and the target image frame and the mapping relationship, the target number of divisions corresponding to the distance is determined. This target number of divisions is then used as the second number to further divide the reference image frame into the second number of second image blocks.

[0083] In some embodiments, the server may also determine a second number of divisions for the reference image frame by: obtaining multiple distance intervals corresponding to the reference image frame, each distance interval being associated with a corresponding number of divisions, which is used for the division of the reference image frame; determining the target distance interval from the multiple distance intervals, and using the number of divisions corresponding to the target distance interval as the second number.

[0084] In practical applications, multiple distance intervals can be pre-set, each associated with a corresponding number of divisions. When dividing a reference image frame, the target distance interval can be determined from the multiple distance intervals, and the number of divisions corresponding to the target distance interval can be used as a second number to further divide the reference image frame into a second number of second image blocks.

[0085] Step 103: For each first image block in the target image frame, with reference to the second image blocks included in the target image frame and the reference image frame, perform feature extraction on the first image block to obtain the image block features of the first image block.

[0086] Based on step 101 above, a first number of first image blocks included in the target image frame are obtained. Based on step 102 above, a second number of second image blocks included in the reference image frame are obtained. In step 103, the following processing can be performed on each first image block in the target image frame to obtain the image block features of each first image block: taking each first image block included in the target image frame and each second image block included in the reference image frame as references, feature extraction is performed on the first image block to obtain the image block features of the first image block.

[0087] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the video processing method provided in the embodiments of this application. Figure 4 It shows Figure 3 The step 103, "using the target image frame and the second image block included in the reference image frame as references, performing feature extraction on the first image block to obtain the image block features of the first image block," can be achieved through steps 1031-1034: Step 1031, obtaining the block vector of the first image block included in the target image frame, and obtaining the block vector of the second image block included in the reference image frame; Step 1032, determining the vector similarity between each block vector and the block vector of the first image block; Step 1033, performing a first aggregation process on multiple block vectors based on the vector similarity corresponding to each block vector to obtain the aggregated features of the first image block; Step 1034, performing a second aggregation process on the block vector and aggregated features of the first image block to obtain the image block features of the first image block.

[0088] In step 1031, the block vectors of the first image block included in the target image frame and the block vectors of the second image blocks included in each reference image frame are obtained. In step 1032, the block vectors of the first image block are compared with the obtained block vectors (including the block vectors of the first image block themselves) to calculate the vector similarity between each block vector and the block vector of the first image block. This vector similarity can be characterized by cosine similarity, vector distance, etc. In step 1033, based on the vector similarity corresponding to each block vector, a first aggregation process is performed on multiple block vectors to obtain the aggregated features of the first image block. Specifically, the vector similarity corresponding to each block vector can be used as the weight of the corresponding block vector, and multiple block vectors can be processed and weighted to obtain the aggregated features of the first image block. In step 1034, the block vectors of the first image block and the aggregated features obtained in step 1033 are subjected to a second aggregation process to obtain the image block features of the first image block. For example, the block vectors of the first image block and the aggregated features can be concatenated, added, multiplied, etc., to obtain the image block features of the first image block.

[0089] In some embodiments, Figure 4 Step 1033 shown can be implemented through steps 10331-10332: Step 10331, normalize the vector similarity corresponding to each block vector and use the result as the vector weight of the corresponding block vector; Step 10332, based on the vector weight of each block vector, perform weighted summation on multiple block vectors to obtain the aggregated features of the first image block.

[0090] In practical applications, the vector similarity of multiple block vectors can be normalized, and the result can be used as the vector weight of the corresponding block vector. Based on the vector weight of each block vector, multiple block vectors can be weighted and summed to obtain the aggregated features of the first image block.

[0091] In some embodiments, the block vector includes a query vector, a key vector, and a value vector; based on this... Figure 4 Step 1032 shown can be implemented as follows: determine the key vector of each image block and the similarity between it and the query vector of the first image block, and use the similarity as the vector similarity; correspondingly, step 1033 can be implemented as follows: based on the vector similarity corresponding to each block vector, perform a first aggregation process on multiple value vectors to obtain the aggregated features of the first image block.

[0092] In practical applications, each image patch's block vector includes a query vector, a key vector, and a value vector. These vectors can be obtained by mapping the features of each image patch into three new vector spaces. When calculating the vector similarity between the block vectors of each image patch and the block vector of the first image patch, the similarity between the key vectors of each image patch and the query vector of the first image patch can be calculated, and this similarity can be used as the vector similarity. Correspondingly, when performing the first aggregation process on the block vectors of multiple image patches, the value vectors of the multiple image patches can be aggregated to obtain the aggregated features of the first image patch.

[0093] In some embodiments, the block vector includes a query vector, a key vector, and a value vector; based on this, see [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating the video processing method provided in the embodiments of this application. Figure 5 It shows Figure 4 Step 1031 can be implemented through steps 201-205: Step 201, perform vector transformation on the second image block included in the reference image frame to obtain the image block vector of the second image block; Step 202, perform linear projection processing on the image block vector to obtain the first intermediate query vector, the first intermediate key vector, and the first intermediate value vector of the second image block; Step 203, perform first pooling processing on the first intermediate query vector to obtain the query vector; Step 204, perform second pooling processing on the first intermediate key vector, and process the result of the second pooling processing based on distance to obtain the key vector; Step 205, perform third pooling processing on the first intermediate value vector, and process the result of the third pooling processing based on distance to obtain the value vector.

[0094] In practical applications, the block vector of each image block includes a query vector, a key vector, and a value vector. Taking the acquisition of the block vector of the second image block included in the reference image frame as an example, the process of obtaining the query vector, key vector, and value vector is explained. In step 201, the second image block is vector-transformed to obtain the image block vector of the second image block. For example, feature extraction is performed on the second image block to obtain the image block token, and then the image block token is embedded to obtain the image block vector. In step 202, the image block vector is linearly projected to obtain the first intermediate query vector, the first intermediate key vector, and the first intermediate value vector of the second image block. For example, the features of the image block are mapped to three new vector spaces through linear projection to obtain the first intermediate query vector, the first intermediate key vector, and the first intermediate value vector. In step 203, the first intermediate query vector is first pooled to obtain the query vector. For example, the first intermediate query vector is pooled using a preset first pooling factor through a first pooling layer. In step 204, the first intermediate key vector is subjected to a second pooling process, and the result of the second pooling process is processed based on the distance to obtain a key vector. For example, the first intermediate key vector is subjected to a second pooling process based on a preset second pooling factor through a second pooling layer, and the result of the second pooling process is processed based on the distance between the reference image frame and the target image frame to obtain a key vector. In step 205, the first intermediate value vector is subjected to a third pooling process, and the result of the third pooling process is processed based on the distance to obtain a value vector. For example, the first intermediate value vector is subjected to a third pooling process based on a preset third pooling factor through a third pooling layer, and the result of the third pooling process is processed based on the distance between the reference image frame and the target image frame to obtain a value vector.

[0095] In some embodiments, Figure 5 Step 204 shown can be implemented through steps 2041-2042: Step 2041, when the distance reaches the distance threshold, the result obtained by the second pooling process is subjected to the first feature extraction process to obtain the key vector; Step 2042, when the distance does not reach the distance threshold, the result obtained by the second pooling process is subjected to the second feature extraction process to obtain the key vector; wherein, the density of the features obtained by the first feature extraction process is lower than the density of the features obtained by the second feature extraction process.

[0096] In practical applications, reference image frames that are at a distance threshold from the target image frame have a lower correlation with the target image frame than the correlation threshold itself. Therefore, these reference image frames can undergo a first feature extraction process with lower density. Conversely, reference image frames that are not at a distance threshold from the target image frame have a higher correlation with the target image frame than the correlation threshold itself. Therefore, these reference image frames can undergo a second feature extraction process with higher density. This approach ensures the accuracy of feature extraction while saving time and improving its efficiency.

[0097] In step 2041, when the distance reaches a distance threshold, the result of the second pooling process is subjected to a first feature extraction process, such as global feature extraction, to obtain a key vector. Specifically, global pooling can be used for global feature extraction. In step 2042, when the distance does not reach the distance threshold, the result of the second pooling process is subjected to a second feature extraction process, such as downsampling, to obtain a downsampled result. Then, based on a neighborhood shift operation, feature extraction is performed on the downsampled result to obtain a key vector.

[0098] In some embodiments, Figure 5 Step 205 shown can be implemented through steps 2051-2052: Step 2051, when the distance reaches the distance threshold, the result obtained by the third pooling process is subjected to the first feature extraction process to obtain the value vector; Step 2052, when the distance does not reach the distance threshold, the result obtained by the third pooling process is subjected to the second feature extraction process to obtain the value vector; wherein, the density of the features obtained by the first feature extraction process is lower than the density of the features obtained by the second feature extraction process.

[0099] In some embodiments, with reference to the target image frame and the second image block included in the reference image frame, the server can extract features from the first image block in the following manner to obtain the image block features of the first image block: with reference to the target image frame and the second image block included in the reference image frame, perform feature extraction on the first image block at least twice to obtain intermediate image block features obtained from each feature extraction; and stitch together the intermediate image block features obtained from each feature extraction to obtain the image block features of the first image block.

[0100] In practical applications, to make the extracted features more accurate, the feature extraction process can be repeated. Specifically, the repetition of feature extraction can be preset, and then, using the target image frame and the second image block included in the reference image frame as references, feature extraction is performed on the first image block a certain number of times to obtain the intermediate image block features obtained in each feature extraction; the intermediate image block features obtained in each feature extraction are then stitched together to obtain the image block features of the first image block.

[0101] Step 104: Perform feature aggregation on the image block features of the first image block included in each image frame to obtain the video features of the video to be processed.

[0102] After obtaining the image block features of each first image block included in each image frame of the video to be processed in step 103, in step 104, the image block features of the first image blocks included in each image frame of the video to be processed are aggregated to obtain the video features of the video to be processed. For example, the image block features of the first image blocks included in each image frame can be concatenated to obtain video features; the image block features of the first image blocks included in each image frame can be added together to obtain video features; the image block features of the first image blocks included in each image frame can be multiplied together to obtain video features; and so on.

[0103] In some embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the video processing method provided in the embodiments of this application. Figure 6 It shows Figure 3 Step 104 can be implemented through steps 1041-1043: Step 1041, for each image frame, the image block features of at least two adjacent first image blocks in the image frame are stitched together to obtain multiple intermediate image block features corresponding to the image frame; Step 1042, for each intermediate image block feature, the intermediate image block features are extracted with reference to the multiple intermediate image block features and the features of the second image block included in the reference image frame to obtain the stitched image block features corresponding to the image frame; Step 1043, the stitched image block features corresponding to each image frame are aggregated to obtain the video features of the video.

[0104] In step 1041, for each image frame, after feature extraction for each first image block, the image block features of at least two adjacent first image blocks in the image frame are concatenated to obtain multiple intermediate image block features corresponding to the image frame. For example, the image block features of a preset number (greater than or equal to 2, such as 2, 4, 6, 9, etc.) of first image blocks can be concatenated. In step 1042, for each intermediate image block feature, using the multiple intermediate image block features and the features of the second image blocks included in the reference image frame as references, feature extraction is performed on the intermediate image block features again to obtain the concatenated image block features corresponding to that image frame. In step 1043, feature aggregation is performed on the concatenated image block features corresponding to each image frame, such as feature concatenation, feature addition, feature multiplication, etc., to obtain the video features of the video.

[0105] See Figure 7 , Figure 7This is a flowchart illustrating the video processing method provided in this application embodiment. Here, the video processing method is implemented using a machine learning model, which includes a feature extraction model. This feature extraction model includes sub-feature extraction models corresponding to three stages, namely, a temporal transformation module (Transformer Block) for Shrinking Temporal Attention (STA). In actual implementation, the number of stages in this sub-feature extraction model is not limited (e.g., it can be three, four, etc.). This feature extraction model is used to extract features from image blocks included in image frames in the video, obtaining image block features for each image block. The machine learning model also includes a vector transformation layer and an image block feature aggregation layer.

[0106] When processing the video using the machine learning model, step 11) divides each image frame in the video to be processed into a first number of first image blocks, and for the target image frame in the video to be processed, divides the reference image frame into a second number of second image blocks based on the distance between the reference image frame and the target image frame.

[0107] Step 12) Input the first image block and the second image block of the video to be processed into the vector transformation layer of the machine learning model.

[0108] Step 13) Through the vector transformation layer, the first image block is vector transformed to obtain the image block vector of the first image block; the second image block included in the reference image frame is vector transformed to obtain the image block vector of the second image block.

[0109] Step 14) For each first image block in the target image frame, the feature extraction model is used to extract features from the image block vector of the first image block using multiple image block vectors as references, so as to obtain the image block features of the first image block, and output the image block features of each first image block included in the video to be processed through the feature extraction model.

[0110] In step 14), each feature extraction model is set with a corresponding number of feature extractions. The number of feature extractions can be once or multiple times. Based on this, the feature extraction model can use multiple image block vectors as references to extract features from the image block vector of the first image block for the specified number of feature extractions, thereby obtaining the intermediate image block features obtained from each feature extraction. Then, the intermediate image block features obtained from each feature extraction are concatenated to obtain the image block features of the first image block.

[0111] In step 14), the size of the first image patch can be different or the same for different stages of the feature extraction model. For example, in the first stage of the feature extraction model, the size of the first image patch is a*b (a and b are both integers greater than 0), and in the second stage of the feature extraction model, the size of the first image patch can be na*nb (a and b are both integers greater than 0, and n is greater than 0), and so on.

[0112] Step 15) Through the image block feature aggregation layer, the image block features of each first image block are aggregated to obtain the video features of the video to be processed.

[0113] In practical applications, when the above machine learning model is used for video tasks (such as video classification and video action recognition), the machine learning model can also include a feature prediction layer, which is used for the final prediction of the video task. For example, when the video task is a video classification task, the feature prediction layer predicts the classification result of the video to be processed based on the extracted video features of the video to be processed; when the video task is a video action recognition task, the feature prediction layer predicts the action recognition result of the video to be processed based on the extracted video features of the video to be processed, and so on.

[0114] See Figure 8 , Figure 8 This is a schematic diagram of the feature extraction model provided in the embodiments of this application. Here, the feature extraction model includes: (1) a linear projection layer: namely, the sub-linear projection layers corresponding to the three new vector spaces, including the first sub-linear projection layer (Linear1), the second sub-linear projection layer (Linear2), and the third sub-linear projection layer (Linear3); (2) a pooling layer: including three pooling layers, namely the first pooling layer (Pool... Q ), second pooling layer (Pool) K ) and the third pooling layer (Pool V (3) First feature extraction layer: namely, the short-long integrated attention module (SLIA) used to extract the key vector. K (4) Second feature extraction layer: Short-Long Integrated Attention (SLIA) module used to extract the value vector. V(5) Fourth pooling layer; (6) Similarity processing layer: similarity is calculated through matrix multiplication (MatMul) and scaling (Scale); (7) Normalization processing layer: for example, a Softmax function can be set; (8) First feature aggregation layer: feature aggregation can be performed through feature multiplication (MatMul); (9) Second feature aggregation layer: the block vector and aggregated features of the first image block are added and regularized to obtain the image block features. The process of feature extraction of the first image block based on the first machine learning model will be explained next.

[0115] Step 21) Perform vector transformation on the first image block to obtain the image block vector of the first image block; perform vector transformation on the second image block included in the reference image frame to obtain the image block vector of the second image block.

[0116] Step 22) The image patch vector is linearly projected through the first sub-linear projection layer (Linear1) to obtain the first intermediate query vector; the image patch vector is linearly projected through the second sub-linear projection layer (Linear2) to obtain the first intermediate key vector; the image patch vector is linearly projected through the third sub-linear projection layer (Linear3) to obtain the first intermediate value vector.

[0117] Step 23) Through the first pooling layer (Pool) Q The first intermediate query vector is subjected to the first pooling process to obtain the query vector; then, it is processed by the second pooling layer. K The first intermediate bond vector is subjected to a second pooling process; then, a third pooling layer is applied. V The first intermediate value vector is then subjected to a third pooling process.

[0118] Step 24) The result of the second pooling process is processed by the first feature extraction layer based on distance to obtain the key vector; the result of the third pooling process is processed by the second feature extraction layer based on distance to obtain the value vector.

[0119] In step 24), when the distance reaches the distance threshold, the result of the second pooling process is subjected to the first feature extraction process to obtain the key vector; when the distance does not reach the distance threshold, the result of the second pooling process is subjected to the second feature extraction process to obtain the key vector; wherein, the density of the features obtained by the first feature extraction process is lower than the density of the features obtained by the second feature extraction process.

[0120] Taking the extraction of value vectors through the second feature extraction layer (which is the same as the first feature extraction layer) as an example, see [link to documentation]. Figure 9 , Figure 9 This is a schematic diagram of the structure of the second feature extraction layer provided in the embodiments of this application. Here, the second feature extraction layer includes: (1) a feature segmentation layer (Token Split); (2) a global pooling layer: global feature extraction is performed by using the Repeat function through a feature copying layer (Token Repeat); (3) a convolutional layer (Conv3D); (4) a neighborhood transformation operation layer: the downsampling results of the convolutional layer (which are the downsampling results of all image blocks in the reference image frame whose distance from the target image frame does not reach the distance threshold) are temporally segmented by a temporal segmentation layer (Temporal Split) to obtain multiple temporal features corresponding to the image blocks of the target image frame. Then, the multiple temporal features obtained by segmentation are filtered by a local temporal window (LocalTemporal window). Then, the temporal features of the neighboring image frames of the target image frame where the filtered image blocks are located are concatenated by a temporal concatenation layer (TemporalConcat) to obtain the value vector of the image blocks; (5) a block vector concatenation layer (Token Concat).

[0121] The process involves several steps. First, a feature segmentation layer determines whether the distance between the reference and target image frames containing the image block vector reaches a distance threshold. If the distance reaches the threshold, a global pooling layer performs global feature extraction on the result of the third pooling process to obtain the image block's value vector. If the distance does not reach the threshold, a convolutional layer downsamples the result of the third pooling process to obtain a downsampled result. Then, a neighborhood transformation layer, based on a neighborhood shift operation, performs feature extraction on the downsampled result to obtain the image block's value vector. Finally, a feature concatenation layer concatenates the value vectors of each image block to obtain the value vectors of each image frame in the video to be processed.

[0122] Step 25) The block vector of the first image block is pooled through the fourth pooling layer to obtain the pooling vector of the first image block.

[0123] Step 26) Through the similarity processing layer, determine the similarity between the key vector of each image block and the query vector of the first image block.

[0124] Step 27) The vector similarity of each image patch is normalized through the normalization processing layer, and the result is used as the vector weight of the corresponding image patch.

[0125] Step 28) Through the first feature aggregation layer, based on the vector weights corresponding to each image block, multiple value vectors are weighted and summed to obtain the aggregated features of the first image block.

[0126] Step 29) Through the second feature aggregation layer, the block vector and aggregated features of the first image block are subjected to a second aggregation process to obtain the image block features of the first image block.

[0127] Applying the above embodiments of this application, each image frame in the video to be processed is divided into a first number of first image blocks. For the target image frame in the video to be processed, the distance between the reference image frame and the target image frame is obtained, and the reference image frame is divided into a second number of second image blocks based on the distance. 1) In this way, for each first image block in the target image frame, with the target image frame and the second image blocks included in the reference image frame as references, feature extraction can be performed on the first image block to obtain the image block features of the first image block. The image block features of the first image blocks included in each image frame are then aggregated to obtain the video features of the video to be processed, ensuring the accuracy of video feature extraction. 2) The reference image frame is an image frame in the video to be processed other than the target image frame. The second number is negatively correlated with the distance between the reference image frame and the target image frame. That is, as the distance between the reference image frame and the target image frame increases, the number of image blocks included in the reference image frame decreases. In this way, when performing feature extraction on the image blocks of the target image frame with the image blocks included in the reference image frame as references, the amount of computation in the feature extraction process is reduced, and the efficiency of video feature extraction is improved.

[0128] The following describes an exemplary application of the embodiments of this application in a real-world application scenario.

[0129] In related technologies, video feature extraction is typically achieved using 3D convolution, but 3D convolution is computationally intensive, resulting in low efficiency. In this embodiment, a temporal transformer can be used, where features from multiple image frames in the video are used as tokens and input into the transformer model to obtain video features. Specifically, the reference image frame (at least one image frame in the video other than the current image frame) is divided into the same number of image blocks. Then, each image block in both the current and reference image frames is used as a reference to perform self-attention processing on each block, thereby obtaining video features. However, when using each image block in both the current and reference image frames as a reference for self-attention processing, the large number of image blocks in the reference frame still consumes significant computational power, and the improvement in video feature extraction efficiency is not substantial.

[0130] Based on this, embodiments of this application provide an optimized temporal transformer structure, through a decaying temporal attention module (i.e. Figure 7 The transformer block shown processes image frames (or video frames) at different distances, significantly reducing computational load and eliminating interference from redundant image frames, thus improving the efficiency and accuracy of feature extraction. Specifically, the current image frame is divided into a first number of patches (each patch having a first size), and the reference image frame (at least one image frame in the video other than the current image frame) is divided into a second number of patches (each patch having a second size). Temporally, the farther the reference image frame is from the current image frame, the fewer patches it contains; that is, as the distance between the reference image frame and the current image frame increases, the number of patches contained in the reference image frame decreases.

[0131] In practical applications, the video feature extraction model (i.e., the optimized temporal transformer structure) provided in this application embodiment can serve as a basic video feature extraction model, applicable to any video task, such as video action recognition, video classification, etc. 1) It can be applied to real-time human action recognition scenarios in video-on-demand / live streaming, i.e., given a video and a list of action labels, it identifies the current human action for subsequent operations such as video structuring; for movies and TV shows, it can identify key action segments in the video, thus outputting them as highlights; for UGC videos, it can identify dangerous actions in the video and determine whether the video violates regulations. 2) It can also be used as a basic model (i.e., a feature extraction model) for general label classification tasks such as events and scenes in videos.

[0132] See Figure 7 The core of this machine learning model lies in the feature extraction model, namely the temporal Transformer Block of Shrinking Temporal Attention (STA).

[0133] Here, the input is any video to be processed (T*H*W*3, where T is the time dimension, H is the height of the image frame, W is the width of the image frame, and 3 is the number of image channels). First, all image frames are divided into multiple image blocks, which are then passed through a vector transformation layer. Features are extracted from each image block to obtain tokens, and the tokens of the image frames are processed to obtain a tensor (i.e., a block vector): Nt*(Nh*Nw+1)*Nc, where Nt represents frame t, Nc represents the c-dimensional feature of each token, Nh and Nw represent the height and width of the token, and 1 represents the tensor after additional global pooling. Next, the final temporal feature Nt*Nc is obtained through three layers of STA modules with different resolutions (i.e., the size of the first image block is different). Finally, after passing through an image block feature aggregation layer, a 1*8C video feature is obtained. Figure 7 Furthermore, the extracted video features can be processed through a feature prediction layer to perform predictions for video classification tasks, yielding prediction results. Figure 7 The "*2" in stage 1 indicates that the sub-feature extraction model in stage 1 repeats the self-attention process (i.e., the number of feature extractions) 2 times. The same applies to "*11" in stage 2 and "*2" in stage 3.

[0134] See Figure 8 , Figure 8 This is a schematic diagram of the feature extraction model provided in this application embodiment. Here, in the STA module, different degrees of temporal attention are applied to the current image frame, short sequence frames (reference image frames that are relatively close to the current image frame, i.e., the distance has not reached the distance threshold) and long sequence frames (reference image frames that are relatively close to the current image frame, i.e., the distance has reached the distance threshold). Specifically, in the self-attention of the current image frame, for the input tensor ( That is, the block vector of the image patch), through linear projection, yields three tensors (block tensors or block vectors), including: query tensor (first intermediate query tensor). ), key tensor (first intermediate key tensor) ) and value tensor (first intermediate value tensor) The query tensor is then obtained through a pooling layer, a first feature extraction layer, and a second feature extraction layer. Key tensor Sum tensor Thus, the self-attention softmax(Q·K)·V of the current frame is obtained.

[0135] See Figure 9 , Figure 9This is a schematic diagram of the feature extraction model provided in this application embodiment. Here, in terms of self-attention for short sequence frames, this application embodiment provides a denser attention mechanism to obtain fine-grained features of the short sequence frames. Specifically, firstly, for... and After further downsampling, we obtained and Then, a finer-grained attention is obtained based on a neighborhood shift operation. This not only reduces computational cost but also minimizes redundant operations between pixels. For self-attention in long sequence frames, traditional methods directly use the same attention as the current frame. However, since long sequence frames have minimal impact on action details, only global semantic features are needed. Therefore, this application employs a global pooling operation to reduce the attention computation between each token. Specifically, this can be implemented as follows:

[0136]

[0137]

[0138] K LA =Repeat(Expand(K))

[0139] V LA =Repeat(Expand(V))

[0140] Among them, Expand(x) shrinks the original token to 1 dimension through global pooling, and repeat copies it nt times to become nt dimensions.

[0141] Compared to traditional transformer methods, such as mVit, its computational complexity is O(n log n). The present invention is significantly reduced to Furthermore, by applying different operations to both long and short sequences, redundant attention operations are effectively reduced. See also Figure 10 , Figure 10 This diagram illustrates the experimental data provided in the embodiments of this application. On a publicly available dataset, compared to the traditional transformer method, this application not only achieves better action recognition accuracy but also significantly reduces the computational cost to half of the previous method. Compared to the traditional 3D convolution method, its computational cost is even reduced by 1 / 10.

[0142] In the embodiments of this application, the transformer network structure is not limited. More complex networks can be used to obtain more effective results, but any similar network structure that conforms to the network idea can be used.

[0143] By applying the above embodiments of this application, the attenuated temporal attention module processes image frames (or video frames) at different distances, which greatly reduces the amount of computation, improves computational efficiency, eliminates the interference of redundant image frames, and improves the efficiency and accuracy of feature extraction.

[0144] It is understood that in the embodiments of this application, data such as user information are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0145] The following description continues to illustrate the exemplary structure of the video processing apparatus 553 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules in the video processing device 553 stored in the memory 550 may include: a first partitioning module 5531, used to partition each image frame in the video to be processed into a first number of first image blocks; a second partitioning module 5532, used to obtain the distance between a reference image frame and the target image frame for a target image frame in the video to be processed, and to partition the reference image frame into a second number of second image blocks based on the distance; wherein the reference image frame is an image frame in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance; a feature extraction module 5533, used to extract features from each first image block in the target image frame, with reference to the target image frame and the second image blocks included in the reference image frame, to obtain image block features of the first image blocks; and a feature aggregation module 5534, used to aggregate the image block features of the first image blocks included in each image frame to obtain video features of the video to be processed.

[0146] In some embodiments, the second segmentation module 5532 is further configured to obtain the number of image frames between the reference image frame and the target image frame, and use the number of frames as the distance between the reference image frame and the target image frame.

[0147] In some embodiments, the second segmentation module 5532 is further configured to obtain the playback time point of the reference image frame and the time interval between the playback time point of the target image frame, and use the time interval as the distance between the reference image frame and the target image frame.

[0148] In some embodiments, the second segmentation module 5532 is further configured to obtain a mapping relationship between distance and segmentation quantity, wherein the segmentation quantity is used for segmenting the reference image frame; based on the distance and the mapping relationship, determine the target segmentation quantity corresponding to the distance, and use the target segmentation quantity as the second quantity.

[0149] In some embodiments, the second segmentation module 5532 is further configured to obtain multiple distance intervals corresponding to the reference image frame, each distance interval being associated with a corresponding segmentation number, the segmentation number being used for segmenting the reference image frame; from the multiple distance intervals, to determine the target distance interval in which the distance is located, and to use the segmentation number corresponding to the target distance interval as the second number.

[0150] In some embodiments, the feature extraction module 5533 is further configured to obtain the block vector of the first image block included in the target image frame, and obtain the block vector of the second image block included in the reference image frame; determine the vector similarity between each block vector and the block vector of the first image block; perform a first aggregation process on the multiple block vectors based on the vector similarity corresponding to each block vector to obtain the aggregated feature of the first image block; and perform a second aggregation process on the block vector of the first image block and the aggregated feature to obtain the image block feature of the first image block.

[0151] In some embodiments, the feature extraction module 5533 is further configured to normalize the vector similarity corresponding to each block vector and use the result of the processing as the vector weight of the corresponding block vector; based on the vector weight of each block vector, perform weighted summation processing on multiple block vectors to obtain the aggregated features of the first image block.

[0152] In some embodiments, the block vector includes a query vector, a key vector, and a value vector; the feature extraction module 5533 is further configured to determine the similarity between the key vector of each image block and the query vector of the first image block, and use the similarity as the vector similarity; the feature extraction module 5533 is further configured to perform a first aggregation process on the multiple value vectors based on the vector similarity corresponding to each block vector to obtain the aggregated features of the first image block.

[0153] In some embodiments, the block vector includes a query vector, a key vector, and a value vector; the feature extraction module 5533 is further configured to perform vector transformation on the second image block included in the reference image frame to obtain an image block vector of the second image block; perform linear projection processing on the image block vector to obtain a first intermediate query vector, a first intermediate key vector, and a first intermediate value vector of the second image block; perform a first pooling process on the first intermediate query vector to obtain the query vector; perform a second pooling process on the first intermediate key vector, and process the result obtained by the second pooling process based on the distance to obtain the key vector; perform a third pooling process on the first intermediate value vector, and process the result obtained by the third pooling process based on the distance to obtain the value vector.

[0154] In some embodiments, the feature extraction module 5533 is further configured to perform a first feature extraction process on the result obtained by the second pooling process to obtain the key vector when the distance reaches a distance threshold; and to perform a second feature extraction process on the result obtained by the second pooling process to obtain the key vector when the distance does not reach the distance threshold; wherein the density of the features obtained by the first feature extraction process is lower than the density of the features obtained by the second feature extraction process.

[0155] In some embodiments, the feature extraction module 5533 is further configured to perform feature extraction on the first image block at least twice, with reference to the target image frame and the second image block included in the reference image frame, to obtain intermediate image block features obtained from each feature extraction; and to concatenate the intermediate image block features obtained from each feature extraction to obtain the image block features of the first image block.

[0156] In some embodiments, the feature aggregation module 5534 is further configured to, for each image frame, stitch together the image block features of at least two adjacent first image blocks in the image frame to obtain a plurality of intermediate image block features corresponding to the image frame; for each intermediate image block feature, perform feature extraction on the intermediate image block feature with reference to the plurality of intermediate image block features and the features of the second image block included in the reference image frame to obtain the stitched image block features corresponding to the image frame; and perform feature aggregation on the stitched image block features corresponding to each image frame to obtain the video features of the video to be processed.

[0157] Applying the above embodiments of this application, each image frame in the video to be processed is divided into a first number of first image blocks. For the target image frame in the video to be processed, the distance between the reference image frame and the target image frame is obtained, and the reference image frame is divided into a second number of second image blocks based on the distance. 1) In this way, for each first image block in the target image frame, with the target image frame and the second image blocks included in the reference image frame as references, feature extraction can be performed on the first image block to obtain the image block features of the first image block. The image block features of the first image blocks included in each image frame are then aggregated to obtain the video features of the video, ensuring the accuracy of video feature extraction. 2) The reference image frame is an image frame in the video to be processed other than the target image frame. The second number is negatively correlated with the distance between the reference image frame and the target image frame. That is, as the distance between the reference image frame and the target image frame increases, the number of image blocks included in the reference image frame decreases. In this way, when performing feature extraction on the image blocks of the target image frame with the image blocks included in the reference image frame as references, the amount of computation in the feature extraction process is reduced, and the efficiency of video feature extraction is improved.

[0158] This application also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video processing method provided in this application.

[0159] This application also provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, will cause the processor to execute the video processing method provided in this application.

[0160] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0161] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0162] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0163] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0164] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A video processing method, characterized in that, The method includes: Each image frame in the video to be processed is divided into a first number of first image blocks; For a target image frame in the video to be processed, the distance between a reference image frame and the target image frame is obtained, and based on the distance, the reference image frame is divided into a second number of second image blocks; The reference image frame refers to the image frames in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance. For each first image block in the target image frame, with reference to the target image frame and the second image blocks included in the reference image frame, feature extraction is performed on the first image block to obtain the image block features of the first image block; Feature aggregation is performed on the image block features of the first image block included in each of the image frames to obtain the video features of the video to be processed.

2. The method as described in claim 1, characterized in that, The step of obtaining the distance between the reference image frame and the target image frame includes: The number of image frames between the reference image frame and the target image frame is obtained, and the number of frames is used as the distance between the reference image frame and the target image frame.

3. The method as described in claim 1, characterized in that, The step of obtaining the distance between the reference image frame and the target image frame includes: The playback time point of the reference image frame and the time interval between the playback time point of the target image frame are obtained, and the time interval is used as the distance between the reference image frame and the target image frame.

4. The method as described in claim 1, characterized in that, Before dividing the reference image frame into a second number of second image blocks based on the distance, the method further includes: Obtain the mapping relationship between distance and number of divisions, wherein the number of divisions is used for dividing the reference image frame; Based on the distance and the mapping relationship, the number of target partitions corresponding to the distance is determined, and the number of target partitions is used as the second number.

5. The method as described in claim 1, characterized in that, Before dividing the reference image frame into a second number of second image blocks based on the distance, the method further includes: Multiple distance intervals corresponding to the reference image frame are obtained, and each distance interval is associated with a corresponding number of divisions, which are used to divide the reference image frame. From the plurality of distance intervals, determine the target distance interval in which the distance is located, and use the number of divisions corresponding to the target distance interval as the second number.

6. The method as described in claim 1, characterized in that, The step of extracting features from the first image block using the target image frame and the second image block included in the reference image frame as references to obtain the image block features of the first image block includes: Obtain the block vector of the first image block included in the target image frame, and obtain the block vector of the second image block included in the reference image frame; Determine the vector similarity between each block vector and the block vector of the first image block; Based on the vector similarity corresponding to each block vector, a first aggregation process is performed on the multiple block vectors to obtain the aggregation features of the first image block; The block vector of the first image block and the aggregated features are subjected to a second aggregation process to obtain the image block features of the first image block.

7. The method as described in claim 6, characterized in that, The first aggregation process, based on the vector similarity corresponding to each block vector, is performed on multiple block vectors to obtain the aggregated features of the first image block, including: The vector similarity corresponding to each block vector is normalized, and the result is used as the vector weight of the corresponding block vector. Based on the vector weights of each block vector, a weighted summation is performed on the multiple block vectors to obtain the aggregated features of the first image block.

8. The method as described in claim 6, characterized in that, The block vector includes a query vector, a key vector, and a value vector; determining the vector similarity between each block vector and the block vector of the first image block includes: The key vector of each image block and the similarity between it and the query vector of the first image block are determined respectively, and the similarity is used as the vector similarity. The first aggregation process, based on the vector similarity corresponding to each block vector, is performed on multiple block vectors to obtain the aggregated features of the first image block, including: Based on the vector similarity corresponding to each block vector, a first aggregation process is performed on multiple value vectors to obtain the aggregated features of the first image block.

9. The method as described in claim 6, characterized in that, The block vector includes a query vector, a key vector, and a value vector; obtaining the block vector of the second image block included in the reference image frame includes: The second image block included in the reference image frame is vector-transformed to obtain the image block vector of the second image block; The image block vector is linearly projected to obtain the first intermediate query vector, the first intermediate key vector, and the first intermediate value vector of the second image block. The first intermediate query vector is subjected to a first pooling process to obtain the query vector; The first intermediate key vector is subjected to a second pooling process, and the result of the second pooling process is processed based on the distance to obtain the key vector; The first intermediate value vector is subjected to a third pooling process, and the result of the third pooling process is processed based on the distance to obtain the value vector.

10. The method as described in claim 9, characterized in that, The process of processing the result obtained from the second pooling process based on the distance to obtain the key vector includes: When the distance reaches the distance threshold, the result obtained from the second pooling process is subjected to the first feature extraction process to obtain the key vector. When the distance does not reach the distance threshold, the result obtained by the second pooling process is subjected to a second feature extraction process to obtain the key vector. The density of features obtained by the first feature extraction process is lower than the density of features obtained by the second feature extraction process.

11. The method as described in claim 1, characterized in that, The step of extracting features from the first image block using the target image frame and the second image block included in the reference image frame as references to obtain the image block features of the first image block includes: Using the target image frame and the second image block included in the reference image frame as references, feature extraction is performed on the first image block at least twice to obtain the intermediate image block features obtained from each feature extraction. The intermediate image patch features obtained from each feature extraction are concatenated to obtain the image patch features of the first image patch.

12. The method as described in claim 1, characterized in that, The step of performing feature aggregation on the image block features of the first image blocks included in each of the image frames to obtain the video features of the video to be processed includes: For each of the image frames, the image block features of at least two adjacent first image blocks in the image frame are stitched together to obtain multiple intermediate image block features corresponding to the image frame; For each of the intermediate image block features, with reference to the features of the plurality of intermediate image block features and the features of the second image block included in the reference image frame, feature extraction is performed on the intermediate image block features to obtain the stitched image block features corresponding to the image frame; Feature aggregation is performed on the spliced ​​image block features corresponding to each of the image frames to obtain the video features of the video to be processed.

13. A video processing apparatus, characterized in that, The device includes: The first partitioning module is used to divide each image frame in the video to be processed into a first number of first image blocks; The second segmentation module is used to obtain the distance between the reference image frame and the target image frame in the video to be processed, and to divide the reference image frame into a second number of second image blocks based on the distance. The reference image frame refers to the image frames in the video to be processed other than the target image frame, and the second number is negatively correlated with the distance. The feature extraction module is used to extract features from each first image block in the target image frame, taking the target image frame and the second image blocks included in the reference image frame as references, to obtain the image block features of the first image block. The feature aggregation module is used to perform feature aggregation on the image block features of the first image block included in each of the image frames to obtain the video features of the video to be processed.

14. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the video processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing executable instructions, characterized in that, When the executable instructions are executed by the processor, they implement the video processing method according to any one of claims 1 to 12.

Images