Video frame processing method, device, electronic device, storage medium and program product

Abstract

The embodiments of the present invention disclose a video frame processing method, apparatus, electronic device, storage medium, and program product. The method obtains multiple modal images corresponding to each video frame in a video to be processed, performs modal feature extraction on the modal images to obtain image features of each modal image, determines nodes corresponding to the image features in the image feature space of each modality, determines valid neighbor nodes of each node, performs node clustering based on each node and its valid neighbor nodes to obtain at least one node cluster set, generates a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set, determines and fuses video content sets corresponding to multiple modalities of the same object, and obtains a processing result of at least one object in the video to be processed. The present invention can determine the video content set of the same object from a video, thereby improving the accuracy of processing video frames.

Description

Technical Field

[0001] The present invention relates to the field of video processing technology, and in particular to a video frame processing method, device, electronic device, storage medium and program product. Background Art

[0002] Videos typically contain rich content information, and the same video may contain frames representing multiple objects. With the recent rise of artificial intelligence, processing video frames based on different objects has gained widespread application in media and search scenarios, such as celebrity search and automatic video editing.

[0003] Currently, the main method for identifying video frames corresponding to different objects in a video is to cluster the frames based on the objects' facial features. However, this method relies on single-modality facial features, which results in a large number of valid video frames, such as those with the head down, distant views, and back views, being unable to be processed. Therefore, the effectiveness of identifying video frames corresponding to different objects in a video using related techniques is less than ideal, and processing accuracy is low. Summary of the Invention

[0004] Embodiments of the present invention provide a video frame processing method, apparatus, electronic device, storage medium, and program product, which can combine multimodal features to determine a set of video content of the same object from a video, thereby improving the accuracy of determining the video frames of each object in the video.

[0005] An embodiment of the present invention provides a video frame processing method, including:

[0006] Based on the video to be processed, obtaining multiple modal images corresponding to each video frame in the video to be processed, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities;

[0007] Performing modal feature extraction on the modal images to obtain image features corresponding to the modal images of each modality, and determining nodes corresponding to the image features under each modality in the image feature space corresponding to each modality;

[0008] Determining valid neighbor nodes of each node in each of the image feature spaces, wherein a node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must satisfy a preset node association condition;

[0009] performing node clustering on each of the nodes in each of the image feature spaces according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality;

[0010] Generating a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set;

[0011] A video content set corresponding to multiple modalities of the same object is determined, and the video content set corresponding to the same object is fused to obtain a processing result of at least one object in the video to be processed.

[0012] Accordingly, an embodiment of the present invention further provides a video frame processing device, comprising:

[0013] An image acquisition unit is configured to acquire, based on a video to be processed, a plurality of modal images corresponding to each video frame in the video to be processed, wherein the plurality of modal images of the same video frame respectively represent video contents corresponding to the video frame in different modalities;

[0014] a feature extraction unit, configured to extract modal features from the modal images to obtain image features corresponding to the modal images of each modality, and determine nodes corresponding to the image features under each modality in the image feature space corresponding to each modality;

[0015] a node determination unit, configured to determine, in each of the image feature spaces, valid neighbor nodes of each node, wherein a node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, a distance between the node and the other node must satisfy a preset node association condition;

[0016] a node clustering unit, configured to perform node clustering on each of the nodes in each of the image feature spaces according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality;

[0017] A set generating unit, configured to generate a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set;

[0018] The video processing unit is used to determine a video content set corresponding to multiple modalities of the same object, fuse the video content set corresponding to the same object, and obtain a processing result of at least one object in the video to be processed.

[0019] Optionally, the node clustering unit is configured to generate, based on each node and the valid neighbor nodes corresponding to the node, a subgraph with each node in the image feature space as a central node and the valid neighbor nodes corresponding to the node as other nodes;

[0020] Obtain the graph clustering network corresponding to each modality;

[0021] Each of the subgraphs is subjected to graph clustering processing using the graph clustering network of the corresponding modality to obtain at least one node clustering set under each modality.

[0022] Optionally, the node clustering unit is configured to update node features of each node according to each node and the valid neighboring nodes corresponding to the node, to obtain updated nodes;

[0023] Based on the similarity between each updated node and other updated nodes in the image feature space, node clustering is performed on each updated node in each image feature space to obtain at least one node cluster set under each modality.

[0024] Optionally, the node clustering unit is configured to calculate, based on each node and the valid neighboring node corresponding to the node, a spatial distance between each node and the corresponding valid neighboring node in the corresponding image feature space;

[0025] Determining, based on each of the spatial distances, an association weight between each of the nodes and the corresponding valid neighbor node;

[0026] Node feature update calculation is performed based on the node features corresponding to each of the nodes, the node features of the valid neighbor nodes corresponding to each of the nodes, and each of the association weights to obtain each updated node.

[0027] Optionally, the node clustering unit is configured to determine, based on each of the updated nodes, in each of the image feature spaces, a preset number of updated nodes as cluster center nodes;

[0028] Obtaining the similarity between each updated node and each cluster center node in each image feature space;

[0029] Dividing each of the updated nodes into a cluster where the corresponding target central node is located, wherein the similarity between the updated node and the corresponding target central node is not less than a preset similarity threshold;

[0030] Based on the updated nodes in each cluster, select a new cluster center node of each cluster, and return to the step of obtaining the similarity between each updated node in each image feature space and each cluster center node until a clustering end condition is met;

[0031] The updated nodes in each of the clusters are respectively determined as a node cluster set.

[0032] Optionally, the video frame processing device provided by an embodiment of the present invention further includes a graph network training unit, configured to perform graph clustering processing on sample subgraphs in the feature space of each sample image using a graph clustering network to be trained corresponding to each modality, to obtain at least one training node cluster set under each modality, wherein each of the sample subgraphs is annotated with a reference cluster set result;

[0033] Based on the training node cluster set and the reference cluster set results under each modality, respectively calculating the loss of the graph clustering network to be trained corresponding to each modality;

[0034] According to the loss, the network parameters of each graph clustering network to be trained are adjusted to obtain a trained graph clustering network corresponding to each modality.

[0035] Optionally, the feature extraction unit is used to map the modal image into the modal feature vector space corresponding to different modalities through the feature mapping parameters of the shared modal feature extraction model, and obtain the image features corresponding to the modal images of each modality based on the mapping results. The shared modal feature extraction model is trained based on a multimodal image sample set, and the multimodal image sample set includes multiple sample modal images that characterize the image content under different modalities.

[0036] Optionally, the video frame processing device provided by an embodiment of the present invention further includes a model training unit, configured to obtain a multimodal image sample set, wherein the multimodal image sample set includes a plurality of sample modality images representing image content under different modalities, each of the sample modality images being annotated with a reference modality;

[0037] Performing modal feature extraction on each of the sample modal images in the multimodal image sample set using the shared modal feature extraction model to be trained to obtain sample image features corresponding to each of the sample modal images;

[0038] Performing modality classification on each of the sample image features using a modality classification model to obtain a training modality corresponding to each of the sample image features;

[0039] Calculating the loss of the shared modality feature extraction network to be trained based on the training modality and the reference modality of each of the sample image features;

[0040] According to the loss, the model parameters of the shared modality feature extraction model to be trained are adjusted to obtain a trained shared modality feature extraction network.

[0041] Optionally, the model training unit is configured to calculate the corresponding loss of the shared modality feature extraction network to be trained in each modality based on the training modality and the reference modality of each sample image feature;

[0042] Based on the corresponding losses under each modality, the total loss of the shared modality feature extraction network is calculated as the loss of the shared modality feature extraction network to be trained.

[0043] Optionally, the video processing unit is configured to determine video time information of each of the video content sets according to video frame sorting information of the video frames corresponding to each of the video content sets in the video to be processed;

[0044] Matching is performed according to the video time information of each of the video content sets, and the successfully matched video content sets of different modalities are used as the video content sets corresponding to multiple modalities of the same object.

[0045] Optionally, the image acquisition unit is configured to acquire, based on the video to be processed, each video frame in the video to be processed and audio information corresponding to each video frame;

[0046] Extracting facial images and limb images from each of the video frames to obtain facial modal images and limb modal images corresponding to each of the video frames;

[0047] Performing voiceprint analysis on the audio information corresponding to each of the video frames to obtain an audio modal image corresponding to each of the audio information;

[0048] The facial modality image, the limb modality image, and the audio modality image corresponding to each of the video frames are used as multiple modality images corresponding to each video frame.

[0049] Optionally, the set generation unit is configured to determine, based on the nodes in the node clustering set under the facial modality, a facial modality extraction region corresponding to each facial modality image corresponding to each node in the corresponding video frame;

[0050] Based on the nodes in each of the node cluster sets under the limb modality, determining a limb modality extraction region corresponding to each of the limb modality images corresponding to each of the nodes in the corresponding video frame;

[0051] Calculating an intersection-over-union ratio between each of the facial modality extraction regions and each of the limb modality extraction regions;

[0052] Determining a set of video frames of the same object based on the intersection-over-union ratio, the video frames corresponding to the facial modality images, and the video frames corresponding to the limb modality images;

[0053] Determining, based on the video frames corresponding to the nodes in each of the node cluster sets in the audio modality, an audio information set corresponding to each of the node cluster sets;

[0054] The video frame set and the audio information set are taken as a video content set.

[0055] Correspondingly, an embodiment of the present invention also provides an electronic device, including a memory and a processor; the memory stores an application, and the processor is used to run the application in the memory to execute the steps in any video frame processing method provided in the embodiment of the present invention.

[0056] Accordingly, an embodiment of the present invention further provides a computer-readable storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the steps in any one of the video frame processing methods provided in the embodiments of the present invention.

[0057] In addition, an embodiment of the present invention further provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps of any one of the video frame processing methods provided in the embodiments of the present invention.

[0058] By adopting the scheme of the embodiment of the present invention, a plurality of modal images corresponding to each video frame in the video to be processed can be obtained based on the video to be processed, wherein the plurality of modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities, modal feature extraction is performed on the modal image to obtain the image features corresponding to the modal images of each modality, the nodes corresponding to the image features under each modality are determined in the image feature space corresponding to each modality, and the valid neighbor nodes of each node are determined in each of the image feature spaces, wherein the node and its valid neighbor nodes are associated nodes with each other, and if a node is an associated node of another node, the distance between the node and the other node must meet the preset node association condition, and the node association condition is determined for each node according to the valid neighbor nodes corresponding to the node. Each node in the image feature space is clustered to obtain at least one node cluster set under each modality, and based on the video frames corresponding to the nodes in each node cluster set, a video content set corresponding to each node cluster set is generated, and the video content sets corresponding to multiple modalities of the same object are determined. The video content sets corresponding to the same object are fused to obtain a processing result of at least one object in the video to be processed. In the embodiment of the present invention, due to the combination of multimodal features, the nodes in the same image feature space under different modalities are preliminarily selected based on the node association conditions, which reduces unreliable erroneous connections between nodes in the clustering process. Therefore, the video content set of the same object can be determined from the video, thereby improving the accuracy of processing the video frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0060] Figure 1 Schematic diagram of a video frame processing method according to an embodiment of the present invention;

[0061] Figure 2 is a flowchart of a video frame processing method provided by an embodiment of the present invention;

[0062] Figure 3 Schematic diagram of the training process of the shared feature extraction model provided by an embodiment of the present invention;

[0063] Figure 4 This is a schematic diagram of determining valid neighbor nodes provided by an embodiment of the present invention;

[0064] Figure 5 Schematic diagram of the technical implementation of video frame processing provided by an embodiment of the present invention;

[0065] Figure 6 This is a flow chart of user authorization login provided by an embodiment of the present invention;

[0066] Figure 7 is a structural diagram of a video frame processing device provided by an embodiment of the present invention;

[0067] Figure 8 is another structural diagram of a video frame processing device provided by an embodiment of the present invention;

[0068] Figure 9 It is a structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0069] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of the present invention.

[0070] Embodiments of the present invention provide a video frame processing method, apparatus, electronic device, and computer-readable storage medium. Specifically, embodiments of the present invention provide a video frame processing method applicable to a video frame processing apparatus, which can be integrated into an electronic device.

[0071] The electronic device can be a terminal or other device, including but not limited to a mobile terminal and a fixed terminal. For example, a mobile terminal includes but is not limited to a smart phone, a smart watch, a tablet computer, a laptop computer, a smart car terminal, etc., wherein a fixed terminal includes but is not limited to a desktop computer, a smart TV, etc.

[0072] The electronic device can also be a server or other device. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms, but is not limited to these.

[0073] The video frame processing method of the embodiment of the present invention can be implemented by a server, or can be implemented by a terminal and a server together.

[0074] The method is described below by taking the example of a terminal and a server jointly implementing the video frame processing method.

[0075] like Figure 1 As shown, the video frame processing system provided by an embodiment of the present invention includes a terminal 10 and a server 20, etc.; the terminal 10 and the server 20 are connected via a network, for example, via a wired or wireless network connection, etc., wherein the terminal 10 can be a terminal for a user to initiate a video frame processing request, and is used to send a video frame processing request to the server 20.

[0076] Alternatively, the terminal 10 may exist as a terminal that sends a video to be processed to the server 20 .

[0077] The server 20 can be used to obtain multiple modal images corresponding to each video frame in the video to be processed based on the video to be processed, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities, perform modal feature extraction on the modal image, obtain image features corresponding to the modal image of each modality, determine the nodes corresponding to the image features under each modality in the image feature space corresponding to each modality, and determine the effective neighbor nodes of each node in each image feature space, wherein the node and its effective neighbor nodes are mutually associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition, and perform node clustering on each node in each image feature space according to each node and the effective neighbor node corresponding to the node to obtain at least one node clustering set under each modality.

[0078] Server 20 can be used to generate a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set, determine the video content set corresponding to multiple modalities of the same object, and fuse the video content set corresponding to the same object to obtain the processing result of at least one object in the video to be processed.

[0079] It is understandable that, in some embodiments, the video frame processing step performed by the server 20 may also be performed by the terminal 10, which is not limited in this embodiment of the present invention.

[0080] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0081] The embodiment of the present invention will be described from the perspective of a video frame processing device, which can be integrated into a server or a terminal.

[0082] like Figure 2 As shown, the specific process of the video frame processing method of this embodiment can be as follows:

[0083] 201. Based on a video to be processed, obtain multiple modal images corresponding to each video frame in the video to be processed, wherein the multiple modal images of the same video frame respectively represent video contents corresponding to the video frame in different modalities.

[0084] The video to be processed is a video that requires video content corresponding to at least one object in the video. The video to be processed may include video frames related to at least one object. For example, the video to be processed includes two different objects: a person and a cat.

[0085] Specifically, the content in the video to be processed may be people, animals, plants, etc., and the embodiment of the present invention does not limit the content type of the video to be processed.

[0086] In the embodiments of the present invention, modality can be understood as a dimension that can represent some or all of the characteristics of an object in the processed video. For example, for an animal, its modality can be the animal's head, torso, limbs, sound, etc.; for a plant, its modality can be the trunk, flowers, fruits, leaves, etc.

[0087] The modality image is an image representing the video content of a video frame of a to-be-processed video in a certain modality. Generally, the same modality image only represents the content in one modality.

[0088] The video content may be the content in the video frame of the video to be processed, or the audio content in the video to be processed.

[0089] It should be noted that one or more video frames in the video being processed may correspond to only one modal image. For example, if the video being processed is based on people, but some video frames contain only scenery and no people, the modal image corresponding to that video frame may only be a single modal image derived from the audio corresponding to the video frame.

[0090] In some examples, taking the example of modalities including facial modality, body modality, and sound modality, the step of "obtaining, based on the video to be processed, multiple modal images corresponding to each video frame in the video to be processed" may specifically include:

[0091] Based on the video to be processed, obtaining each video frame in the video to be processed and audio information corresponding to each video frame;

[0092] Perform voiceprint analysis on the audio information corresponding to each video frame to obtain the audio modal image corresponding to each audio information;

[0093] Extracting facial images and limb images from each video frame to obtain facial modal images and limb modal images corresponding to each video frame;

[0094] The facial modality image, the limb modality image, and the audio modality image corresponding to each video frame are used as multiple modality images corresponding to each video frame.

[0095] Among them, the facial modality image is an image representing the video content of the video frame of the video to be processed in the facial modality; the body modality image is an image representing the video content of the video frame of the video to be processed in the body modality; and the audio modality image is an image representing the video content corresponding to the video frame of the video to be processed in the audio modality.

[0096] For example, Figure 3 As shown, a facial modality image for facial modality, a limb modality image for limb modality, and an audio modality image corresponding to the video frame can be separated from the video frame.

[0097] Specifically, voiceprint analysis of the audio information corresponding to each video frame can be performed by extracting the voiceprint using a voiceprint analysis program, converting the obtained voiceprint into a logarithmic Mel-spectrogram, and obtaining an audio modal image in the form of image data. Alternatively, the extracted sound spectrum can be processed by framing, windowing, filtering, Fourier transform, and other methods before being converted into an image format.

[0098] It is understandable that several facial modality images and several limb modality images may be extracted from the same video frame, and the number of facial modality images and limb modality images extracted from the same video frame may be the same or different.

[0099] For example, the same video frame may include the full bodies of multiple characters, and facial modal images and limb modal images corresponding to the multiple characters may be obtained; or the same video frame may only include the limbs of one character, and only the limb modal image corresponding to one character may be obtained.

[0100] In embodiments of the present invention, artificial intelligence (AI) technology can be applied to process video frames. Artificial intelligence (AI) refers to theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI is the study of the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0101] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0102] 202. Perform modal feature extraction on the modal image to obtain image features corresponding to the modal image of each modality, and determine nodes corresponding to the image features under each modality in the image feature space corresponding to each modality.

[0103] The image feature space is generally understood as a vector space constructed based on the modality of the modal image. In some optional examples, the image feature space can be a vector space shared by multiple modalities. In other examples, each image feature space can correspond to a modality.

[0104] In some embodiments, the step of "extracting modal features from the modal image" may be applying a scale-invariant feature transform (SIFT) feature extraction method, a Histogram of Oriented Gradient (HOG) feature extraction method, and the like.

[0105] In other embodiments, the step of "extracting modal features from the modal images" can be implemented using a modal feature extraction model. For example, different modal feature extraction models can be set for modal images of different modalities to extract features from the modal images.

[0106] For example, the modal feature extraction model may include a facial modal feature extraction model that extracts features from modal images of facial modality, a limb modal feature extraction model that extracts features from modal images of limb modality, an audio modal feature extraction model that extracts features from modal images of audio modality, and so on.

[0107] It is understandable that if a corresponding modal feature extraction model is set for each modality, it will occupy a large amount of storage space in actual application, and the requirements for computing resources when performing feature extraction will also be relatively high. Therefore, in an embodiment of the present invention, a method for extracting features from images of different modalities using the same shared modal feature extraction model is proposed, that is, the step of "extracting modal features from modal images to obtain image features corresponding to modal images of each modality" includes:

[0108] By sharing the feature mapping parameters of the modal feature extraction model, the modal images are mapped into the modal feature vector space corresponding to different modalities. Based on the mapping results, the image features corresponding to the modal images of each modality are obtained. The shared modal feature extraction model is trained based on a multimodal image sample set, which includes multiple sample modal images that represent the image content under different modalities.

[0109] Among them, the shared modal feature extraction model is a model that can realize modal feature extraction for different modalities.

[0110] The feature extraction scheme involved in this embodiment focuses on extracting features from modal images of different modalities (for example, face, body, voiceprint, etc.) using a unified shared modality feature extraction model, maintaining feature discriminability while making the overall model parameters fewer and more convenient to deploy.

[0111] In actual application, the shared modality feature extraction model is obtained through pre-training. Through the pre-training process, the parameters of the shared modality feature extraction model can be adjusted to enable the shared modality feature extraction model to achieve better feature extraction performance. Therefore, before the step of "mapping the modal image to the modal feature vector space corresponding to different modalities using the feature mapping parameters of the shared modality feature extraction model", the video frame processing method provided by the embodiment of the present invention may also include:

[0112] Acquire a multimodal image sample set, where the multimodal image sample set includes multiple sample modality images representing image content under different modalities, and each sample modality image is annotated with a reference modality;

[0113] Performing modal feature extraction on each sample modal image in the multimodal image sample set using the shared modal feature extraction model to be trained, and obtaining sample image features corresponding to each sample modal image;

[0114] Through the modality classification model, the features of each sample image are classified into modalities to obtain the training modalities corresponding to the features of each sample image;

[0115] Based on the training modality and reference modality of each sample image feature, calculate the loss of the shared modality feature extraction network to be trained;

[0116] According to the loss, the model parameters of the shared modality feature extraction model to be trained are adjusted to obtain the trained shared modality feature extraction network.

[0117] The multimodal image sample set includes sample modality images of different modalities.

[0118] Specifically, the shared modality feature extraction model may include a normalization layer, a multi-head attention layer, a forward mapping layer, and the like. Figure 3 As shown in Figure 2, during the model training phase, each mini-batch of multimodal image samples can contain samples from three modalities at the same time. The shared modality feature extraction model can divide the input sample modality images x into K non-overlapping patches, and then map them to another high-dimensional space to obtain K tokens, denoted as {E k , k∈[1,k]}.

[0119] The shared modality feature extraction model can concatenate a category tag Z of the same dimension to K tokens. cls , and add the position code p to get the feature z to be mapped.

[0120] Where z=[Z cls , E1, E2, ..., E k ]+p.

[0121] Input z into a model composed of multiple layers of Transformer blocks, such as Figure 3 As shown in the figure, each block consists of multi-head attention, layer normalization, and feed forward layers. The mathematical operation process can be expressed as:

[0122] yI =MSA(LN(z I ))+z I

[0123] z I+1 =FF(LN(y I ))+y I

[0124] Among them, MSA stands for multi-head attention, LN stands for layer normalization, and FF stands for forward mapping.

[0125] Optionally, the loss of the shared modality feature extraction model can be solved by a cross entropy function, a gradient descent method, etc., which is not limited in this embodiment of the present invention.

[0126] To achieve the goal of extracting representations of data from multiple modalities using a single model, embodiments of the present invention propose a model-aware training strategy. This allows the proposed shared modality feature extraction model, after joint training on multimodal data, to achieve the same or even exceed the performance of models trained on each modality independently. Specifically, the step of "calculating the loss of the shared modality feature extraction network to be trained based on the training modality and reference modality of each sample image feature" may include:

[0127] Based on the training modality and reference modality of each sample image feature, the corresponding loss of the shared modality feature extraction network to be trained in each modality is calculated respectively;

[0128] Based on the corresponding losses under each modality, the total loss of the shared modality feature extraction network is calculated as the loss of the shared modality feature extraction network to be trained.

[0129] The training modality is the modality corresponding to the sample modality image obtained after the modality classification model performs modality classification on the features of each sample image. The reference modality is the modality actually annotated in the sample modality image.

[0130] In other words, in this embodiment of the present invention, the classifier prediction results for different modalities can be separated, and a corresponding loss function can be used for each modality. Finally, the total loss of the shared modality feature extraction model is calculated based on the losses of the different modalities, and the shared modality feature extraction model is adjusted based on the total loss.

[0131] Specifically, taking three different modalities of face, body, and audio as an example, the total loss of the shared modality feature extraction model can be expressed as follows:

[0132] L=λ f *L f +λ b *L b +λ v *L v

[0133] Among them, L f 、L b and L v are the losses corresponding to facial modality, body modality and audio modality, respectively, f ,λ b and λ v Weights are calculated for the losses corresponding to facial modality, body modality, and audio modality respectively.

[0134] In the embodiment of the present invention, in the image feature space corresponding to each modality, the nodes corresponding to the image features under each modality can be determined by directly using the image features as nodes, or by using features obtained by mapping the image features as nodes. Technicians can choose the node based on actual application conditions.

[0135] 203. Determine the valid neighbor nodes of each node in each image feature space, wherein the node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition.

[0136] For a node, other nodes in the same image feature space can be called neighbor nodes of the node.

[0137] Currently, the most commonly used methods for clustering nodes are graph-based supervised clustering algorithms or unsupervised KNN algorithms, Kmeans algorithms, etc.

[0138] Generally, graph-based supervised clustering algorithms use the k-nearest neighbor graphing method to construct a relationship graph between nodes. The k-nearest neighbor graphing method selects a fixed number of k neighbor nodes for each node. Figure 4 As shown in Figure 1, for nodes a, b, and c, k = 3. The circles in the figure represent the possible neighborhoods of each node, and the line segment between two points represents a connection between them. However, a fixed number of k neighbor nodes will introduce many incorrect connections. For example, Figure 4 In the example, a is the central node, b and c are both 3-neighbor nodes of a, but a may connect to node c which is actually far away from it in space, thus forming an unreliable connection.

[0139] As for the unsupervised KNN algorithm and Kmeans algorithm, they only rely on the feature vectors of each node for clustering during the node clustering process. However, the amount of information contained in the node itself is limited, and information such as the relationship between nodes does not play a role in the clustering process.

[0140] To address the impact of unreliable connections on graph clustering and the ineffectiveness of similarity relationships between nodes in the clustering process, embodiments of the present invention explore constructing a node relationship graph using a variable number of valid neighbor nodes. In other words, if two nodes are both neighbors of each other that meet the node association conditions, they are considered valid neighbors of each other.

[0141] In other words, a node's valid neighbor nodes are the node's neighbor nodes that are associated with the node. That is, the node is an associated node of its valid neighbor nodes, and its valid neighbor nodes are also associated nodes of the node.

[0142] If, for node B, the distance between node A and node B meets a preset node association condition, then node A is the associated node of node B.

[0143] Specifically, the node association condition can be the N nodes closest to a node among its neighbor nodes, or the node association condition can be the N nodes closest to a node among its neighbor nodes whose distance to the node does not exceed a certain range, and so on.

[0144] For example, when the node association condition is the three nodes closest to a node among its neighbor nodes, assume that the neighbor nodes of node 1 include nodes 2 to 10, among which the three nodes closest to node 1 are node 2, node 4, and node 7; the neighbor nodes of node 2 include node 1 and nodes 3 to 10, among which the three nodes closest to node 2 are node 3, node 5, and node 6; the neighbor nodes of node 4 include nodes 1 to 3 and nodes 5 to 10, among which the three nodes closest to node 4 are node 1, node 3, and node 8.

[0145] At this point, node 1's associated nodes include node 2, node 4, and node 7; node 2's associated nodes include node 3, node 5, and node 6; and node 4's associated nodes include node 1, node 3, and node 8. In other words, node 2 is an associated node of node 1, but node 1 is not an associated node of node 2. Therefore, node 1 and node 2 are not associated with each other, and node 2 is not a valid neighbor node of node 1.

[0146] However, node 4 is an associated node of node 1, and node 1 is also an associated node of node 4. Therefore, node 1 and node 4 are associated nodes to each other. At this time, node 4 is a valid neighbor node of node 1, and node 1 is also a valid neighbor node of node 4.

[0147] It is understandable that the fact that node A is an associated node of node B does not necessarily mean that node B is also an associated node of node A. Figure 4For example, assuming that the node association condition is the three nodes closest to a node among its neighbor nodes, then node c is the associated node of node a, but node a is not the associated node of node c.

[0148] In actual applications, if a node has no valid neighboring nodes that are mutually associated with it, its associated nodes can be directly used as its valid neighboring nodes. For example, node m's only associated node is node n, but node m is not a related node of node n. In this case, node m has no valid neighboring nodes that are mutually associated with it. Therefore, node n can also be considered as a valid neighboring node of node m and participate in subsequent node clustering.

[0149] Therefore, through the embodiment of the present invention, the neighbor nodes of each node can be processed, so that the effective neighbor nodes can play a role in the graph clustering process.

[0150] 204. Perform node clustering on each node in each image feature space according to each node and its corresponding valid neighboring nodes, and obtain at least one node clustering set under each modality.

[0151] In some optional embodiments, such as Figure 5 As shown, node clustering can be achieved through a supervised clustering algorithm of the graph. In this case, the image feature space can be a graph network space, and step 204 can specifically include:

[0152] According to each node and its corresponding valid neighbor nodes, a subgraph is generated with each node in the network space of each graph as the central node and the valid neighbor nodes corresponding to the node as other nodes;

[0153] Obtain the graph clustering network corresponding to each modality;

[0154] For each subgraph, graph clustering processing is performed through the graph clustering network of the corresponding modality to obtain at least one node cluster set under each modality.

[0155] Specifically, a subgraph is a graph network that connects only some of its nodes. For example, a node in the graph network space can be used as the central node, and some of the nodes in the graph network space other than the central node can be used as neighbor nodes. By connecting the central node with all of its neighbor nodes, a subgraph is formed.

[0156] For example, there is a node A and its corresponding effective neighbor nodes B and C. A can be used as the central node, B and C as other nodes, and A, B and A, C can be connected to obtain a subgraph.

[0157] Graph clustering networks can be implemented based on graph convolutional networks (GCNs). GCNs can be used in scenarios such as supervised node classification, link prediction, and recommendation systems. Given a graph, GCNs can use graph convolution operations to obtain node embeddings layer by layer: at each layer, to obtain the embedding of a node, it is necessary to collect the embeddings of adjacent nodes, then perform one or more layers of linear transformations and nonlinear activations, and finally obtain the embedding vector in the final layer for some final task. For example, in a node classification problem, the embedding vector of the last layer is passed to the classifier to predict the node label and achieve node classification.

[0158] The graph clustering network can be obtained through training. Through the training process, the parameters of the graph clustering network can be adjusted so that the graph clustering network can achieve better clustering performance. Before the step of "obtaining the graph clustering network corresponding to each modality", the video frame processing method provided by the embodiment of the present invention may also include:

[0159] Perform graph clustering on the sample subgraphs in the feature space of each sample image using the graph clustering network to be trained corresponding to each modality, and obtain at least one training node cluster set under each modality, wherein each sample subgraph is annotated with a reference cluster set result;

[0160] Based on the training node clustering set and reference clustering set results under each modality, the loss of the graph clustering network to be trained corresponding to each modality is calculated respectively;

[0161] According to the loss, the network parameters of each graph clustering network to be trained are adjusted to obtain the trained graph clustering network corresponding to each modality.

[0162] Optionally, the loss of the graph clustering network can be solved by a cross entropy function, a gradient descent method, etc., which is not limited in this embodiment of the present invention.

[0163] Specifically, the network parameters of the graph clustering network may include the number of convolutional layers used for node embedding in the graph clustering network, parameters of linear or nonlinear transformation functions, etc. For example, if the graph clustering network includes convolutional layers, the network parameters of the graph clustering network may include the number of convolutional layers, the size of the convolution kernel in the convolutional layer, and / or the number of input channels corresponding to each convolutional layer, etc.

[0164] In other optional embodiments, such as Figure 5 As shown in Figure 2, node clustering can be achieved through unsupervised clustering algorithms. Related technologies such as the KNN algorithm and the Kmeans algorithm rely solely on the feature vectors of each node for clustering. However, the amount of information contained in the nodes themselves is limited, and information such as the relationships between nodes does not play a role in the clustering process.

[0165] In order to include more information in the features of each node and enhance the influence of node relationships during node clustering, nodes can be updated based on their valid neighbor nodes and clustered using the updated nodes. In other words, the step of "clustering each node in each image feature space based on each node and its corresponding valid neighbor nodes to obtain at least one node cluster set under each modality" can specifically include:

[0166] According to each node and its corresponding valid neighbor nodes, the node features of each node are updated to obtain each updated node;

[0167] Based on the similarity between each updated node and other updated nodes in the image feature space, each updated node in each image feature space is clustered to obtain at least one node cluster set under each modality.

[0168] The similarity can be obtained by calculating the cosine distance, Euclidean distance, etc. between each updated node and other updated nodes in the image feature space.

[0169] In some examples, updating each node may be performed by calculating the distance value between each node and its valid neighbor nodes, concatenating the vector composed of each distance value with the feature vector corresponding to the node, and obtaining an updated node.

[0170] In other examples, the weights between each node and its valid neighbor nodes can be calculated based on the distances between each node and its valid neighbor nodes, and then a weighted calculation is performed on the nodes and their valid neighbor nodes based on the weights to obtain updated nodes. That is, the step of "updating the node features of each node based on the valid neighbor nodes corresponding to the node to obtain each updated node" can specifically include:

[0171] According to each node and its corresponding valid neighbor node, the spatial distance between each node and its corresponding valid neighbor node in the corresponding image feature space is calculated respectively;

[0172] Based on each spatial distance, determine the association weight between each node and the corresponding valid neighbor node;

[0173] Node feature update calculation is performed based on the node features corresponding to each node, the node features of the valid neighbor nodes corresponding to each node, and each associated weight to obtain each updated node.

[0174] Specifically, assume that the nodes in the image feature space are X = [x1, x2...x N ]∈R N*D , where N and D refer to the number and dimension of nodes respectively. For each node x i(1≤i≤N), calculate the cosine similarity between the node and other nodes, and filter out its k valid neighbor nodes. The node update process can be expressed by the following formula:

[0175]

[0176] Among them, w i，j Refers to node x i The weight coefficient between it and its k valid neighbor nodes, d i，j Refers to x i and x j After all nodes are updated, the updated nodes will be used for unsupervised clustering.

[0177] Optionally, the inverse of the spatial distance can be used as the weight coefficient to play a role in determining the distance from node x. i The farther away the valid neighbor node is, the lower its weight coefficient is, and so on.

[0178] In some examples, to improve computational efficiency and conserve computing resources, the step of "clustering each updated node in each image feature space based on the similarity between each updated node and other updated nodes in the image feature space to obtain at least one node cluster set under each modality" may specifically include:

[0179] Based on each updated node, in each image feature space, determining a preset number of updated nodes as cluster center nodes;

[0180] Obtain the similarity between each updated node in each image feature space and the center node of each cluster;

[0181] Divide each updated node into the cluster where the corresponding target center node is located, wherein the similarity between the updated node and the corresponding target center node is not less than a preset similarity threshold;

[0182] Based on the updated nodes in each cluster, a new cluster center node is selected for each cluster, and the process returns to the step of obtaining the similarity between each updated node in each image feature space and each cluster center node until a clustering end condition is met;

[0183] The updated nodes in each cluster are respectively determined as a node cluster set.

[0184] The cluster is a set of data objects generated based on a clustering process. In the embodiment of the present invention, the cluster is a set of nodes generated based on a clustering process.

[0185] The cluster center nodes are cluster centers determined when clustering the updated nodes in each image feature space. The number of cluster center nodes can be a fixed number that is preset. The cluster center nodes selected during the first clustering are generally randomly selected. Each time a new cluster center node is selected in a cluster, another updated node can be selected in each cluster as the new cluster center node.

[0186] In another example, the clustering process may be implemented in the form of a clustering model. The clustering model may cluster the updated nodes in each image feature space by adjusting the number of cluster center nodes multiple times to determine the most accurate cluster.

[0187] Optionally, the clustering end condition may be that the updated nodes in each cluster no longer change, or that the cluster center node corresponding to each cluster no longer changes, or that the number of times the step of obtaining the similarity between each updated node in each image feature space and each cluster center node is returned in the clustering process is pre-set, and so on.

[0188] Specifically, the similarity between each updated node and each cluster center node can be obtained by calculating the distance between each updated node and each cluster center node. Alternatively, the similarity between each updated node and each cluster center node can be obtained by calculating the association weight between each updated node and each cluster center node, and so on.

[0189] The preset similarity threshold may be set by technical personnel according to actual needs, and is not limited in this embodiment of the present invention.

[0190] 205. Generate a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set.

[0191] The video content set is a set of video content obtained from the video to be processed based on the video frames corresponding to each node. Specifically, the video content set may include at least one of a video frame set and an audio information set.

[0192] For example, for the node clustering set under the audio modality, since the audio frames in the video correspond to the video frames, when obtaining the audio modality image, it can be obtained by obtaining a set of audio frames corresponding to a certain group of continuous video frames in the video to be processed and processing them. Therefore, there is a natural correspondence between the nodes corresponding to each audio modality image and the video frames in the video to be processed. Through this correspondence, the audio frames corresponding to each node can be obtained from the original video to be processed, and then an audio information set can be generated as a video content set.

[0193] In actual application, taking the example of modalities including face, body, and audio, since face and body are usually in one-to-one correspondence, the video content sets corresponding to the face modality and body modality can be merged based on the nodes under the face modality and body modality. In some embodiments, step 205 can specifically include:

[0194] Based on the nodes in the cluster set of each node under the facial modality, determining the facial modality extraction region corresponding to each facial modality image corresponding to each node in the corresponding video frame;

[0195] Based on the nodes in the node clustering set under the limb modality, determining the limb modality extraction region corresponding to each limb modality image corresponding to each node in the corresponding video frame;

[0196] Calculate the intersection-over-union ratio between each facial modality extraction region and each limb modality extraction region;

[0197] determining a set of video frames of the same object based on the intersection-over-union ratio, the video frames corresponding to each facial modality image, and the video frames corresponding to each limb modality image;

[0198] Determine the audio information set corresponding to each node cluster set according to the video frames corresponding to the nodes in each node cluster set under the audio modality;

[0199] The video frame set and the audio information set are taken as the video content set.

[0200] The Intersection over Union (IoU) function calculates the ratio of the intersection and union of two bounding boxes. IoU measures the relative size of the overlap between two bounding boxes.

[0201] In this embodiment, the coordinates between the facial modality extraction region and the limb modality extraction region are used to calculate the intersection over union (IoU) between the face and limb. The region with the largest IoU is found to identify the matching face and limb. This allows the video frames corresponding to the facial and limb modalities to be merged.

[0202] Specifically, the video frames corresponding to the facial modality and the body modality are merged by directly splicing the video frames corresponding to the facial modality and the video frames corresponding to the body modality according to the order of the video frames in the video to be processed.

[0203] Since the video frames corresponding to some facial modality images and the video frames corresponding to the limb modality images may be the same video frames, in order to avoid duplicate video frames in the obtained video frame set, a union operation can be performed on the video frames corresponding to the facial modality images and the video frames corresponding to the limb modality images to remove duplicate video frames.

[0204] 206. Determine a video content set corresponding to multiple modalities of the same object, fuse the video content set corresponding to the same object, and obtain a processing result of at least one object in the video to be processed.

[0205] In some optional embodiments, if the multimodality only includes facial and body modalities, the IoU (Interference over Union) can be calculated for the video frames corresponding to the facial and body modalities based on the nodes in the facial and body modalities to determine the corresponding video content sets for the same subject in the facial and body modalities. The specific steps for determining the video content sets for the same subject in the facial and body modalities based on IoU are similar to the aforementioned steps for determining the video content sets for the same subject based on IoU, and will not be further described in this embodiment of the present invention.

[0206] In other embodiments, due to environmental factors, video content of the same object may be clustered into different video content sets. In this case, video content sets of different modalities can be matched based on the video time information corresponding to the video content sets. If a match is successful, the video content sets are determined to be the same object. That is, the step of "determining video content sets corresponding to multiple modalities of the same object" may specifically include:

[0207] Determining video time information of each video content set according to video frame sorting information of video frames corresponding to each video content set in the video to be processed;

[0208] Matching is performed based on the video time information of each video content set, and the successfully matched video content sets of different modalities are used as the video content sets corresponding to multiple modalities of the same object.

[0209] The video time information may be a set of time values corresponding to the video content in the video content set, or a time corresponding to the video content in the video content set.

[0210] For example, for a video content set corresponding to a certain facial modality, its video time information may be the video time from 3 minutes 01 seconds to 3 minutes 15 seconds; for a video content set corresponding to another facial modality, its video time information may be the video time from 3 minutes 23 seconds to 3 minutes 40 seconds; and for a video content set corresponding to a certain audio modality, its video time information may be the video time from 2 minutes 59 seconds to 3 minutes 47 seconds.

[0211] At this time, the video time information of the video content sets corresponding to the two facial modalities can be considered to match the video time information of the video content set corresponding to the audio state, and the video content sets corresponding to the two facial modalities and the video content set corresponding to the audio state can be regarded as video content sets corresponding to multiple modalities of the same object.

[0212] Taking the three modalities of face, body and audio as an example, due to the huge differences in external factors, such as lighting, resolution, etc., the video content of the same object may be clustered into different video content sets. The audio features are not affected by these factors and can play a complementary role. At this time, a high-confidence set can be selected from the audio information set, and the audio information of each audio information set can be spliced according to the time dimension to obtain multiple time series. Objects that appear in the same time series in different scenes and at different time points are likely to be the same object. Based on this principle, the video frame sets that are fused with faces and bodies can be re-fused, so that video content sets of different types but with the same identity can be merged.

[0213] The confidence level of an audio information set can be calculated based on the node clusters corresponding to the audio information set. For example, the compactness of the node clusters can be assessed using metrics such as Normalized Mutual Information (NMI) or Pairwise F-score, with the assessment result used as the confidence level. Alternatively, a high-confidence audio information set can be obtained by obtaining audio information corresponding to a group of nodes within the node cluster that are relatively close to the cluster center.

[0214] The embodiment of the present invention experimentally compares the effects of various algorithms on face clustering on the multimodal character clustering dataset VideoX. The compared algorithms include unsupervised clustering algorithms such as K-means, DBSCAN, HDBSCAN, SpectralClustering, Agglomerative Clustering, and supervised clustering algorithms such as GCN-D, L-GCN, GCN-V, and STAR-FC. For the algorithm adopted in this solution, L-GCN is used as the baseline model, and then the traditional k-nearest neighbor composition method is gradually replaced by a composition method based on mutual k-nearest neighbors, and the original features (from VGGFace2) are replaced by features generated by the shared modality feature extraction model. The results are shown in the following table.

[0215] Note: rk-NN refers to the composition method based on mutual k-nearest neighbors, and T refers to the features generated by the shared modality feature extraction model.

[0216]

[0217] In the table below, the first and second parts show the results of unsupervised and supervised clustering algorithms. Unsupervised clustering algorithms often rely on specific data assumptions, which leads to poor performance on VideoX. In general, supervised algorithms can learn the characteristics of data under different data distributions, and therefore perform better than unsupervised algorithms. L-GCN combined with the mutual k-nearest neighbor-based composition method surpasses all other methods in the table. On this basis, replacing the features generated by UMT greatly improves the performance of the algorithm, achieving 77.2F in three indicators. P , 83.6F B , with an excellent performance of 82.5NMI. This experiment strongly demonstrates the great potential of this method in unimodal clustering.

[0218]

[0219] As shown in the table above, for the unsupervised approach, this experiment used the K-means algorithm to cluster the three modal features generated by UMT. For the supervised approach, this experiment trained STAR-FC on the VideoX training set and reported the results on its test set. The experimental results show that the performance of the fused multimodal clustering results is significantly better than that of any single modality, demonstrating that the fusion algorithm invented in this solution can effectively combine the clustering results of each modality.

[0220] As can be seen from the above, an embodiment of the present invention can obtain multiple modal images corresponding to each video frame in the video to be processed based on the video to be processed, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities, perform modal feature extraction on the modal image, obtain image features corresponding to the modal images of each modality, determine the nodes corresponding to the image features under each modality in the image feature space corresponding to each modality, and determine the valid neighbor nodes of each node in each image feature space, wherein the node and its valid neighbor nodes are mutually associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition, perform node clustering on each node in each image feature space according to each node and the valid neighbor nodes corresponding to the node, obtain at least one node cluster set under each modality, generate a video content set corresponding to each node cluster set based on the video frame corresponding to the node in each node cluster set, determine the video content sets corresponding to multiple modalities of the same object, and fuse the video content sets corresponding to the same object to obtain a processing result of at least one object in the video to be processed.

[0221] In the embodiment of the present invention, multimodal features are combined to perform preliminary selection of nodes in the same image feature space under different modalities based on node association conditions, thereby reducing unreliable erroneous connections between nodes during the clustering process. Therefore, the video content set of the same object can be determined from the video, thereby improving the accuracy of processing the video frames.

[0222] The method described in the above embodiments will be further described in detail below with examples.

[0223] In this embodiment, the Figure 1 The system is described in terms of modalities including facial modality, body modality and audio modality.

[0224] like Figure 6 As shown, the video frame processing method of this embodiment may have the following specific process:

[0225] 601. The terminal receives a user's operation for viewing a processed video and sends a request for viewing a processed video to a server. The request for viewing a processed video indicates a video to be processed.

[0226] 602. The server determines a video to be processed and obtains multiple modal images corresponding to each video frame in the video to be processed, wherein the multiple modal images of the same video frame respectively represent video contents corresponding to the video frame in different modalities.

[0227] Specifically, step 602 may include: based on the video to be processed, obtaining each video frame in the video to be processed and audio information corresponding to each video frame;

[0228] Extracting facial images and limb images from each video frame to obtain facial modal images and limb modal images corresponding to each video frame;

[0229] Perform voiceprint analysis on the audio information corresponding to each video frame to obtain the audio modal image corresponding to each audio information;

[0230] The facial modality image, the limb modality image, and the audio modality image corresponding to each video frame are used as multiple modality images corresponding to each video frame.

[0231] 603. The server maps the modal image to the modal feature vector space corresponding to different modalities by using the feature mapping parameters of the shared modal feature extraction model, and obtains the image features corresponding to the modal image of each modality based on the mapping result.

[0232] Among them, the shared modality feature extraction model is trained based on a multimodal image sample set, and the multimodal image sample set includes multiple sample modality images that represent image content under different modalities.

[0233] Specifically, the shared modality feature extraction model can be trained through the following steps:

[0234] Acquire a multimodal image sample set, where the multimodal image sample set includes multiple sample modality images representing image content under different modalities, and each sample modality image is annotated with a reference modality;

[0235] Performing modal feature extraction on each sample modal image in the multimodal image sample set using the shared modal feature extraction model to be trained, and obtaining sample image features corresponding to each sample modal image;

[0236] Through the modality classification model, the features of each sample image are classified into modalities to obtain the training modalities corresponding to the features of each sample image;

[0237] Based on the training modality and reference modality of each sample image feature, the corresponding loss of the shared modality feature extraction network to be trained in each modality is calculated respectively;

[0238] Based on the corresponding losses under each modality, the total loss of the shared modality feature extraction network is calculated as the loss of the shared modality feature extraction network to be trained;

[0239] According to the loss, the model parameters of the shared modality feature extraction model to be trained are adjusted to obtain the trained shared modality feature extraction network.

[0240] 604. The server clusters each node in each image feature space according to each node and its corresponding valid neighboring nodes, and obtains at least one node cluster set under each modality.

[0241] In the embodiment of the present invention, nodes may be clustered using a supervised graph clustering algorithm or an unsupervised graph clustering algorithm.

[0242] 605. The server calculates an intersection-over-union ratio between each facial modality extraction region and each limb modality extraction region.

[0243] Specifically, the server can determine the facial modality extraction area corresponding to each facial modality image of each node in the corresponding video frame based on the nodes in the node clustering set under the facial modality, and determine the limb modality extraction area corresponding to each limb modality image of each node in the corresponding video frame based on the nodes in the node clustering set under the limb modality.

[0244] 606. The server determines a set of video frames of the same object based on the intersection-over-union ratio, the video frames corresponding to each facial modality image, and the video frames corresponding to each limb modality image.

[0245] In this embodiment, the coordinates between the facial modality extraction region and the limb modality extraction region are used to calculate the intersection over union (IoU) between the face and limb. The region with the largest IoU is found to obtain the matching face and limb. This allows the video content sets corresponding to the facial and limb modalities to be merged.

[0246] 607. The server determines an audio information set corresponding to each node cluster set according to the video frames corresponding to the nodes in each node cluster set in the audio mode.

[0247] Taking the three modalities of face, body and audio as an example, due to the huge differences in external factors, such as lighting, resolution, etc., the video content of the same object may be clustered into different video content sets. The audio features are not affected by these factors and can play a complementary role. At this time, a high-confidence set can be selected from the audio information set, and the audio information of each audio information set can be spliced according to the time dimension to obtain multiple time series. Objects that appear in the same time series in different scenes and at different time points are likely to be the same object. Based on this principle, the video frame sets that are fused with faces and bodies can be re-fused, so that video content sets of different types but with the same identity can be merged.

[0248] 608. The server merges the video content set corresponding to the same object to obtain a processing result of at least one object in the video to be processed.

[0249] In some optional examples, the processing results of the video to be processed can directly determine the identity of the object corresponding to each processing result. For example, the video to be processed may include only one object, and the identity of the object is known when the video to be processed is processed, so the identity of the object in the processing result can be directly determined.

[0250] Alternatively, object feature information of each object in the video to be processed can be obtained, and the processing results can be matched with the object features of each object to determine the identity of the object corresponding to each processing result. For example, after obtaining the processing results, the server can search for relevant information about the object, such as images, from the Internet to obtain the object feature information, etc.

[0251] To verify the effectiveness of the effective neighbor node algorithm, this experiment retrained and tested several baseline models on VideoX face images and two other face clustering datasets—CASIA and IJB-B. The results are shown in the table below.

[0252]

[0253] As can be seen from the table: (1) For unsupervised algorithms, the effective neighbor node algorithm brings significant improvements in most cases, which to some extent proves that it improves the representation used for clustering. (2) The L-GCN algorithm uses a graph convolutional neural network to model the contextual information of the subgraph and the confidence of the predicted nodes. The effective neighbor node algorithm helps L-GCN build a more sparse and reliable subgraph, thereby reducing computational overhead and improving the performance of the algorithm. (3) The STAR-FC algorithm aims to cluster large-scale face data through graph cropping and graph updating. The effective neighbor node algorithm helps STAR-FC filter out the noise nodes in the subgraph, thereby achieving better results than the traditional k-nearest neighbor composition method. The above experimental results show that the effective neighbor node algorithm can bring significant performance improvements to both supervised and unsupervised clustering methods.

[0254] From the above, it can be seen that the embodiments of the present invention can combine multimodal features and perform preliminary selection of nodes in the same image feature space under different modalities based on node association conditions, thereby reducing unreliable erroneous connections between nodes during the clustering process. Therefore, the video content set of the same object can be determined from the video, thereby improving the accuracy of processing the video frames.

[0255] In order to better implement the above method, accordingly, an embodiment of the present invention further provides a video frame processing device.

[0256] refer to Figure 7 , the device comprises:

[0257] The image acquisition unit 701 may be configured to acquire, based on the video to be processed, multiple modal images corresponding to each video frame in the video to be processed, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities;

[0258] The feature extraction unit 702 may be configured to extract modal features from the modal images, obtain image features corresponding to the modal images of each modality, and determine nodes corresponding to the image features under each modality in the image feature space corresponding to each modality;

[0259] The node determination unit 703 may be configured to determine the valid neighbor nodes of each node in each image feature space, wherein a node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must satisfy a preset node association condition;

[0260] The node clustering unit 704 may be configured to cluster the nodes in each image feature space according to the nodes and their corresponding valid neighbor nodes, to obtain at least one node cluster set under each modality;

[0261] The set generation unit 705 may be configured to generate a video content set corresponding to each node cluster set based on the video frames corresponding to the nodes in each node cluster set;

[0262] The video processing unit 706 can be used to determine the video content sets corresponding to multiple modalities of the same object based on the video time overlap of each video content set, fuse the video content sets corresponding to the same object, and obtain the processing result of at least one object in the video to be processed.

[0263] In some optional embodiments, the node clustering unit 704 may be configured to generate, based on each node and its corresponding valid neighbor nodes, a subgraph with each node in each image feature space as a central node and its corresponding valid neighbor nodes as other nodes;

[0264] Obtain the graph clustering network corresponding to each modality;

[0265] For each subgraph, graph clustering processing is performed through the graph clustering network of the corresponding modality to obtain at least one node cluster set under each modality.

[0266] In some optional embodiments, the node clustering unit 704 may be configured to update node features of each node based on each node and its corresponding valid neighbor nodes to obtain updated nodes.

[0267] Based on the similarity between each updated node and other updated nodes in the image feature space, each updated node in each image feature space is clustered to obtain at least one node cluster set under each modality.

[0268] In some optional embodiments, the node clustering unit 704 may be configured to calculate, based on each node and the valid neighboring nodes corresponding to the node, the spatial distance between each node and the corresponding valid neighboring node in the corresponding image feature space;

[0269] Based on each spatial distance, determine the association weight between each node and the corresponding valid neighbor node;

[0270] Node feature update calculation is performed based on the node features corresponding to each node, the node features of the valid neighbor nodes corresponding to each node, and each associated weight to obtain each updated node.

[0271] In some optional embodiments, the node clustering unit may be configured to determine, based on each updated node, in each image feature space, a preset number of updated nodes as cluster center nodes;

[0272] Obtain the similarity between each updated node in each image feature space and the center node of each cluster;

[0273] Divide each updated node into the cluster where the corresponding target center node is located, wherein the similarity between the updated node and the corresponding target center node is not less than a preset similarity threshold;

[0274] Based on the updated nodes in each cluster, a new cluster center node is selected for each cluster, and the process returns to the step of obtaining the similarity between each updated node in each image feature space and each cluster center node until a clustering end condition is met;

[0275] The updated nodes in each cluster are respectively determined as a node cluster set.

[0276] In some optional embodiments, such as Figure 8 As shown, the video frame processing device provided by the embodiment of the present invention may further include a graph network training unit 707, which can be used to perform graph clustering processing on the sample subgraphs in the feature space of each sample image through the graph clustering network to be trained corresponding to each modality, to obtain at least one training node cluster set under each modality, wherein each sample subgraph is annotated with a reference cluster set result;

[0277] Based on the training node clustering set and reference clustering set results under each modality, the loss of the graph clustering network to be trained corresponding to each modality is calculated respectively;

[0278] According to the loss, the network parameters of each graph clustering network to be trained are adjusted to obtain the trained graph clustering network corresponding to each modality.

[0279] In some optional embodiments, the feature extraction unit 702 can be used to map the modal image into the modal feature vector space corresponding to different modalities through the feature mapping parameters of the shared modal feature extraction model, and obtain the image features corresponding to the modal image of each modality based on the mapping results. The shared modal feature extraction model is trained based on a multimodal image sample set, and the multimodal image sample set may include multiple sample modal images that represent the image content under different modalities.

[0280] In some optional embodiments, the video frame processing apparatus provided by the embodiments of the present invention may further include a model training unit 708, which may be used to obtain a multimodal image sample set. The multimodal image sample set may include multiple sample modality images representing image content under different modalities, and each sample modality image is annotated with a reference modality;

[0281] Performing modal feature extraction on each sample modal image in the multimodal image sample set using the shared modal feature extraction model to be trained, and obtaining sample image features corresponding to each sample modal image;

[0282] Through the modality classification model, the features of each sample image are classified into modalities to obtain the training modalities corresponding to the features of each sample image;

[0283] Based on the training modality and reference modality of each sample image feature, calculate the loss of the shared modality feature extraction network to be trained;

[0284] According to the loss, the model parameters of the shared modality feature extraction model to be trained are adjusted to obtain the trained shared modality feature extraction network.

[0285] In some optional embodiments, the model training unit may be configured to calculate the corresponding loss of the shared modality feature extraction network to be trained under each modality based on the training modality and the reference modality of each sample image feature;

[0286] Based on the corresponding losses under each modality, the total loss of the shared modality feature extraction network is calculated as the loss of the shared modality feature extraction network to be trained.

[0287] In some optional embodiments, the video processing unit 706 may be configured to determine video time information of each video content set based on video frame sorting information of the video frames corresponding to each video content set in the video to be processed;

[0288] Matching is performed based on the video time information of each video content set, and the successfully matched video content sets of different modalities are used as the video content sets corresponding to multiple modalities of the same object.

[0289] In some optional embodiments, the image acquisition unit may be configured to acquire, based on the video to be processed, each video frame in the video to be processed and audio information corresponding to each video frame;

[0290] Extracting facial images and limb images from each video frame to obtain facial modal images and limb modal images corresponding to each video frame;

[0291] Perform voiceprint analysis on the audio information corresponding to each video frame to obtain the audio modal image corresponding to each audio information;

[0292] The facial modality image, the limb modality image, and the audio modality image corresponding to each video frame are used as multiple modality images corresponding to each video frame.

[0293] In some optional embodiments, the set generation unit 705 may be configured to cluster nodes in the set based on each node under the facial modality, and determine a facial modality extraction region corresponding to each facial modality image corresponding to each node in the corresponding video frame;

[0294] Based on the nodes in the node clustering set under the limb modality, determining the limb modality extraction region corresponding to each limb modality image corresponding to each node in the corresponding video frame;

[0295] Calculate the intersection-over-union ratio between each facial modality extraction region and each limb modality extraction region;

[0296] determining a set of video frames of the same object based on the intersection-over-union ratio, the video frames corresponding to each facial modality image, and the video frames corresponding to each limb modality image;

[0297] Determine the audio information set corresponding to each node cluster set according to the video frames corresponding to the nodes in each node cluster set under the audio modality;

[0298] The video frame set and the audio information set are taken as the video content set.

[0299] From the above, it can be seen that through the video frame processing device, multimodal features can be combined to perform preliminary selection of nodes in the same image feature space under different modalities based on node association conditions, thereby reducing unreliable erroneous connections between nodes during the clustering process. Therefore, the video content set of the same object can be determined from the video, thereby improving the accuracy of video frame processing.

[0300] In addition, an embodiment of the present invention further provides an electronic device, which may be a terminal or a server, etc. Figure 9 , which shows a schematic structural diagram of an electronic device involved in an embodiment of the present invention, specifically:

[0301] The electronic device may include components such as a radio frequency (RF) circuit 901, a memory 902 including one or more computer-readable storage media, an input unit 903, a display unit 904, a sensor 905, an audio circuit 906, a wireless fidelity (WiFi) module 907, a processor 908 including one or more processing cores, and a power supply 909. It will be understood by those skilled in the art that Figure 9 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0302] The RF circuit 901 can be used to receive and send signals during information transmission or calls. In particular, after receiving downlink information from the base station, it is handed over to one or more processors 908 for processing; in addition, uplink data is sent to the base station. Generally, the RF circuit 901 includes but is not limited to an antenna, at least one amplifier, a tuner, one or more oscillators, a subscriber identity module (SIM) card, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 901 can also communicate with the network and other devices via wireless communication. Wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc.

[0303] The memory 902 can be used to store software programs and modules, and the processor 908 executes various functional applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory 902 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 902 may also include a memory controller to provide the processor 908 and the input unit 903 with access to the memory 902.

[0304] The input unit 903 can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical, or trackball signal input related to user settings and function control. Specifically, in one embodiment, the input unit 903 may include a touch-sensitive surface and other input devices. A touch-sensitive surface, also known as a touch display or touchpad, can detect user touch operations on or near it (for example, operations performed by a user using a finger, stylus, or any other suitable object or accessory on or near the touch-sensitive surface) and drive corresponding connected devices according to a pre-set program. Optionally, the touch-sensitive surface may include a touch detection device and a touch controller. The touch detection device detects the user's touch direction and detects signals generated by the touch operation, transmitting the signals to the touch controller. The touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 908. It can also receive and execute commands from the processor 908. In addition, touch-sensitive surfaces can be implemented using various types, such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface, the input unit 903 may also include other input devices. Specifically, other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0305] The display unit 904 can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 904 may include a display panel. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like. Furthermore, the touch-sensitive surface may cover the display panel, and when the touch-sensitive surface detects a touch operation on or near it, it is transmitted to the processor 908 to determine the type of touch event, and then the processor 908 provides corresponding visual output on the display panel according to the type of touch event. Although in Figure 9 In the embodiment, the touch-sensitive surface and the display panel are used as two independent components to realize input and output functions, but in some embodiments, the touch-sensitive surface and the display panel can be integrated to realize input and output functions.

[0306] The electronic device may also include at least one sensor 905, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the electronic device is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the electronic device can also be configured with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be described here.

[0307] Audio circuit 906, speakers, and microphones provide an audio interface between the user and the electronic device. Audio circuit 906 converts received audio data into electrical signals and transmits them to the speaker, which then converts them into sound signals for output. The microphone, on the other hand, converts collected sound signals into electrical signals, which are then received by audio circuit 906 and converted into audio data. The audio data is then processed by output processor 908 and transmitted via RF circuit 901 to, for example, another electronic device. Alternatively, the audio data is output to memory 902 for further processing. Audio circuit 906 may also include an earphone jack to allow communication between external headphones and the electronic device.

[0308] WiFi is a short-range wireless transmission technology. Electronic devices can help users send and receive emails, browse web pages, and access streaming media through WiFi modules 907. It provides users with wireless broadband Internet access. Figure 9 A WiFi module 907 is shown, but it is understandable that it is not an essential component of the electronic device and can be omitted as needed without changing the essence of the invention.

[0309] The processor 908 is the control center of the electronic device. It connects all parts of the phone using various interfaces and circuits. It executes the various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 902 and accessing data stored in the memory 902. Optionally, the processor 908 may include one or more processing cores. Preferably, the processor 908 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into the processor 908.

[0310] The electronic device also includes a power supply 909 (e.g., a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 908 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 909 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0311] Although not shown, the electronic device may also include a camera, a Bluetooth module, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 908 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 908 will run the application programs stored in the memory 902 to implement various functions as follows:

[0312] Based on the video to be processed, multiple modal images corresponding to each video frame in the video to be processed are obtained, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities;

[0313] Perform modal feature extraction on the modal image to obtain the image features corresponding to the modal image of each modality, and determine the nodes corresponding to the image features under each modality in the image feature space corresponding to each modality;

[0314] Determine the valid neighbor nodes of each node in each image feature space, wherein the node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition;

[0315] Clustering each node in each image feature space according to each node and its corresponding valid neighboring nodes to obtain at least one node clustering set under each modality;

[0316] Based on the video frames corresponding to the nodes in each node cluster set, generating a video content set corresponding to each node cluster set;

[0317] According to the video time overlap of each video content set, the video content sets corresponding to multiple modalities of the same object are determined, and the video content sets corresponding to the same object are fused to obtain the processing result of at least one object in the video to be processed.

[0318] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0319] To this end, an embodiment of the present invention provides a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute the steps of any of the video frame processing methods provided in the embodiments of the present invention. For example, the instructions can execute the following steps:

[0320] Based on the video to be processed, multiple modal images corresponding to each video frame in the video to be processed are obtained, wherein the multiple modal images of the same video frame respectively represent the video content corresponding to the video frame in different modalities;

[0321] Perform modal feature extraction on the modal image to obtain the image features corresponding to the modal image of each modality, and determine the nodes corresponding to the image features under each modality in the image feature space corresponding to each modality;

[0322] Determine the valid neighbor nodes of each node in each image feature space, wherein the node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition;

[0323] Clustering each node in each image feature space according to each node and its corresponding valid neighboring nodes to obtain at least one node clustering set under each modality;

[0324] Based on the video frames corresponding to the nodes in each node cluster set, generating a video content set corresponding to each node cluster set;

[0325] According to the video time overlap of each video content set, the video content sets corresponding to multiple modalities of the same object are determined, and the video content sets corresponding to the same object are fused to obtain the processing result of at least one object in the video to be processed.

[0326] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0327] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0328] Since the instructions stored in the computer-readable storage medium can execute the steps in any video frame processing method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any video frame processing method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0329] According to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the methods provided in various optional implementations of the above embodiments.

[0330] The video frame processing method, device, electronic device, storage medium and program product provided in the embodiments of the present invention are introduced in detail above. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A video frame processing method, characterized in that: include: Based on a video to be processed, obtaining multiple modal images corresponding to each video frame in the video to be processed, wherein the multiple modal images of the same video frame respectively represent video content corresponding to the video frame in different modalities, and the multiple modal images include facial modal images, body modal images, and audio modal images; Performing modal feature extraction on the modal images to obtain image features corresponding to the modal images of each modality, and determining nodes corresponding to the image features under each modality in the image feature space corresponding to each modality; Determining valid neighbor nodes of each node in each of the image feature spaces, wherein a node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, the distance between the node and the other node must meet a preset node association condition; performing node clustering on each of the nodes in each of the image feature spaces according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality; Determining, based on the nodes in the node cluster set under the facial modality, a facial modality extraction region corresponding to each facial modality image corresponding to each node in the corresponding video frame; Based on the nodes in each of the node cluster sets under the limb modality, determining a limb modality extraction region corresponding to each of the limb modality images corresponding to each of the nodes in the corresponding video frame; Calculating an intersection-over-union ratio between each of the facial modality extraction regions and each of the limb modality extraction regions; Determining a set of video frames of the same object based on the intersection-over-union ratio, the video frames corresponding to the facial modality images, and the video frames corresponding to the limb modality images; Determining, based on the video frames corresponding to the nodes in each of the node cluster sets in the audio modality, an audio information set corresponding to each of the node cluster sets; Taking the video frame set and the audio information set as a video content set; A video content set corresponding to multiple modalities of the same object is determined, and the video content set corresponding to the same object is fused to obtain a processing result of at least one object in the video to be processed.

2. The video frame processing method according to claim 1, wherein: The node clustering is performed on each node in each image feature space according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality, including: According to each node and the valid neighbor nodes corresponding to the node, a subgraph is generated with each node in the image feature space as a central node and the valid neighbor nodes corresponding to the node as other nodes; Obtain the graph clustering network corresponding to each modality; Each of the subgraphs is subjected to graph clustering processing using the graph clustering network of the corresponding modality to obtain at least one node clustering set under each modality.

3. The video frame processing method according to claim 1, wherein: The node clustering is performed on each node in each image feature space according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality, including: According to each node and the valid neighboring nodes corresponding to the node, node features of each node are updated to obtain updated nodes; Based on the similarity between each updated node and other updated nodes in the image feature space, node clustering is performed on each updated node in each image feature space to obtain at least one node cluster set under each modality.

4. The video frame processing method according to claim 3, wherein: The updating of node features of each node according to each node and the valid neighboring nodes corresponding to the node to obtain each updated node includes: Calculating the spatial distance between each node and the corresponding valid neighbor node in the corresponding image feature space according to each node and the corresponding valid neighbor node; Determining, based on each of the spatial distances, an association weight between each of the nodes and the corresponding valid neighbor node; Node feature update calculation is performed based on the node features corresponding to each of the nodes, the node features of the valid neighbor nodes corresponding to each of the nodes, and each of the association weights to obtain each updated node.

5. The video frame processing method according to claim 3, wherein: The step of clustering the updated nodes in each image feature space based on the similarity between each updated node and other updated nodes in the image feature space to obtain at least one node cluster set under each modality includes: Based on each of the updated nodes, in each of the image feature spaces, determining a preset number of updated nodes as cluster center nodes; Obtaining the similarity between each updated node and each cluster center node in each image feature space; Dividing each of the updated nodes into a cluster where the corresponding target central node is located, wherein the similarity between the updated node and the corresponding target central node is not less than a preset similarity threshold; Based on the updated nodes in each cluster, select a new cluster center node of each cluster, and return to the step of obtaining the similarity between each updated node in each image feature space and each cluster center node until a clustering end condition is met; The updated nodes in each of the clusters are respectively determined as a node cluster set.

6. The video frame processing method according to claim 2, wherein: Before obtaining the graph clustering network corresponding to each modality, the method further includes: Performing graph clustering on the sample subgraphs in the feature space of each sample image using the graph clustering network to be trained corresponding to each modality to obtain at least one training node cluster set under each modality, wherein each sample subgraph is annotated with a reference cluster set result; Based on the training node cluster set and the reference cluster set results under each modality, respectively calculating the loss of the graph clustering network to be trained corresponding to each modality; According to the loss, the network parameters of each graph clustering network to be trained are adjusted to obtain a trained graph clustering network corresponding to each modality.

7. The video frame processing method according to any one of claims 1 to 6, characterized in that: The extracting modal features of the modal images to obtain image features corresponding to the modal images of each modality includes: The modal image is mapped into the modal feature vector space corresponding to different modalities by using the feature mapping parameters of the shared modal feature extraction model, and the image features corresponding to the modal images of each modality are obtained based on the mapping results. The shared modal feature extraction model is trained based on a multimodal image sample set, and the multimodal image sample set includes multiple sample modal images that represent the image content under different modalities.

8. The video frame processing method according to claim 7, wherein: Before mapping the modal image into a modal feature vector space corresponding to a different modality by using the feature mapping parameters of the shared modal feature extraction model, the method further includes: Acquire a multimodal image sample set, wherein the multimodal image sample set includes a plurality of sample modality images representing image content under different modalities, each of the sample modality images being annotated with a reference modality; Performing modal feature extraction on each of the sample modal images in the multimodal image sample set using the shared modal feature extraction model to be trained to obtain sample image features corresponding to each of the sample modal images; Performing modality classification on each of the sample image features using a modality classification model to obtain a training modality corresponding to each of the sample image features; Calculating the loss of the shared modality feature extraction network to be trained based on the training modality and the reference modality of each of the sample image features; According to the loss, the model parameters of the shared modality feature extraction model to be trained are adjusted to obtain a trained shared modality feature extraction network.

9. The video frame processing method according to claim 8, wherein: The calculating the loss of the shared modality feature extraction network to be trained based on the training modality and the reference modality of each of the sample image features includes: Based on the training modality and the reference modality of each of the sample image features, respectively calculating the corresponding loss of the shared modality feature extraction network to be trained under each modality; Based on the corresponding losses under each modality, the total loss of the shared modality feature extraction network is calculated as the loss of the shared modality feature extraction network to be trained.

10. The video frame processing method according to claim 1, wherein: The determining of a video content set corresponding to multiple modalities of the same object includes: determining video time information of each video content set according to video frame sorting information of the video frames corresponding to each video content set in the video to be processed; Matching is performed according to the video time information of each of the video content sets, and the successfully matched video content sets of different modalities are used as the video content sets corresponding to multiple modalities of the same object.

11. The video frame processing method according to any one of claims 1 to 10, characterized in that: The acquiring, based on the video to be processed, a plurality of modal images corresponding to each video frame in the video to be processed includes: Based on the video to be processed, obtaining each video frame in the video to be processed and audio information corresponding to each video frame; Extracting facial images and limb images from each of the video frames to obtain facial modal images and limb modal images corresponding to each of the video frames; Performing voiceprint analysis on the audio information corresponding to each of the video frames to obtain an audio modal image corresponding to each of the audio information; The facial modality image, the limb modality image, and the audio modality image corresponding to each of the video frames are used as multiple modality images corresponding to each video frame.

12. A video frame processing device, characterized in that: include: an image acquisition unit, configured to acquire, based on a video to be processed, a plurality of modal images corresponding to each video frame in the video to be processed, wherein the plurality of modal images of the same video frame respectively represent video content corresponding to the video frame in different modalities, and the plurality of modal images include facial modal images, body modal images, and audio modal images; a feature extraction unit, configured to extract modal features from the modal images to obtain image features corresponding to the modal images of each modality, and determine nodes corresponding to the image features under each modality in the image feature space corresponding to each modality; a node determination unit, configured to determine, in each of the image feature spaces, valid neighbor nodes of each node, wherein a node and its valid neighbor nodes are associated nodes, and if a node is an associated node of another node, a distance between the node and the other node must satisfy a preset node association condition; a node clustering unit, configured to perform node clustering on each of the nodes in each of the image feature spaces according to each node and the valid neighboring nodes corresponding to the node, to obtain at least one node cluster set under each modality; a set generation unit for determining, based on the nodes in each of the node cluster sets under facial modality, a facial modality extraction region corresponding to each of the facial modality images corresponding to each of the nodes in the corresponding video frame; determining, based on the nodes in each of the node cluster sets under limb modality, a limb modality extraction region corresponding to each of the limb modality images corresponding to each of the nodes in the corresponding video frame; calculating an intersection-over-union (IoU) between each of the facial modality extraction regions and each of the limb modality extraction regions; determining, based on the IoU and the video frames corresponding to each of the facial modality images and the video frames corresponding to each of the limb modality images, a set of video frames of the same object; determining, based on the video frames corresponding to the nodes in each of the node cluster sets under audio modality, a set of audio information corresponding to each of the node cluster sets; and using the video frame set and the audio information set as a video content set; The video processing unit is used to determine a video content set corresponding to multiple modalities of the same object, fuse the video content set corresponding to the same object, and obtain a processing result of at least one object in the video to be processed.

13. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application, and the processor is used to run the application in the memory to execute the steps in the video frame processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the video frame processing method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the video frame processing method according to any one of claims 1 to 11 are implemented.

Images