Multimedia data processing method, device, server and storage medium

Abstract

The present invention discloses a multimedia data processing method, device, server, and storage medium. The method includes: obtaining content similarity between multimedia data in a target multimedia data set; constructing a multimedia data relationship network with multimedia data as nodes and content similarity as edges; extracting connected components in the multimedia data relationship network and determining a node sequence of the connected components; traversing the nodes in the node sequence starting from the tail node, and during the traversal, determining the parent node corresponding to the node that meets a first preset condition as a first key node to obtain a first key node set; and determining a second key node set, i.e., a key multimedia data set, based on the first key node in the first key node set and the neighboring nodes of the first key node in the multimedia data relationship network. The key multimedia data set of the present invention can represent the target multimedia data set, thereby reducing the computational overhead of related services and improving service processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of multimedia technology, and in particular to a multimedia data processing method, device, server and storage medium. Background Art

[0002] With the development of the video fragmentation trend in recent years, a large number of fragmented short videos or small videos edited from long videos are flooding the video database. Due to their short length, these videos may be very different from their original videos in semantics, but visually they are very similar to the original videos in some segments. As a result, there are a large number of videos with similar visual content in the video database, which greatly increases the computational overhead of business processing based on the video database, such as video recommendation, video copyright detection, and video deduplication, and reduces business processing efficiency. Summary of the Invention

[0003] In order to solve the problems of the prior art, the embodiments of the present invention provide a multimedia data processing method, device, server and storage medium. The technical solution is as follows:

[0004] In one aspect, a method for processing multimedia data is provided, the method comprising:

[0005] Obtaining content similarity between multimedia data in a target multimedia data set;

[0006] Constructing a multimedia data relationship network corresponding to the target multimedia data set by taking the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges;

[0007] Extracting connected components from the multimedia data relationship network and determining a node sequence corresponding to the connected components; the connected components include at least one node connected through the edge;

[0008] Traversing the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determining the parent node corresponding to the node that meets the first preset condition as the first key node to obtain a first key node set;

[0009] A second key node set is determined based on the first key node in the first key node set and the neighbor nodes of the first key node in the multimedia data relationship network; the multimedia data corresponding to the second key node in the second key node set is used as the key multimedia data of the target multimedia data set.

[0010] In another aspect, a multimedia data processing device is provided, the device comprising:

[0011] A first acquisition module is used to obtain content similarity between multimedia data in a target multimedia data set;

[0012] A first construction module is configured to construct a multimedia data relationship network corresponding to the target multimedia data set, using the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges;

[0013] a node sequence determination module, configured to extract connected components from the multimedia data relationship network and determine a node sequence corresponding to the connected components; the connected components include at least one node connected via the edge;

[0014] a first determining module, configured to traverse the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determine the parent node corresponding to the node that meets the first preset condition as the first key node, to obtain a first key node set;

[0015] The second determination module is used to determine the second key node set based on the first key node in the first key node set and the neighbor nodes of the first key node in the multimedia data relationship network; the multimedia data corresponding to the second key node in the second key node set is used as the key multimedia data of the target multimedia data set.

[0016] Optionally, the first determining module includes a first traversal module, and the first traversal module is used to traverse the nodes in the node sequence starting from the tail node of the node sequence;

[0017] The first traversal module includes:

[0018] A third determining module is used to determine the parent node corresponding to the node according to the position of the node in the node sequence;

[0019] A first judgment module, configured to judge whether the node and the parent node meet a first preset condition;

[0020] a fourth determining module, configured to, when the node satisfies the first preset condition and the parent node satisfies the first preset condition, determine that the parent node is a first key node and add the parent node to a first key node set;

[0021] a fifth determining module, configured to determine nodes connected to the parent node in the connected components corresponding to the node sequence, and mark the nodes connected to the parent node as covered nodes;

[0022] The first traversal termination module is configured to terminate the traversal of the nodes in the node sequence when the covered nodes in the connected component corresponding to the node sequence meet a second preset condition.

[0023] Optionally, the first judgment module includes:

[0024] A first judgment submodule, configured to judge whether there is a first key node matching the node in the first key node set;

[0025] a second judgment submodule, configured to judge whether there is a first key node matching the parent node in the first key node set when the result of the judgment by the first judgment submodule is negative;

[0026] The sixth determination module is configured to determine, when the result of the determination by the second determination submodule is no, whether the node satisfies the first preset condition and whether the parent node satisfies the first preset condition.

[0027] Optionally, the first traversal module further includes:

[0028] A second judgment module is used to judge whether all nodes in the connected components corresponding to the node sequence are marked as covered nodes;

[0029] The seventh determination module is configured to determine, when the result of the determination by the second determination module is yes, whether the covered nodes in the connected components corresponding to the node sequence meet a second preset condition.

[0030] Optionally, the second determining module includes:

[0031] A first neighbor node determination module, configured to determine first-order neighbor nodes of a first key node in the first key node set in the multimedia data relationship network;

[0032] A first acquisition module is configured to acquire the content similarity between the first key node and the corresponding first-order neighbor node to obtain a first content similarity;

[0033] an extended node determination module, configured to determine a first-order neighbor node whose first content similarity is less than a first similarity threshold, and obtain an extended key node;

[0034] An eighth determining module is configured to determine a second key node set according to the first key node in the first key node set and the extended key node.

[0035] Optionally, the eighth determining module includes:

[0036] A ninth determining module is configured to use the first key node and the extended key node as third key nodes to obtain a third key node set; and determine content similarity between the third key nodes in the third key node set;

[0037] A second building module is used to build a key node relationship network based on the third key nodes in the third key node set and the content similarity between the third key nodes;

[0038] A key node sequence determination module, configured to determine a key node sequence based on the key node relationship network;

[0039] a tenth determining module, configured to traverse the key nodes in the key node sequence starting from the tail key node of the key node sequence, and when traversing the key nodes in the key node sequence, determine the parent key node corresponding to the key node that meets the third preset condition as a fourth key node, to obtain a fourth key node set;

[0040] An eleventh determining module is configured to determine a second key node set based on a fourth key node in the fourth key node set and neighboring key nodes of the fourth key node in the key node relationship network.

[0041] Optionally, the eleventh determining module includes:

[0042] A second neighbor node determination module is used to determine a first-order neighbor key node of the fourth key node in the fourth key node set in the key node relationship network;

[0043] A second acquisition module is configured to acquire the content similarity between the fourth key node and the first-order neighbor key node to obtain a second content similarity;

[0044] a target neighbor node determination module, configured to determine a first-order neighbor key node whose second content similarity is less than a second similarity threshold, and obtain a target neighbor key node;

[0045] The twelfth determining module is configured to use the fourth key node in the fourth key node set and the target neighbor key node as second key nodes to obtain a second key node set.

[0046] On the other hand, a server is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the above-mentioned multimedia data processing method.

[0047] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored. The at least one instruction or the at least one program is loaded and executed by a processor to implement the multimedia data processing method as described above.

[0048] The embodiment of the present invention constructs a multimedia data relationship network corresponding to a target multimedia data set based on the content similarity between multimedia data, determines a node sequence corresponding to the connected components in the multimedia data relationship network based on the connected components, and then traverses the nodes in the node sequence starting from the tail node of the node sequence. During the traversal, the parent node corresponding to the node that meets the first preset condition is determined as the first key node to obtain a first key node set, and a second key node set is determined based on the first key node in the first key node set and the neighbor nodes of the first key node in the multimedia data relationship network. The multimedia data corresponding to the second key node in the second key node set is used as the key multimedia data of the multimedia data set, thereby obtaining a key multimedia data set. Since the key multimedia data set obtained by the above technical solution can not only represent all the content appearing in the target multimedia data set, but also greatly reduce the duplicate content between the key multimedia data, in actual applications, corresponding business processing can be directly carried out based on the key multimedia data set, such as using the key video set as a video pool for video recommendation, video copyright detection, and video duplication elimination, which can greatly reduce the computational overhead of these business processing and improve business processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0050] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of a multimedia data processing method provided by an embodiment of the present invention;

[0052] Figure 3 1 is a flow chart of a video data processing method provided by an embodiment of the present invention;

[0053] Figure 4 is a schematic diagram of a connected component provided by an embodiment of the present invention;

[0054] Figure 5It is a local topology diagram of the video relationship network provided by an embodiment of the present invention;

[0055] Figure 6 is a flow chart of another video data processing method provided by an embodiment of the present invention;

[0056] Figure 7 is a flow chart of another video data processing method provided by an embodiment of the present invention;

[0057] Figure 8 is a structural diagram of a multimedia data processing device provided by an embodiment of the present invention;

[0058] Figure 9 This is a hardware structure diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0060] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way are interchangeable where appropriate so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0061] See also Figure 1 , which is a schematic diagram of an implementation environment of a multimedia data processing method provided by an embodiment of the present invention. The implementation environment may include multiple servers 100. Any of the multiple servers 100 can be a node device in the blockchain system, which can share the acquired and generated information with other node devices in the blockchain system, thereby realizing information sharing among multiple node devices.

[0062] Optionally, the multiple servers 100 may include: a business server, a multimedia data processing server, and a database. The multimedia data may include, but is not limited to, video data, image data, audio data, etc. The business server is used to provide business services involving multimedia data, such as, but not limited to, video recommendation services, video copyright detection services, and video duplicate elimination services. The database is used to store the multimedia data. Optionally, the database may group the multimedia data according to its source platform, with each group of multimedia data constituting a multimedia data set.

[0063] The multimedia data processing server is used to process the multimedia data stored in the database to obtain a key multimedia data set corresponding to the corresponding multimedia data set, and send the key multimedia data set to the corresponding service server, which then provides the corresponding service based on the key multimedia data set. The multimedia data processing server can be one or more multimedia data processing servers. When there are multiple multimedia data processing servers, at least two multimedia data processing servers are used to provide different services, and / or at least two multimedia data processing servers are used to provide the same service, such as providing the same service in a load balancing manner or providing the same service in a primary server and mirror server manner. This is not limited in the embodiments of the present invention.

[0064] The multiple servers 100 may be connected via a wireless network or a wired network, optionally using standard communication technologies and / or protocols. The network is typically the Internet, but may be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network, or any combination of a virtual private network.

[0065] See also Figure 2 , which is a flow chart of a multimedia data processing method provided by an embodiment of the present invention, which can be applied to Figure 1 Implementation environment. It should be noted that this specification provides method operation steps as described in the embodiments or flow charts, but may include more or fewer operation steps based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps, and does not represent the only execution order. When the actual system or product is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include:

[0066] S201: Obtain content similarity between multimedia data in a target multimedia data set.

[0067] The target multimedia data set is a multimedia data set to be processed, and the multimedia data set to be processed can be any multimedia data set in the database. The content similarity between multimedia data refers to the similarity between the multimedia data in terms of content.

[0068] S203 : Constructing a multimedia data relationship network corresponding to the target multimedia data set by using the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges.

[0069] In the embodiments of this specification, based on graph network embedding technology, a multimedia relationship network corresponding to the target multimedia data set is constructed with the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges.

[0070] Among them, graph embedding is currently the main graph network analysis technology. It uses the relationship between nodes and edges to design a graph network embedding method, embedding nodes into feature vectors, so that topological problems can be converted into numerical problems, thereby using powerful numerical calculation tools to achieve clustering, recognition, sorting and other functions for use in video systems.

[0071] S205: extracting connected components from the multimedia data relationship network, and determining a node sequence corresponding to the connected components.

[0072] The maximal connected subgraph in an undirected graph is called a connected component. Any connected graph has only one connected component, which is itself. A non-connected undirected graph has multiple connected components. The multimedia data relationship network in the embodiment of this specification can be considered as a weighted undirected graph, and the weight of the edge is the content similarity between the multimedia data. The undirected graph may include at least one connected component, each connected component includes at least one node connected by an edge, and any two nodes in a connected component can be connected by an edge, that is, there is a path between any pair of nodes in a connected component, but there is no connected path between two nodes located in different connected components.

[0073] S207 , traversing the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determining the parent node corresponding to the node that meets the first preset condition as the first key node to obtain a first key node set.

[0074] S209 : Determine a second key node set according to the first key node in the first key node set and the neighboring nodes of the first key node in the multimedia data relationship network.

[0075] The multimedia data corresponding to the second key node in the second key node set is used as the key multimedia data of the target multimedia data set, thereby obtaining the key multimedia data set corresponding to the target multimedia data set.

[0076] In order to locate key multimedia data, the embodiments of this specification introduce the concept of minimum dominating set in graph network analysis, that is, to find a minimum existing set of nodes in the network so that all points not in this set are connected to the points in this set. The first key node set obtained in the above step S207 is the initial minimum dominating set, and the second key node set in step S209 is the target minimum dominating set determined based on the initial minimum dominating set and the multimedia data relationship network. The embodiments of this specification will respectively elaborate on steps S207 and S209 in the subsequent content.

[0077] The key multimedia data set obtained in the embodiment of the present invention represents all the contents contained in the target multimedia data set, and the duplicate content between the key multimedia data is greatly reduced. In other words, the key multimedia data set can be considered as a refined set of the target multimedia data set, and the subsequent business processing for the target multimedia data set can be directly based on the corresponding key multimedia data set, thereby greatly reducing the computational overhead of business processing and improving business processing efficiency.

[0078] It should be noted that the multimedia data processing method of the embodiment of the present invention can be applied to the processing of video data, image data, audio data, etc. In the following description of the embodiment of this specification, the multimedia data is described in detail as video data.

[0079] The embodiments of this specification can process video data based on artificial intelligence computer vision and machine learning technologies.

[0080] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0081] Computer vision (CV) is the science of making machines "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying, tracking, and measuring objects. This involves further processing images, transforming them into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0082] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0083] See also Figure 3 , which is a flow chart of a video data processing method provided by an embodiment of the present invention, which can be applied to Figure 1 implementation environment.

[0084] Generally speaking, the video data processing method of the embodiment of the present invention can include two parts. The first part is to construct a video relationship network of the video data set and extract its connected components. The second part is to analyze the video relationship network to locate key videos, and obtain a key video set to represent the video data set. The key video set here represents all video clips that appear in the video data set, while minimizing the number of repeated video clips between key videos. In other words, the key video set meets the following objective function:

[0085]

[0086] in, Represents the key video set, v k Indicates a key video.

[0087] In practical applications, key video sets can be placed in the video pool instead of video data sets, thereby effectively improving the overall efficiency of business processing and reducing computing overhead by reducing the size of the video pool.

[0088] Specifically, such as Figure 3 As shown, the video data processing method of the embodiment of the present invention may include:

[0089] S301: Obtain content similarity between videos in a target video data set.

[0090] The target video data set is a video set to be processed, and the videos in the target video data set can be long videos, short videos, or short videos clipped from long videos.

[0091] In actual applications, the server may process the target video data set in a preset period, and the preset period may be set according to actual conditions, for example, the preset period may be 1 week, 1 month, etc.

[0092] The content similarity between videos refers to the degree of similarity in video content. The server can obtain image feature information for each video and determine the content similarity between videos based on the image feature information of the videos in the target video dataset. A video is a video frame sequence consisting of a series of continuous video frames, i.e., video images. Therefore, the image feature information of a video can be an image feature vector of the video frame.

[0093] In a possible implementation, the server may obtain the image feature information of the video by obtaining an image feature vector of each video frame in all video frames constituting the video.

[0094] In another possible implementation, the server may obtain the image feature information of the video by obtaining image feature vectors of key video frames among all video frames constituting the video. A key video frame refers to a video frame that contains a key action in the motion or transformation of a character or object. In a specific implementation, when decoding the video, the server may obtain an intra-frame coded frame. This intra-frame coded frame is an independent frame that carries all the information. The server may identify the intra-frame coded frame obtained during video decoding as a key video frame, thereby obtaining a key video frame sequence corresponding to the video.

[0095] The server may obtain the image feature vector of the video frame by: inputting the video frame into a pre-trained image feature extraction model, and extracting the image feature vector through the image feature extraction model. The image feature extraction model may be a deep learning network model, for example, a convolutional neural network (CNN) model VGG16 (a 16-layer CNN model). The video frame is used as the input of VGG16, and the output vector of the preset convolutional layer in VGG16 is taken as the image feature vector representing the video frame. The preset convolutional layer can be set according to actual needs. For example, the preset convolutional layer is a 1000-dimensional vector output by the penultimate convolutional layer.

[0096] To reduce computational complexity, the server may further perform dimensionality reduction on the image feature vector output by the image feature extraction network, and use the reduced dimensionality image feature vector as the image feature vector of the corresponding video frame. The dimensionality reduction method may be, but is not limited to, principal component analysis (PCA).

[0097] Based on the above description, each video in the target video data set can be represented by the image feature vectors of the video frames in its entire video frame sequence, or by the image feature vectors of the key video frames in its key video frame sequence.

[0098] In one possible implementation, the content similarity between videos can be characterized by the distance between them. The smaller the distance, the more similar the videos' contents are; conversely, the larger the distance, the greater the difference in their contents. This means that content similarity and distance are inversely correlated. Based on this, the server can determine the distance between videos based on the image feature vectors of the videos in the target video dataset, thereby determining the content similarity between the videos. In other words, content similarity is the inverse of distance.

[0099] Taking the image feature vectors of key video frames in a key video frame sequence as an example to represent a video, in one possible implementation, the server may determine the distance between videos based on the image feature vectors of videos in the target video data set, including the following steps:

[0100] (1) For any video i in the target video data set, determine the distance between each key video frame of video i and the remaining video j in the target video data set, where the remaining video j is the video in the target video data set other than video i.

[0101] In one possible implementation, for any key video frame i_k of video i, the distance between key video frame i_k and the remaining video j can be represented based on the minimum value of the distances from all key video frames corresponding to each remaining video j to key video frame i_k. The distance from the key video frame of the remaining video j to the key video frame i_k can be determined by the vector distance between the image feature vectors of the key video frames, and this vector distance can be, but is not limited to, a Euclidean distance.

[0102] (2) Determine the target key video frame of video i based on the distance between each key video frame of video i and the remaining videos j in the target video data set.

[0103] Specifically, the server can determine whether the distance between key video frame i_k and the remaining video j is greater than a preset distance threshold. If so, it indicates that key video frame i_k has no similar video frames in the remaining video j and is not the target key video frame. If not, it indicates that key video frame i_k has a similar key video frame in the remaining video j and is the target key video frame. In this case, the key video frame of the remaining video j corresponding to the minimum distance is determined as the similar key video frame similar to key video frame i_k. In a specific implementation, the server also records the frame position information of the similar key video frame in the remaining video j. The preset distance threshold can be set according to actual needs.

[0104] (3) Determine the distance between video i and the remaining video j based on the distance from the target key video frame of video i to the remaining video j in the target video data set.

[0105] Specifically, the server may calculate the sum of the distances between each target key video frame of video i and the remaining video j, and use the sum as the distance between video i and the remaining video j. It is understood that the average of the sums may also be used as the distance between video i and the remaining video j.

[0106] In practical applications, before calculating the sum of the distances from each target key video frame in video i to the remaining video j, the server may further include the following steps: for each remaining video j, the server may obtain a similar key video frame sequence corresponding to the target key video frame sequence in video i. Based on the frame position information of each similar key video frame in the similar key video frame sequence in the remaining video j, the server may obtain the positional relationship of the similar key video frames in the remaining video j. When the similar key video frame sequence matches the positional relationship, or the temporal relationship is consistent, it indicates that there is a video segment in the remaining video i with content similar to that of video i. At this time, the server may calculate the sum of the distances from each target key video frame in video i to the remaining video j.

[0107] In the embodiments of this specification, after determining the content similarity between the videos based on the distance between them, it is determined whether the content similarity is less than a preset content similarity threshold. If so, it is determined that there is no similarity between the videos, and the content similarity between the videos can be assigned a value of zero. If not, it is determined that there are video segments with similar content between the videos, and the videos are similar. The preset content similarity threshold can be set according to actual needs.

[0108] It should be noted that the above only uses the distance between videos to represent the content similarity between videos as an example to illustrate how the server obtains the content similarity between videos. In actual applications, other forms can also be used to represent the content similarity between videos, and the embodiments of the present invention do not specifically limit this.

[0109] In actual applications, videos are often modified during the distribution stage, such as coding loss, cropping, logo addition, etc., which causes large deviations in the image feature vectors of video frames with similar visual content, thereby reducing the accuracy of the content similarity between videos determined based on the image feature vectors of the video frames. Based on this, in an embodiment of this specification, when training the image feature extraction model, the training sample data is processed according to the various modifications that occur during the video distribution stage, and then the image feature extraction model is trained based on the processed sample data, so that the trained image feature extraction model has good robustness to various modifications that occur during the video distribution stage, ensuring the similarity of the image feature vectors of video frames with similar visual content, thereby improving the accuracy of the content similarity between videos determined based on the image feature vectors of the video frames.

[0110] S303 : Constructing a video relationship network corresponding to the target video data set by using the videos in the target video data set as nodes and the content similarities between the videos as edges.

[0111] In the embodiments of this specification, based on graph network embedding technology, a video relationship network corresponding to the target video data set is constructed with videos in the target video data set as nodes and content similarities between videos as edges.

[0112] The video relationship network constructed in the embodiment of this specification can be expressed as G(V,E), where V represents a node, E represents an edge, and the edge has no direction. Each node represents a video data. Edges between nodes Represents the similarity between videos. The edge weight is the inverse of the content similarity between the videos. When the content similarity is zero, the corresponding edge weight is infinite, and the edge no longer exists. Therefore, the higher the edge weight, the lower the similarity between the two videos associated with the edge. If there are no similar video clips between two videos, there will be no edge connecting them in the video relationship network.

[0113] Existing video relationship networks are typically constructed based on video semantic information, such as manual tags, video classification information, or video title analysis, or based on user operation records such as sharing, viewing, and clicking. These networks can only represent abstract connections between videos, such as similar videos and the degree of match between videos and users. When a video collection includes fragmented short videos or mini-videos edited from a long video, these videos may differ significantly from the original long video in semantics due to their shorter duration, yet their visual content may be very similar to the original long video in some segments. In this case, representing the relationships between videos based on coarse-grained semantic tags or user operations cannot fully express these relationships.

[0114] The embodiment of this specification constructs a video relationship network based on the similarity of the video visual content, which can well characterize the relationship between the above-mentioned videos. That is, when the videos in the video collection include fragmented short videos or small videos edited from long videos, the video relationship network of the embodiment of this specification can more accurately characterize the relationship between the videos in the video collection, which is conducive to improving the accuracy of subsequent key video determination, and further conducive to improving the efficiency and accuracy of business processing.

[0115] S305: extracting connected components from the video relationship network, and determining a node sequence corresponding to the connected components.

[0116] The video relationship network constructed in the embodiments of this specification is essentially a weighted undirected graph. The maximal connected subgraph of this undirected graph is the connected component of the undirected graph. Each connected component includes at least one node connected by an edge. Any two nodes in a connected component can be connected by an edge. In other words, there is a path between any pair of nodes in a connected component, but there is no path between two nodes in different connected components.

[0117] There is at least one connected component in an undirected graph. Starting from any node in each connected component, you can reach any node in the connected component by walking along the edges in the graph. That is to say, the video relationship network constructed in the embodiment of this specification includes at least one connected component. Since the subsequent determination of key videos needs to be carried out in a connected graph (a connected graph has only one connected component, that is, an undirected graph with a path between any two nodes is a connected graph), it is necessary to extract the connected components in the video relationship network in step S305.

[0118] For multiple connected components in the video relationship network, the videos in each connected component have certain content duplication or similarity with each other, while the video contents in different connected components do not overlap. The union of multiple connected components is the entire video relationship network.

[0119] In a possible implementation, the server extracting connected components in the video relationship network may include: (1) selecting a first node from the video relationship network, where the first node is any node in the video relationship network; (2) starting from the first node, performing a breadth-first traversal of the nodes in the video relationship network along the edges in the video relationship network, and adding the node identifiers of the traversed nodes to a traversed node identifier set, wherein the node identifiers are used to uniquely identify the nodes in the video relationship network; (3) when the node corresponding to the node identifier in the traversed node identifier set is a first-order neighbor node (also referred to as an adjacent node) in the video relationship network, and there is a matching node identifier in the traversed node identifier set, It indicates that there are no untraversed nodes among the adjacent nodes of all traversed nodes, and the above-mentioned breadth-first traversal can be stopped at this time; (4) the node corresponding to the node identifier in the traversed node identifier set is taken as a connected component; (5) the nodes in the video relationship network other than the nodes corresponding to the node identifier in the traversed node identifier set are taken as remaining nodes, and the video relationship network containing only the remaining nodes can be called the remaining video relationship network, and the above-mentioned traversed node identifier set is cleared at the same time; (6) the remaining video relationship network is taken as the above-mentioned video relationship network, and the above-mentioned steps (1) to (4) are executed until all nodes in the video relationship network are traversed, thereby obtaining multiple connected components contained in the video relationship network.

[0120] Among them, breadth-first traversal is similar to the level-order traversal of a tree. It starts from the root node and traverses layer by layer. Only after all the nodes of a layer are completely traversed will it enter the traversal of the next layer.

[0121] In step S305 , for the extracted connected components, a node sequence corresponding to each connected component is determined respectively, and two adjacent nodes in the node sequence are in an adjacency relationship with each other in the video relationship network.

[0122] In one possible implementation, when determining the node sequence of a connected component, the server may traverse the connected component using a depth-first algorithm, using the node traversal order as the node sequence of the connected component. Depth-first traversal starts at an unvisited node and moves along the edges of the current node to unvisited nodes. When there are no unvisited nodes, the server returns to the previous node and continues to explore other nodes until all nodes have been visited.

[0123] For example, Figure 4 The diagram shows a connected component. Figure 4 Perform a depth-first traversal. Figure 4 Each node in the traversal has a preset storage order. Assume that the preset storage order is ABCDEFGH. When a node has multiple adjacent nodes during the traversal process, the adjacent nodes with the earlier storage order are accessed first, such as Figure 4 The adjacent nodes of node A are BDF. According to the preset storage order, B is ranked before D and F, so node B is visited first. The specific depth-first traversal process is assumed to start from node A. The visiting process of each node is: visit node A; visit node B; visit node G; visit node E; visit node C; visit node D; since node D has no unvisited adjacent nodes, it backtracks to node G and then visits another neighbor node H of node G; visit F. At this time, all nodes have been visited and the traversal ends. The traversal order of the above nodes is taken as Figure 4 The node sequence of the connected component shown is: A->B->G->E->C->D->H->F.

[0124] Understandably, Figure 4 This is only an example of extracting a node sequence of connected components and does not constitute a specific limitation to the embodiments of the present invention.

[0125] In any connected component obtained in step S305, nodes with similar video clips will be connected together, and nodes of some longer videos will be connected with nodes of many videos, such as Figure 5As shown in the local topology diagram of the video relationship network provided, node a of the long video is connected to node b of many independent videos. Through analysis, it is found that although the video quality and format may be different, the video content represented by node b is included in node a. In other words, node a can represent a series of nodes a connected to it, that is, node a can be considered as a key node. These key nodes will be located from the video relationship network based on the node sequence of each connected component.

[0126] S307 , traversing the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determining the parent node corresponding to the node that meets the first preset condition as the first key node to obtain a first key node set.

[0127] This specification embodiment is intended to locate Figure 5 The node a shown is a key node, which introduces the concept of minimum dominating set in graph network analysis, that is, finding a minimum set of nodes in the network so that all points not in this set are connected to the points in this set. Among them, the first key node set in step S307 can be considered as the minimum dominating set of the video relationship network.

[0128] Specifically, step S307 traverses each node sequence starting from the tail node of the node sequence, that is, the end node of the sequence. When traversing the nodes in the node sequence, the following steps may be performed:

[0129] (1) According to the position of the node in the node sequence, determine the parent node corresponding to the node.

[0130] The parent node is the node that precedes the currently visited node in the node sequence. For example, in the node sequence A->B->G->E->C->D->H->F, if the currently visited node is F, then the parent node of F is H. Similarly, the parent node of H is D, and so on.

[0131] (2) Determine whether the node and the parent node meet a first preset condition.

[0132] The first preset condition may be that there is no matching first key node in the first key node set. The initial value of the first key node set may be set to empty.

[0133] Based on this, judging whether a node and a parent node satisfy the first preset condition may include: judging whether there is a first key node in the first key node set that matches the currently visited node, that is, judging whether the currently visited node is in the first key node set; when the result of the judgment is no, that is, the currently visited node is not in the first key node set, further judging whether there is a first key node in the first key node set that matches the corresponding parent node, that is, judging whether the parent node is in the first key node set; when the result of the judgment is no, that is, the parent node is not in the first key node set, it can be determined that the currently visited node satisfies the above-mentioned first preset condition, and its parent node also satisfies the first preset condition, and step (3) can be executed.

[0134] When the result of the judgment is that the currently visited node is in the first key node set, it is determined that the currently visited node does not meet the first preset condition. At this time, the access to the current node can be directly terminated and the next node in the node sequence can be accessed.

[0135] (3) When the node satisfies the first preset condition and the parent node satisfies the first preset condition, the parent node is determined to be a first key node, and the parent node is added to the first key node set.

[0136] After adding the parent node to the first key node set, the access to the current node can be terminated and the next node can be accessed, that is, the above steps (1) to (3) are performed on the next node in the node sequence until the traversal of the head node in the node sequence is completed. The above can obtain the first key node set corresponding to each node sequence.

[0137] In order to improve processing efficiency, after step (3), the following steps may be further included:

[0138] (4) Determine the nodes connected to the parent node in the connected components corresponding to the node sequence, and mark the nodes connected to the parent node as covered nodes.

[0139] (5) When the covered nodes in the connected components corresponding to the node sequence meet the second preset condition, the traversal of the nodes in the node sequence is terminated.

[0140] Among them, the second preset condition may be that the covering node covers all nodes in the connected component corresponding to the node sequence. Based on this, when the covering node in the connected component corresponding to the node sequence meets the second preset condition, ending the traversal of the nodes in the node sequence may include: judging whether the nodes in the connected component corresponding to the node sequence are all marked as covering nodes; if the nodes in the connected component corresponding to the node sequence are all marked as covering nodes, then the traversal of the nodes in the node sequence can be ended, thereby improving processing efficiency.

[0141] The following is the aforementioned Figure 4 The above step S307 is explained by taking the connected components and their corresponding node sequences as an example. It can be understood that this example does not constitute a specific limitation on the embodiments of the present invention.

[0142] For the node sequence A->B->G->E->C->D->H->F, traverse starting from node F.

[0143] For node F: the parent node of node F is H, the initial value of the first key node set is empty, at this time node F is not in the first key node set, and its parent node H is also not in the first key node set, the parent node H is added to the first key node set, and the first key node set is {H}. In addition, Figure 4 It is known that the nodes connected to H are F and G, so F and G are marked as covered nodes. At this time, the covered nodes are {F, G}, which are not covered. Figure 4 All nodes in the connected component, so we can continue to traverse the node sequence; then traverse the next node in the node sequence, that is, H.

[0144] For node H: Since the first key node set is {H}, it is obvious that node H is in the first key node set, so node H does not meet the first preset condition, and the access to the current node H is ended. At this time, the covered node is {F, G}, which is not covered. Figure 4 All nodes in the connected component can therefore continue to traverse the node sequence and enter the visit to the next node, namely node D.

[0145] For node D: the parent node of point D is C. Node D is not in the first key node set {H}, and its parent node C is also not in the first key node set {H}. Add the parent node C to the first key node set, and the first key node set is {H, C}. In addition, Figure 4 It is known that the nodes connected to C are D and E, so D and E are marked as covered nodes. At this time, the covered nodes are {F, G, D, E}, which are not covered. Figure 4 All nodes in the connected component, so we can continue to traverse the node sequence; then traverse the next node in the node sequence, that is, C.

[0146] For node C: Since the first key node set is {H, C}, it is obvious that node C is in the first key node set, so node C does not meet the first preset condition, and the current access to node C ends. At this time, the covered nodes are {F, G, D, E}, which is not covered. Figure 4 All nodes in the connected component can therefore continue to traverse the node sequence and enter the next node, namely node E.

[0147] For node E: the parent node of point E is G, node E is not in the first key node set {H, C}, and its parent node G is also not in the first key node set {H, C}, add the parent node G to the first key node set, and the first key node set is {H, C, G}. In addition, through Figure 4 It is known that the nodes connected to G are E, H and B, so E, H and B are marked as covered nodes. At this time, the covered nodes are {F, G, D, E, H, B}, which are not covered. Figure 4 All nodes in the connected component, so we can continue to traverse the node sequence; then traverse the next node in the node sequence, that is, G.

[0148] For node G: Since the first key node set is {H, C, G}, it is obvious that node G is in the first key node set, so node G does not meet the first preset condition, and the current access to node G ends. At this time, the covered nodes are {F, G, D, E, H, B}, which is not covered. Figure 4 All nodes in the connected component can therefore continue to traverse the node sequence and enter the next node, namely node B.

[0149] For node B: the parent node of point B is A, node B is not in the first key node set {H, C, G}, and its parent node A is also not in the first key node set {H, C, G}, add the parent node A to the first key node set, and the first key node set is {H, C, G, A}. In addition, through Figure 4 It is known that the nodes connected to A are F, B and D, so F, B and D are marked as covered nodes. At this time, the covered nodes are {F, G, D, E, H, B}, which are not covered. Figure 4 All nodes in the connected component, so we can continue to traverse the node sequence; then traverse the next node in the node sequence, that is, A.

[0150] For node A: Point A is the root node, node A is in the first key node set {H, C, G, A}, and the access to the node sequence ends.

[0151] After the traversal of the above node sequence is completed, it can be obtained that the first key node set corresponding to the node sequence is {H, C, G, A}.

[0152] As can be understood, after traversing each node sequence, the first key node set corresponding to each node can be obtained. The first key node set of each node is combined to form the first key node set of the video relationship network. Each key node in the first key node set of the video relationship network can represent other nodes connected to it in the video relationship network. In other words, all nodes in the video relationship network that are not in the first key node set are connected to the key nodes in the first key node set.

[0153] S309 : Determine a second key node set according to the first key node in the first key node set and the neighboring nodes of the first key node in the video relationship network.

[0154] The video corresponding to the second key node in the second key node set is used as the key video of the target video data set, and then the key video set corresponding to the target video data set can be obtained.

[0155] In the embodiment of this specification, the first key node set can ensure that other video nodes in the corresponding connected component overlap with the first key node set, but cannot ensure that the first key node set covers all video clips in the connected component. Based on this, in the embodiment of this specification, the first key node set obtained in step S307 is used as the initial minimum dominating set, and the initial minimum dominating set is expanded to obtain the target minimum dominating set, namely the key video set.

[0156] In a possible implementation, the server may determine the second key node set based on the first key node in the first key node set and the neighboring nodes of the first key node in the video relationship network, including the following steps:

[0157] (1) Determine the first-order neighbor nodes of the first key node in the video relationship network in the first key node set, wherein the first-order neighbor nodes are nodes directly connected to the first key node in the video relationship network.

[0158] (2) Obtain the content similarity between the first key node and the corresponding first-order neighbor node to obtain the first content similarity.

[0159] The weight values ​​of the edges in the video relationship network of the embodiment of this specification are determined based on the content similarity between the nodes (videos). Based on this, the content similarity between the first key node and the corresponding first-order neighbor node can be directly obtained according to the edge connecting the first key node and the corresponding first-order neighbor node.

[0160] (3) Determine the first-order neighbor nodes whose first content similarity is less than the first similarity threshold, and obtain the extended key nodes.

[0161] Among them, the first similarity threshold can be set according to historical experience. Generally, the smaller the first similarity threshold, the lower the similarity between the expanded key node and the first key node. While ensuring the comprehensiveness of the content coverage of the subsequent key video set, the size of the key video set is reduced as much as possible.

[0162] (4) Determine a second key node set based on the first key node and the extended key node in the first key node set.

[0163] In a possible implementation, the server may directly add the extended key node to the first key node set to obtain an expanded first key node set, and use the expanded first key node set as the second key node set.

[0164] In practical applications, considering that there may still be duplicate video clips in the expanded first key node set, in order to make the target minimum dominating set, i.e., the key video set, as small as possible and improve the efficiency of business processing, such as Figure 6 A flow chart of another video data processing method is provided, in which the server may include the following steps when determining the second key node set based on the first key node and the extended key node in the first key node set:

[0165] S601 : Taking the first key node and the extended key node as third key nodes to obtain a third key node set, and determining content similarity between the third key nodes in the third key node set.

[0166] Specifically, the method for determining the content similarity between the third key nodes can refer to the relevant description of determining the content similarity between videos (nodes) in the aforementioned step S301, which will not be repeated here.

[0167] S603: Construct a key node relationship network based on the third key nodes in the third key node set and the content similarities between the third key nodes.

[0168] Specifically, the third key nodes in the third key node set are nodes of the key node relationship network, the content similarity between the third key nodes is an edge in the key node relationship network, and the weight value of the edge is the inverse of the content similarity between the third key nodes.

[0169] S605: Determine a key node sequence according to the key node relationship network.

[0170] Specifically, the server may traverse the key node relationship network using a depth-first algorithm, and use the traversal order of the third key node as the key node sequence of the key node relationship network. For specific implementation, please refer to the relevant description in the aforementioned step S305, which will not be repeated here.

[0171] S607, starting from the tail key node of the key node sequence, traversing the key nodes in the key node sequence, when traversing the key nodes in the key node sequence, determining the parent key node corresponding to the key node that meets the third preset condition as the fourth key node, and obtaining a fourth key node set.

[0172] When executing step S607, reference may be made to the relevant description in the aforementioned step S307, wherein the third preset condition is that there is no matching key node in the fourth key node set. The specific implementation details are not repeated here.

[0173] S609 : Determine a second key node set according to the fourth key node in the fourth key node set and the neighboring key nodes of the fourth key node in the key node relationship network.

[0174] In one possible implementation, step S609 may include the following steps:

[0175] (1) Determine the first-order neighbor key nodes of the fourth key node in the key node relationship network in the fourth key node set.

[0176] (2) Obtain the content similarity between the fourth key node and the first-order neighbor key node to obtain the second content similarity.

[0177] The weight values ​​of the edges in the key node relationship network of the embodiment of this specification are determined based on the content similarity between the third key nodes (videos). Based on this, the content similarity between the fourth key node and the corresponding first-order neighbor key node can be directly obtained according to the edge connecting the third key node and the corresponding first-order neighbor key node.

[0178] (3) Determine the first-order neighbor key node whose second content similarity is less than the second similarity threshold, and obtain the target neighbor key node.

[0179] The second similarity threshold may be set according to historical experience. Optionally, the second similarity threshold may be set to 100%.

[0180] (4) The fourth key node and the target neighbor key node in the fourth key node set are used as the second key nodes to obtain the second key node set.

[0181] In order to facilitate the understanding of the above-mentioned video data processing method of the embodiment of the present invention, the following Figure 7 This method will be described.

[0182] like Figure 7 As shown, the video data processing method of an embodiment of the present invention includes two parts framed by dotted lines, wherein the first part is to construct a video relationship network of a video data set and extract its connected components, and the second part is to analyze the video relationship network to locate key videos to obtain a key video set to represent the video data set.

[0183] Specifically, when implementing the first part, the content similarity between videos is first determined, and then a video relationship network is constructed with videos as nodes and content similarity as edges, and the connected components in the video relationship network are extracted to obtain at least one connected component such as connected component 1, connected component 2, and connected component 3.

[0184] After obtaining at least one connected component, the second part can be implemented. When implementing the second part, the minimum dominating set is first calculated according to the node sequence corresponding to the connected component to obtain the first key node set (i.e., the initial minimum dominating set), and then the first key node set is expanded, and a key node relationship network is constructed based on the expanded first key node set. Then, the minimum dominating set is calculated again based on the key node sequence in the key node relationship network to obtain the fourth key node set, and the target minimum dominating set is calculated based on the recalculated minimum dominating set, i.e., the fourth key node set, to obtain the second key node set, that is, the key video set.

[0185] It can be seen from the above technical solutions of the embodiments of the present invention that the video data processing method of the embodiments of the present invention constructs a video relationship network of a video data set based on the content similarity between videos, and further determines the key videos in the video data set through analysis of the video relationship network, thereby obtaining a key video set that can represent the video data set. In practical applications, the key video set can be used as a video pool for related services such as video recommendation, video copyright detection, and video rearrangement, thereby greatly reducing the size of the video pool, reducing the computational overhead of these business processing, and improving business processing efficiency.

[0186] See also Figure 8 , which is a schematic diagram of the structure of a multimedia data processing device provided by an embodiment of the present invention, the device has the function of implementing the multimedia data processing method in the above method embodiment, and the function can be implemented by hardware or by hardware executing corresponding software. Figure 8 As shown, the device may include:

[0187] A first acquisition module 810 is configured to acquire content similarities between multimedia data in a target multimedia data set;

[0188] A first construction module 820 is configured to construct a multimedia data relationship network corresponding to the target multimedia data set, using the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges;

[0189] A node sequence determination module 830 is configured to extract connected components from the multimedia data relationship network and determine a node sequence corresponding to the connected components; the connected components include at least one node connected via the edge;

[0190] A first determining module 840 is configured to traverse the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determine the parent node corresponding to the node that meets the first preset condition as the first key node, to obtain a first key node set;

[0191] The second determination module 850 is used to determine the second key node set based on the first key node in the first key node set and the neighbor nodes of the first key node in the multimedia data relationship network; the multimedia data corresponding to the second key node in the second key node set is used as the key multimedia data of the target multimedia data set.

[0192] As an optional implementation, the first determining module 840 may include a first traversal module, the first traversal module being configured to traverse the nodes in the node sequence starting from the tail node of the node sequence;

[0193] The first traversal module includes:

[0194] A third determining module is used to determine the parent node corresponding to the node according to the position of the node in the node sequence;

[0195] A first judgment module, configured to judge whether the node and the parent node meet a first preset condition;

[0196] a fourth determining module, configured to, when the node satisfies the first preset condition and the parent node satisfies the first preset condition, determine that the parent node is a first key node and add the parent node to a first key node set;

[0197] a fifth determining module, configured to determine nodes connected to the parent node in the connected components corresponding to the node sequence, and mark the nodes connected to the parent node in the connected components corresponding to the node sequence as covered nodes;

[0198] The first traversal termination module is configured to terminate the traversal of the nodes in the node sequence when the covered nodes in the connected component corresponding to the node sequence meet a second preset condition.

[0199] As an optional implementation, the first judgment module may include:

[0200] A first judgment submodule, configured to judge whether there is a first key node matching the node in the first key node set;

[0201] a second judgment submodule, configured to judge whether there is a first key node matching the parent node in the first key node set when the result of the judgment by the first judgment submodule is negative;

[0202] The sixth determination module is configured to determine, when the result of the determination by the second determination submodule is no, whether the node satisfies the first preset condition and whether the parent node satisfies the first preset condition.

[0203] As an optional implementation, the first traversal module may further include:

[0204] A second judgment module is used to judge whether all nodes in the connected components corresponding to the node sequence are marked as covered nodes;

[0205] The seventh determination module is configured to determine, when the result of the determination by the second determination module is yes, whether the covered nodes in the connected components corresponding to the node sequence meet a second preset condition.

[0206] As an optional implementation, the second determining module 850 may include:

[0207] A first neighbor node determination module, configured to determine first-order neighbor nodes of a first key node in the first key node set in the multimedia data relationship network;

[0208] A first acquisition module is configured to acquire the content similarity between the first key node and the corresponding first-order neighbor node to obtain a first content similarity;

[0209] an extended node determination module, configured to determine a first-order neighbor node whose first content similarity is less than a first similarity threshold, and obtain an extended key node;

[0210] An eighth determining module is configured to determine a second key node set according to the first key node in the first key node set and the extended key node.

[0211] As an optional implementation, the eighth determining module may include:

[0212] A ninth determining module is configured to use the first key node and the extended key node as third key nodes to obtain a third key node set; and determine content similarity between the third key nodes in the third key node set;

[0213] A second building module is used to build a key node relationship network based on the third key nodes in the third key node set and the content similarity between the third key nodes;

[0214] A key node sequence determination module, configured to determine a key node sequence based on the key node relationship network;

[0215] a tenth determining module, configured to traverse the key nodes in the key node sequence starting from the tail key node of the key node sequence, and when traversing the key nodes in the key node sequence, determine the parent key node corresponding to the key node that meets the third preset condition as a fourth key node, to obtain a fourth key node set;

[0216] An eleventh determining module is configured to determine a second key node set based on a fourth key node in the fourth key node set and neighboring key nodes of the fourth key node in the key node relationship network.

[0217] As an optional implementation, the eleventh determining module may include:

[0218] A second neighbor node determination module is used to determine a first-order neighbor key node of the fourth key node in the fourth key node set in the key node relationship network;

[0219] A second acquisition module is configured to acquire the content similarity between the fourth key node and the first-order neighbor key node to obtain a second content similarity;

[0220] a target neighbor node determination module, configured to determine a first-order neighbor key node whose second content similarity is less than a second similarity threshold, and obtain a target neighbor key node;

[0221] The twelfth determining module is configured to use the fourth key node in the fourth key node set and the target neighbor key node as second key nodes to obtain a second key node set.

[0222] It should be noted that the apparatus provided in the above embodiments, when implementing its functions, is only illustrated by the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0223] The multimedia data processing device of the embodiment of the present invention constructs a video relationship network of a video data set based on the content similarity between videos, and further determines the key videos in the video data set by analyzing the video relationship network, thereby obtaining a key video set that can represent the video data set. In practical applications, the key video set can be used as a video pool for related services such as video recommendation, video copyright detection, and video re-arrangement, thereby greatly reducing the size of the video pool, reducing the computational overhead of these business processing, and improving business processing efficiency.

[0224] An embodiment of the present invention provides a server, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the multimedia data processing method provided in the above method embodiment.

[0225] Figure 9 This is a hardware structure diagram of a server running a multimedia data processing method provided by an embodiment of the present invention, such as Figure 9 As shown, the server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be temporary storage or permanent storage. The program stored in the storage medium 1020 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 to execute a series of instruction operations in the storage medium 920 on the server 900. The server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0226] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the aforementioned network may include a wireless network provided by a communications provider of the server 900. In one embodiment, the input / output interface 940 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one embodiment, the input / output interface 940 can be a radio frequency (RF) module for wirelessly communicating with the Internet.

[0227] An embodiment of the present invention also provides a computer-readable storage medium, which can be set in a server to store at least one instruction or at least one program related to implementing a multimedia data processing method. The at least one instruction or the at least one program is loaded and executed by the processor to implement the multimedia data processing method provided by the above method embodiment.

[0228] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0229] It should be noted that the order in which the embodiments of the present invention are described above is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The above description is of specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0230] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences from other embodiments. In particular, the device embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0231] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0232] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A multimedia data processing method, characterized in that: The method comprises: Obtaining content similarity between multimedia data in a target multimedia data set; Constructing a multimedia data relationship network corresponding to the target multimedia data set by taking the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges; Extracting connected components from the multimedia data relationship network and determining a node sequence corresponding to the connected components; the connected components include at least one node connected by an edge; Traversing the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determining the parent node corresponding to the node that meets the first preset condition as the first key node to obtain a first key node set; Determine the first-order neighbor nodes of the first key node in the first key node set in the multimedia data relationship network; obtain the content similarity between the first key node and the corresponding first-order neighbor node to obtain the first content similarity; determine the first-order neighbor nodes whose first content similarity is less than the first similarity threshold to obtain the extended key node; determine the second key node set based on the first key node in the first key node set and the extended key node; and use the multimedia data corresponding to the second key node in the second key node set as the key multimedia data of the target multimedia data set.

2. The multimedia data processing method according to claim 1, wherein: The traversing of the nodes in the node sequence starting from the tail node of the node sequence, determining the parent node corresponding to the node that meets the first preset condition as the first key node when traversing the nodes in the node sequence, and obtaining the first key node set includes: The nodes in the node sequence are traversed starting from the tail node of the node sequence, and the following steps are performed when traversing the nodes in the node sequence: Determine the parent node corresponding to the node according to the position of the node in the node sequence; Determining whether the node and the parent node meet a first preset condition; When the node satisfies the first preset condition and the parent node satisfies the first preset condition, determining the parent node as a first key node, and adding the parent node to a first key node set; Determine the nodes connected to the parent node in the connected components corresponding to the node sequence, and mark the nodes connected to the parent node as covered nodes; When the covered nodes in the connected components corresponding to the node sequence meet a second preset condition, the traversal of the nodes in the node sequence is terminated.

3. The multimedia data processing method according to claim 2, wherein: The determining whether the node and the parent node meet a first preset condition includes: Determine whether there is a first key node matching the node in the first key node set; When the result of the judgment is no, determining whether there is a first key node matching the parent node in the first key node set; When the result of the judgment is no, it is determined that the node and the parent node meet the first preset condition.

4. The multimedia data processing method according to claim 2, wherein: When the covered nodes in the connected components corresponding to the node sequence meet the second preset condition, before terminating the traversal of the nodes in the node sequence, the method further includes: Determine whether all nodes in the connected components corresponding to the node sequence are marked as covered nodes; When the result of the judgment is yes, it is determined that the covering node in the connected component corresponding to the node sequence meets the second preset condition.

5. The multimedia data processing method according to claim 1, wherein: The determining of the second key node set according to the first key node in the first key node set and the extended key node includes: Taking the first key node and the extended key node as third key nodes, to obtain a third key node set; Determining content similarity between third key nodes in the third key node set; Constructing a key node relationship network based on the third key nodes in the third key node set and the content similarities between the third key nodes; Determining a key node sequence according to the key node relationship network; Traversing the key nodes in the key node sequence starting from the tail key node of the key node sequence, and when traversing the key nodes in the key node sequence, determining a parent key node corresponding to a key node that meets a third preset condition as a fourth key node, to obtain a fourth key node set; A second key node set is determined according to the fourth key node in the fourth key node set and neighboring key nodes of the fourth key node in the key node relationship network.

6. The multimedia data processing method according to claim 5, characterized in that: The determining of the second key node set according to the fourth key node in the fourth key node set and the neighboring key nodes of the fourth key node in the key node relationship network includes: Determine a first-order neighbor key node of the fourth key node in the fourth key node set in the key node relationship network; Obtaining a content similarity between the fourth key node and the first-order neighbor key node to obtain a second content similarity; Determine a first-order neighbor key node whose second content similarity is less than a second similarity threshold, and obtain a target neighbor key node; The fourth key node in the fourth key node set and the target neighbor key node are used as second key nodes to obtain a second key node set.

7. A multimedia data processing device, characterized in that: The device comprises: A first acquisition module is used to obtain content similarity between multimedia data in a target multimedia data set; A first construction module is configured to construct a multimedia data relationship network corresponding to the target multimedia data set, using the multimedia data in the target multimedia data set as nodes and the content similarities between the multimedia data as edges; a node sequence determination module, configured to extract connected components from the multimedia data relationship network and determine a node sequence corresponding to the connected components; the connected components include at least one node connected via an edge; a first determining module, configured to traverse the nodes in the node sequence starting from the tail node of the node sequence, and when traversing the nodes in the node sequence, determine the parent node corresponding to the node that meets the first preset condition as the first key node, to obtain a first key node set; The second determination module is used to determine the first-order neighbor nodes of the first key node in the first key node set in the multimedia data relationship network; obtain the content similarity between the first key node and the corresponding first-order neighbor node to obtain the first content similarity; determine the first-order neighbor nodes whose first content similarity is less than the first similarity threshold to obtain the extended key node; determine the second key node set based on the first key node and the extended key node in the first key node set; and use the multimedia data corresponding to the second key node in the second key node set as the key multimedia data of the target multimedia data set.

8. A server, characterized in that: The method comprises a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the multimedia data processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, wherein the at least one instruction or the at least one program is loaded and executed by a processor to implement the multimedia data processing method according to any one of claims 1 to 6.

Images