Video processing method and device, electronic equipment, computer program product and computer readable storage medium

Abstract

The invention provides a video processing method and device, electronic equipment, a computer program product and a computer readable storage medium. The method comprises the steps that at least one to-be-detected video and a plurality of first clusters are acquired, the first clusters are clusters of a plurality of detected videos, and the plurality of first clusters are provided with category marks; acquiring a multi-modal feature vector of the to-be-detected video; based on the multi-modal feature vectors, the multiple first clusters are retrieved, and retrieval results are obtained; in response to the retrieval result representing that the to-be-detected video does not belong to any one of the first clusters, updating the plurality of first clusters based on the to-be-detected video to obtain a plurality of second clusters; periodically querying unmarked clusters from the plurality of second clusters; and obtaining a text feature vector of the to-be-detected video in the unmarked cluster, and performing classification processing on the unmarked cluster based on the text feature vector to obtain a category mark of the unmarked cluster. Through the video detection method and device, the real-time performance and accuracy of large-scale video detection can be ensured.

Description

Technical Field

[0001] The present application relates to artificial intelligence technology and big data technology, and in particular to a video processing method, device, electronic device, computer program product, and computer-readable storage medium. Background Art

[0002] Posting videos on social networks is a typical form of interaction, such as selfies and live streams. Video classification is necessary, for example, to identify anomalous videos and prevent their posting or dissemination. As social networks grow in size and frequency, their limited backend resources—and the difficulty of allocating hardware resources to scale linearly with the volume of videos—have made it increasingly difficult to accurately and timely detect videos.

[0003] With limited hardware resources, there is a lack of effective solutions in related technologies to ensure the real-time and accuracy of detection. Summary of the Invention

[0004] The embodiments of the present application provide a video processing method, device, electronic device, computer program product, and computer-readable storage medium, which can ensure the real-time and accuracy of large-scale video detection.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a video processing method, which includes:

[0007] Acquire at least one video to be detected and a plurality of first clusters, wherein the first cluster is a cluster of a plurality of detected videos, and the plurality of first clusters all have category labels;

[0008] Obtaining a multimodal feature vector of the video to be detected;

[0009] Retrieving the plurality of first clusters based on the multimodal feature vector to obtain a retrieval result, wherein the number of the plurality of first clusters does not exceed a cluster number threshold, and the retrieval result is used to indicate whether the video to be detected belongs to any of the first clusters;

[0010] In response to the retrieval result indicating that the video to be detected does not belong to any of the first clusters, updating the multiple first clusters based on the video to be detected to obtain multiple second clusters, wherein the multiple second clusters are used to replace the first clusters, the multiple first clusters are partially identical to the multiple second clusters, and the number of the multiple second clusters does not exceed the cluster number threshold;

[0011] Periodically querying an unlabeled cluster from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected;

[0012] A text feature vector of the video to be detected in the unlabeled cluster is obtained, and the unlabeled cluster is classified based on the text feature vector to obtain a category label of the unlabeled cluster.

[0013] An embodiment of the present application provides a video processing device, the device comprising:

[0014] An acquisition module is configured to acquire at least one video to be detected and a plurality of first clusters, wherein the first clusters are clusters of a plurality of detected videos, and the plurality of first clusters all have category labels; and acquire a multimodal feature vector of the video to be detected;

[0015] a retrieval module, configured to retrieve the plurality of first clusters based on the multimodal feature vector, respectively, to obtain retrieval results, wherein the number of the plurality of first clusters does not exceed a cluster number threshold, and the retrieval results are used to indicate whether the video to be detected belongs to any of the first clusters;

[0016] an updating module, configured to, in response to the retrieval result indicating that the video to be detected does not belong to any of the first clusters, update the plurality of first clusters based on the video to be detected to obtain a plurality of second clusters, wherein the plurality of second clusters are used to replace the first clusters, the plurality of first clusters are partially identical to the plurality of second clusters, and the number of the plurality of second clusters does not exceed the cluster number threshold;

[0017] a query module, configured to periodically query an unlabeled cluster from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected;

[0018] The classification module is used to obtain the text feature vector of the video to be detected in the unlabeled cluster, and classify the unlabeled cluster based on the text feature vector to obtain a category label of the unlabeled cluster.

[0019] An embodiment of the present application provides an electronic device, comprising:

[0020] a memory for storing computer-executable instructions;

[0021] The processor is configured to implement the video processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0022] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the video processing method provided in the embodiment of the present application when executed by a processor.

[0023] An embodiment of the present application provides a computer program product, including a computer program or computer-executable instructions. When the computer program or computer-executable instructions are executed by a processor, the video processing method provided in the embodiment of the present application is implemented.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] By combining multimodal features with text features, on the one hand, the multimodal features reflect the characteristics of the video to be detected from multiple different dimensions, ensuring the comprehensiveness of the retrieval clustering. On the other hand, when the video to be detected does not belong to any of the first clusters, that is, when the video to be detected is classified into the second cluster, the text feature vector, which is more lightweight than the multimodal feature vector, is used to determine the category label of the unlabeled cluster in the second cluster, taking into account the efficiency of the classification of the video to be detected, and overall taking into account both accuracy and efficiency.

[0026] When searching multiple first clusters, the size of the cluster is controlled by the cluster number threshold. Compared with the situation in which the size of the cluster in the existing technology continues to grow, the scale of computing power consumption of the cluster calculation is controlled. Combined with the periodic classification and processing of the videos to be detected in the unlabeled cluster classification, the videos to be detected belonging to the unlabeled cluster can be classified in time, and the real-time classification of the videos to be detected is ensured from the two dimensions of computing power consumption and execution time. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 1 is a schematic diagram of the architecture of a video processing system 100 provided in an embodiment of the present application;

[0028] Figure 2 2 is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application;

[0029] Figure 3A This is a schematic diagram of a first flow chart of a video processing method provided in an embodiment of the present application;

[0030] Figure 3B 2 is a schematic diagram of a second flow chart of the video processing method provided in an embodiment of the present application;

[0031] Figure 3C 3 is a schematic diagram of a third flow chart of the video processing method provided in an embodiment of the present application;

[0032] Figure 3D 4 is a schematic diagram of a fourth flow chart of a video processing method provided in an embodiment of the present application;

[0033] Figure 3E 5 is a schematic diagram of a fifth flow chart of a video processing method provided in an embodiment of the present application;

[0034] Figure 3F 1 is a sixth flow chart of the video processing method provided in an embodiment of the present application;

[0035] Figure 3G 7 is a schematic diagram of a seventh flow chart of a video processing method provided in an embodiment of the present application;

[0036] Figure 3H This is an eighth flow chart of the video processing method provided in an embodiment of the present application;

[0037] Figure 3I This is a ninth flow chart of the video processing method provided in an embodiment of the present application;

[0038] Figure 3J 10 is a schematic diagram of a video processing method according to an embodiment of the present application;

[0039] Figure 3K This is a schematic diagram of the eleventh flow chart of the video processing method provided in an embodiment of the present application;

[0040] Figure 3L This is a twelfth flow chart of the video processing method provided in an embodiment of the present application;

[0041] Figure 3M This is a thirteenth flow chart of the video processing method provided in an embodiment of the present application;

[0042] Figure 3N This is a fourteenth flow chart of the video processing method provided in an embodiment of the present application;

[0043] Figure 3O This is a fifteenth flow chart of the video processing method provided in an embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of ResNet provided in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the principle of obtaining similar clusters provided in an embodiment of the present application;

[0046] Figure 6 This is a schematic diagram of the structure of a graph constructed based on HNSW provided in an embodiment of the present application;

[0047] Figure 7 This is a schematic diagram of an application of suppressing abnormal videos provided by an embodiment of the present application;

[0048] Figure 8AThis is a schematic diagram of the structure of the index provided in the embodiment of the present application;

[0049] Figure 8B This is a schematic diagram of an application for adding an index provided in an embodiment of the present application;

[0050] Figure 8C This is a schematic diagram of an application for deleting an index provided in an embodiment of the present application;

[0051] Figure 9 Schematic diagram of the structure of the language understanding model with training initialization provided in the embodiment of the present application;

[0052] Figure 10 Schematic diagram of the principle of the supervised SimCSE method provided in the embodiment of the present application;

[0053] Figure 11 Schematic diagram of the principle of the classification model for training initialization provided in the embodiment of the present application;

[0054] Figure 12 This is an architecture diagram of abnormal video detection provided by an embodiment of the present application;

[0055] Figure 13 This is a logical diagram of the real-time clustering service provided by an embodiment of the present application;

[0056] Figure 14 This is a schematic diagram of the architecture of multi-machine distributed parallel clustering provided by an embodiment of the present application;

[0057] Figure 15 This is a schematic diagram of the principle of the manual operation solution provided in the embodiment of the present application;

[0058] Figure 16 This is a schematic diagram of the principle of the real-time feature calculation service provided by the embodiment of the present application;

[0059] Figure 17 This is an application diagram of the detection and alarm service provided in an embodiment of the present application.

[0060] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of distinction between the advantages and disadvantages of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0062] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0063] If similar descriptions of "first / second" appear in the application documents, the following explanation is added. In the following description, the terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It can be understood that "first / second / third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0064] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0065] Unless otherwise specified, "at least one" mentioned below refers to one or more situations, and "a plurality of" may refer to two or more situations.

[0066] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0067] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0068] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0069] 1) Clustering: This is the grouping of multiple videos with identical or similar content. For example, the content can be at least one of images, audio, and text, with the text including information such as the video title and description. Clustering can be represented by data, such as an array. The videos belonging to the cluster are the set of feature vectors where the similarity between any two feature vectors exceeds a similarity threshold. Feature vectors in the same cluster belong to the same category.

[0070] It should be pointed out that the first cluster and the second cluster mentioned below do not refer to a specific cluster, but are used to distinguish between the clusters that already existed before detection (i.e., the first cluster) and the clusters that are updated after clustering the video to be detected (i.e., the second cluster) when classifying at least one current video to be detected.

[0071] For example, before detecting the video to be detected, assuming that clusters 1-3 exist, clusters 1-3 are called the first cluster. During the detection process of video 1, a new cluster 4 is obtained through clustering, then clusters 1-3 and cluster 4 are both called the second cluster.

[0072] It can be understood that the first cluster and the second cluster are relative concepts. For example, after detecting video 1, when continuing to detect video 2, the above-mentioned clusters 1-4 are regarded as the first cluster. In the process of detecting video 2, the new cluster 5 is obtained through clustering. Then clusters 1-4 and cluster 5 are both called second clusters.

[0073] 2) Category tags: These are used to identify the category corresponding to the cluster. Category tags can be divided in a variety of ways. For example, they can indicate whether the video being tested is normal or abnormal, allowing for suppression of abnormal videos; or they can indicate whether the user prefers or dislikes the video being tested, allowing for targeted recommendations.

[0074] 3) Inverted index: A data structure that represents the relationship between cluster identifiers and keywords. Keywords extracted from the text of the video to be tested are mapped to corresponding cluster identifiers. When querying the inverted index, cluster identifiers related to the keyword can be quickly found by keyword.

[0075] 4) Forward index: This data structure represents the relationship between cluster identifiers and cluster pointers. Cluster pointers allow for quick location of cluster identifiers in the inverted index, facilitating fast searches. Forward indexes are often used in conjunction with inverted indexes.

[0076] 5) Distributed locks: A mechanism used to coordinate multiple clustering services. When assigning new cluster identifiers to multiple new clusters, due to their large size, distributed locks are used to ensure that only one new cluster can be assigned a new cluster identifier at a time, thus preventing a new cluster from being assigned a new cluster identifier multiple times. Distributed locks achieve this by establishing a mutually exclusive relationship between clustering services. Once a new cluster obtains a new cluster identifier, other clustering services cannot obtain it again until the lock is released.

[0077] 6) Contrastive learning, an unsupervised learning method, learns text representations by comparing the similarities between different text feature vectors. In contrastive learning, the first text feature vector and the second text feature vector of the language understanding model initialized by calling the text of the video to be detected are used as positive examples, and the third feature text vector obtained by calling the language understanding model initialized by calling the text extracted from the videos included in the remaining clusters is used as a negative example. This method can help the language understanding model learn the intrinsic structure and semantic information of the data, thereby improving the generalization ability of the language understanding model. The main advantage of contrastive learning is that it does not require labeled text, so it can be applied to a large number of unlabeled text scenarios.

[0078] 7) Undetected videos refer to videos whose clusters have not yet been detected. For example, when a user uploads a video to a social network, the video is considered to be undetected.

[0079] 8) Detected videos are videos whose clusters have been detected. The clusters are represented by the corresponding cluster category labels. For example, when a user uploads a video to a social network, the video is considered to be detected. When the cluster to which it belongs is detected, the video inherits the category label of the cluster, such as normal video or abnormal video. At this time, the video uploaded by the user is considered a detected video.

[0080] 9) Text vector generation algorithms convert text into numerical vectors, typically using word embedding techniques to represent each word as a vector, and then combining these vectors into a text vector. Text vector generation can help machine learning models better understand and process text data, improving model performance.

[0081] 10) The Bag of Words (BoW) model is a commonly used text representation method that treats text as a bag of words, ignoring the order and frequency of word appearance, and only focusing on whether a word appears. Specifically, the Bag of Words model converts text into a vector consisting of words. The length of the vector is the size of the vocabulary, and each element of the vector indicates whether a word appears in the text. If it appears, the value is 1, otherwise it is 0. One advantage of the model is that it is simple and easy to use, and can quickly convert text into a numerical vector, facilitating subsequent machine learning modeling. However, one of its disadvantages is that it ignores the order and frequency of word appearance, resulting in the loss of grammatical and semantic information in the text.

[0082] 11) ClickHouse, a column-oriented database management system (DBMS) for online analytical processing (OLAP). It supports a wide range of data types, including integers, floating-point numbers, strings, dates, and times, and has rich data manipulation capabilities. With its high performance, high availability, and high scalability, ClickHouse is suitable for processing large datasets such as internet advertising data, mobile application data, and real-time data analysis.

[0083] The applicant discovered that the related art for video classification suffers from a lack of real-time performance and accuracy, as shown in the following aspects:

[0084] 1. Related technologies use offline clustering algorithms to implement video clustering, which has poor recognition timeliness and cannot recognize newly emerging categories in real time.

[0085] 2. Related technologies rely on pure text content for recall and are unable to cope with multimodal data such as video images, video voice, etc. in short video scenes, resulting in low classification accuracy.

[0086] 3. When calculating text similarity, the accuracy and coverage of related technologies are not high, and omissions are prone to occur due to malicious text confrontation.

[0087] Based on the above analysis, the applicant found that the video processing method of the related technology cannot discover new cluster categories in a timely manner, resulting in low efficiency in detecting abnormal videos. To address the above problem, an embodiment of the present application provides a video processing method that can classify large-scale videos to be detected in real time.

[0088] The embodiments of the present application provide a video processing method, apparatus, electronic device, computer program product, and computer-readable storage medium that can ensure real-time and accurate detection of large-scale videos. The following describes exemplary applications of the video processing device provided by the embodiments of the present application. The device provided by the embodiments of the present application can be implemented as various types of terminals, such as laptops, tablet computers, desktop computers, set-top boxes, smartphones, smart speakers, smart watches, smart TVs, and in-vehicle terminals, and can also be implemented as a server. The following describes exemplary applications of the device implemented as a server.

[0089] See also Figure 1 , Figure 1 This is an architectural diagram of the video processing system 100 provided in an embodiment of the present application. To support a video processing application, terminals (terminal 400-1 and terminal 400-2 are shown as examples) are connected to the server 200 via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0090] Terminal 400 is used to publish the video to be detected and display it on a graphical interface (graphic interface 410-1 and graphical interface 410-2 are shown as examples). Server 200 is used to obtain the video to be detected and search it in multiple first clusters. Based on the video to be detected, multiple first clusters are updated to obtain multiple second clusters. Through classification processing, category labels of unlabeled clusters in the multiple second clusters are obtained.

[0091] In some embodiments, in the scenario of abnormal video detection, a user posts a video he or she has shot on a social network. After obtaining the video to be detected uploaded by the user, the server 200 detects the video to be detected. If the category label of the unlabeled cluster of the video to be detected indicates that the video to be detected is an abnormal video, the video to be detected is suppressed and the account of the user who posted the video to be detected is blocked.

[0092] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiments of the present application.

[0093] The embodiments of the present application can be implemented through artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0094] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0095] See also Figure 2 , Figure 2 is a structural diagram of the server 200 provided in an embodiment of the present application, Figure 2 The server 200 shown includes: at least one processor 210, a memory 230 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2 Various buses are labeled as bus system 240 .

[0096] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0097] The memory 230 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 230 may optionally include one or more storage devices that are physically remote from the processor 210.

[0098] The memory 230 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 230 described in the embodiments of the present application is intended to include any suitable type of memory.

[0099] In some embodiments, memory 230 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0100] Operating system 231, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0101] A network communication module 232 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0102] In some embodiments, the apparatus provided in the embodiments of the present application may be implemented in software. Figure 2 The video processing device 233 stored in the memory 230 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 2331, a retrieval module 2332, an update module 2333, a query module 2334, and a classification module 2335. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0103] In some embodiments, the server can implement the video processing method provided in the embodiments of the present application by running various computer-executable instructions or computer programs. For example, the computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. The computer program can be a native program or software module in the operating system. In short, the above-mentioned computer-executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module, or plug-in in any form.

[0104] The video processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0105] See also Figure 3A , Figure 3A This is a first flow chart of the video processing method provided by the embodiment of the present application, which is based on the server and combines Figure 3A The steps shown are explained.

[0106] In step 101, at least one video to be detected and a plurality of first clusters are obtained, wherein the first cluster is a cluster of a plurality of detected videos, and the plurality of first clusters all have category labels.

[0107] The server can receive the video to be detected in the form of a complete file, for example, receiving at least one video file pre-shot by the user; it can also receive the video to be detected in the form of a data stream, for example, receiving a live video stream of at least one anchor.

[0108] It should be noted that steps 101 to 106 can be executed periodically, for example, executing steps 101 to 106 for the videos to be detected received every minute, or they can be executed in batches according to a preset number, for example, storing the received videos to be detected in a cache queue, and when the number of videos in the queue reaches a preset number, executing steps 101 to 106 based on the videos to be detected in the cache queue.

[0109] In some embodiments, the first cluster and the second cluster mentioned below do not specifically refer to a certain cluster, but are used to distinguish between the cluster that already existed before detection (i.e., the first cluster) and the cluster that is updated after clustering the video to be detected (i.e., the second cluster) when classifying at least one current video to be detected.

[0110] As an example, after the server initialization is completed, before the video to be detected is detected, assuming that clusters 1-3 exist, clusters 1-3 are called the first cluster. During the detection of video 1, a new cluster 4 is obtained through clustering, then clusters 1-3 and cluster 4 are both called second clusters.

[0111] It can be understood that the first cluster and the second cluster are relative concepts. For example, after detecting video 1, when continuing to detect video 2, the above-mentioned clusters 1-4 are regarded as the first cluster. In the process of detecting video 2, the new cluster 5 is obtained through clustering. Then clusters 1-4 and cluster 5 are both called second clusters.

[0112] In some embodiments, the first cluster is a cluster of videos that have been detected before step 101 , and the first cluster includes text feature vectors of the detected videos.

[0113] In some embodiments, the types of category tag representations can be diverse, and can represent normal videos or abnormal videos so that the server can detect abnormal videos and suppress them; can represent preferences or non-preferences so that the server can make recommendations based on preferences.

[0114] In step 102, a multimodal feature vector of the video to be detected is obtained.

[0115] In some embodiments, see Figure 3B , Figure 3B This is a second flow chart of the video processing method provided in an embodiment of the present application. Figure 3A Step 102 can be achieved by Figure 3B Steps 1021 to 1023 are implemented as described below.

[0116] In step 1021, text, video frames and audio are extracted from the video to be detected.

[0117] In some embodiments, the introduction, subtitles, or title of the video to be detected can be used as text; or the key frames of the video can be extracted first, and the text can be obtained by performing speech recognition on the key frames. The average pixel intensity of each frame of the video to be detected is calculated, and the difference between the average pixel intensity of the current frame and the previous frame is calculated, and the frame with the largest difference is used as the video frame of the video to be detected; or video frames are periodically collected from the video to be detected as the video frame of the video to be detected. The sound data corresponding to the video frame is obtained, and the obtained sound data is converted into an electrical signal by a recording device, and then the electrical signal is digitized to obtain the audio frame corresponding to the video frame as the audio of the video to be detected.

[0118] In step 1022, a text feature vector corresponding to the text, a video feature vector corresponding to the video frame, and an audio feature vector corresponding to the audio are obtained.

[0119] In some embodiments, the text feature vector corresponding to the text can be obtained through a method based on a bag-of-words model, a word embedding method, or adversarial learning, the video feature vector corresponding to the video frame can be obtained based on a deep residual network (ResidualNetwork, ResNet) model, and the audio feature vector corresponding to the audio can be obtained based on a constant Q transform method (Constant Q Transform, CQTNet) based on a neural network.

[0120] For example, we use the word embedding model to obtain text feature vectors. We segment the text of the video to be detected, represent each word as a vector of fixed length, and learn the vector representation of the word by training the neural network. We concatenate the vectors of all the words to obtain the text feature vector corresponding to the text. For example, we use ResNet to obtain video feature vectors. Figure 4 , Figure 4 This is a schematic diagram of the structure of ResNet provided in the embodiment of the present application. Figure 4In the network structure of ResNet, there are seven parts. The first part does not contain residual blocks and mainly performs convolution, regularization, activation function, and maximum pooling calculations on the input. The second, third, fourth, and fifth parts all contain residual blocks. In the network structure of Resnet, the residual blocks all have three layers of convolution, so the network has a total of 1+3×(3+4+6+3)=49 convolution layers, and the final fully connected layer has a total of 50 layers. The input video frame is converted into a video feature vector through the convolution calculations of the first five parts and the pooling operation of the pooling layer in the sixth part. Taking CQTNet to obtain audio feature vectors as an example, the audio is preprocessed, such as normalization, framing, and windowing. Based on the preprocessed audio, the trained CQTNet model is called to obtain the audio feature vector.

[0121] In step 1023 , the text feature vector, the video feature vector, and the audio feature vector are combined into a multimodal feature vector.

[0122] In some embodiments, the text feature vector, the video feature vector, and the audio feature vector are sequentially concatenated to form a multimodal feature vector.

[0123] For example, if the text feature vector is a, the video feature vector is b, and the audio feature vector is c, then a+b+c is taken as the multimodal feature vector.

[0124] As an example of steps 1021 to 1023, see Figure 5 , Figure 5 This is a schematic diagram of the principle of obtaining similar clusters provided by the embodiment of this application. Figure 5 In this paper, text, video frames and audio are first extracted from the video to be detected; then, the text feature vector is generated based on the Simple Contrastive Learning of Sentence Embeddings (SimCSE), the video feature vector is generated based on ResNet, and the audio feature vector is generated based on CQTNet. The text feature vector, video feature vector and audio feature vector are combined into a multimodal feature vector.

[0125] Continue to see Figure 3A In step 103, multiple first clusters are retrieved based on the multimodal feature vector to obtain retrieval results, wherein the number of the multiple first clusters does not exceed the cluster number threshold, and the retrieval results are used to indicate whether the video to be detected belongs to any first cluster.

[0126] In some embodiments, each first cluster includes a plurality of text feature vectors, and the plurality of text feature vectors correspond to a plurality of detected videos, see Figure 3C , Figure 3CThis is a third flow chart of the video processing method provided in an embodiment of the present application. Figure 3A Step 103 of “retrieve multiple first clusters based on the multimodal feature vector to obtain retrieval results” can be performed by executing Figure 3C Steps 1031 to 1033 are implemented as described below.

[0127] In step 1031 , a similar cluster is retrieved from the plurality of first clusters, wherein the similar cluster is a first cluster corresponding to the plurality of text feature vectors having the highest first similarity to the multimodal feature vector, and the highest first similarity exceeds a first similarity threshold.

[0128] In some embodiments, by setting the first similarity threshold, the introduction of low-similarity noise can be avoided.

[0129] For example, the first cluster has three text feature vectors, and the first similarities of these three text feature vectors with the multimodal feature vector are 0.3, 0.5 and 0.8 respectively. At this time, the first similarity threshold is 0.4, so the first cluster corresponding to the text feature vector with a first similarity of 0.8 is taken as the similarity cluster.

[0130] As an example of step 1031, see Figure 5 , calculating the first similarity between the multimodal feature vector and the multiple text feature vectors in the first cluster, so as to obtain the first cluster corresponding to the text feature vector when the first similarity is the highest, as the similarity cluster.

[0131] In some embodiments, see Figure 3D , Figure 3D This is a fourth flow chart of the video processing method provided in an embodiment of the present application. Figure 3C Step 1031 of "retrieve similar clusters from multiple first clusters" can be achieved by Figure 3D Steps 10311 to 10313 are implemented as described below.

[0132] In step 10311, the node corresponding to the multimodal feature vector in the graph is determined as the target node.

[0133] In some embodiments, the graph is constructed by a vector search technique based on a Hierarchical Navigable SmallWorld (HNSW) network.

[0134] For example, see Figure 6 , Figure 6 This is a schematic diagram of the structure of a graph constructed based on HNSW provided in the embodiment of this application. Figure 6 , the target node 610 is the node corresponding to the multimodal feature vector in the graph.

[0135] In step 10312, the shortest edge among the edges connected to the target node is found.

[0136] For example, see Figure 6 There are four edges connected to the target node 610, namely the edge between the target node 610 and the adjacent node 611, the edge between the target node 610 and the adjacent node 612, the edge between the target node 610 and the adjacent node 613, and the edge between the target node 610 and the adjacent node 614. Among them, the edge between the target node 610 and the adjacent node 612 is the shortest edge.

[0137] In step 10313, adjacent nodes connected to the shortest edge are grouped as similar clusters.

[0138] For example, the adjacent nodes 612 connected to the shortest edge are grouped as similar clusters.

[0139] In some embodiments, see Figure 3E , Figure 3E This is a fifth flow chart of the video processing method provided by the embodiment of the present application. Before step 1031 "retrieve similar clusters from multiple first clusters", execute Figure 3E Steps 201 to 204 are described in detail below.

[0140] In step 201 , a plurality of text feature vectors corresponding to a plurality of detected videos in a first cluster are used as nodes.

[0141] For example, see Figure 6 , Figure 6 There are multiple nodes in , each of which is a text feature vector corresponding to a detected video in the first cluster.

[0142] In step 202 , nodes whose distances to the node are less than a distance threshold are regarded as neighboring nodes of the node, wherein the distance represents the similarity between the node and the neighboring nodes.

[0143] Distance and similarity are positively correlated, that is, the smaller the distance, the higher the similarity between the node and the adjacent nodes; the larger the distance, the lower the similarity between the node and the adjacent nodes.

[0144] For example, see Figure 6 , assuming that the distances between node 615, node 616 and node 613 are 3 and 4 respectively, and the distance threshold is 5, Figure 6 The distances between the remaining nodes in and node 615 are all greater than 5, so node 616 and node 613 are taken as adjacent nodes of node 615. Among them, since the distance between node 615 and node 616 is less than the distance between node 615 and node 613, the similarity between node 615 and node 616 is the highest.

[0145] In step 203, edges are established between each node and its adjacent nodes.

[0146] For example, see Figure 6 , node 615 establishes edges with adjacent node 616 and adjacent node 613 respectively.

[0147] In step 204 , a graph is constructed based on nodes and edges, wherein the graph includes multiple levels, the bottom level includes the text feature vector of each detected video in the first cluster, and the number of nodes included in the multiple levels decreases from the bottom level to the top level.

[0148] For example, see Figure 6 ,The graph includes three levels. The 0th level (i.e., the bottom level) includes the text feature vectors of each detected video in the ,first cluster. From the 0th level to the 2nd level, the number of nodes decreases.

[0149] Continue to see Figure 3C In step 1032, if a similar cluster is retrieved, a retrieval result is generated indicating that the video to be detected belongs to the similar cluster.

[0150] In step 1033 , if no similar cluster is found, a search result is generated indicating that the video to be detected does not belong to any of the first clusters.

[0151] In some embodiments, step 103 "retrieve multiple first clusters based on the multimodal feature vector to obtain retrieval results" can also be achieved by performing the following operations: determine the first similarities between the multiple text feature vectors included in the first cluster and the multimodal feature vector; if the highest first similarity exceeds the first similarity threshold, the first cluster corresponding to the highest first similarity is used as a similar cluster; if a similar cluster is retrieved, a retrieval result is generated that the video to be detected belongs to the similar cluster; if no similar cluster is retrieved, a retrieval result is generated that the video to be detected does not belong to any first cluster.

[0152] As an example, the first similarity can be calculated by cosine similarity, calculating the vector product of the text feature vector included in the first cluster and the multimodal feature vector, calculating the product between the length of the text feature vector included in the first cluster and the length of the multimodal feature vector, and calculating the ratio of the vector product and the product as the first similarity.

[0153] The embodiment of the present application can minimize the clustering calculation for the video to be detected by first searching in the first cluster and then re-clustering if no results are found. Since the computational complexity of retrieval is lower than the computational complexity of clustering, the computational consumption is reduced and the efficiency of detection is improved.

[0154] In some embodiments, the types of category labels include abnormal videos and normal videos, see Figure 3F , Figure 3F This is a sixth flow chart of the video processing method provided in the embodiment of the present application. After step 103, execute Figure 3F Steps 301 to 302 are described in detail below.

[0155] In step 301 , in response to the retrieval result indicating that the video to be detected belongs to any first cluster, the first cluster to which the video to be detected belongs is taken as a target cluster, and a category label of the target cluster is obtained.

[0156] In some embodiments, the cluster label of the target cluster is the category label of the first cluster to which the to-be-detected video belongs.

[0157] For example, if the type of the category label of the first cluster to which the video to be detected belongs is normal video, then the type of the category label of the target cluster is normal video.

[0158] In step 302 , in response to the category label of the target cluster indicating that the video to be detected is an abnormal video, suppression processing is performed on the video to be detected.

[0159] In some embodiments, the suppression process may include shielding the video to be detected and freezing the account of the person who posted the video to be detected.

[0160] For example, see Figure 7 , Figure 7 This is a schematic diagram of an application for suppressing abnormal video provided by an embodiment of the present application. Figure 7 The left side of the figure shows the video screen of the video to be detected. If the video to be detected is detected as an abnormal video, Figure 7 In the right figure of , the video to be detected is suppressed, that is, the video to be detected is no longer displayed.

[0161] The embodiment of the present application performs retrieval in the first cluster through multimodal feature vectors, reflecting the characteristics of the video to be detected from three different dimensions: text, video frame, and audio, thereby enhancing the accuracy of retrieval and improving the recall rate.

[0162] Continue to see Figure 3A In step 104, in response to the retrieval result indicating that the video to be detected does not belong to any first cluster, multiple first clusters are updated based on the video to be detected to obtain multiple second clusters, wherein the multiple second clusters are used to replace the first clusters, the multiple first clusters are partially identical to the multiple second clusters, and the number of the multiple second clusters does not exceed a cluster number threshold.

[0163] In some embodiments, see Figure 3G , Figure 3GThis is the seventh flow chart of the video processing method provided in the embodiment of the present application. Figure 3A Step 104 of "updating multiple first clusters based on the video to be detected to obtain multiple second clusters" can be performed by executing Figure 3G Steps 1041 to 1043 are implemented as described below.

[0164] In step 1041 , a new cluster is created, wherein the new cluster includes the text feature vector of the video to be detected.

[0165] In step 1042 , in response to the total number of clusters of the multiple first clusters and the new cluster being less than a cluster number threshold, the new cluster and the multiple first clusters are both used as second clusters to form multiple second clusters.

[0166] For example, taking cluster size as the evaluation indicator, if there are 3 first clusters in total, the number of new clusters is 1, and the cluster number threshold is 5, and the total number of clusters of multiple first clusters and new clusters is less than the cluster number threshold, then the new cluster and the 3 first clusters will be taken together as the second cluster.

[0167] In step 1043, in response to the total number of clusters of the multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, any first cluster among the multiple first clusters is deleted, and the remaining first clusters and the new cluster are used as second clusters to form multiple second clusters.

[0168] In some embodiments, when deleting any one of the multiple first clusters, the selection criteria of any one are to calculate the evaluation index according to at least one of the following dimensions: cluster size and exposure, and delete the first cluster with the highest evaluation index.

[0169] For example, taking cluster size as the evaluation indicator, if there are 3 first clusters in total, the number of new clusters is 1, the cluster number threshold is 4, and the cluster sizes of the 3 first clusters, that is, the number of text feature vectors included in the 3 first clusters are 6, 7, and 9 respectively, then the first cluster with a cluster size of 6 is deleted, and the first cluster with a cluster size of 7, the first cluster with a cluster size of 9, and the new cluster are collectively used as the second cluster.

[0170] In some embodiments, when deleting any one of the plurality of first clusters, a selection criterion for any one of the first clusters may be: using a least recently used (LRU) algorithm, preferentially deleting the first cluster with the least number of new clusters added within a preset time.

[0171] In this way, when there is no upper limit on the number of clusters, the clustering efficiency will decrease. The embodiment of the present application controls the scale of the cluster through the cluster number threshold, so that it can be deployed online with lower resource occupancy. Each time a video is detected, the cluster can be updated in real time, so that new types of videos can be detected in time, ensuring the real-time nature of the clustering and improving the efficiency of clustering and detection.

[0172] In some embodiments, each first cluster includes multiple text feature vectors and multiple indexes, the multiple text feature vectors correspond to multiple detected videos, and the index includes an inverted index, see Figure 3H , Figure 3H This is the eighth flow chart of the video processing method provided in the embodiment of the present application. Figure 3G Before step 1041, execute Figure 3H Steps 401 to 404 are described in detail below.

[0173] As an example of indexing, see Figure 8A , Figure 8A This is a schematic diagram of the structure of the index provided in the embodiment of the present application. Figure 8A In the table, the index includes a forward index and an inverted index. Cluster 1, Cluster 2, Cluster 3, and Cluster 4 are all first cluster identifiers. Word A and Word B are keywords. Ptrl1-1, Ptrl2-1, and Ptrl2-2 are all pointers to the first cluster. Word A is associated with Cluster 1, Cluster 2, and Cluster 4, Word B is associated with Cluster 2 and Cluster 3, Ptrl1-1 is associated with Cluster 1, and Ptrl2-1 and Ptrl2-2 are associated with Cluster 2.

[0174] In step 401 , a candidate set is retrieved according to an inverted index, wherein the candidate set includes part of the plurality of first clusters.

[0175] In some embodiments, the inverted index represents the association between the first cluster identifier and the keyword, the first cluster identifier is used to represent the first cluster, and the keyword is extracted from the detected video, see Figure 3I , Figure 3I This is the ninth flow chart of the video processing method provided in the embodiment of the present application. Figure 3H Step 401 can be achieved by Figure 3I Steps 4011 to 4014 are implemented as described below.

[0176] In step 4011, a plurality of candidate words are extracted from the video to be detected.

[0177] In some embodiments, text is extracted from the video to be detected, and the text is segmented to obtain multiple candidate words.

[0178] For example, if the text extracted from the video to be detected is "I like to eat mangoes", the multiple candidate words obtained after word segmentation are "I", "like", "eat" and "mango".

[0179] In step 4012, the keyword identical to the candidate word is searched in the inverted index to be used as the target word.

[0180] For example, see Figure 8A , the keywords in the inverted index include word A and word B. If the candidate word is also word A, then word A is used as the target word.

[0181] In step 4013, the first cluster identifier associated with the target word is retrieved in the inverted index as the target identifier.

[0182] In some embodiments, when there are multiple first cluster identifiers associated with the target word in the inverted index, any one of the first cluster identifiers is selected as the target identifier.

[0183] For example, see Figure 8A Cluster 1, cluster 2 and cluster 4 are all first cluster identifiers associated with the target word (word A). Any one of cluster 1, cluster 2 and cluster 4 can be used as the target identifier.

[0184] In step 4014, a first cluster corresponding to the target identifier among the plurality of first clusters is determined to form a candidate set.

[0185] For example, see Figure 8A , if cluster 1 is the target identifier, the first cluster corresponding to cluster 1 is determined as the candidate set.

[0186] In step 402, a second similarity between the text feature vector of the video to be detected and any first cluster in the candidate set is determined.

[0187] In some embodiments, a text feature vector corresponding to a cluster center of any first cluster in the candidate set is obtained, and a second similarity between the text feature vector and the text feature vector of the video to be detected is calculated. The second similarity can be determined by calculating Euclidean distance, cosine similarity, edit distance, or Jaccard similarity.

[0188] For example, taking cosine similarity as an example, the vector product of the text feature vector of the video to be detected and the text feature vector corresponding to the cluster center of any first cluster in the candidate set is calculated, the product of the lengths of the two text feature vectors is calculated, and the ratio of the vector product and the product of the lengths is calculated as the second similarity.

[0189] In step 403, if the second similarity is greater than the second similarity threshold, the first cluster corresponding to the second similarity is used as the belonging cluster, the text feature vector of the video to be detected is added to the belonging cluster, and multiple first clusters including the belonging cluster are used as multiple second clusters.

[0190] For example, if the candidate set includes two first clusters, the second similarities between the text feature vector of the video to be detected and the two first clusters are 0.3 and 0.6 respectively, and the second similarity threshold is 0.5. At this time, the second similarity between the text feature vector of the video to be detected and a first cluster in the candidate set is greater than the second similarity threshold. At this time, the first cluster corresponding to the second similarity of 0.6 is used as the belonging cluster, the text feature vector of the video to be detected is added to the belonging cluster, and multiple first clusters including the belonging cluster are used as multiple second clusters.

[0191] In some embodiments, in response to the category label of the cluster to which the video belongs indicating that the video to be detected is an abnormal video, suppression processing is performed on the video to be detected.

[0192] In step 404 , if the second similarity between the text feature vector of the video to be detected and any first cluster in the candidate set is less than the second similarity threshold, the process proceeds to establishing a new cluster.

[0193] For example, if the candidate set includes two first clusters, the second similarities between the text feature vector of the video to be detected and the two first clusters are 0.3 and 0.4 respectively, and the second similarity threshold is 0.5, at this time, the second similarities between the text feature vector of the video to be detected and the two first clusters in the candidate set are both less than the second similarity threshold, then the process proceeds to execute step 1041 "Establish a new cluster".

[0194] In some embodiments, the index further includes a forward index, which includes an association relationship between the first cluster identifier and the pointer to the first cluster. Figure 3J , Figure 3J The tenth flow chart of the video processing method provided by the embodiment of the present application is shown in FIG. The update of the above index can be performed periodically. Figure 3J Steps 501 to 502 are implemented as described below.

[0195] In step 501, in response to the total number of clusters of multiple first clusters and the new cluster being less than the cluster number threshold, a new cluster identifier is assigned to the new cluster, the association between the new cluster identifier and the pointer of the new cluster is added to the forward index, and the association between the new cluster identifier and the keywords extracted from the video to be detected is added to the inverted index.

[0196] For example, see Figure 8B , Figure 8BThis is a schematic diagram of the application of adding indexes provided by the embodiment of the present application. If the number of the first cluster is 4, the number of the new cluster is 1, and the cluster number threshold is 6, a new cluster identifier is assigned to the new cluster. Figure 8B In the figure, cluster 1, cluster 2, cluster 3 and cluster 4 are all the first cluster identifiers corresponding to the first cluster, cluster 5 is the new cluster identifier corresponding to the new cluster, and the keyword extracted from the video to be detected is word B. At this time, the association between cluster 5 and Ptr5-1 (the pointer to the new cluster) is added to the forward index, and the association between cluster 5 and word B is added to the inverted index.

[0197] In some embodiments, when the number of new clusters is multiple, see Figure 3K , Figure 3K This is the eleventh flow chart of the video processing method provided in the embodiment of the present application. Figure 3J Step 501 of “assigning a new cluster identifier to a new cluster” can be accomplished by Figure 3K Steps 5011 to 5012 are implemented as described below.

[0198] In step 5011, a distributed lock is established between multiple cluster services, wherein the distributed lock is for an unassigned new cluster identifier.

[0199] In some embodiments, the clustering service is a clustering operation for updating a first cluster to obtain a second cluster, and different clustering services are executed on different servers.

[0200] In step 5012, different new cluster identifiers are sequentially allocated to the multiple new clusters based on the distributed lock.

[0201] In some embodiments, if the current server allocates a new cluster identifier for a new cluster, it needs to wait until the previous server completes the operation of allocating a new cluster identifier for the new cluster before it can be executed. The new cluster identifier allocated to each new cluster is different.

[0202] The embodiment of the present application establishes a distributed lock between multiple clustering services to ensure that new cluster identifiers that are not assigned at the same time can only be locked by one clustering service, so that different new clusters are classified with different new cluster identifiers, avoiding clustering errors caused by two new clusters being assigned to the same new cluster identifier.

[0203] In some embodiments, allocating new cluster identifiers may be converted from an offline timed allocation to an online resident memory allocation.

[0204] The embodiment of the present application saves index update time by allocating new cluster identifiers online, achieving updates in minutes, so as to quickly discover new clusters and enhance the detection speed of abnormal clusters, thereby enabling faster warning and suppression of abnormal videos, while also improving cluster coverage and identification coverage.

[0205] Continue to see Figure 3J In step 502, in response to the total number of clusters of the multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, any first cluster deleted from the multiple first clusters is used as a cluster to be deleted, the first cluster identifier corresponding to the cluster to be deleted is used as the cluster identifier to be deleted, the keywords extracted from the video belonging to the cluster to be deleted are used as the words to be deleted, the association between the cluster identifier to be deleted and the pointer to the cluster to be deleted in the forward index is deleted, and the association between the cluster identifier to be deleted and the words to be deleted in the inverted index is deleted.

[0206] For example, see Figure 8C , Figure 8C This is an application diagram of deleting indexes provided by the embodiment of the present application. If the number of first clusters is 4, the number of new clusters is 1, and the cluster number threshold is 5, any first cluster deleted from the multiple first clusters is used as the cluster to be deleted. Figure 8C In the parent graph, cluster 1, cluster 2, cluster 3 and cluster 4 are all the first cluster identifiers corresponding to the first cluster. If cluster 2 is used as the cluster identifier to be deleted and word B is used as the word to be deleted, Figure 8C In the lower graph of , delete the association between cluster 2 and the pointers of the cluster to be deleted (ie, Ptr2-1 and Ptr2-2) in the forward index, and delete the association between cluster 2 and word B in the inverted index.

[0207] In some embodiments, when there are multiple new clusters, clustering services can be performed on different servers. At this time, a global cluster number threshold can be set. Only when the sum of the total number of clusters on multiple servers is greater than or equal to the global cluster number threshold, a new cluster identifier will be assigned to the new cluster.

[0208] The embodiment of the present application sets a cluster number threshold to ensure an upper limit on the number of clusters. Compared with the case where the number of clusters is unlimited, it can ensure the real-time retrieval and update of the first cluster, thereby improving the efficiency of retrieval and cluster update.

[0209] In step 105 , an unlabeled cluster is periodically queried from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected.

[0210] In some embodiments, the value of the period is not limited and depends on the timeliness requirement for video processing in the application scenario. The higher the requirement, the smaller the period. The period is negatively correlated with the timeliness requirement.

[0211] For example, the period may be set to 2 hours, and unlabeled clusters may be queried from the plurality of second clusters every 2 hours.

[0212] In step 106 , the text feature vector of the video to be detected in the unlabeled cluster is obtained, and the unlabeled cluster is classified based on the text feature vector to obtain a category label of the unlabeled cluster.

[0213] In some embodiments, see Figure 3L , Figure 3L This is the twelfth flow chart of the video processing method provided in an embodiment of the present application. Figure 3A Step 106 of "obtaining the text feature vector of the video to be detected in the unlabeled cluster" can be achieved by Figure 3L Steps 1061 to 1068 are implemented as described below.

[0214] In step 1061, an initialized language understanding model is obtained.

[0215] In some embodiments, the language understanding model can be a bag-of-words model, a word embedding model, or a convolutional neural network (CNN) model. Initialization represents randomly assigning values ​​to the model parameters, for example, assigning all model parameters to 0 or all to 1.

[0216] In step 1062 , the initialized language understanding model is called based on the text extracted from the video to be detected to obtain a first text feature vector.

[0217] For example, see Figure 9 , Figure 9 This is a schematic diagram of the structure of the language understanding model for training initialization provided in the embodiment of the present application. Figure 9 In the example, the encoder of the initialized language understanding model is called based on the text extracted from the video to be detected to obtain the first text feature vector.

[0218] In step 1063 , the initialized language understanding model is called based on the text extracted from the video to be detected to obtain a second text feature vector.

[0219] For example, see Figure 9 , based on the text extracted from the video to be detected, the encoder of the initialized language understanding model is called to obtain the second text feature vector.

[0220] In some embodiments, the encoder invoked to obtain the first text feature vector and the encoder invoked to obtain the second text feature vector can be the same encoder. Because the overfitting prevention strategy (Dropout) for the same text in the same encoder may vary, the first text feature vector and the second text feature vector obtained based on the text extracted from the video to be detected are different.

[0221] In step 1064 , the first text feature vector and the second text feature vector are combined into a positive example.

[0222] In some embodiments, the first text feature vector and the second text feature vector are concatenated to obtain a positive example.

[0223] For example, if the first text feature vector is C and the second text feature vector is D, then the positive example is C+D.

[0224] In step 1065 , the initialized language understanding model is called based on the text extracted from the video included in any remaining cluster to obtain a third feature text vector as a negative example, wherein the remaining cluster is the second cluster excluding the unlabeled cluster.

[0225] For example, see Figure 9 , based on the text extracted from the video included in any remaining cluster, the encoder of the initialized language understanding model is called to obtain a third text feature vector as a negative example, where the remaining cluster is the second cluster except the unlabeled cluster.

[0226] In step 1066, a second loss is determined based on the positive examples and the negative examples, and the parameters of the initialized language understanding model are updated based on the second loss to obtain a pre-trained language understanding model.

[0227] In some embodiments, a second loss between positive examples and negative examples is determined by a loss function, and the second loss is back-propagated to update the parameters of the initialized language understanding model. The process of calculating the second loss and updating the parameters is iterated multiple times until the second loss no longer increases or decreases, and the iterative process is stopped to form a pre-trained language understanding model.

[0228] As examples, the loss function may be a mean square error loss function, a cross entropy loss function, a multi-label classification loss function, and a triplet loss function.

[0229] For example, backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer. The weights and biases of the neurons are updated based on the gradient, and the parameters are continuously updated using gradient descent to reduce the loss value. Gradient descent can use various gradient descent algorithms, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0230] In step 1067 , the text to be detected in the video to be detected in the unlabeled cluster is extracted.

[0231] In some embodiments, the introduction, subtitles, or titles of the videos to be detected in the unlabeled cluster may be used as the text to be detected.

[0232] In step 1068, a pre-trained language understanding model is called based on the text to be detected to obtain a text feature vector of the video to be detected.

[0233] In some embodiments, an encoder of a pre-trained language understanding model is called based on the text to be detected to obtain a text feature vector of the video to be detected.

[0234] In some embodiments, the text feature vector of the video to be detected can also be obtained by a supervised SimCSE method. Figure 10 , Figure 10 This is a schematic diagram of the principle of the supervised SimCSE method provided in the embodiment of the present application. In a general text database, two texts with semantic similarity greater than the semantic similarity threshold are selected as positive examples, and the text with semantic similarity less than the semantic similarity threshold is selected as a negative example. Figure 10 In the example, "two dogs are running" and "pets are sitting on the sofa" are positive examples, "there are animals outside the door" is a negative example; "a child is on a skateboard" and "a child is playing with a skateboard" are positive examples, and "there is a toolbox in the house" is a negative example. Using different combinations of positive and negative examples, the initialized language understanding model is trained to obtain a pretrained language understanding model. This pretrained language understanding model is then invoked based on the text to be tested extracted from the video to obtain the text feature vector of the video to be tested.

[0235] The embodiment of the present application obtains the text feature vector of the video to be detected based on the contrastive learning method, so that the text feature vector of the video to be detected is more evenly distributed in space, and the correlation between the texts corresponding to the text feature vectors can be reflected by the distance between different text feature vectors.

[0236] The embodiment of the present application combines multimodal features with text features. On the one hand, the multimodal features reflect the characteristics of the features of the video to be detected from multiple different dimensions, ensuring the comprehensiveness of the retrieval clustering. On the other hand, when the video to be detected does not belong to any of the first clusters, that is, when the video to be detected is classified into the second cluster, the category label of the unlabeled cluster in the second cluster is determined by using a text feature vector that is more lightweight than the multimodal feature vector, taking into account the efficiency of the classification of the video to be detected, and overall taking into account both accuracy and efficiency.

[0237] In some embodiments, the types of category labels include abnormal videos and normal videos. When there are multiple unlabeled clusters, see Figure 3M , Figure 3M This is the thirteenth flow chart of the video processing method provided in the embodiment of the present application. Figure 3A Step 106 of "classifying the unlabeled clusters based on the text feature vector to obtain the category labels of the unlabeled clusters" can be achieved by Figure 3M Steps 601A to 605A are implemented as described below.

[0238] In step 601A, classification processing is performed based on at least one text feature vector included in the unlabeled cluster to obtain the abnormal probability of the unlabeled cluster.

[0239] In some embodiments, when there are multiple text feature vectors, the average value of the multiple abnormality probabilities is calculated as the abnormality probability of the unlabeled cluster.

[0240] For example, multiple abnormal probabilities are obtained as 0.3, 0.4 and 0.8 respectively, then the abnormal probability of the unlabeled cluster is (0.3+0.4+0.8) / 3=0.5.

[0241] In step 602A, the cluster size of the unlabeled cluster and the video exposure amount of the unlabeled cluster are obtained.

[0242] In some embodiments, the cluster size of the unlabeled cluster is the number of videos to be detected in the unlabeled cluster, and the video exposure of the unlabeled cluster is the exposure of the videos to be detected in the unlabeled cluster.

[0243] For example, if the number of videos to be detected in the unlabeled cluster is 5, the cluster size of the unlabeled cluster is 5; if the exposure of the videos to be detected in the unlabeled cluster is 10, the video exposure of the unlabeled cluster is 10.

[0244] In step 603A, the cluster size, video exposure and abnormal probability are weighted and summed according to preset weights to obtain a ranking parameter.

[0245] For example, if the cluster size is 5, the video exposure is 10, the anomaly probability is 0.5, and the preset weights of the cluster size, video exposure and anomaly probability are 0.3, 0.3 and 0.4 respectively, then the weighted sum value is 0.3*5+0.3*10+0.4*0.5=4.7, that is, the sorting parameter is 4.7.

[0246] In step 604A, the multiple unlabeled clusters are sorted in descending order according to the sorting parameters corresponding to the unlabeled clusters.

[0247] For example, if there are three unlabeled clusters, namely unlabeled cluster A, unlabeled cluster B and unlabeled cluster C, and the sorting parameters corresponding to unlabeled cluster A, unlabeled cluster B and unlabeled cluster C are 20, 30 and 10 respectively, then the results of descending sorting are: unlabeled cluster C, unlabeled cluster A and unlabeled cluster B.

[0248] In step 605A, the descending sorted results are sent to a manual review agent, so that the manual review agent generates category labels for the unlabeled clusters according to the order of the descending sorted results.

[0249] In some embodiments, a preset proportion of unlabeled clusters may be selected starting from the first place in the descending order result to generate category labels for the unlabeled clusters.

[0250] For example, the results of descending sorting are: unlabeled cluster C, unlabeled cluster A and unlabeled cluster B. The preset ratio is 2 / 3, so unlabeled cluster C and unlabeled cluster A are selected and are marked with category labels.

[0251] The embodiment of the present application obtains the sorting parameters by performing weighted summation on the cluster size, video exposure and anomaly probability, so that unlabeled clusters can be sorted from multiple dimensions, thereby enhancing the reliability of the sorting results.

[0252] In some embodiments, the unlabeled cluster is a cluster of at least one video to be detected, each unlabeled cluster includes at least one text feature vector, and each text feature vector corresponds to a video to be detected, see Figure 3N , Figure 3N This is the fourteenth flow chart of the video processing method provided in the embodiment of the present application. Figure 3A Step 106 of "classifying the unlabeled clusters based on the text feature vector to obtain the category labels of the unlabeled clusters" can also be done by Figure 3N Steps 601B to 603B are implemented as described below.

[0253] In step 601B, feature information of the object corresponding to the video to be detected in the unlabeled cluster is obtained.

[0254] In some embodiments, feature information of objects corresponding to the to-be-detected videos in the unlabeled cluster may be obtained according to a fixed time window.

[0255] For example, feature information of objects corresponding to the to-be-detected videos in the unlabeled cluster is obtained every 2 hours.

[0256] In step 602B, an aggregation operation is performed on the feature information to obtain aggregated features.

[0257] In some embodiments, streaming aggregation is performed through Flink to aggregate feature information with similarity greater than a similarity threshold to obtain aggregated features.

[0258] In step 603B, the aggregated features are mapped to category labels of the unlabeled clusters.

[0259] In some embodiments, the aggregated features may represent the category of feature information of the object corresponding to the video to be detected, and the aggregated features may be mapped to the category label of the unlabeled cluster according to the mapping relationship between the aggregated features and the video to be detected in the unlabeled cluster.

[0260] In some embodiments, the classification process is implemented by training a classification model, see Figure 3O , Figure 3O This is the fifteenth flow chart of the video processing method provided in the embodiment of the present application. Figure 3A Before step 106 "Classify unlabeled clusters based on text feature vectors", execute Figure 3O Steps 701 to 705 are used to train the classification model, as described in detail below.

[0261] In step 701 , an initialized classification model and a plurality of labeled clusters are obtained, wherein each labeled cluster includes at least one text feature vector.

[0262] In some embodiments, the initialization representation randomly assigns values ​​to the parameters of the model, for example, all the parameters of the model are assigned values ​​of 0 or all are assigned values ​​of 1.

[0263] In step 702, a plurality of text feature vectors are extracted from a plurality of labeled clusters as a training set.

[0264] In step 703 , the category label of the labeled cluster corresponding to each text feature vector is used as the true label of the text feature vector.

[0265] For example, if the category of the labeled cluster corresponding to the text feature vector is labeled as abnormal video, the true label is abnormal video.

[0266] In step 704 , the initialized classification model is called based on the training set to obtain a predicted label, wherein the predicted label represents a predicted category label of the text feature vector in the training set.

[0267] For example, see Figure 11 , Figure 11 This is a schematic diagram of the principle of the classification model for training initialization provided in the embodiment of the present application. Figure 11 In

[15] , the training set is input into the initialized classification model. Since the training set includes text feature vectors, no convolution operation is required. Therefore, the training set is input into the fully connected layer, and then the predicted label is obtained through the fully connected layer.

[0268] In step 705, a first loss is determined based on the true label and the predicted label, and the parameters of the initialized classification model are updated based on the first loss to obtain a trained classification model.

[0269] In some embodiments, a first loss between the true label and the predicted label is determined by a loss function, and the first loss is back-propagated to update the parameters of the initialized classification model. The process of calculating the first loss and updating the parameters is iterated multiple times until the first loss no longer increases or decreases, and the iterative process is stopped to form a trained classification model.

[0270] As examples, the loss function may be a mean square error loss function, a cross entropy loss function, a multi-label classification loss function, and a triplet loss function.

[0271] For example, backpropagation is implemented using the backpropagation algorithm, which calculates the gradient of each neuron from the output layer to the input layer. The weights and biases of the neurons are updated based on the gradient, and the parameters are continuously updated using gradient descent to reduce the loss value. Gradient descent can use various gradient descent algorithms, such as batch gradient descent, stochastic gradient descent, adaptive gradient descent, and momentum gradient descent.

[0272] The embodiment of the present application classifies unlabeled clusters through a classification model, which improves classification efficiency and enhances classification accuracy compared to related technologies that perform classification manually in sequence.

[0273] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0274] Taking the scenario of detecting abnormal videos as an example, when detecting abnormal videos, it is necessary to first extract the text of the abnormal video, then convert it into a text feature vector, and then cluster the text feature vector. The clustering methods used by related technologies mainly include the k-means-based text clustering algorithm and the simhash-based text clustering algorithm.

[0275] The k-means-based text clustering algorithm performs the following clustering steps: First, the text is preprocessed, including removing stop words, punctuation, and numbers. The text is then converted into a vector representation, using methods such as the bag-of-words model, TF-IDF, and Word2Vec. The text feature vectors are then clustered using k-means. Based on the similarity of the text feature vectors within a cluster, clusters with lower similarity are merged until the desired number of clusters is reached. The k-means-based text clustering algorithm is suitable for clustering analysis of small to medium-sized text data.

[0276] The SIMHash-based text clustering method's clustering process is as follows: First, the text is preprocessed, including removing stop words, punctuation, and numbers, and performing operations such as stemming and part-of-speech restoration to convert the text into a bag-of-words model. A hash function is then selected to convert the text into a fixed-length hash value. Common hash functions include SIMHASH and MINHASH. The Hamming distance between the hash values ​​of two texts is calculated, which is the number of different binary bits in the two hash values. The smaller the Hamming distance, the more similar the two texts are. Based on the similarity between the texts, they are divided into different clusters. For each cluster, the average similarity of all the texts within the cluster is calculated as the cluster's similarity. The average similarity between any two clusters is calculated as the similarity between the two clusters. Based on the inter-cluster similarity, clusters with lower similarity are merged until the desired number of clusters is reached. The SIMHash-based text clustering algorithm is suitable for cluster analysis of large-scale text data.

[0277] The text clustering methods of related technologies all complete clustering in an offline manner and are unable to detect the category of the video in time to process the abnormal video in real time. Through the video processing method provided by the embodiment of the present application, the multimodal feature vector of the video to be detected is obtained to retrieve multiple first clusters, wherein the first cluster is a cluster of multiple detected videos. When the retrieval result indicates that the multimodal feature vector does not belong to any first cluster, the multiple first clusters are updated based on the video to be detected to obtain multiple second clusters. Unlabeled clusters are periodically queried from the multiple second clusters to obtain the text feature vector of the video to be detected in the unlabeled cluster. The unlabeled cluster is classified based on the text feature vector to obtain the category label of the unlabeled cluster. When the category label of the unlabeled cluster indicates that the video to be detected is an abnormal video, the video to be detected is suppressed, thereby improving the efficiency of abnormal video detection.

[0278] As Figure 1 An exemplary architecture of the server in Figure 12 , Figure 12 This is a diagram of the architecture of abnormal video detection provided by the embodiment of this application. Figure 12In the paper, the architecture of abnormal video detection is divided into feature layer, service layer, operation layer, strategy layer, application layer and storage layer. Among them, the feature layer is responsible for calculating the text feature vector and multimodal feature vector of the video to be detected, clustering through the text feature vector, and multimodal recall through multimedia features; the service layer is the core part of the architecture, including vector retrieval service and real-time clustering service, which are used to complete real-time cluster discovery and cluster similarity recall. In order to be able to discover new clusters in real time, a global cluster identity is implemented. document, ID) (i.e., the first cluster identifier) ​​is updated in real time; the operation layer includes heat operation services, cluster labeling services, real-time feature calculation and detection alarm services. In order to improve the detection efficiency of abnormal videos, a weighted sorting mechanism is introduced in the cluster labeling service. After classification, the clusters (i.e., unlabeled clusters) are weightedly scored based on the cluster size and video exposure, and then manual review agents complete the detection of the videos to be detected; the strategy layer is a specific suppression strategy for abnormal videos, for example, abnormal videos in abnormal clusters are discovered through multimodal recall; real-time feature calculation of account dimensions is completed based on the streaming computing system to achieve gradient suppression ; Dynamic multiple suppression upgrades account suppression, that is, if multiple videos published by a user are detected as abnormal videos, the video suppression will be upgraded to account suppression; automatic retrospective suppression, that is, for clusters that are reviewed as abnormal (that is, the type of category labeling includes unlabeled clusters of abnormal videos), the incremental short videos of the cluster are suppressed in real time, and the existing short videos of the cluster are retrospectively suppressed; the application layer includes dynamic and account-dimensional submission or suppression, as well as cluster automatic alarms; the storage layer implements cluster information storage through the MySQL database, provides operational support through the ClickHouse database, and implements real-time feature storage through a high-performance key-value database storage system (Feature Key-Value, FeatureKV).

[0279] For the feature layer, it is first necessary to obtain the text feature vector of the video to be detected. The underlying layer of the text clustering service strongly relies on the algorithm for generating text feature vectors. The higher the quality of the generated text feature vectors, the better the similarity recall results. Therefore, in order to address the problem of insufficient clustering hit accuracy, it is necessary to focus on optimizing the text feature vector generation algorithm. The text feature vector generation models used in related technologies are mostly based on pre-trained language models, such as the Bidirectional Encoder Representations from Transformers (BERT) model. Although BERT has achieved good results in many downstream tasks, the text feature vectors generated by the native BERT are not effective. The effect of calculating the similarity of text feature vectors of different texts generated by the native BERT is lower than the effect of calculating the similarity of text feature vectors of different texts generated by the unsupervised word representation model (Global Vectors for Word Representation, Glove) based on the word co-occurrence matrix. The main reason is that the training goal of BERT pre-training does not consider how to generate a text feature vector suitable for vector search or similarity calculation. Since the BERT text feature vector space is anisotropic, the text feature vectors generated by BERT are unevenly distributed in space, and the distance between text feature vectors cannot well represent the correlation between texts. Therefore, a method based on simple contrastive learning to implement sentence embeddings (Simple Contrastive Learning of Sentence Embeddings, SimCSE) is used to generate text feature vectors. This method uses the idea of ​​contrastive learning, in which the distance between text feature vectors with the same semantics is close enough, and the distance between text feature vectors with different semantics is as far as possible. The SimCSE model achieves better results than previous methods for both unsupervised and supervised tasks, and has significant improvements. Refer to the above process of obtaining the text feature vector of the video to be detected, which will not be elaborated here.

[0280] Secondly, it is necessary to obtain multimodal feature vectors. Video content has very rich features, including dynamic descriptions, dynamic optical character recognition (OCR), and dynamic automatic speech recognition (ASR) in the text dimension, as well as cover images, video images, and audio features in the multimedia dimension. If clustering discovery is based solely on the text dimension, it will be difficult to deal with situations where there is no text, text dissimilarities, and malicious text confrontation, and it is easy for abnormal groups to bypass detection. Therefore, multimodal feature vectors are used for joint recall. Based on the use of text feature vectors generated based on the SimCSE model, video feature vectors extracted based on the deep residual network (Residual Network, ResNet) model and audio feature vectors extracted based on the constant Q transform method (CQTNet) based on the neural network are combined. During real-time cluster recall, multiple features are combined for joint recall. Among them, only the text feature vector is used for clustering, while the video feature vector and audio feature vector are used for recall, which plays the role of multi-way recall and does not participate in clustering. Specifically, after clustering the text feature vectors, cluster recall is performed using multimodal feature vectors. The vector search module stores the text feature vectors, video feature vectors, and audio feature vectors of the videos in the historical cluster (i.e., the first cluster). During recall, hits are made from multiple feature dimensions, i.e., dynamic content that meets the multimodal similarity threshold (i.e., the first similarity threshold) is also included in the corresponding cluster while ensuring the recall and accuracy of the cluster. After using multimodal feature vectors, cluster recall can be significantly improved. Cluster recall uses a method based on vector similarity. If the similarity of two vectors exceeds a similarity threshold (e.g., 0.9), they are considered similar. The similarity calculation uses cosine similarity. The text feature vector, video feature vector, and audio feature vector of the current video are compared for similarity with the corresponding feature vectors (i.e., the text feature vectors included in the first cluster) of the candidate cluster (i.e., the first cluster). Those that exceed the corresponding similarity threshold (i.e., the first similarity threshold) are considered to belong to the same cluster. The similarity search here is implemented using the vector search service described below.

[0281] In the vector search service at the service layer, for each vector, it is necessary to search for the most similar cluster in the existing clustering results (i.e., the first cluster). Since the number of clusters may reach millions, it is necessary to index the existing cluster vectors to speed up the search efficiency. To this end, a vector search technology based on the Hierarchical Navigable Small World (HNSW) network is adopted. HNSW is a graph-based vector search method. Its basic idea is to form a graph structure by building edges between similar vectors in a vector set, thereby improving the query speed. This method uses a hierarchical graph structure to further accelerate the search speed. The main challenges faced in building an online similarity search service based on HNSW are high memory overhead, high search request volume, and time-consuming index update synchronization. Since HNSW itself is a stand-alone search engine, it is limited by the performance of a single machine. Therefore, in order to cope with large data volume scenarios, it needs to be expanded to a distributed deployment to address the problems of high memory overhead, high search request volume, and time-consuming index update synchronization. The solution adopted here is to split a complete index into multiple parts (shards) and deploy them on multiple machines. This can effectively solve the problems of memory bottlenecks and high search request volume. At the same time, multiple shards can be deployed on each machine for disaster recovery and read performance scaling. A real-time clustering service is used to support real-time vector similarity retrieval. For each video, after generating a text feature vector, it is input into the vector search service to find the most similar cluster center vector. If the cosine similarity between the two exceeds the specified threshold (i.e., the first similarity threshold), then the vector is considered to have hit the existing cluster (i.e., the first cluster), and the corresponding cluster ID (i.e., the first cluster identifier) ​​will be returned. Otherwise, it is considered that there is no hit, and the logic of the real-time clustering service will continue.

[0282] In real-time clustering services, see Figure 13 , Figure 13 This is a logical diagram of the real-time clustering service provided by the embodiment of this application. Figure 13In , for a new short video (i.e., the video to be detected), first a candidate set is retrieved based on the inverted index, and then the cosine similarity is used to further search for the most similar cluster from the candidate set. Then, we judge one by one whether the vector of the current short video is similar enough to a cluster in the candidate set. If it is similar enough (i.e., the second similarity is greater than the second similarity threshold), then it is classified into this category (i.e., the belonging cluster); if no sufficiently similar cluster is found from the candidate set, then we judge whether the current number of clusters is greater than the given single-machine cluster number threshold (i.e., the cluster number threshold), where the single-machine cluster number threshold represents the upper limit of the total number of clusters that can be accommodated in a single clustering service machine. If it is less than the single-machine cluster number threshold, then we divide the short video into a new cluster. If it is greater than or equal to the single-machine cluster number threshold, the least recently used algorithm (LRU) is used to preferentially delete the cluster that has been hit the least number of times within the preset time (i.e., the first cluster). At the same time, Figure 13 During the update, the cluster index is also updated. The cluster index is used to index existing clusters. The index ensures efficient sample comparison and supports fast query, addition, and deletion. The index is established after the cluster (i.e., the second cluster) is generated. Because it uses similarity search rather than precise sample matching, the cluster is flexible and resistant to change. To achieve fast query capabilities, an inverted index is used. A short video is selected as the center of the cluster (i.e., the first cluster). The text content in the short video is preprocessed and tokenized to obtain a list of words. The mapping relationship between words and clusters (i.e., the association between the first cluster identifier and the keyword) is then obtained and stored in the inverted index. This allows for quick search of relevant clusters based on the word. To support addition and deletion, a two-level indexing approach is used. First, a forward index is used to index cluster-cluster pointers (i.e., the association between the first cluster identifier and the pointer to the first cluster). The cluster is used as the key, and the value stores a pointer to the cluster in the inverted index list. In this way, the position of the cluster in the inverted index can be quickly located based on the cluster center, making it easier to add and delete clusters.

[0283] In the real-time clustering service, in order to generate a unique global cluster ID (i.e., the first cluster identifier), we adopt the method of discovering new clusters online in real time and updating the global cluster (i.e., the first cluster) offline. The offline timed and single-instance update method can ensure that the generated cluster ID has the characteristics of sequential self-increment, non-conflict, and global uniqueness. The generated cluster ID is bound to a short video cluster with specific semantics. The short videos in the same cluster are similar in content. Since the cluster ID is persistent, it can be considered that the cluster ID is bound to these homogeneous short video contents. The cluster ID can be used to detect the cluster change curve, and at the same time, the cluster can be continuously effective and utilized online after one labeling.

[0284] To ensure cluster coverage, a global threshold for the number of new clusters is set. This threshold represents the minimum number of similar short videos required to generate a global cluster. A lower threshold makes it easier to generate clusters, allowing more short videos to be assigned to cluster IDs and achieving higher cluster coverage. A higher threshold emphasizes high-frequency top clusters and reduces the focus on smaller, long-tail clusters. The threshold is set and adjusted by the business side and cannot be adjusted automatically. Increasing the number of clusters without limit—that is, when the number of clusters becomes very large—can reduce clustering efficiency. In the real-time clustering service, single-machine clustering is temporary clustering and only takes effect on the current single machine. The number of clusters in a single machine is subject to the single-machine cluster number threshold. Only when the clusters in multiple single machines are reported and aggregated, and the sum of the number of clusters in multiple single machines is greater than or equal to the global new cluster number threshold, a new global cluster (i.e., a new cluster) is generated and a global cluster ID (i.e., a new cluster identifier) ​​is assigned to the new global cluster. Online traffic is distributed across multiple machines, multiple computer rooms, and even multiple Internet Data Centers (IDCs). In order to achieve unified cluster ID numbering and statistics, the clustering results of each machine need to be reported and summarized, and the latest clustering results need to be updated to the vector search service. See [ 15 ] for more information. Figure 14 , Figure 14 It is a schematic diagram of the architecture of multi-machine distributed parallel clustering provided by an embodiment of the present application. The lower the global threshold for the number of new clusters, the easier it is to generate clusters. After clustering, each machine reports the clusters that appear more frequently. After receiving the report, the aggregation machine will perform secondary aggregation on the reported data. When the cluster size reaches the specified upper threshold, it will be reported to the offline database. Then, in an offline state, a new cluster ID (i.e., a new cluster identifier) ​​will be uniformly assigned to the newly added clusters (i.e., new clusters), and synchronized to the online vector search service at regular intervals. The cluster center of a cluster will not change. When a new cluster (i.e., a new cluster) appears, a new cluster ID will be assigned. According to Figure 14After long-term operation of this architecture, some scenarios, due to the large volume of data, generate a large number of clusters. Even with the introduction of a mechanism to eliminate expired clusters, the number of clusters can still reach millions. When the cluster size is large, the offline scheduled loading of existing clusters and indexing takes a long time. In some scenarios, loading can take as long as 30 to 40 minutes, exceeding the pre-set 5-minute update delay. The timely detection of new clusters directly impacts the timeliness of malicious content alerts and the cluster coverage.

[0285] To optimize slow updates, the offline cluster index building process has been moved online, keeping the existing global clusters resident in memory to eliminate the time required for each offline load. Unlike typical online services, the global cluster update process is single-instance, scheduled, and has a maximum number of clusters. To maintain this single-instance feature, it is necessary to ensure that only one online machine is executing the global cluster update for each scenario at a time. Therefore, each scenario is first assigned to a single online machine based on the scenario ID, and then a distributed lock is used to ensure single-instance performance. This single-instance feature ensures unique cluster ID generation. Since the online real-time clustering process is concurrent and distributed, multiple clusters are generated simultaneously. To avoid errors caused by different clusters being assigned the same cluster ID, a single-instance cluster ID assignment process is required. The optimized cluster service global cluster ID update method switches from offline scheduled updates to online, in-memory updates, eliminating the time required to load the global cluster index each time and achieving true minute-level updates. After optimization, new clusters can be discovered more quickly, early warnings can be issued more quickly, and cluster coverage and label coverage can be improved.

[0286] For the operation layer, we first carry out manual operation services and clustering labeling services. The real-time clustering anti-anomaly service is an unsupervised task. The clustering results are hot short video content, which requires manual review for continuous operation. Therefore, a labeling system that is compatible with the real-time clustering service is designed to operate the clustering results. Figure 15 , Figure 15 This is a schematic diagram of the principle of the manual operation solution provided in the embodiment of the present application. Figure 15In the process of real-time clustering, after the short videos have passed the vector retrieval service and completed real-time clustering, the results of the online real-time clustering are reported to the Clickhouse database. Based on Clickhouse, the clustering results are analyzed, and the top popular clusters in the current time period are pulled regularly (for example, every hour). The definition of the top popular cluster can be measured from the dimensions of cluster size, total exposure of short videos in the cluster (i.e., video exposure), and the overall abnormality level (abnormality probability) of short videos in the cluster. The top popular clusters are imported into the cluster annotation service, and the operator labels the clusters detected by the system (i.e., generates category labels for unlabeled clusters). The obtained short video labels (i.e., category labels) are promptly fed back to the online platform after submission. The online business logic processes the cluster accordingly based on the labeling results. For example, if a short video cluster is marked as abnormal, all short videos in the cluster will be blocked, and the accounts that send these abnormal short videos will also be suppressed.

[0287] Currently, when clustering is applied to business operations for suppression, it often relies on human review results to implement policy processing. The processing path is: cluster discovery -> manual review -> policy suppression. However, business practice has found that in most scenarios, the proportion of anomalous content is relatively low, with a general probability of anomaly below 10%. This means that the majority of cluster review results are non-anomalous, and human resources are not effectively utilized. Furthermore, due to limited human resources, a large number of clusters remain unreviewed, and anomalous content in cluster results cannot be promptly addressed. To improve human review efficiency, clusters for review are pre-ranked, with highly suspicious clusters receiving higher rankings and prioritized for review.

[0288] Priority is given to clusters with large cluster size, high probability of anomalies, and high dynamic exposure (i.e., video exposure) for review. Cluster size and dynamic exposure can be directly calculated. To identify content with a high probability of anomalies within a cluster, a classification model is introduced for pre-sorting. For the large amount of existing manually annotated cluster data, multiple annotated clusters are selected from the cluster data, and multiple different texts are extracted from each cluster. A classification model is then trained. The trained classification model has an accuracy rate greater than 92% and a recall rate greater than 86%. Manual review results for clusters are subject to error, and the review of video account scenarios is often influenced by the judgment of video content in addition to text. Furthermore, the machine review results are not used for direct judgment, but only as a basis for ranking. After obtaining the classification model, it is used to classify and predict the unlabeled cluster texts before sending them for review. For example, if there are 10 texts in an unlabeled cluster, and 8 of them are anomalous, the unlabeled cluster is considered anomalous. A weighted score is then assigned to the cluster based on the cluster size and dynamic exposure before submission for review. After adding this sorting strategy, the human review rate for malicious clustering increased from 17% to 52%, effectively improving the review efficiency.

[0289] In the real-time feature calculation service, in addition to manually labeling the clustering results, when manpower is insufficient, cluster label recognition can also be performed automatically. Figure 16 , Figure 16 This is a schematic diagram of the principle of the real-time feature calculation service provided by the embodiment of this application. Figure 16 In this paper, a comprehensive object feature (i.e., object characteristic information) is first constructed based on data from multiple dimensions. Streaming computing is used to perform feature aggregation analysis on the user identification number (UIN) in the real-time clustering results. This indirectly derives the cluster label (i.e., category identifier) ​​for appropriate action. A real-time computing system based on a distributed big data computing engine (Flink) performs real-time cluster data analysis, and a policy platform is used to implement suppression actions. The specific process of streaming computing aggregation analysis is as follows: object features of all UINs in each cluster are continuously aggregated according to a fixed time window, summarizing and analyzing the object features within the cluster. These aggregated object features can be used as the cluster label (i.e., category label) to assist in cluster action decisions. Furthermore, a real-time detection system is used to observe the clustering results and perform real-time ranking detection, enabling real-time alerts for clusters with abnormal curve fluctuations.

[0290] In the detection and alarm service, the detection system is used to help operators understand the current clustering situation. Figure 17 , Figure 17 This is an application diagram of the detection and alarm service provided by the embodiment of this application. Based on the data analysis function of Clickhouse, the detection system displays the current clustering situation in real time (for example, displaying the most popular short video content at the moment). Figure 17 In the text center alert, a trend curve of the magnitude change of suspicious clusters is generated, and abnormal videos containing abnormal text are displayed in the video account popularity risk review alert. For clusters with abnormal magnitude changes, automatic warnings are issued based on the anomaly detection algorithm. For abnormal clusters discovered in the detection system, operators can mark and suppress them.

[0291] The embodiment of the present application combines multimodal features and text features. On the one hand, the multimodal features reflect the characteristics of the features of the video to be detected from multiple different dimensions, thereby ensuring the comprehensiveness of the retrieval clustering. On the other hand, when the video to be detected does not belong to any of the first clusters, that is, when the video to be detected is classified into the second cluster, the category label of the unlabeled cluster in the second cluster is determined by the text feature vector that is more lightweight than the multimodal feature vector, taking into account the efficiency of the classification of the video to be detected, and taking into account both accuracy and efficiency as a whole; when searching multiple first clusters, the scale of the cluster is controlled by the cluster number threshold, compared with the situation where the scale of the cluster in the existing technology continues to grow, the scale of the computing power consumption of the cluster calculation is controlled, and combined with the periodic classification processing of the videos to be detected in the unlabeled cluster classification, the videos to be detected belonging to the unlabeled cluster can be classified in time, and the real-time classification of the videos to be detected is ensured from the two dimensions of computing power consumption and execution time. By establishing distributed locks across multiple clustering services, a new cluster identifier can only be locked by one clustering service at a time, ensuring its uniqueness. A two-level index structure, comprised of forward and reverse indexes, enables efficient retrieval, insertion, and deletion of clusters. Classifying unlabeled clusters using a classification model allows for rapid identification of anomalous videos, improving both the efficiency of detecting anomalous videos and the accuracy of suppressing them.

[0292] The following continues to describe the exemplary structure of the video processing device 233 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the video processing device 233 of the memory 230 may include:

[0293] The acquisition module 2331 is used to acquire at least one video to be detected and multiple first clusters, wherein the first cluster is a cluster of multiple detected videos, and the multiple first clusters all have category labels; and acquire a multimodal feature vector of the video to be detected.

[0294] The retrieval module 2332 is used to retrieve multiple first clusters based on the multimodal feature vector to obtain retrieval results, wherein the number of the multiple first clusters does not exceed the cluster number threshold, and the retrieval results are used to indicate whether the video to be detected belongs to any first cluster.

[0295] The updating module 2333 is used to update multiple first clusters based on the video to be detected in response to the retrieval result indicating that the video to be detected does not belong to any first cluster, and obtain multiple second clusters, wherein the multiple second clusters are used to replace the first clusters, the multiple first clusters are partially identical to the multiple second clusters, and the number of the multiple second clusters does not exceed the cluster number threshold.

[0296] The query module 2334 is configured to periodically query an unlabeled cluster from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected.

[0297] The classification module 2335 is configured to obtain text feature vectors of the video to be detected in the unlabeled cluster, and classify the unlabeled cluster based on the text feature vectors to obtain category labels of the unlabeled cluster.

[0298] In some embodiments, the acquisition module 2331 is also used to extract text, video frames and audio from the video to be detected; obtain the text feature vector corresponding to the text, the video feature vector corresponding to the video frame and the audio feature vector corresponding to the audio; and combine the text feature vector, the video feature vector and the audio feature vector into a multimodal feature vector.

[0299] In some embodiments, each first cluster includes multiple text feature vectors, and the multiple text feature vectors correspond to multiple detected videos. The retrieval module 2332 is also used to retrieve similar clusters from the multiple first clusters, wherein the similar cluster is the first cluster corresponding to the multiple text feature vectors with the highest first similarity to the multimodal feature vector, and the highest first similarity exceeds the first similarity threshold; if a similar cluster is retrieved, a retrieval result is generated that the video to be detected belongs to the similar cluster; if no similar cluster is retrieved, a retrieval result is generated that the video to be detected does not belong to any first cluster.

[0300] In some embodiments, the retrieval module 2332 is further used to determine that the node corresponding to the multimodal feature vector in the graph is the target node; find the shortest edge among the edges connected to the target node; and cluster the adjacent nodes connected to the shortest edge as similarity.

[0301] In some embodiments, the retrieval module 2332 is further used to take multiple text feature vectors corresponding to multiple detected videos in the first cluster as nodes; take nodes whose distance to the node is less than a distance threshold as adjacent nodes of the node, wherein the distance represents the similarity between the node and the adjacent nodes; establish edges between each node and the adjacent nodes; construct a graph based on the nodes and edges, wherein the graph includes multiple levels, the bottom level includes the text feature vector of each detected video in the first cluster, and the number of nodes included in the multiple levels decreases from the bottom level to the top level.

[0302] In some embodiments, the types of category tags include abnormal videos and normal videos. The retrieval module 2332 is also used to respond to the retrieval result characterizing that the video to be detected belongs to any first cluster, take the first cluster to which the video to be detected belongs as the target cluster, and obtain the category tag of the target cluster; respond to the category tag of the target cluster characterizing that the video to be detected is an abnormal video, and suppress the video to be detected.

[0303] In some embodiments, the update module 2333 is further used to establish a new cluster, wherein the new cluster includes the text feature vector of the video to be detected; in response to the total number of clusters of multiple first clusters and the new cluster being less than a cluster number threshold, the new cluster and the multiple first clusters are both used as second clusters to form multiple second clusters; in response to the total number of clusters of multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, any one of the multiple first clusters is deleted, and the remaining first clusters and the new cluster are both used as second clusters to form multiple second clusters.

[0304] In some embodiments, each first cluster includes multiple text feature vectors and multiple indexes, the multiple text feature vectors correspond to multiple detected videos, the index includes an inverted index, and the update module 2333 is also used to retrieve a candidate set based on the inverted index, wherein the candidate set includes part of the multiple first clusters; determine the second similarity between the text feature vector of the video to be detected and any one of the first clusters in the candidate set; if the second similarity is greater than the second similarity threshold, then the first cluster corresponding to the second similarity is used as the belonging cluster, and the text feature vector of the video to be detected is added to the belonging cluster, and the multiple first clusters including the belonging cluster are used as multiple second clusters; if the second similarity between the text feature vector of the video to be detected and any one of the first clusters in the candidate set is less than the second similarity threshold, then enter the process of establishing a new cluster.

[0305] In some embodiments, the inverted index represents the association between the first cluster identifier and the keyword, the first cluster identifier is used to represent the first cluster, the keyword is extracted from the detected video, and the update module 2333 is also used to extract multiple candidate words from the video to be detected; search for the same keyword as the candidate word in the inverted index as the target word; retrieve the first cluster identifier associated with the target word in the inverted index as the target identifier; determine the first cluster corresponding to the target identifier in multiple first clusters to form a candidate set.

[0306] In some embodiments, the index also includes a forward index, which includes an association between a first cluster identifier and a pointer to the first cluster. The update module 2333 is also used to, in response to the total number of clusters of multiple first clusters and the new cluster being less than a cluster number threshold, assign a new cluster identifier to the new cluster, add the association between the new cluster identifier and the pointer to the new cluster to the forward index, and add the association between the new cluster identifier and the keyword extracted from the video to be detected to the inverted index; in response to the total number of clusters of multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, use any first cluster deleted from the multiple first clusters as the cluster to be deleted, use the first cluster identifier corresponding to the cluster to be deleted as the cluster identifier to be deleted, use the keyword extracted from the video belonging to the cluster to be deleted as the word to be deleted, delete the association between the cluster identifier to be deleted and the pointer to the cluster to be deleted in the forward index, and delete the association between the cluster identifier to be deleted and the word to be deleted in the inverted index.

[0307] In some embodiments, when there are multiple new clusters, the update module 2333 is also used to establish a distributed lock between multiple clustering services, wherein the distributed lock is for unassigned new cluster identifiers; and different new cluster identifiers are assigned to multiple new clusters in turn based on the distributed lock.

[0308] In some embodiments, the classification module 2335 is also used to obtain an initialized language understanding model; call the initialized language understanding model based on the text extracted from the video to be detected to obtain a first text feature vector; call the initialized language understanding model based on the text extracted from the video to be detected to obtain a second text feature vector; combine the first text feature vector and the second text feature vector into a positive example; call the initialized language understanding model based on the text extracted from the video included in any remaining cluster to obtain a third feature text vector as a negative example, wherein the remaining cluster is the second cluster other than the unlabeled cluster; determine a second loss based on the positive example and the negative example, and update the parameters of the initialized language understanding model based on the second loss to obtain a pre-trained language understanding model; extract the text to be detected of the video to be detected in the unlabeled cluster; call the pre-trained language understanding model based on the text to be detected to obtain a text feature vector of the video to be detected.

[0309] In some embodiments, the types of category labels include abnormal videos and normal videos. When there are multiple unlabeled clusters, the classification module 2335 is further used to perform classification processing based on at least one text feature vector included in the unlabeled cluster to obtain the abnormal probability of the unlabeled cluster; obtain the cluster size of the unlabeled cluster and the video exposure of the unlabeled cluster; perform weighted summation of the cluster size, video exposure and abnormal probability according to preset weights to obtain sorting parameters; sort multiple unlabeled clusters in descending order according to the sorting parameters corresponding to the unlabeled clusters; send the results of the descending sorting to the manual review seat, so that the manual review seat generates category labels for the unlabeled clusters in the order of the results of the descending sorting.

[0310] In some embodiments, the unlabeled cluster is a cluster of at least one video to be detected, each unlabeled cluster includes at least one text feature vector, each text feature vector corresponds to a video to be detected, and the classification module 2335 is also used to obtain feature information of the object corresponding to the video to be detected in the unlabeled cluster; perform aggregation operations on the feature information to obtain aggregated features; and map the aggregated features to category labels of the unlabeled cluster.

[0311] In some embodiments, the classification processing is implemented by a trained classification model. The classification module 2335 is also used to obtain an initialized classification model and multiple labeled clusters, wherein each labeled cluster includes at least one text feature vector; extract multiple text feature vectors from the multiple labeled clusters as a training set; use the category label of the labeled cluster corresponding to each text feature vector as the true label of the text feature vector; call the initialized classification model based on the training set to obtain a predicted label, wherein the predicted label represents the predicted category label of the text feature vector in the training set; determine a first loss based on the true label and the predicted label, and update the parameters of the initialized classification model based on the first loss to obtain a trained classification model.

[0312] The present invention provides a computer program product including a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the video processing method described in the present invention.

[0313] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the video processing method provided by the embodiment of the present application, for example, Figure 3A The video processing method is shown.

[0314] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0315] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0316] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0317] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0318] In summary, the multimodal features and text features are processed in conjunction with each other through the embodiment of the present application. On the one hand, the multimodal features reflect the characteristics of the features of the video to be detected from multiple different dimensions, thereby ensuring the comprehensiveness of the retrieval clustering. On the other hand, when the video to be detected does not belong to any of the first clusters, that is, when the video to be detected is classified into the second cluster, the category label of the unlabeled cluster in the second cluster is determined by the text feature vector that is more lightweight than the multimodal feature vector, taking into account the efficiency of the classification of the video to be detected, and taking into account both accuracy and efficiency as a whole. When searching multiple first clusters, the size of the cluster is controlled by the cluster number threshold. Compared with the situation where the size of the cluster in the existing technology continues to grow, the scale of the computing power consumption of the cluster calculation is controlled. Combined with the periodic classification of the videos to be detected in the unlabeled cluster classification, the videos to be detected belonging to the unlabeled cluster can be classified in time, and the real-time classification of the videos to be detected is ensured from the two dimensions of computing power consumption and execution time. By establishing distributed locks across multiple clustering services, a new cluster identifier can only be locked by one clustering service at a time, ensuring its uniqueness. A two-level index structure, comprised of forward and reverse indexes, enables efficient retrieval, insertion, and deletion of clusters. Classifying unlabeled clusters using a classification model allows for rapid identification of anomalous videos, improving both the efficiency of detecting anomalous videos and the accuracy of suppressing them.

[0319] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A video processing method, characterized in that: The method comprises: Acquire at least one video to be detected and a plurality of first clusters, wherein the first cluster is a cluster of a plurality of detected videos, and the plurality of first clusters all have category labels; Obtaining a multimodal feature vector of the video to be detected; Retrieving the plurality of first clusters based on the multimodal feature vector to obtain a retrieval result, wherein the number of the plurality of first clusters does not exceed a cluster number threshold, and the retrieval result is used to indicate whether the video to be detected belongs to any of the first clusters; In response to the retrieval result indicating that the video to be detected does not belong to any of the first clusters, updating the multiple first clusters based on the video to be detected to obtain multiple second clusters, wherein the multiple second clusters are used to replace the first clusters, the multiple first clusters are partially identical to the multiple second clusters, and the number of the multiple second clusters does not exceed the cluster number threshold; Periodically querying an unlabeled cluster from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected; A text feature vector of the video to be detected in the unlabeled cluster is obtained, and the unlabeled cluster is classified based on the text feature vector to obtain a category label of the unlabeled cluster.

2. The method according to claim 1, characterized in that The updating of the plurality of first clusters based on the video to be detected to obtain a plurality of second clusters includes: The following processing is performed for each video to be detected: Establishing a new cluster, wherein the new cluster includes the text feature vector of the video to be detected; In response to the total number of clusters of the multiple first clusters and the new cluster being less than the cluster number threshold, the new cluster and the multiple first clusters are both used as second clusters to form a plurality of second clusters; In response to the total number of clusters of the multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, any one of the multiple first clusters is deleted, and the remaining first clusters and the new cluster are used as second clusters to form multiple second clusters.

3. The method according to claim 2, characterized in that Each of the first clusters includes a plurality of text feature vectors and a plurality of indexes, wherein the plurality of text feature vectors correspond to a plurality of detected videos, and the index includes an inverted index; Before establishing the new cluster, the method further includes: Retrieving a candidate set according to the inverted index, wherein the candidate set includes part of the plurality of first clusters; Determine a second similarity between the text feature vector of the video to be detected and any one of the first clusters in the candidate set; If the second similarity is greater than a second similarity threshold, the first cluster corresponding to the second similarity is used as the belonging cluster, the text feature vector of the video to be detected is added to the belonging cluster, and the multiple first clusters including the belonging cluster are used as the multiple second clusters; If the second similarity between the text feature vector of the video to be detected and any one of the first clusters in the candidate set is less than the second similarity threshold, the process proceeds to the process of establishing a new cluster.

4. The method according to claim 3, characterized in that The inverted index represents an association between a first cluster identifier and a keyword, the first cluster identifier is used to represent the first cluster, and the keyword is extracted from the detected video; The retrieving a candidate set according to the inverted index includes: Extracting multiple candidate words from the video to be detected; Searching the inverted index for the keyword identical to the candidate word as the target word; Retrieving the first cluster identifier associated with the target word in the inverted index as a target identifier; The first cluster corresponding to the target identifier among the multiple first clusters is determined to form a candidate set.

5. The method according to claim 3, characterized in that The index further includes a forward index, and the forward index includes an association relationship between the first cluster identifier and a pointer to the first cluster; The method further comprises: The following processing is performed periodically: In response to the total number of clusters of the multiple first clusters and the new cluster being less than the cluster number threshold, assigning a new cluster identifier to the new cluster, adding an association between the new cluster identifier and a pointer to the new cluster to the forward index, and adding an association between the new cluster identifier and a keyword extracted from the video to be detected to the inverted index; In response to the total number of clusters of the multiple first clusters and the new cluster being greater than or equal to the cluster number threshold, any one of the multiple first clusters deleted is used as a cluster to be deleted, the first cluster identifier corresponding to the cluster to be deleted is used as the cluster identifier to be deleted, the keywords extracted from the video belonging to the cluster to be deleted are used as the words to be deleted, the association between the cluster identifier to be deleted and the pointer to the cluster to be deleted in the forward index is deleted, and the association between the cluster identifier to be deleted and the words to be deleted in the inverted index is deleted.

6. The method according to claim 5, characterized in that When there are multiple new clusters, allocating new cluster identifiers to the new clusters includes: Establishing a distributed lock between multiple cluster services, wherein the distributed lock is for an unassigned new cluster identifier; Different new cluster identifiers are sequentially allocated to the multiple new clusters based on the distributed lock.

7. The method according to claim 1, characterized in that The types of the category labels include abnormal videos and normal videos; After respectively searching the plurality of first clusters based on the multimodal feature vector to obtain search results, the method further includes: In response to the retrieval result indicating that the video to be detected belongs to any one of the first clusters, taking the first cluster to which the video to be detected belongs as a target cluster, and obtaining the category label of the target cluster; In response to the category label of the target cluster indicating that the video to be detected is an abnormal video, suppression processing is performed on the video to be detected.

8. The method according to claim 1, characterized in that The types of the category labels include abnormal videos and normal videos; When there are multiple unlabeled clusters, performing classification processing based on the multiple text feature vectors included in the unlabeled clusters to obtain category labels of the unlabeled clusters includes: Performing classification processing based on the at least one text feature vector included in the unlabeled cluster to obtain an abnormality probability of the unlabeled cluster; Obtaining a cluster size of the unlabeled cluster and a video exposure amount of the unlabeled cluster; Performing weighted summation of the cluster size, the video exposure, and the abnormality probability according to preset weights to obtain a ranking parameter; sorting the plurality of unlabeled clusters in descending order according to the sorting parameters respectively corresponding to the unlabeled clusters; The descending sorting result is sent to a manual review seat, so that the manual review seat generates the category labels of the unlabeled clusters according to the order of the descending sorting result.

9. The method according to any one of claims 1 to 8, characterized in that The unlabeled cluster is a cluster of at least one of the videos to be detected, each of the unlabeled clusters includes at least one text feature vector, and each of the text feature vectors corresponds to one of the videos to be detected; The performing classification processing based on the multiple text feature vectors included in the unlabeled cluster to obtain the category label of the unlabeled cluster includes: Obtaining feature information of an object corresponding to the video to be detected in the unlabeled cluster; Performing an aggregation operation on the feature information to obtain aggregated features; The aggregated features are mapped to category labels of the unlabeled clusters.

10. The method according to any one of claims 1 to 8, characterized in that The classification process is achieved through a trained classification model; Before classifying the unlabeled clusters based on the text feature vectors, the method further includes: The classification model is trained by: Obtaining an initialized classification model and a plurality of labeled clusters, wherein each of the labeled clusters includes at least one text feature vector; Extracting multiple text feature vectors from the multiple labeled clusters as a training set; Using the category label of the labeled cluster corresponding to each of the text feature vectors as the true label of the text feature vector; Calling the initialized classification model based on the training set to obtain a predicted label, wherein the predicted label represents a predicted category label of the text feature vector in the training set; A first loss is determined based on the true label and the predicted label, and parameters of the initialized classification model are updated based on the first loss to obtain the trained classification model.

11. The method according to any one of claims 1 to 8, characterized in that Each of the first clusters includes a plurality of text feature vectors, and the plurality of text feature vectors correspond to the plurality of detected videos; The step of respectively searching the plurality of first clusters based on the multimodal feature vector to obtain search results includes: The following processing is performed for each of the first clusters: Retrieving a similar cluster from the plurality of first clusters, wherein the similar cluster is the first cluster corresponding to the plurality of text feature vectors having the highest first similarity to the multimodal feature vector, and the highest first similarity exceeds a first similarity threshold; If the similar cluster is retrieved, generating a retrieval result indicating that the video to be detected belongs to the similar cluster; If the similar cluster is not retrieved, a retrieval result is generated indicating that the video to be detected does not belong to any of the first clusters.

12. The method according to claim 11, characterized in that Before retrieving similar clusters from the plurality of first clusters, the method further includes: Taking multiple text feature vectors corresponding to the multiple detected videos in the first cluster as nodes; Taking nodes whose distance to the node is less than a distance threshold as neighboring nodes of the node, wherein the distance represents the similarity between the node and the neighboring nodes; establishing an edge between each of the nodes and the adjacent nodes; Constructing a graph based on the nodes and the edges, wherein the graph includes multiple levels, a bottom level includes the text feature vector of each of the detected videos in the first cluster, and the number of the nodes included in the multiple levels decreases from the bottom level to the top level; The retrieving similar clusters from the plurality of first clusters comprises: Determine a node corresponding to the multimodal feature vector in the graph as a target node; Finding the shortest edge among the edges connected to the target node; The adjacent nodes connected to the shortest edge are grouped as similar clusters.

13. The method according to any one of claims 1 to 8, characterized in that The obtaining of the multimodal feature vector of the video to be detected includes: Extracting text, video frames and audio from the video to be detected; Obtaining a text feature vector corresponding to the text, a video feature vector corresponding to the video frame, and an audio feature vector corresponding to the audio; The text feature vector, the video feature vector, and the audio feature vector are combined into a multimodal feature vector.

14. The method according to any one of claims 1 to 8, characterized in that The obtaining of the text feature vector of the video to be detected in the unlabeled cluster includes: Get the initialized language understanding model; Calling the initialized language understanding model based on the text extracted from the video to be detected to obtain a first text feature vector; Calling the initialized language understanding model based on the text extracted from the video to be detected to obtain a second text feature vector; combining the first text feature vector and the second text feature vector into a positive example; Calling the initialized language understanding model based on text extracted from the video included in any remaining cluster to obtain a third feature text vector as a negative example, wherein the remaining cluster is the second cluster excluding the unlabeled cluster; Determining a second loss based on the positive example and the negative example, and updating parameters of the initialized language understanding model based on the second loss to obtain a pre-trained language understanding model; Extracting the text to be detected from the video to be detected in the unlabeled cluster; The pre-trained language understanding model is called based on the text to be detected to obtain a text feature vector of the video to be detected.

15. A video processing device, characterized in that: The device comprises: An acquisition module is configured to acquire at least one video to be detected and a plurality of first clusters, wherein the first clusters are clusters of a plurality of detected videos, and the plurality of first clusters all have category labels; and acquire a multimodal feature vector of the video to be detected; a retrieval module, configured to retrieve the plurality of first clusters based on the multimodal feature vector, respectively, to obtain retrieval results, wherein the number of the plurality of first clusters does not exceed a cluster number threshold, and the retrieval results are used to indicate whether the video to be detected belongs to any of the first clusters; an updating module, configured to, in response to the retrieval result indicating that the video to be detected does not belong to any of the first clusters, update the plurality of first clusters based on the video to be detected to obtain a plurality of second clusters, wherein the plurality of second clusters are used to replace the first clusters, the plurality of first clusters are partially identical to the plurality of second clusters, and the number of the plurality of second clusters does not exceed the cluster number threshold; a query module, configured to periodically query an unlabeled cluster from the plurality of second clusters, wherein the unlabeled cluster is a second cluster without a corresponding category label, and the unlabeled cluster includes the video to be detected; The classification module is used to obtain the text feature vector of the video to be detected in the unlabeled cluster, and classify the unlabeled cluster based on the text feature vector to obtain the category label of the unlabeled cluster.

16. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; The processor is configured to implement the video processing method according to any one of claims 1 to 14 when executing the computer-executable instructions stored in the memory.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the video processing method according to any one of claims 1 to 14 is implemented.

18. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the video processing method according to any one of claims 1 to 14 is implemented.

Images