Video detection method, device and computer-readable storage medium

Abstract

The present application provides a video detection method, device and computer-readable storage medium; the method includes: obtaining a video to be detected; based on the video to be detected, recalling a video from a video resource library to obtain a video to be compared; separating the audio information corresponding to the video to be detected to obtain the audio to be detected, extracting the features of the audio to be detected in terms of audio characteristics, and obtaining an audio fingerprint to be detected, where the audio characteristics are the auditory characteristics of the audio information; separating the audio information corresponding to the video to be compared to obtain the audio to be compared, extracting the features of the audio to be compared in terms of audio characteristics, and obtaining an audio fingerprint to be compared; comparing the audio fingerprint to be detected with the audio fingerprint to be compared, and determining a video detection result of the video to be detected for the video to be compared based on the comparison result, where the video detection result is a detection result of whether the video to be detected is a duplicate video for the video to be compared. Through the present application, the accuracy of video detection can be improved based on artificial intelligence technology.

Description

Technical Field

[0001] The present application relates to video processing technology in the field of artificial intelligence, and in particular to a video detection method, device, and computer-readable storage medium. Background Art

[0002] With the rapid development of social networks, video has gradually become one of the dominant content formats on the mobile internet. Due to its high engagement and dissemination value, video uploads are increasing in volume, necessitating rapid video review and publication.

[0003] Generally speaking, to review videos, artificial intelligence technology is often used to detect videos based on their images and titles to determine whether they are duplicates of previously uploaded videos. However, because the detection is based on video images and titles, for example, if a video is similar to a "series of lecture videos," the images and titles between the videos are similar. As a result, the detection results often mistakenly identify "series of lecture videos" as duplicates. As a result, video detection accuracy is low. Summary of the Invention

[0004] The embodiments of the present application provide a video detection method, device, and computer-readable storage medium, which can improve the accuracy of video detection.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a video detection method, including:

[0007] Get the video to be tested;

[0008] Based on the video to be detected, recall videos from a video resource library to obtain a video to be compared;

[0009] Separating the audio information corresponding to the video to be detected to obtain the audio to be detected, and extracting the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected, wherein the audio characteristics are auditory characteristics of the audio information;

[0010] Separating the audio information corresponding to the video to be compared to obtain the audio to be compared, and extracting the features of the audio to be compared based on the audio characteristics to obtain the audio fingerprint to be compared;

[0011] The audio fingerprint to be detected is compared with the audio fingerprint to be compared, and a video detection result of the video to be detected relative to the video to be compared is determined based on the comparison result, wherein the video detection result is a detection result of whether the video to be detected is a duplicate video relative to the video to be compared.

[0012] The present invention provides a video detection device, including:

[0013] Video acquisition module, used to acquire the video to be detected;

[0014] A video recall module is used to recall videos from a video resource library based on the video to be detected to obtain a video to be compared;

[0015] a feature acquisition module, configured to separate the audio information corresponding to the video to be detected to obtain the audio to be detected, and extract the audio characteristic features of the audio to be detected to obtain the audio fingerprint to be detected, wherein the audio characteristic features are auditory characteristics of the audio information;

[0016] The feature acquisition module is further configured to separate the audio information corresponding to the video to be compared to obtain the audio to be compared, and extract the features of the audio to be compared based on the audio characteristics to obtain the audio fingerprint to be compared;

[0017] The video detection module is used to compare the audio fingerprint to be detected with the audio fingerprint to be compared, and determine a video detection result of the video to be detected relative to the video to be compared based on the comparison result, wherein the video detection result is a detection result of whether the video to be detected is a duplicate video relative to the video to be compared.

[0018] In an embodiment of the present application, the feature acquisition module is further configured to extract multiple frames of sub-audio to be detected from the audio to be detected based on preset frame units; extract features of each frame of sub-audio to be detected in the multiple frames of sub-audio to be detected in terms of the audio characteristics to obtain an initial sub-audio fingerprint to be detected; perform dimensionality reduction on the initial sub-audio fingerprint to be detected to obtain a sub-audio fingerprint to be detected, thereby obtaining multiple frames of sub-audio fingerprints to be detected corresponding to the audio to be detected, wherein the audio fingerprint to be detected includes the multiple frames of sub-audio fingerprints to be detected.

[0019] In an embodiment of the present application, the feature acquisition module is further configured to perform pre-emphasis processing on the audio to be detected to obtain the audio to be framed.

[0020] In an embodiment of the present application, the feature acquisition module is further configured to extract the multiple frames of sub-audio to be detected from the audio to be framed based on the preset frame unit.

[0021] In an embodiment of the present application, the feature acquisition module is further used to sample the audio to be detected based on a preset sampling frequency to obtain multiple sampling points; among the multiple sampling points, starting from the first sampling point, sequentially select a preset number of sampling points to form a frame of sub-audio to be detected, and continue to select the preset number of sampling points from the position corresponding to the preset number of overlapping sampling points before the end position of the selection of the sampling point as the next frame of sub-audio to be detected, until the multiple sampling points are selected and processed to obtain the multiple frames of sub-audio to be detected, wherein the preset frame unit is determined based on the preset sampling frequency and the preset number of sampling points.

[0022] In an embodiment of the present application, the feature acquisition module is further used to perform windowing processing on each frame of the sub-audio to be detected in the multiple frames of sub-audio to be detected to obtain a sub-audio to be transformed; transform the sub-audio to be transformed into an energy distribution in the frequency domain to obtain a sub-spectrum to be detected, obtain the power spectrum of the sub-spectrum to be detected, and obtain a sub-power spectrum to be detected; smooth the sub-power spectrum to be detected to obtain a sub-smoothed power spectrum; inversely transform the logarithmic energy of the sub-smoothed power spectrum and obtain audio characteristic parameters of a preset order of the inverse transformation result; obtain differential parameters of the audio characteristic parameters, and the frame energy of each frame of the sub-audio to be detected, thereby obtaining the initial sub-audio fingerprint to be detected including one or more of the audio characteristic parameters, the differential parameters and the frame energy.

[0023] In an embodiment of the present application, the feature acquisition module is further configured to remove the lowest frequency feature for each initial sampling point audio fingerprint in the initial sub-audio fingerprint to be detected to obtain an S-1 dimensional feature, wherein the initial sub-audio fingerprint to be detected includes the initial sampling point audio fingerprints of the preset number of sampling points, and each initial sampling point audio fingerprint includes an S-dimensional feature, where S is a positive integer greater than 1; clustering and dimensionality reduction are performed on the S-1 dimensional feature based on a preset number of categories to obtain the preset number of cluster categories; and cluster center information of the cluster category is determined as the sampling point audio fingerprint of each initial sampling point audio fingerprint, thereby obtaining the sub-audio fingerprint to be detected corresponding to the initial sub-audio fingerprint to be detected, wherein the sub-audio fingerprint to be detected includes the audio fingerprints of the sampling points of the preset number of sampling points.

[0024] In an embodiment of the present application, the video detection module is further configured to compare each frame of the audio fingerprint to be detected in the audio fingerprint to be detected with each frame of the audio fingerprint to be compared, respectively, to obtain the comparison results corresponding to each frame of the audio fingerprint to be detected and each frame of the audio fingerprint to be compared; when preset regularity information exists in the comparison results, determining the video detection result that the video to be detected is a duplicate video for the video to be compared, wherein the preset regularity information is a similarity trend between each frame of the audio fingerprint to be detected and each frame of the audio fingerprint to be compared; when the preset regularity information does not exist in the comparison results, determining the video detection result that the video to be detected is a non-duplicate video for the video to be compared.

[0025] In an embodiment of the present application, the video detection module is further used to use the audio fingerprint of each frame to be detected as a one-dimensional attribute of the matrix, the audio fingerprint of each frame to be compared as another dimensional attribute of the matrix, and the comparison result as an element of the matrix to construct a similarity matrix; based on the correspondence between the preset similarity value and the display color, the similarity matrix is ​​converted into a similarity matrix diagram; when the color difference between each display color at the diagonal position in the similarity matrix diagram and the preset color is less than the color difference threshold, it is determined that the preset regular information exists in the comparison result; when the color difference between each display color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that the preset regular information does not exist in the comparison result.

[0026] In an embodiment of the present application, the video recall module is also used to obtain video recall features corresponding to the video to be detected, wherein the video recall features include one or more of content semantic features, text semantic features, title semantic features, body semantic features, frame image semantic features and cover image semantic features; obtain the features to be recalled corresponding to each video in the video resource library, wherein the features to be recalled correspond to the video recall features in feature type; based on the recall similarity values ​​between the video recall features and each feature to be recalled, determine the target feature to be recalled that is similar to the video recall feature from each feature to be recalled, wherein the recall similarity value includes one or more of Euclidean distance, vector dot product value and cosine similarity value; use the video in the video resource library that corresponds to the target feature to be recalled as the recalled video, thereby obtaining the video to be compared that belongs to the recalled video.

[0027] In an embodiment of the present application, the video recall module is also used to obtain the recall feature index corresponding to the video recall feature, and the respective recall feature indexes corresponding to the respective features to be recalled, wherein the respective features to be recalled correspond one-to-one to the respective recall feature indexes; the matching degree between the recall feature index and the respective recall feature indexes is used as the recall similarity value, and based on the recall similarity value, the target feature to be recalled that is similar to the video recall feature is determined from the respective features to be recalled.

[0028] In an embodiment of the present application, the video acquisition module is also used to receive a video detection request sent by a task scheduling device, wherein the video detection request is generated by the task scheduling device in response to a video upload request sent by a video production end device; in response to the video detection request, the video to be detected is obtained from a content storage device.

[0029] In an embodiment of the present application, the video detection device also includes a result processing module, which is used to send the video detection result to a subsequent detection device when the video detection result is that the video to be detected is a duplicate video for the video to be compared, so that the subsequent detection device generates a subsequent detection request for the video detection result, and obtains the target detection result of the video to be detected in response to the subsequent detection request.

[0030] In an embodiment of the present application, the video detection device is also used to send the video to be detected to a task scheduling device when the video detection result is that the video to be detected is a non-duplicate video with respect to the video to be compared, so that the task scheduling device pushes the video to be detected to the content consumption end device through the content distribution device based on the obtained recommendation information, so that the content consumption end device plays the video to be detected.

[0031] The present invention provides a video detection device, including:

[0032] a memory for storing executable instructions;

[0033] The processor is used to implement the video detection method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0034] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the video detection method provided in the embodiment of the present application.

[0035] The embodiments of the present application have at least the following beneficial effects: when comparing the video to be detected with the recalled video to be compared, the video detection result of whether the video to be detected is a duplicate video is determined by comparing the features of the audio information of the video to be detected with the features of the audio information of the video to be compared in terms of audio characteristics; and the audio characteristics are the auditory features of the audio information, such as volume, sound quality and timbre, and videos such as "series of videos of lectures" have differences in audio characteristics; therefore, it is possible to accurately identify whether videos with similar pictures and titles such as "series of videos of lectures" are duplicate videos, thereby improving the accuracy of video detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is an exemplary video detection process diagram;

[0037] Figure 2 This is a schematic diagram of an optional architecture of a video detection system provided in an embodiment of the present application;

[0038] Figure 3 This embodiment of the present application provides Figure 2 A schematic diagram of the composition structure of a server in FIG;

[0039] Figure 4 This is an optional flowchart of the video detection method provided in the embodiment of the present application;

[0040] Figure 5 This is another optional flowchart of the video detection method provided in the embodiment of the present application;

[0041] Figure 6 This is an optional interactive flow diagram of the video detection method provided in the embodiment of the present application;

[0042] Figure 7 is a structural diagram of an exemplary video detection system provided in an embodiment of the present application;

[0043] Figure 8 This is a schematic diagram of an exemplary process for obtaining an audio fingerprint provided in an embodiment of the present application;

[0044] Figure 9 This is a schematic diagram of an exemplary linear relationship between Mel frequency and audio frequency provided in an embodiment of the present application;

[0045] Figure 10 1 is a schematic diagram of filtering results of a set of exemplary Mel-scale triangular filters provided in an embodiment of the present application;

[0046] Figure 11is a schematic diagram of an exemplary matrix similarity graph provided in an embodiment of the present application;

[0047] Figure 12 This is a schematic diagram of another exemplary matrix similarity graph provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0051] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0052] 1) Artificial Intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0053] 2) Machine Learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning typically includes techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, and inductive learning.

[0054] 3) Artificial neural network is a mathematical model that imitates the structure and function of biological neural networks. Exemplary structures of artificial neural networks in the embodiments of the present application include deep neural networks (DNN), convolutional neural networks (CNN) and recurrent neural networks (RNN).

[0055] 4) In response, it is used to indicate the conditions or states on which the executed processing depends. When the dependent conditions or states are met, one or more requests executed can be real-time or have a set delay; unless otherwise specified, there is no restriction on the order in which the multiple requests executed are executed.

[0056] 5) Feeds stream, also known as news source, also known as source material, feed, information provider, feed, summary, source, news subscription, network source, etc., is a data format. The website disseminates the latest information to users through the feeds stream, that is, the feeds stream is an information stream that is continuously updated and presented to users. Usually, the feeds stream is arranged in a timeline manner, and the timeline is the most basic display form of the feeds stream. The prerequisite for users to subscribe to a website is that the website provides a news source. In addition, converging the feeds stream into one place is called aggregation, and the software used for aggregation is called an aggregator; among them, an aggregator is software specifically used to subscribe to information on a website, such as an RSS (Really Simple Syndication) reader, a "feed" reader, a news reader, etc.

[0057] 6) MCN (Multi-Channel Network) is used to ensure the continuous output of PGC content with the strong support of capital, thereby ultimately achieving stable commercial realization. In other words, MCN helps content producers (including the video production terminal device in the embodiment of this application) focus on content creation on the one hand, and connects with platforms and fans to package, strengthen promotion and promote realization on the other hand.

[0058] 7) PGC (Professional Generated Content, professionally produced content (for example, video websites) or expert-produced content (for example, Weibo)) refers to content generators with personalized content, diversified perspectives, and virtualized social relationships. It is also called PPC (Professionally-produced Content).

[0059] 8) Short videos, also known as short clips, are a form of internet content dissemination, generally referring to video content under five minutes long and distributed on new media platforms. These videos are typically broadcast on various new media platforms, suitable for viewing on the go and in short, casual moments, and are frequently pushed, ranging from a few seconds to several minutes. These content incorporates themes such as skill sharing, humor and quirkiness, fashion trends, social issues, street interviews, public welfare education, advertising creativity, and commercial customization. Due to their short nature, short videos can be produced as standalone pieces or as part of a series. Furthermore, unlike micro-films and live broadcasts, short video production does not require specific formats or team requirements. They offer a simple production process, low barriers to entry, and high levels of engagement, while also possessing greater dissemination value than live broadcasts. The emergence of short videos has enriched the forms of native advertising in new media. Therefore, short videos will gradually become a dominant content format on the mobile internet, replacing graphic content consumption to a certain extent and gradually gaining a dominant position in graphic media such as news and social media platforms. Short videos are usually displayed in the form of feeds for users to quickly refresh. For example, the "News Feed" on the Facebook homepage (a new type of aggregator) has a subscription source of friends or people they follow, and the video content is the dynamics publicly released by friends or people they follow. When there are a large number of friends and they are active, you can receive continuously updated content. In addition, Weibo, Zhihu, and QQ Kandian are similar. In the embodiment of the present application, the video to be detected includes a short video.

[0060] 9) Videos, which are content recommended to users of content-consuming devices, include short vertical and horizontal videos, typically sourced from PGC, MCN, or UGC (User-generated Content) and provided as feeds. Content recommended to users of content-consuming devices also includes text and images. These texts are typically created and published by self-media platforms after they have established a public account, including short vertical and horizontal texts.

[0061] 10) Clustering, an unsupervised classification method; for example, K-means clustering algorithm (K-means clustering algorithm), an iterative cluster analysis algorithm, the steps include: if the data is divided into K groups, first randomly select K objects as the initial cluster centers, then calculate the distance between each object and each seed cluster center, and assign each object to the cluster center closest to it; the cluster centers and the objects assigned to them represent a cluster; each time a sample is assigned, the cluster center is recalculated based on the existing objects in the cluster; this process is repeated until a termination condition is met. The termination condition can be that no (or a minimum number of) objects are reassigned to different clusters, no (or a minimum number of) cluster centers change again, or the sum of squared errors reaches a local minimum.

[0062] 11) FFmpeg, an open source computer program that can be used to record and convert digital audio and video and convert them into streams; in the embodiment of the present application, FFmpeg can be used to separate the audio information from the video.

[0063] 12) Faiss, an approximate nearest neighbor search library, provides efficient similarity search and clustering for dense vectors, supporting large-scale vector searches. Given a search vector, Faiss retrieves a list of database objects that are closest to it in Euclidean distance, have the highest vector dot product, or have the largest cosine similarity. Faiss typically implements clustering and retrieval using key technologies such as OpenMP, heap sorting, vector quantization (PQ), inverted indexing, K-means clustering, and principal component analysis.

[0064] It's important to note that artificial intelligence (AI) is a comprehensive technology within computer science that seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions.

[0065] Furthermore, artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0066] With the research and advancement of artificial intelligence technology, it has been studied and applied in many fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart healthcare, and smart customer service. As technology develops, artificial intelligence technology will be applied in more fields and play an increasingly important role. In the embodiments of this application, the application of artificial intelligence in the field of video processing will be described.

[0067] It's also worth noting that social networks originated from online social networking, which began with email. The internet is essentially a network of computers. Early email solved the problem of remote email transmission and became the most popular application on the internet. It was also the starting point for online social networking. BBS (Bulletin Board System) normalized "group messaging" and "forwarding," theoretically enabling the ability to publish information and discuss topics to anyone, becoming an early platform for the spontaneous generation of internet content. Recently, with the widespread adoption of smartphones, the ubiquity of Wi-Fi (Wireless Fidelity) facilities, and the generally lower mobile phone rates, in the current context of the mobile internet, users' demand for information is transitioning from the era of text and images to the era of video.

[0068] Currently, with the rapid development of the internet and the lowering of barriers to content production, the upload and publication volume of videos (including short videos) is growing exponentially. These videos come from a variety of content creation organizations, such as PGC and UGC content from self-media and organizations, as well as the "kandian" (a popular video site) business based on QQ public accounts and browsers. With this significant increase in video uploads, in order to ensure the security and timeliness of distributed content and the interests of the copyright holders of the video content itself, video content must be reviewed within a very short period of time. For example, the content's content must be identified and processed for pornography, gambling, and drug-related content, sensitivity, content quality, and security. In addition, in order to encourage content creation, video platforms have relevant subsidies and incentive mechanisms for video content. In order to increase their own income, video content creators will upload a large number of similar videos (simple editing and modification of videos, such as video titles, watermarks, adding advertising opening and ending credits, modifying audio such as voice changing, adjusting audio playback speed, etc.) or directly copying, modifying or replacing the cover, or making some deletions and modifications to the video content; the above-mentioned video processing methods prevent the activation of normal main content of the account, while squeezing out a large amount of traffic, which is not conducive to the healthy development of the entire video content ecosystem. Therefore, it is necessary to review small videos, short videos and other videos, and to review videos accurately in order to complete the release of small videos, short videos and other videos.

[0069] Generally speaking, video review is primarily based on similar titles and images (including cover images and video content). For example, based on the salient features of the video title, an exact string and regular expression matching algorithm are used to automatically extract the video title, generating a video extension represented by a regular expression. When a new video file arrives, the regular expression matching algorithm is used again to determine whether it has appeared before, thereby achieving deduplication of online videos. When the video titles are similar but the images are dissimilar, they represent different content, such as different stages of an accident handling incident captured by different people. When the titles are different but the images are the same, it is likely that the title has been modified and republished, and thus considered duplicate content. However, there are cases where the titles and images are the same but the audio is different, such as videos of a singer singing different songs at a concert, videos of different chapters of a teacher training lecture, videos of a broadcaster delivering a weather forecast, or videos of calligraphy written in black and white on paper, etc. These videos are all different videos. When video detection is performed mainly based on the picture and title, video content with similar pictures and titles but different audio is often mistakenly identified as a duplicate video. That is, for video content with similar pictures and titles but different audio, the recognition effect and efficiency are low.

[0070] In addition, when reviewing a video, in addition to checking the title and image, it can also be based on audio features; such as the "chromaprint" feature. Figure 1 , Figure 1 is an exemplary video detection process diagram; Figure 1 As shown, the exemplary video detection process includes audio extraction 1-1, audio fingerprint extraction 1-2, and audio similarity calculation 1-3; wherein: during audio extraction 1-1, audio 1-12 is extracted from video 1-11 and stored in a storage service 1-13 (e.g., COS (Cloud Object Storage)). During audio fingerprint extraction 1-2, first, audio 1-12 is read from storage service 1-13, audio 1-12 is split into overlapping segments 1-21, and the overlapping segments 1-21 are converted into spectrograms using STFT (Short-Time Fourier Transform). Figure 1-2 2, and the sound spectrum Figure 1-2 2Convert to musical notes Figure 1-2 3; Then, use filter 1-24 to filter the notes Figure 1-23 performs binary filtering to obtain a filtering result 1-25. Here, the filter 1-24 is trained using the audio training sample 1-26 and the "Asymmetric Pairwise Boosting Algorithm" technology. Finally, the audio fingerprint 1-27 of the filtering result 1-25 is obtained and stored in the storage service 1-28 (for example, the storage service CKV). When calculating the audio similarity 1-3, first, the audio fingerprint pair 1-31 is read from the storage service 1-28 (wherein one of the audio fingerprints in the audio fingerprint pair 1-31 is the audio fingerprint 1-27, and the other audio fingerprint is the audio fingerprint of the video to be compared, and the acquisition method is the same as the acquisition method of the audio fingerprint 1-27). Then, the edit distance of the audio fingerprint pair 1-31 is calculated, and the audio similarity 1-32 is obtained based on the edit distance, see formula (1):

[0071] similarity=1-d / (l1+l2) (1)

[0072] Where d is the edit distance of the audio fingerprint pair 1-31 (here, the distance of the replacement operation is 2), l1 and l2 are the feature lengths corresponding to the two audio fingerprints in the audio fingerprint pair 1-31, and similarity is the audio similarity 1-32.

[0073] Finally, it is determined whether the audio similarity 1-32 is greater than a similarity threshold, so as to obtain a detection result 1-33 of whether the video 1-11 and the video to be compared are duplicate videos based on the determination result.

[0074] Based on the exemplary video detection process described above, since the audio fingerprint obtained is a "chromaprint" feature, and the "chromaprint" feature mainly uses music theory to segment the signal spectrum, the difference in the audio part of lectures, training, a large number of TV dramas, factual videos, etc. lies mainly in the human voice part, not the music theory part. Therefore, it is still impossible to identify videos that are essentially the same (title and picture) but have different degrees of difference in audio volume, sound quality, and timbre; therefore, it is impossible to accurately detect videos with similar pictures and titles but different audio. Therefore, the accuracy of video detection is low.

[0075] Based on this, the embodiments of the present application provide a video detection method, apparatus, device, and computer-readable storage medium that can improve the accuracy of video detection. The following describes an exemplary application of the video detection device provided by the embodiments of the present application. The video detection device provided by the embodiments of the present application can be implemented as various types of user terminals such as laptop computers, tablet computers, desktop computers, set-top boxes, mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices), and can also be implemented as a server. Below, an exemplary application of the video detection device when implemented as a server will be described.

[0076] See also Figure 2 , Figure 2 This is an optional architectural diagram of a video detection system provided in an embodiment of the present application; Figure 2 As shown, to support a video detection application, in the video detection system 100, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples, wherein terminal 400-1 is a video production terminal device and terminal 400-2 is a content consumption terminal device) is connected to the server 200 (video detection device) via a network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. In addition, the video detection system 100 also includes a database 500 for providing data support to the server 200 when the server 200 performs video detection.

[0077] The terminal 400 - 1 is used to send the video to be detected to the server 200 through the network 300 when receiving the video to be detected through the publishing control; and is also used to receive the video detection result sent by the server 200 through the network 300.

[0078] Server 200 is configured to obtain a video to be detected sent by terminal 400-1 via network 300; retrieve videos from a video resource library based on the video to be detected via database 500 to obtain a video to be compared; separate audio information corresponding to the video to be detected to obtain the audio to be detected, and extract audio characteristics of the audio to be detected to obtain an audio fingerprint to be detected, wherein audio characteristics are auditory characteristics of the audio information; separate audio information corresponding to the video to be compared to obtain the audio to be compared, and extract audio characteristics of the audio to be compared to obtain an audio fingerprint to be compared; compare the audio fingerprint to be detected with the audio fingerprint to be compared, and determine a video detection result of the video to be detected relative to the video to be compared based on the comparison result, wherein the video detection result is a detection result of whether the video to be detected is a duplicate of the video to be compared. Server 200 is further configured to send the video to be detected to terminal 400-2 via network 300 based on the video detection result, and send the video detection result to terminal 400-1 via network 300 based on the video detection result.

[0079] The terminal 400 - 2 is configured to receive the video to be detected sent by the server 200 through the network 300 and play the video to be detected on a graphical interface.

[0080] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present invention.

[0081] See also Figure 3 , Figure 3 This embodiment of the present application provides Figure 2 A schematic diagram of the composition structure of a server in Figure 3 The server 200 shown includes: at least one processor 210, a memory 250, at least one network interface 220, and a user interface 230. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to including a data bus, the bus system 240 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 3 Various buses are labeled as bus system 240 .

[0082] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0083] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0084] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0085] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0086] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0087] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0088] A network communication module 252 for reaching other computing devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, Wi-Fi, and Universal Serial Bus (USB);

[0089] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0090] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0091] In some embodiments, the video detection device provided in the embodiments of the present application can be implemented in a software manner. Figure 3The video detection device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: a video acquisition module 2551, a video recall module 2552, a feature acquisition module 2553, a video detection module 2554, and a result processing module 2555. These modules are logical and can be arbitrarily combined or further separated according to the functions implemented. The functions of each module will be described below.

[0092] In other embodiments, the video detection device provided in the embodiments of the present application can be implemented in hardware. As an example, the video detection device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the video detection method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0093] Below, the video detection method provided in the embodiment of the present application will be described in combination with the exemplary application and implementation of the server provided in the embodiment of the present application.

[0094] See also Figure 4 , Figure 4 This is an optional flow chart of the video detection method provided in the embodiment of the present application, which will be combined with Figure 4 The steps shown are explained.

[0095] S401: Obtain a video to be detected.

[0096] In an embodiment of the present application, when a video production end device receives a video posted by a user, such as a video produced by PGC, MCN or UGC, the video detection device also obtains the video to be detected.

[0097] It should be noted that the video to be detected is a video to be deduplicated, and deduplication detection is to detect whether the video to be detected is a duplicate video of the published video; and the video to be detected includes audio information; and the video to be detected can be a short video type video, or other non-short video type video, etc., and the embodiment of the present application does not make specific restrictions on this. In addition, a communication connection is established between the video production end device and the video detection device. The communication connection can be a direct communication connection or a communication connection established through an intermediate device. The embodiment of the present application does not make specific restrictions on this.

[0098] S402: Based on the video to be detected, recall videos from a video resource library to obtain a video to be compared.

[0099] In an embodiment of the present application, the video detection device can obtain videos previously posted by the user (before posting the video to be detected, that is, before obtaining the video to be detected), or the video detection device stores videos previously posted by the user, that is, a video resource library; thus, after the video detection device obtains the video to be detected, in order to perform deduplication detection on the video to be detected, the video that meets the similarity condition with the video to be detected is recalled from the video resource library, and the video to be compared is obtained based on the recalled video.

[0100] It should be noted that the video resource library is a collection of videos obtained before the video to be detected is obtained; in addition, when performing video recall, it can be based on the title and / or picture of the video to be detected (in this case, the video to be compared is a video that satisfies the similarity condition in the title and / or picture with the video to be detected), or it can be based on other content of the video to be detected; and, the video to be compared can be a recalled video, or any one of the recalled videos, or a portion of the recalled video, and this embodiment of the application does not specifically limit this. In addition, the similarity condition is, for example, greater than the recall similarity threshold, or less than or equal to the recall similarity threshold.

[0101] S403: Separate the audio information corresponding to the video to be detected to obtain the audio to be detected, and extract the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected.

[0102] In an embodiment of the present application, the recalled video and the video to be detected are relatively similar (for example, the similarity corresponding to the title and / or picture is less than or equal to the recall similarity threshold). In order to deduplicate the video to be detected, the video detection device performs video detection processing from the audio aspect. Therefore, the video detection device separates the audio information in the video to be detected (for example, using FFmpeg to achieve separation of audio information), and the separated audio information is the audio to be detected. Next, the video detection device extracts the features of the audio to be detected in the audio characteristics, and the extracted features are the audio fingerprint to be detected.

[0103] It should be noted that audio characteristics are the auditory characteristics of audio information, including one or more of volume, sound quality (or pitch) and timbre; while features in audio characteristics refer to the auditory perception characteristics of sound, which are acoustic features, such as MFCC (Mel-scale Frequency Cepstral Coefficients) features. In terms of hearing, different sound quality / pitch correspond to different auditory sensitivities. For example, the human ear has different auditory sensitivities to sound waves of different frequencies. Speech signals ranging from 200Hz to 5000Hz have different effects on speech clarity. Different volume levels also correspond to different auditory sensitivities. For example, when two sounds of different loudness act on the human ear, the presence of the louder frequency component will affect the perception of the lower frequency component, making the lower frequency component less noticeable (a phenomenon known as masking). In addition, lower-frequency sounds travel a greater distance across the basilar membrane of the cochlea than higher-frequency sounds. Therefore, low-frequency sounds generally mask high-frequency sounds more easily, while high-frequency sounds have a harder time masking low-frequency sounds. Furthermore, the critical bandwidth for sound masking at low frequencies is smaller than that at high frequencies. Therefore, the input signal is filtered by passing it through a set of bandpass filters within the frequency band from low to high frequencies, in descending order of critical bandwidth. The signal energy output by each bandpass filter is used as the basic signal feature, which, after further processing, can be used as the input feature of speech. Furthermore, the human ear perceives sounds of different frequencies differently. MFCC features make no assumptions or restrictions on the input signal and leverage auditory characteristics. Therefore, video detection methods based on "chromaprint" features struggle to recognize audio with varying degrees of volume, sound quality, and timbre. Furthermore, due to the wide variety of video types, beyond simple music, the main difference in the audio portion of many TV dramas and factual videos lies in the human voice. Therefore, MFCC features are widely used in the field of speech recognition. Furthermore, because MFCC features are based on Mel frequencies, the extracted features tend to focus more on low-frequency, low-amplitude components, which better align with the frequency distribution of human voices. In summary, auditory features such as MFCC features can accurately identify audio with varying degrees of volume, sound quality, and timbre.

[0104] S404: Separate the audio information corresponding to the video to be compared to obtain the audio to be compared, and extract the audio characteristics of the audio to be compared to obtain a fingerprint of the audio to be compared.

[0105] It should be noted that the features corresponding to the video to be detected and used for comparison with the video to be compared are the features of the audio characteristics of the audio to be detected. Therefore, the video detection device needs to perform similar processing to S403 on the video to be compared to obtain the features of the audio characteristics corresponding to the video to be compared and used for comparison with the video to be detected. The embodiments of this application will not be repeated here.

[0106] Here, the audio to be compared is the audio information of the video to be compared, and the audio fingerprint to be compared is the audio characteristics of the audio to be compared. In addition, S403 and S404 are not executed in a particular order.

[0107] S405 : Compare the audio fingerprint to be detected with the audio fingerprint to be compared, and determine a video detection result of the video to be detected with respect to the video to be compared based on the comparison result.

[0108] In an embodiment of the present application, the video detection device obtains the audio characteristics of the video to be detected and the video to be compared respectively: after the audio fingerprint to be detected and the audio fingerprint to be compared, the comparison of the video to be detected and the video to be compared is completed by comparing the audio fingerprint to be detected with the audio fingerprint to be compared, and the video detection result of the video to be detected for the video to be compared is determined.

[0109] It should be noted that the comparison result refers to the comparison result between the audio fingerprint to be detected and the audio fingerprint to be compared, and the comparison result represents the similarity between the audio fingerprint to be detected and the audio fingerprint to be compared; the video detection result is the detection result of whether the video to be detected is a duplicate video relative to the video to be compared, which can be the detection result that the video to be detected is a duplicate video relative to the video to be compared, or it can be the detection result that the video to be detected is not a duplicate video relative to the video to be compared.

[0110] It can be understood that when comparing the video to be detected with the recalled video to be compared, the video detection result of whether the video to be detected is a duplicate video is determined by comparing the audio characteristics of the audio information of the video to be detected with the audio characteristics of the audio information of the video to be compared; and the audio characteristics are the auditory characteristics of the audio information, such as volume, sound quality and timbre, and videos such as "series of videos of lectures" have differences in audio characteristics; therefore, it is possible to accurately identify whether videos with similar pictures and titles such as "series of videos of lectures" are duplicate videos, thereby improving the accuracy of video detection.

[0111] It can also be understood that the video detection method provided in the embodiment of the present application also realizes the layering of video recall and video deduplication, which can increase the recall volume of videos, reduce the number of subsequent reviews (for example, manual reviews), and improve the efficiency of video reviews; it is also beneficial to optimize the video recall and video deduplication separately.

[0112] In the embodiment of the present application, in S403, the video detection device extracts the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected, including S4031-S4033. Each step is described below.

[0113] S4031 . Extract multiple sub-frames of audio to be detected from the audio to be detected based on a preset frame unit.

[0114] It should be noted that a preset frame unit is pre-set in the video detection device, or the video detection device can obtain the preset frame unit. The preset frame unit refers to the size of a frame of audio, which can be time (for example, 1 second) or the number of sampling points included, etc. The embodiment of the present application does not make specific limitations on this.

[0115] In an embodiment of the present application, the video detection device performs frame processing on the audio to be detected based on preset frame units, thereby extracting multiple frames of sub-audio to be detected from the audio to be detected; that is, the audio to be detected includes multiple frames of sub-audio to be detected.

[0116] S4032: Extract audio characteristics of each sub-frame of audio to be detected from the multiple sub-frames of audio to be detected to obtain an initial sub-frame of audio to be detected fingerprint.

[0117] In an embodiment of the present application, when the video detection device performs feature extraction on audio characteristics, it is performed at a frame granularity. Thus, after obtaining multiple frames of sub-frame audio to be detected, the audio characteristics of each frame of sub-frame audio to be detected in the multiple frames of sub-frame audio to be detected are extracted, and the audio characteristics of one frame of sub-frame audio to be detected, that is, the initial sub-frame audio fingerprint to be detected, are obtained.

[0118] Here, when the video detection device completes the feature extraction of the audio characteristics of each sub-audio to be detected, it obtains multiple frames of initial sub-audio fingerprints corresponding to the multiple frames of sub-audio to be detected, and the multiple frames of sub-audio to be detected correspond one-to-one to the multiple frames of initial sub-audio fingerprints.

[0119] S4033: Perform dimensionality reduction on the initial sub-audio fingerprint to be detected to obtain a sub-audio fingerprint to be detected, thereby obtaining multiple frames of sub-audio fingerprints to be detected corresponding to the audio to be detected.

[0120] It should be noted that, considering that the space occupied by the initial sub-audio fingerprint to be detected in each frame is relatively large, the video detection device performs dimensionality reduction processing on the initial sub-audio fingerprint to be detected. The initial sub-audio fingerprint to be detected after dimensionality reduction is the sub-audio fingerprint to be detected. When the sub-audio fingerprint to be detected corresponding to the initial sub-audio fingerprint to be detected in each frame is obtained, the multi-frame sub-audio fingerprint to be detected corresponding to the multi-frame initial sub-audio fingerprint to be detected is obtained, which is also the multi-frame sub-audio fingerprint to be detected corresponding to the audio to be detected. Here, the multi-frame initial sub-audio fingerprint to be detected corresponds one-to-one to the multi-frame sub-audio fingerprint to be detected, and the audio fingerprint to be detected includes the multi-frame sub-audio fingerprint to be detected. In addition, the video detection device can perform dimensionality reduction processing on the initial sub-audio fingerprint to be detected by clustering, extracting key features, etc.

[0121] It can be understood that by dividing the audio to be detected into frames, video detection is performed at a frame granularity, and the granularity of video detection is finer, thus improving the accuracy of video detection.

[0122] In the embodiment of the present application, S4034 is included before S4031; that is, before the video detection device extracts multiple frames of sub-audio to be detected from the audio to be detected, the video detection method also includes S4034. These steps are explained below.

[0123] S4034: Perform pre-emphasis processing on the audio to be detected to obtain audio to be framed.

[0124] It should be noted that pre-emphasis processing is to pass the audio to be detected through a high-pass filter to enhance the high-frequency part of the audio to be detected. Here, the audio to be framed is the audio to be detected after pre-emphasis processing.

[0125] Exemplarily, the pre-emphasis processing may be implemented by formula (2), which is:

[0126] H(z)=1-μz -1 (2)

[0127] Wherein, z is the audio to be detected, H(z) is the audio to be framed, and μ is the pre-emphasis parameter, which is between 0.9 and 1.0 and is usually 0.97.

[0128] Accordingly, in an embodiment of the present application, the video detection device extracts multiple sub-frames of audio to be detected from the audio to be detected based on the preset frame units in S4031, including: the video detection device extracts multiple sub-frames of audio to be detected from the audio to be framed based on the preset frame units. In other words, if the video detection device performs pre-emphasis processing on the audio to be detected before framing, the pre-emphasized audio to be framed is framed during framing.

[0129] In the embodiment of the present application, the video detection device in S4031 extracts multiple frames of sub-audio to be detected from the audio to be detected based on a preset frame unit, including S40311 and S40312. Each step is described below.

[0130] S40311. Sample the audio to be detected based on a preset sampling frequency to obtain multiple sampling points.

[0131] In an embodiment of the present application, a preset sampling frequency is pre-set in the video detection device, or the video detection device can obtain the preset sampling frequency, which is the sampling frequency for the audio to be detected when performing frame processing; here, the video detection device samples the audio to be detected based on the preset sampling frequency, and the sampling result obtained when the sampling is completed is multiple sampling points.

[0132] S40312. Starting from the first sampling point, sequentially select a preset number of sampling points from the multiple sampling points to form a sub-frame of audio to be detected, and continuously select a preset number of sampling points from positions corresponding to a preset number of overlapping sampling points before the end position of the sampling point selection to form the next sub-frame of audio to be detected, until all the sampling points are selected and processed, thereby obtaining multiple sub-frames of audio to be detected.

[0133] It should be noted that after the video detection device obtains multiple sampling points, it combines the sampling points from the multiple sampling points to obtain a frame of audio. Here, a preset number of sampling points is pre-set in the video detection device, or the video detection device is able to obtain the preset number of sampling points, which refers to the number of sampling points contained in a frame of video; the video detection device starts from the first sampling point from the multiple sampling points, and each time selects a preset number of sampling points to combine into a frame of sub-audio to be detected, and selects a preset number of sampling points from the position that overlaps with the previous frame of sub-audio to be detected by the preset number of overlapping sampling points to form the next frame of sub-audio to be detected. In this way, the sampling points are continuously combined until all sampling points from the multiple sampling points are selected, and all the sub-audio to be detected obtained are multiple frames of sub-audio to be detected.

[0134] Here, the preset frame unit is determined based on a preset sampling frequency and a preset number of sampling points. For example, when sampling the audio to be detected based on the preset sampling frequency, if the preset number of sampling points is 256 or 512, the preset frame unit is 20 to 30 milliseconds. In addition, the preset number of overlapping sampling points is less than the preset number of sampling points, for example, the preset number of overlapping sampling points is 1 / 2 or 1 / 3 of the preset number of sampling points.

[0135] It can be understood that when the audio to be detected is framed, by dividing the audio to be detected into multiple overlapping sub-audios to be detected, the changes between the multiple frames of sub-audios to be detected are made smooth, so that rich audio special effects features can be extracted; in this way, when video detection is performed based on the extracted features, the accuracy of video detection can be improved.

[0136] In the embodiment of the present application, S4032 can be implemented through S40321-S40325; that is, the video detection device extracts the audio characteristics of each frame of sub-frame audio to be detected in multiple frames of sub-frame audio to be detected, and obtains the initial sub-frame audio fingerprint to be detected, including S40321-S40325. Each step is explained below.

[0137] S40321. Perform windowing processing on each frame of sub-audio to be detected in the multiple frames of sub-audio to be detected to obtain sub-audio to be transformed.

[0138] In an embodiment of the present application, in order to increase the continuity of each frame of sub-audio to be detected, the video detection device performs windowing processing on each frame of sub-audio to be detected, and each frame of sub-audio to be detected after windowing processing is sub-audio to be transformed; here, when the video detection device completes the windowing processing of each frame of sub-audio to be detected, multiple frames of sub-audio to be transformed corresponding to multiple frames of sub-audio to be detected will be obtained, and the multiple frames of sub-audio to be detected correspond one-to-one to the multiple frames of sub-audio to be transformed.

[0139] It should be noted that the windowing process is to make the time domain signal better meet the periodicity requirement of the frequency domain transformation process and reduce leakage.

[0140] For example, the windowing process can be implemented by formula (3), which is:

[0141] S'(n)=S(n)×W(n), n=0,1, ...N-1 (3)

[0142] Where S(n) is the n-th frame of sub-audio to be detected, W(n) is the window function corresponding to the n-th frame of sub-audio to be detected, and S'(n) is the n-th frame of sub-audio to be transformed; where N is the preset number of sampling points. Here, W(n) is shown in formula (4):

[0143]

[0144] Where a is the windowing parameter, for example, 0.46; different values ​​of a will produce different W(n).

[0145] S40322. Convert the sub-audio to be converted into energy distribution in the frequency domain to obtain a sub-spectrum to be detected, acquire the power spectrum of the sub-spectrum to be detected, and obtain a sub-power spectrum to be detected.

[0146] It should be noted that, since it is usually difficult to see the characteristics of the sub-audio to be transformed in the time domain, the video detection device converts the sub-audio to be transformed into an energy distribution in the frequency domain for observation; different energy distributions represent the characteristics of different audios. Therefore, after windowing, the video detection device also performs a frequency domain transform (for example, DFT (Discrete Fourier Transform)) on the sub-audio to be transformed to obtain the energy distribution in the frequency domain, that is, the sub-spectrum to be detected. Next, the video detection device obtains the power spectrum of the sub-spectrum to be detected, and thus obtains the power spectrum of the sub-spectrum to be detected; here, when obtaining the power spectrum of the sub-spectrum to be detected, it can be achieved by taking the modulus square of the sub-spectrum to be detected, or by taking the absolute value of the sub-spectrum to be detected, or by taking the square of the sub-spectrum to be detected.

[0147] For example, the frequency domain transformation can be implemented by equation (5), which is:

[0148]

[0149] Wherein, k is the sub-spectrum to be detected in the k-th frame, and j is the complex imaginary part.

[0150] S40323. Smoothing the sub-to-be-detected power spectrum to obtain a sub-smoothed power spectrum.

[0151] Exemplarily, the video detection device inputs the sub-power spectrum to be detected into a group of Mel-scale triangular filters for smoothing. The frequency response corresponding to the group of Mel-scale triangular filters is shown in formula (6):

[0152]

[0153] Where M is the number of triangular filters; f(m-1), f(m), and f(m+1) are all center frequencies, which are determined based on the highest and lowest frequencies of the sub-power spectrum to be detected. In addition, the corresponding relationship between Mel frequency and audio frequency can be found in formula (7), which is:

[0154] Mel(f) = 2595 × lg (1+ f / 700) (7)

[0155] Among them, Mel(f) is the Mel frequency, f is X a (k) The corresponding frequency.

[0156] H m (k) is the frequency response of the k-th sampling point for the m-th triangular filter, where it satisfies equation (8):

[0157]

[0158] Thus, the sub-smoothed power spectrum is obtained by equation (9), which is:

[0159]

[0160] Among them, y(1)…y(m)…y(M) are sub-smoothed power spectra.

[0161] S40324: Perform an inverse transformation on the logarithmic energy of the sub-smoothed power spectrum, and obtain audio characteristic parameters of a preset order from the inverse transformation result.

[0162] It should be noted that the inverse transform is a transform performed relative to the frequency domain transformation in S40322. That is, the video detection device transforms the sub-smoothed power spectrum into information in the frequency domain. The inverse transform is, for example, DCT (Discrete Cosine Transform). Here, the audio characteristic parameters are, for example, MFCC coefficients.

[0163] For example, the inverse transformation can be implemented by formula (10):

[0164]

[0165] Wherein, c(l) is the l-th order MFCC coefficient in the audio characteristic parameters; L is a preset order, such as 12, 16, etc.

[0166] S40325: Acquire differential parameters of the audio characteristic parameters and frame energy of each frame of the sub-audio to be detected, thereby obtaining an initial sub-audio fingerprint to be detected including one or more of the audio characteristic parameters, differential parameters, and frame energy.

[0167] It should be noted that the audio characteristic parameters are static characteristics of the audio. In order to obtain the characteristics of the audio characteristics with higher accuracy, the video detection device extracts the differential parameters of the audio characteristic parameters to obtain the dynamic characteristics of the audio. In addition, the frame energy is the volume of the sub-audio to be detected. For example, the frame energy can be obtained by taking the logarithm of the sum of the squares of the volume of the sub-audio to be detected with a base of 10 and multiplying it by 10. For example, the initial sub-audio fingerprint to be detected includes: N-dimensional MFCC parameters (N / 3 MFCC coefficients + N / 3 first-order differential parameters + N / 3 second-order differential parameters) + frame energy.

[0168] For example, the differential parameter can be obtained by using formula (11), which is:

[0169]

[0170] Where V is the time difference of the first-order derivative, for example, 1 or 2; d lis the lth first-order difference parameter.

[0171] When the differential parameters include first-order differential parameters and second-order differential parameters, the second-order differential parameters are obtained by referring to formula (11).

[0172] In the embodiment of the present application, S4033 can be implemented through S40331-S40333; that is, the video detection device reduces the dimension of the initial sub-audio fingerprint to be detected to obtain the sub-audio fingerprint to be detected, including S40331-S40333. Each step is described below.

[0173] S40331. For each initial sampling point audio fingerprint in the initial sub-audio fingerprint to be detected, remove the lowest frequency feature to obtain an S-1 dimensional feature.

[0174] It should be noted that the initial sub-audio fingerprint to be detected includes initial sampling point audio fingerprints of a preset number of sampling points, and each initial sampling point audio fingerprint includes S-dimensional features, where S is a positive integer greater than 1.

[0175] S40332. Perform clustering and dimensionality reduction on the S-1 dimensional features based on a preset number of categories to obtain a preset number of cluster categories.

[0176] It should be noted that a preset number of categories is pre-set in the video detection device, or the video detection model can obtain the preset number of categories, which is less than S-1; thus, the video detection device clusters S-1 as a feature and obtains a preset number of cluster categories, thereby achieving dimensionality reduction of the S-1 dimensional feature.

[0177] For example, when each initial sampling point audio fingerprint includes 12-dimensional floating point features (S-dimensional features), removing the lowest frequency features will result in 11-dimensional floating point features, i.e., 11*4*2 8 Bit characteristics; 11*4*2 8 The features of the bit (S-1 dimensional features) are clustered into 256 categories (the number of preset categories), that is, 1-byte features; the corresponding space occupancy of the 1-byte features is reduced compared to the 12-dimensional floating-point features.

[0178] S40333: Determine the cluster center information of the cluster category as the sampling point audio fingerprint of each initial sampling point audio fingerprint, thereby obtaining a sub-audio fingerprint to be detected corresponding to the initial sub-audio fingerprint to be detected.

[0179] It should be noted that after the video detection device obtains a preset number of cluster categories, the cluster center of each cluster category can be used as the sampling point audio fingerprint of each initial sampling point audio fingerprint; in this way, when the video detection device completes the acquisition of the sampling point audio fingerprint of each initial sampling point audio fingerprint, it can also obtain the preset number of sampling point audio fingerprints corresponding to the initial sub-audio fingerprint to be detected; wherein, the sub-audio fingerprint to be detected includes the preset number of sampling point audio fingerprints.

[0180] See also Figure 5 , Figure 5 This is another optional flow chart of the video detection method provided in the embodiment of the present application; Figure 5 As shown, in the embodiment of the present application, S405 can be implemented through S4051-S4053; that is, the video detection device compares the audio fingerprint to be detected with the audio fingerprint to be compared, and determines the video detection result of the video to be detected for the video to be compared based on the comparison result, including S4051-S4053. Each step is explained below.

[0181] S4051. Compare each sub-frame of the audio fingerprint to be detected in the audio fingerprint to be detected with each sub-frame of the audio fingerprint to be compared, and obtain a comparison result corresponding to each sub-frame of the audio fingerprint to be detected and each sub-frame of the audio fingerprint to be compared.

[0182] In the embodiment of the present application, since the audio fingerprint to be detected includes multiple frames of sub-audio fingerprints to be detected, the audio fingerprint to be compared also includes multiple frames of sub-audio fingerprints to be compared (including multiple frames of audio fingerprints to be compared); therefore, the video detection device compares each frame of audio fingerprint to be detected in the multiple frames of sub-audio fingerprints to be detected with each frame of audio fingerprint to be compared in the multiple frames of sub-audio fingerprints to be compared one by one, and the corresponding information of each frame of audio fingerprint to be detected and each frame of audio fingerprint to be compared constitutes the comparison result.

[0183] It should be noted that the number of frames corresponding to the multi-frame sub-audio fingerprints to be detected and the number of frames corresponding to the multi-frame audio fingerprints to be compared can be the same or different. The number of sub-comparison results (comparison information) of the sub-audio fingerprints to be detected and the sub-audio fingerprints to be compared included in the comparison result is the product of the number of frames corresponding to the multi-frame sub-audio fingerprints to be detected and the number of frames corresponding to the multi-frame audio fingerprints to be compared.

[0184] S4052: When preset regularity information exists in the comparison result, determine a video detection result that the video to be detected is a duplicate video for the video to be compared.

[0185] It should be noted that the preset regular information is the similarity trend between the sub-audio fingerprint to be detected in each frame and the sub-audio fingerprint to be compared in each frame, for example, the similarity between the sub-audio fingerprint to be detected and the sub-audio fingerprint to be compared for a consecutive preset number of times is higher than a threshold.

[0186] S4053: When there is no preset regularity information in the comparison result, determine that the video to be detected is a non-duplicate video for the video to be compared.

[0187] It should be noted that S4052 and S4053 are executed in parallel.

[0188] In the embodiment of the present application, after S4051, the video detection method further includes S4054-S4057; that is, after the video detection device obtains the comparison result corresponding to the audio fingerprint to be detected in each frame and the audio fingerprint to be compared in each frame, the video detection method further includes S4054-S4057. Each step is described below.

[0189] S4054: Use the sub-frame audio fingerprint to be detected as one-dimensional attribute of the matrix, the sub-frame audio fingerprint to be compared as another-dimensional attribute of the matrix, and the comparison results as elements of the matrix to construct a similarity matrix.

[0190] In an embodiment of the present application, the video detection device uses the audio fingerprint of each frame to be detected as a one-dimensional attribute of the matrix, and the audio fingerprint of each frame to be compared as the other-dimensional attribute of the matrix, thereby obtaining a two-dimensional matrix of the number of frames corresponding to the multiple-frame sub-audio fingerprints to be detected * the number of frames corresponding to the multiple-frame audio fingerprints to be compared, that is, a similarity matrix; and each element in the two-dimensional matrix is ​​the similarity value between a frame sub-audio fingerprint to be detected and a frame sub-audio fingerprint to be compared.

[0191] S4055. Based on the correspondence between the preset similarity values ​​and the display colors, the similarity matrix is ​​converted into a similarity matrix graph.

[0192] It should be noted that the video detection device is pre-set with a correspondence between preset contrast information and display colors. When a similarity value is given, the display color corresponding to the given similarity value can be determined; thus, the video detection device determines the display color corresponding to each similarity value in the similarity matrix based on the correspondence between the preset similarity value and the display color and displays it, thus obtaining a similarity matrix diagram.

[0193] Here, the similarity value is correlated with the depth of the display color; and the similarity matrix can also be used for display to achieve a visual display of video detection. In addition, the similarity matrix can also be used to display other forms of display corresponding to the similarity value, such as different graphics.

[0194] S4056: When the color difference between each displayed color at the diagonal position in the similarity matrix diagram and the preset color is less than the color difference threshold, it is determined that preset regularity information exists in the comparison result.

[0195] In an embodiment of the present application, after the video detection device obtains the similarity matrix diagram, it judges the color difference between each display color at the diagonal position of the similarity matrix diagram; when the color difference between each display color at the diagonal position and the preset color (for example, 7FFF00 color) is less than the color difference threshold, it is determined that there is preset regular information in the comparison result, that is, the audio fingerprint to be detected is similar to the audio fingerprint to be compared, and then the video to be detected and the video to be compared constitute a duplicate video.

[0196] Here, the detection of preset regular information can be achieved through classification detection through a neural network. For example, the similarity matrix diagram is input into the CNN+"Xgbboost" image classification detection model to obtain the detection results for the preset regular information.

[0197] S4057: When the color difference between the displayed color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that there is no preset regularity information in the comparison result.

[0198] It should be noted that when the color difference between the displayed color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that there is no preset regular information in the comparison result, that is, the audio fingerprint to be detected is not similar to the audio fingerprint to be compared, and then the video to be detected and the video to be compared constitute non-repeating videos.

[0199] It can be understood that by constructing a similarity matrix diagram between the audio fingerprint to be detected and the audio fingerprint to be compared, and by the direction of the lines formed by the feature points in the similarity matrix diagram, it is possible to determine whether the video detection result constitutes a duplicate video, thereby simplifying the video detection process.

[0200] In an embodiment of the present application, S402 can be implemented through S4021-S4024; that is, the video detection device recalls videos from the video resource library based on the video to be detected to obtain the video to be compared, including S4021-S4024. Each step is explained below.

[0201] S4021. Obtain video recall features corresponding to the video to be detected.

[0202] It should be noted that video recall features include one or more of content semantic features, text semantic features, title semantic features, body semantic features, frame image semantic features, and cover image semantic features; wherein, content semantic features are semantic features corresponding to the content expressed in the video to be detected; text semantic features are semantic features corresponding to the text obtained through text recognition of subtitles, title text, and video frame images in the video to be detected; title semantic features are semantic features corresponding to the title of the video to be detected; body semantic features are semantic features corresponding to the content, title, and cover image expressed in the video to be detected; frame image semantic features are semantic features corresponding to the key frame images or all frame images of the video to be detected; cover image semantic features are semantic features corresponding to the cover image of the video to be detected. Here, semantic features are, for example, "Simhash" vectors, "BERT" vectors, "embeeding" vectors, etc.

[0203] S4022: Obtain the features to be recalled corresponding to each video in the video resource library.

[0204] In an embodiment of the present application, the features to be recalled include one or more of semantic features of content to be recalled, semantic features of text to be recalled, semantic features of title to be recalled, semantic features of body text to be recalled, semantic features of frame image to be recalled, and semantic features of cover image to be recalled; and the features to be recalled correspond to the video recall features in terms of feature type. For example, when the video recall features are title semantic features and frame image semantic features, the features to be recalled are title semantic features and frame image semantic features.

[0205] S4023. Based on the recall similarity values ​​between the video recall feature and each feature to be recalled, determine a target feature to be recalled that is similar to the video recall feature from each feature to be recalled.

[0206] In the embodiment of the present application, the target to-be-recalled feature similar to the video recall feature is the to-be-recalled feature corresponding to a recall similarity value greater than (or less than or equal to) a recall similarity threshold. Here, the recall similarity value includes one or more of Euclidean distance, vector dot product value, and cosine similarity value.

[0207] S4024: Use the video corresponding to the target to-be-recalled feature in the video resource library as the recalled video, thereby obtaining the to-be-compared video belonging to the recalled video.

[0208] It should be noted that in the video resource library, the video corresponding to the target feature to be recalled is the video that meets the recall condition (the recall similarity value is greater than (or less than or equal to) the recall similarity threshold) with the video to be detected. Here, the recalled video includes the video to be compared.

[0209] In an embodiment of the present application, S4023 can be implemented through S40231 and S40232; that is, the video detection device determines the target to-be-recalled features that are similar to the video recall features from each to-be-recalled feature based on the recall similarity values ​​between the video recall features and each to-be-recalled feature, including S40231 and S40232. Each step is explained below.

[0210] S40231. Obtain the recall feature index corresponding to the video recall feature, and each recall feature index corresponding to each feature to be recalled.

[0211] It should be noted that each feature to be recalled corresponds to each recall feature index one by one; here, the video detection device constructs the recall feature index corresponding to the video recall feature and each recall feature index corresponding to each feature to be recalled through a preset retrieval library (for example, Faiss).

[0212] S40232. The matching degree between the recall feature index and each recall feature index is used as a recall similarity value, and the recall similarity value is used to determine a target feature to be recalled that is similar to the video recall feature from each feature to be recalled.

[0213] It should be noted that the video detection device compares the video recall features with each feature to be recalled based on the index to improve the recall efficiency of the video.

[0214] See also Figure 6 , Figure 6 FIG. 1 is an optional interactive flow diagram of the video detection method provided in the embodiment of the present application; Figure 6 As shown, in the embodiment of the present application, S401 can be implemented through S4011 and S4012; that is, the video detection device obtains the video to be detected, including S4011 and S4012, and each step is described below.

[0215] S4011. Receive a video detection request sent by a task scheduling device.

[0216] It should be noted that the video to be detected posted by the user and received by the video production end device is realized by the task scheduling device sending the video to be detected to the video detection device; that is, when the video production end device receives the video to be detected posted by the user, the task scheduling device generates a video detection request based on the video upload request sent by the video production end device, and sends the video detection request to the video detection device, so that the video detection device receives the video detection request; here, the video detection request is generated by the task scheduling device in response to the video upload request sent by the video production end device, the video detection request is a request from the task scheduling device to perform deduplication detection on the video to be detected, and the video upload request is a request from the video production end device to publish the video to be detected.

[0217] S4012: In response to the video detection request, obtain the video to be detected from the content storage device.

[0218] It should be noted that after receiving the video detection request, the video detection device responds to the video detection request and starts executing the detection process of the video to be detected, and obtains the video to be detected from the content storage device for deduplication detection based on the instruction of the video detection request.

[0219] Continue to see Figure 6 In the embodiment of the present application, after S405, S406 is also included; that is, after the video detection device compares the audio fingerprint to be detected with the audio fingerprint to be compared, and determines the video detection result of the video to be detected for the video to be compared based on the comparison result, the video detection method also includes S406. This step is explained below.

[0220] S406: When the video detection result indicates that the video to be detected is a duplicate video with respect to the video to be compared, the video detection result is sent to a subsequent detection device.

[0221] It should be noted that the video detection device sends the video detection result to the subsequent detection device, so that the subsequent detection device generates a subsequent detection request for the video detection result, and obtains the target detection result of the video to be detected in response to the subsequent detection request, that is, S407.

[0222] Here, the subsequent detection request can be a request for a re-review of the video to be detected and the approval to be compared using a network model, or a request for a manual review of the video to be detected and the approval to be compared. Furthermore, the target detection result can be a re-review result corresponding to the network model, or a manual review result; and the target detection result is the detection result of whether the video to be detected is a duplicate of the video to be compared, determined through the subsequent review.

[0223] Continue to see Figure 6In the embodiment of the present application, after S405, S408 is also included; that is, after the video detection device compares the audio fingerprint to be detected with the audio fingerprint to be compared, and determines the video detection result of the video to be detected for the video to be compared based on the comparison result, the video detection method also includes S408. This step is explained below.

[0224] S408: When the video detection result shows that the video to be detected is a non-duplicate video with respect to the video to be compared, the video to be detected is sent to the task scheduling device.

[0225] It should be noted that the video detection device sends the video to be detected to the task scheduling device, so that the task scheduling device pushes the video to be detected to the content consumption end device through the content distribution device based on the obtained recommendation information, that is, S409. Here, the task scheduling device pushes the video to be detected to the content consumption end device so that the content consumption end device plays the video to be detected, that is, S410.

[0226] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0227] See also Figure 7 , Figure 7 is a structural diagram of an exemplary video detection system provided in an embodiment of the present application; Figure 7 As shown, in this exemplary video detection system, it includes a content production end 7-101 (video production end device), an uplink and downlink content interface server 7-102, a content database 7-103, a content storage service 7-104 (together with the content database 7-103, a content storage device), a scheduling center service 7-105 (task scheduling device), a video deduplication service 7-106, a retrieval and recall service 7-107, an audio deduplication verification service 7-108, an audio fingerprint and other modal vector generation service 7-109, an audio extraction service 7-110, a video download system 7-111, a manual review system 7-112, a content distribution export service 7-113 (together with 7-106 to 7-111, a video detection device) and a content consumption end 7-114 (content consumption end device). Among them:

[0228] The content production terminal 7-101, including the PGC terminal, the UGC terminal, and the MCN terminal, is used to obtain the uploaded video (the video to be detected) through the video publishing page or the back-end video publishing interface, and communicate with the upstream and downstream content interface server 7-102 to send the uploaded video to the upstream and downstream content interface server 7-102; it is also used to obtain the behavioral data of the uploaded video and send the obtained behavioral data of the uploaded video to the back-end server for statistical analysis. Among them, the uploaded video is usually a short video or small video shot by a shooting terminal. During the shooting process, the video can be beautified by selecting background music, filter templates, and other video features for the shot.

[0229] The uplink and downlink content interface server 7-102 is used to communicate with the content production end 7-101, obtain the uploaded video, and store the metadata of the uploaded video (for example, video source file size, release time, title, author, cover image, category and tag information, etc.) in the content database 7-103, and store the entity of the uploaded video (for example, video source file) in the content storage service 7-104; it is also used to communicate with the dispatch center service 7-105 to send an audit request (video detection request) for the uploaded video; it is also used to provide video index information to the content consumption end 7-114.

[0230] The content database 7-103 is used to store the metadata of the uploaded video sent by the uplink and downlink content interface servers 7-102; it is also used to update the metadata based on the metadata update data sent by the scheduling center service 7-105; it is also used to provide the metadata of the video to the video deduplication service 7-106 (actually providing the metadata of the video to the audio fingerprint and other modal vector generation service 7-109) and the manual review system 7-112 (not shown in the figure).

[0231] The content storage service 7-104 is used to store the entities of the uploaded videos sent by the uplink and downlink content interface servers 7-102; it is also used to provide video source files to the video download system 7-111 and the content consumption terminal 7-114.

[0232] The dispatch center service 7-105 is used to receive the review request for the uploaded video sent by the upstream and downstream content interface server 7-102, and dispatch the manual review system 7-112 and the video deduplication service 7-106 to conduct video review; it is also used to dispatch the content distribution export service 7-113 to distribute the video.

[0233] The video deduplication service 7-106 is used to schedule the retrieval and recall service 7-107 to perform video recall (not shown in the figure), and to schedule the audio deduplication verification service 7-108 to perform audio verification and deduplication (not shown in the figure) to realize the scheduling of the video download system 7-111.

[0234] The retrieval and recall service 7-107 is used to obtain the vector corresponding to each video (audio fingerprint or other modal vector, where other modal vectors include video recall features and features to be recalled) through the audio fingerprint and other modal vector generation service 7-109, and to build a matching retrieval of the vector based on the vector corresponding to each video through Faiss to achieve rapid recall of similar videos, and send the reading information of the recalled video to the audio deduplication verification service 7-108 to obtain the recalled video.

[0235] The audio deduplication verification service 7-108 receives the reading information of the recalled video sent by the retrieval recall service 7-107, reads the recalled video from the content storage service 7-104 through the video download system 7-111, and extracts the audio (audio to be compared) of each video (video to be compared) in the recalled video through the audio extraction service 7-110, and obtains the audio fingerprint (audio fingerprint to be compared) of the extracted audio through the audio fingerprint and other modal vector generation service 7-109; it is also used to respond to the scheduling of the video deduplication service 7-106, and read the uploaded video from the content storage service 7-104 through the video download system 7-111 Entity, and extract the audio (audio to be detected) of the video entity (video to be detected) through the audio extraction service 7-110, and obtain the audio fingerprint (audio fingerprint to be detected) of the extracted audio through the audio fingerprint and other modal vector generation service 7-109; it is also used to compare the audio fingerprints obtained above to realize audio deduplication verification. When it is determined that the two videos are duplicate videos, the manual review system 7-112 is called. When it is determined that the two videos are non-duplicate videos, the video deduplication service 7-106 is used to feedback to the dispatch center service 7-105, so that the dispatch center service 7-105 dispatches the content distribution export service 7-113 to distribute the video.

[0236] Audio fingerprint and other modality vector generation service 7-109, used to generate recall vectors (other modality vectors) and audio fingerprints.

[0237] The audio extraction service 7-110 is used to extract audio information from the video, for example, separating the audio information from the video content through FFmpeg.

[0238] The video downloading system 7-111 is used to read the video source file from the content storage service 7-104.

[0239] The manual review system 7-112 is used to provide the video to the content consumption end 7-114 through the display page of the content distribution export service 7-113; it is also used to review and filter sensitive, pornographic, legally prohibited and other content that cannot be determined by the machine by reading data in the content database 7-103 and the content storage service 7-104, and at the same time, the results and status of the manual review are transmitted back to the content database 7-103; it is also used to label and re-confirm the video content.

[0240] The content distribution export service 7-113 is used to distribute videos to the content consumption end 7-114; the content distribution export service 7-113 is, for example, a recommendation engine, a search engine or an operation platform.

[0241] The content consumption end 7-114 is used to communicate with the upstream and downstream content interface server 7-102 to obtain video index information based on the access request, and then obtain the video source file from the content storage service 7-104 based on the obtained video index information; it is also used to communicate with the content distribution export service 7-113 to obtain the distributed video entity, wherein the video entity is the video source file of the video uploaded by the content production end 7-101, and the source file can be a pushed video or a subscribed video. It is also used to obtain the behavioral data of downloading videos and playing videos (for example, freeze information, loading time, etc.), and send the obtained behavioral data of downloading videos and playing videos to the background server for statistical analysis. It is also used to browse content data in the form of a feed stream, and the content data includes graphics, pictures and videos.

[0242] The following describes Figure 7 The implementation process of obtaining audio fingerprints in the audio fingerprint and other modal vector generation service 7-109. Figure 8 , Figure 8 This is a schematic diagram of an exemplary process of obtaining an audio fingerprint provided in an embodiment of the present application; Figure 8As shown, the exemplary process of obtaining audio fingerprints includes two parts: feature acquisition 8-1 and feature dimensionality reduction 8-2. Here, the audio fingerprint of the uploaded video 8-11 is taken as an example for explanation. Among them, in feature acquisition 8-1, first, the audio information 8-12 is separated from the video 8-11, and the audio information 8-12 is sequentially pre-emphasized (implemented by formula (2)), framed, and windowed (implemented by formulas (3) and (4)) 8-13, and then each frame of audio after windowing is transformed into the frequency domain 8-14 using formula (5), and the absolute value or square value or modulus square 8-15 of each frame of audio after frequency domain transformation is taken to obtain the power spectrum; then, the power spectrum is subjected to Mel filtering 8-16 (implemented by formulas (6) and (7)) to obtain the frequency response. Then, equation (9) is used to perform logarithmic processing on the frequency response and power spectrum 8-17, and DCT (implemented using equation (10)) is used to perform inverse transformation 8-18 on the logarithmic frequency response and power spectrum, thereby obtaining MFCC coefficients; finally, equation (11) is used to obtain the differential parameters of the MFCC system to obtain dynamic features 8-19, thereby obtaining MFCC features including MFCC coefficients and differential parameters. Here, MFCC features can also include logarithmic energy, that is, the logarithmic value of the sum of the squares of the volume of a frame of audio is taken to the base 10 and multiplied by 10. In feature dimensionality reduction 8-2, clustering 8-21 is used to cluster the feature dimensions of each sampling point of each frame of audio, and the cluster center of the cluster category is used as the feature value of each sampling point of each frame of audio, thereby obtaining an audio fingerprint 8-22.

[0243] It should be noted that Figure 8 The Mel filters 8-16 in the figure are implemented by a set of Mel-scale triangular filters. The obtained MFCC coefficients are the inverse spectral parameters of the Mel-calibrated frequency domain. The Mel calibration describes the nonlinear characteristics of the audio frequency. The Mel frequency has a linear relationship with the audio frequency, such as Figure 9 As shown in Figure 9-1, the curve 9-1 describes that the Mel frequency has a positive correlation with the audio frequency. Figure 10 , Figure 10 is a schematic diagram of the filtering results of a set of exemplary Mel-scale triangular filters provided in an embodiment of the present application; see Figure 10 , which shows the Mel filtering processing results including 6 triangular filters with center frequencies of f(0) to f(7), and the frequency responses of the triangular filters are H1(k), H3(k), H5(k) and H6(k).

[0244] The following describes Figure 7The implementation process of audio verification by the audio deduplication verification service 7-108 in the audio verification. After obtaining the audio fingerprint corresponding to each frame through the audio fingerprint and other modal vector generation service 7-109, here, when the audio fingerprint corresponding to 12 frames of audio is obtained for video 8-11, and the audio fingerprint corresponding to 12 frames of audio is also obtained for the video to be compared with video 8-11 (video to be compared), the audio deduplication verification service 7-108 is used to construct the audio fingerprint corresponding to the 12 frames of audio of video 8-11 and the audio fingerprint corresponding to the 12 frames of audio of the video to be compared with video 8-11. Figure 11 The matrix shown is similar Figure 11-1 , and use the neural network model (for example, CNN+"Xgbboost" image classification detection model) to analyze the matrix similarity Figure 11-1 Perform classification detection to determine matrix similarity Figure 11-1 Corresponding similarities and relationships (video detection results); here, due to Figure 11 The matrix similarity in Figure 11-1 In area 11-11 on the image, there is a point that coincides with or is close to the diagonal line, and it is determined that video 8-11 and the video to be compared with video 8-11 are duplicate videos.

[0245] Here, when the audio fingerprint corresponding to 12 frames of audio is obtained for video 8-11, and the audio fingerprint corresponding to 9 frames of audio is obtained for the video to be compared with video 8-11 (video to be compared), the audio deduplication verification service 7-108 is used to construct the following audio fingerprint corresponding to the 12 frames of audio of video 8-11 and the audio fingerprint corresponding to the 9 frames of audio of the video to be compared with video 8-11. Figure 12 The matrix shown is similar Figure 12-1 , and use the neural network model to analyze matrix similarity Figure 11-1 Perform classification detection to determine matrix similarity Figure 12-1 Corresponding similarities and relationships (video detection results); here, due to Figure 11 The matrix similarity in Figure 12-1 On the correlation value distribution graph, as shown in area 12-11, it is determined that video 8-11 and the video to be compared with video 8-11 are non-duplicate videos.

[0246] It can be understood that the embodiment of the present application provides an MFCC feature extraction method based on video audio, and performs clustering and dimensionality reduction processing on the extracted MFCC features, and then performs deduplication verification on the content recalled based on the video content, title or cover image similarity during the deduplication process through the MFCC features after dimensionality reduction, thereby improving the accuracy of video deduplication; finally, for the comparison information of the MFCC features of each frame of audio extracted, an inter-frame similarity alignment map is constructed, and a neural network is used to classify and detect the inter-frame similarity alignment map, making full use of the frame granularity temporal relationship, so that the accuracy of verifying duplicate videos is greatly improved. In addition, the video detection method provided by the embodiment of the present application increases the recall rate of videos with similar titles and / or pictures, and content with obvious differences in audio, can effectively reduce the manpower investment required to review content, greatly reduce the number of duplicate video content entering manual review, and improve the processing capacity of video detection.

[0247] The following continues to describe the exemplary structure of the video detection device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, such as Figure 3 As shown, the software modules stored in the video detection device 255 of the memory 250 may include:

[0248] The video acquisition module 2551 is used to acquire the video to be detected;

[0249] The video recall module 2552 is used to recall videos from a video resource library based on the video to be detected to obtain a video to be compared;

[0250] The feature acquisition module 2553 is configured to separate the audio information corresponding to the video to be detected to obtain the audio to be detected, and extract the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected, wherein the audio characteristics are auditory characteristics of the audio information;

[0251] The feature acquisition module 2553 is further configured to separate the audio information corresponding to the video to be compared to obtain the audio to be compared, and extract the features of the audio to be compared based on the audio characteristics to obtain the audio fingerprint to be compared;

[0252] The video detection module 2554 is used to compare the audio fingerprint to be detected with the audio fingerprint to be compared, and determine a video detection result of the video to be detected relative to the video to be compared based on the comparison result, wherein the video detection result is a detection result of whether the video to be detected is a duplicate video relative to the video to be compared.

[0253] In an embodiment of the present application, the feature acquisition module 2553 is further configured to extract multiple frames of sub-audio to be detected from the audio to be detected based on preset frame units; extract features of each frame of sub-audio to be detected in the multiple frames of sub-audio to be detected in terms of the audio characteristics to obtain an initial sub-audio to be detected fingerprint; perform dimensionality reduction on the initial sub-audio to be detected fingerprint to obtain a sub-audio to be detected fingerprint, thereby obtaining multiple frames of sub-audio to be detected fingerprints corresponding to the audio to be detected, wherein the audio fingerprint to be detected includes the multiple frames of sub-audio to be detected fingerprints.

[0254] In the embodiment of the present application, the feature acquisition module 2553 is further configured to perform pre-emphasis processing on the audio to be detected to obtain audio to be framed.

[0255] In the embodiment of the present application, the feature acquisition module 2553 is further configured to extract the multiple frames of sub-audio to be detected from the audio to be framed based on the preset frame unit.

[0256] In an embodiment of the present application, the feature acquisition module 2553 is further used to sample the audio to be detected based on a preset sampling frequency to obtain multiple sampling points; among the multiple sampling points, starting from the first sampling point, sequentially select a preset number of sampling points as a frame of sub-audio to be detected, and continue to select the preset number of sampling points from the position corresponding to the preset number of overlapping sampling points before the end position of the selection of the sampling point as the next frame of sub-audio to be detected, until the multiple sampling points are selected and processed to obtain the multiple frames of sub-audio to be detected, wherein the preset frame unit is determined based on the preset sampling frequency and the preset number of sampling points.

[0257] In an embodiment of the present application, the feature acquisition module 2553 is further used to perform windowing processing on each frame of the sub-audio to be detected in the multiple frames of sub-audio to be detected to obtain a sub-audio to be transformed; transform the sub-audio to be transformed into an energy distribution in the frequency domain to obtain a sub-spectrum to be detected, obtain the power spectrum of the sub-spectrum to be detected, and obtain a sub-power spectrum to be detected; smooth the sub-power spectrum to be detected to obtain a sub-smoothed power spectrum; inversely transform the logarithmic energy of the sub-smoothed power spectrum and obtain audio characteristic parameters of a preset order of the inverse transformation result; obtain differential parameters of the audio characteristic parameters, and the frame energy of each frame of the sub-audio to be detected, thereby obtaining the initial sub-audio fingerprint to be detected including one or more of the audio characteristic parameters, the differential parameters, and the frame energy.

[0258] In an embodiment of the present application, the feature acquisition module 2553 is further configured to remove the lowest frequency feature for each initial sampling point audio fingerprint in the initial sub-audio fingerprint to be detected to obtain an S-1 dimensional feature, wherein the initial sub-audio fingerprint to be detected includes the initial sampling point audio fingerprints of the preset number of sampling points, and each initial sampling point audio fingerprint includes an S-dimensional feature, where S is a positive integer greater than 1; perform clustering and dimensionality reduction on the S-1 dimensional feature based on a preset number of categories to obtain the preset number of cluster categories; determine the cluster center information of the cluster category as the sampling point audio fingerprint of each initial sampling point audio fingerprint, thereby obtaining the sub-audio fingerprint to be detected corresponding to the initial sub-audio fingerprint to be detected, wherein the sub-audio fingerprint to be detected includes the audio fingerprints of the sampling points of the preset number of sampling points.

[0259] In an embodiment of the present application, the video detection module 2554 is further configured to compare each frame of the audio fingerprint to be detected in the audio fingerprint to be detected with each frame of the audio fingerprint to be compared, respectively, to obtain the comparison result corresponding to each frame of the audio fingerprint to be detected and each frame of the audio fingerprint to be compared; when preset regularity information exists in the comparison result, determining that the video detection result of the video to be detected is a duplicate video for the video to be compared, wherein the preset regularity information is a similarity trend between each frame of the audio fingerprint to be detected and each frame of the audio fingerprint to be compared; when the preset regularity information does not exist in the comparison result, determining that the video to be detected is a non-duplicate video for the video to be compared.

[0260] In an embodiment of the present application, the video detection module 2554 is further used to use the audio fingerprint of each frame to be detected as a one-dimensional attribute of the matrix, the audio fingerprint of each frame to be compared as another dimensional attribute of the matrix, and the comparison result as an element of the matrix to construct a similarity matrix; based on the correspondence between the preset similarity value and the display color, the similarity matrix is ​​converted into a similarity matrix diagram; when the color difference between each display color at the diagonal position in the similarity matrix diagram and the preset color is less than the color difference threshold, it is determined that the preset regular information exists in the comparison result; when the color difference between each display color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that the preset regular information does not exist in the comparison result.

[0261] In an embodiment of the present application, the video recall module 2552 is also used to obtain video recall features corresponding to the video to be detected, wherein the video recall features include one or more of content semantic features, text semantic features, title semantic features, body semantic features, frame image semantic features and cover image semantic features; obtain the features to be recalled corresponding to each video in the video resource library, wherein the features to be recalled correspond to the video recall features in feature type; based on the recall similarity values ​​between the video recall features and each feature to be recalled, determine the target feature to be recalled that is similar to the video recall feature from each feature to be recalled, wherein the recall similarity value includes one or more of Euclidean distance, vector dot product value and cosine similarity value; use the video in the video resource library that corresponds to the target feature to be recalled as the recalled video, thereby obtaining the video to be compared belonging to the recalled video.

[0262] In an embodiment of the present application, the video recall module 2552 is also used to obtain the recall feature index corresponding to the video recall feature, and the respective recall feature indexes corresponding to the respective features to be recalled, wherein the respective features to be recalled correspond one-to-one to the respective recall feature indexes; the matching degree between the recall feature index and the respective recall feature indexes is used as the recall similarity value, and based on the recall similarity value, the target feature to be recalled that is similar to the video recall feature is determined from the respective features to be recalled.

[0263] In an embodiment of the present application, the video acquisition module 2551 is also used to receive a video detection request sent by a task scheduling device, wherein the video detection request is generated by the task scheduling device in response to a video upload request sent by a video production end device; in response to the video detection request, the video to be detected is obtained from a content storage device.

[0264] In an embodiment of the present application, the video detection device 255 also includes a result processing module 2555, which is used to send the video detection result to a subsequent detection device when the video detection result is that the video to be detected is a duplicate video for the video to be compared, so that the subsequent detection device generates a subsequent detection request for the video detection result, and obtains the target detection result of the video to be detected in response to the subsequent detection request.

[0265] In an embodiment of the present application, the video detection device 255 is also used to send the video to be detected to the task scheduling device when the video detection result is that the video to be detected is a non-duplicate video with respect to the video to be compared, so that the task scheduling device pushes the video to be detected to the content consumption end device through the content distribution device based on the obtained recommendation information, so that the content consumption end device plays the video to be detected.

[0266] The present invention provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the video detection method described above in the present invention.

[0267] The embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein the executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the video detection method provided by the embodiment of the present application, for example, Figure 4-6 The video detection method is shown.

[0268] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0269] In some embodiments, executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0270] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0271] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0272] In summary, through the embodiments of the present application, when comparing the video to be detected with the recalled video to be compared, the video detection result of whether the video to be detected is a duplicate video is determined by comparing the audio characteristics of the audio information of the video to be detected with the audio characteristics of the audio information of the video to be compared; and the audio characteristics are the auditory characteristics of the audio information, such as volume, sound quality and timbre, and videos such as "series of lecture videos" have differences in audio characteristics; therefore, it is possible to accurately identify whether videos with similar pictures and titles such as "series of lecture videos" are duplicate videos, thereby improving the accuracy of video detection. In addition, since video recall and video deduplication verification are layered, video recall and video deduplication verification can be tuned separately, and the overall processing capacity of video detection can be improved by recalling more videos, reducing the number of videos for manual review, and improving the efficiency and consumption of video review.

[0273] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A video detection method, characterized in that: include: Get the video to be tested; Based on the video to be detected, video recall is performed from a video resource library to obtain a video to be compared; wherein the video recall features used for the video recall include one or more of content semantic features, text semantic features, title semantic features, body semantic features, frame image semantic features, and cover image semantic features; Separating the audio information corresponding to the video to be detected to obtain the audio to be detected, and extracting the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected, wherein the audio characteristics are auditory characteristics of the audio information; Separating the audio information corresponding to the video to be compared to obtain the audio to be compared, and extracting the features of the audio to be compared based on the audio characteristics to obtain the audio fingerprint to be compared; Comparing each sub-frame of the audio fingerprint to be detected in the audio fingerprint to be detected with each sub-frame of the audio fingerprint to be compared, respectively, to obtain a comparison result corresponding to each sub-frame of the audio fingerprint to be detected and each sub-frame of the audio fingerprint to be compared; Constructing a similarity matrix by using the sub-frame audio fingerprint to be detected as a one-dimensional attribute of a matrix, the sub-frame audio fingerprint to be compared as another-dimensional attribute of the matrix, and the comparison results as elements of the matrix; Based on the corresponding relationship between the preset similarity values ​​and the display colors, the similarity matrix is ​​converted into a similarity matrix graph; When the color difference between each display color at the diagonal position in the similarity matrix and the preset color is less than a color difference threshold, determining that preset regularity information exists in the comparison result, and determining a video detection result that the video to be detected is a duplicate video for the video to be compared, wherein the preset regularity information is a similarity trend between the audio fingerprint of each frame to be detected and the audio fingerprint of each frame to be compared; When the color difference between each displayed color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that the preset regularity information does not exist in the comparison result, and it is determined that the video to be detected is a video detection result of a non-repetitive video for the video to be compared.

2. The method according to claim 1, characterized in that The extracting the audio characteristics of the audio to be detected to obtain the audio fingerprint to be detected includes: Extracting multiple sub-frames of audio to be detected from the audio to be detected based on a preset frame unit; Extracting a feature of each sub-frame of audio to be detected in the multiple sub-frames of audio to be detected based on the audio characteristics to obtain an initial sub-frame of audio to be detected fingerprint; Performing dimensionality reduction on the initial sub-audio fingerprint to be detected to obtain a sub-audio fingerprint to be detected, thereby obtaining multiple frames of sub-audio fingerprints to be detected corresponding to the audio to be detected, wherein the audio fingerprint to be detected includes the multiple frames of sub-audio fingerprints to be detected.

3. The method according to claim 2, characterized in that Before extracting multiple sub-frames of audio to be detected from the audio to be detected based on the preset frame unit, the method further includes: Performing pre-emphasis processing on the audio to be detected to obtain audio to be framed; The extracting a plurality of sub-frames of audio to be detected from the audio to be detected based on the preset frame unit includes: Based on the preset frame unit, the multiple frames of sub-audio to be detected are extracted from the audio to be framed.

4. The method according to claim 2, characterized in that The extracting a plurality of sub-frames of audio to be detected from the audio to be detected based on the preset frame unit includes: Sampling the audio to be detected based on a preset sampling frequency to obtain a plurality of sampling points; Among the multiple sampling points, starting from the first sampling point, a preset number of sampling points are sequentially selected as a sub-frame of audio to be detected, and the preset number of sampling points corresponding to the position before the end position of the sampling point selection are continuously selected as the next sub-frame of audio to be detected, until The plurality of sampling points are selected and processed to obtain the plurality of sub-frames of audio to be detected, wherein the preset frame unit is determined based on a preset sampling frequency and the preset number of sampling points.

5. The method according to any one of claims 2 to 4, characterized in that The extracting the feature of each sub-frame of audio to be detected from the multiple sub-frames of audio to be detected based on the audio characteristics to obtain an initial sub-frame of audio to be detected fingerprint includes: Performing windowing processing on each frame of the sub-audio to be detected in the multiple frames of sub-audio to be detected to obtain a sub-audio to be converted; Convert the sub-audio to be converted into an energy distribution in the frequency domain to obtain a sub-spectrum to be detected, obtain the power spectrum of the sub-spectrum to be detected, and obtain a sub-power spectrum to be detected; Smoothing the sub-to-be-detected power spectrum to obtain a sub-smoothed power spectrum; Performing an inverse transformation on the logarithmic energy of the sub-smoothed power spectrum, and obtaining audio characteristic parameters of a preset order from the inverse transformation result; The differential parameters of the audio characteristic parameters and the frame energy of each frame of the sub-audio to be detected are obtained, thereby obtaining the initial sub-audio fingerprint to be detected including one or more of the audio characteristic parameters, the differential parameters, and the frame energy.

6. The method according to claim 4, characterized in that The step of reducing the dimension of the initial sub-audio fingerprint to be detected to obtain the sub-audio fingerprint to be detected includes: For each initial sampling point audio fingerprint in the initial sub-audio fingerprint to be detected, remove the lowest frequency feature to obtain an S-1-dimensional feature, wherein the initial sub-audio fingerprint to be detected includes the audio fingerprints of the initial sampling points of the preset number of sampling points, and each initial sampling point audio fingerprint includes an S-dimensional feature, where S is a positive integer greater than 1; Performing clustering and dimensionality reduction on the S-1 dimensional features based on a preset number of categories to obtain the preset number of cluster categories; The cluster center information of the cluster category is determined as the sampling point audio fingerprint of each initial sampling point audio fingerprint, thereby obtaining the sub-audio fingerprint to be detected corresponding to the initial sub-audio fingerprint to be detected, wherein the sub-audio fingerprint to be detected includes the audio fingerprints of the sampling points of the preset number of sampling points.

7. The method according to any one of claims 1 to 4, characterized in that The step of retrieving a video from a video resource library based on the video to be detected to obtain a video to be compared includes: Obtaining the video recall feature corresponding to the video to be detected; Obtaining the to-be-recalled features corresponding to each video in the video resource library, wherein the to-be-recalled features correspond to the video recall features in terms of feature type; Determining, from the various features to be recalled, a target feature to be recalled that is similar to the video recall feature based on recall similarity values ​​between the video recall feature and each feature to be recalled, wherein the recall similarity value includes one or more of a Euclidean distance, a vector dot product value, and a cosine similarity value; The video corresponding to the target to-be-recalled feature in the video resource library is used as the recalled video, thereby obtaining the to-be-compared video belonging to the recalled video.

8. The method according to claim 7, characterized in that The determining, based on the recall similarity values ​​between the video recall feature and each to-be-recalled feature, a target to-be-recalled feature similar to the video recall feature from each to-be-recalled feature, includes: Obtaining a recall feature index corresponding to the video recall feature and each recall feature index corresponding to each of the features to be recalled, wherein each of the features to be recalled corresponds one-to-one to each of the recall feature indexes; The matching degree between the recall feature index and each recall feature index is used as the recall similarity value, and based on the recall similarity value, the target feature to be recalled that is similar to the video recall feature is determined from the various features to be recalled.

9. The method according to any one of claims 1 to 4, characterized in that The step of obtaining the video to be detected includes: receiving a video detection request sent by a task scheduling device, wherein the video detection request is generated by the task scheduling device in response to a video upload request sent by a video production end device; In response to the video detection request, the video to be detected is obtained from a content storage device.

10. The method according to any one of claims 1 to 4, characterized in that After determining the video detection result, the method further includes: When the video detection result is that the video to be detected is a duplicate video for the video to be compared, the video detection result is sent to a subsequent detection device so that The subsequent detection device generates a subsequent detection request for the video detection result, and obtains the target detection result of the video to be detected in response to the subsequent detection request.

11. The method according to any one of claims 1 to 4, characterized in that After determining the video detection result, the method further includes: When the video detection result is that the video to be detected is a non-duplicate video for the video to be compared, the video to be detected is sent to the task scheduling device so that The task scheduling device pushes the video to be detected to the content consumption terminal device through the content distribution device based on the obtained recommendation information, so that The content consumption end device plays the video to be detected.

12. A video detection device, characterized in that: The device comprises: Video acquisition module, used to acquire the video to be detected; A video recall module is configured to recall videos from a video resource library based on the video to be detected to obtain a video to be compared; wherein the video recall features used for the video recall include one or more of content semantic features, text semantic features, title semantic features, body semantic features, frame image semantic features, and cover image semantic features; a feature acquisition module, configured to separate the audio information corresponding to the video to be detected to obtain the audio to be detected, and extract the audio characteristic features of the audio to be detected to obtain the audio fingerprint to be detected, wherein the audio characteristic features are auditory characteristics of the audio information; The feature acquisition module is further configured to separate the audio information corresponding to the video to be compared to obtain the audio to be compared, and extract the features of the audio to be compared based on the audio characteristics to obtain the audio fingerprint to be compared; The video detection module is configured to compare each sub-frame audio fingerprint to be detected in the audio fingerprint to be detected with each sub-frame audio fingerprint to be compared in the audio fingerprint to be compared, and obtain a comparison result corresponding to each sub-frame audio fingerprint to be detected and each sub-frame audio fingerprint to be compared; construct a similarity matrix by using each sub-frame audio fingerprint to be detected as a one-dimensional attribute of a matrix, each sub-frame audio fingerprint to be compared as another dimensional attribute of the matrix, and the comparison result as an element of the matrix; convert the similarity matrix into a similarity matrix diagram based on the correspondence between the preset similarity value and the display color; when each display color at the diagonal position in the similarity matrix diagram is equal to the display color, the display color at the diagonal position is equal to the display color at the diagonal position. When the color difference between the displayed color and the preset color is less than the color difference threshold, it is determined that there is preset regular information in the comparison result, and it is determined that the video to be detected is a video detection result of a repeated video for the video to be compared, wherein the preset regular information is the similarity trend between the sub-audio fingerprint to be detected in each frame and the sub-audio fingerprint to be compared in each frame; when the color difference between each displayed color at the diagonal position in the similarity matrix diagram and the preset color is not less than the color difference threshold, it is determined that there is no preset regular information in the comparison result, and it is determined that the video to be detected is a video detection result of a non-repeated video for the video to be compared.

13. A video detection device, characterized in that: include: a memory for storing executable instructions; A processor, configured to implement the method according to any one of claims 1 to 11 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that Executable instructions are stored, and when executed by a processor, they are used to implement the method described in any one of claims 1 to 11.

15. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the method according to any one of claims 1 to 11 is implemented.

Images