Audio copy detection method and apparatus, and electronic device, computer storage medium and computer program product

Abstract

Disclosed in the embodiments of the present application are an audio copy detection method and apparatus, and a device, a storage medium and a program product. The method comprises: extracting a global audio feature corresponding to audio to be processed, and extracting an audio segment feature sequence corresponding to the audio to be processed; retrieving from an audio retrieval library an approximate global audio feature corresponding to the global audio feature; acquiring an audio segment feature sequence corresponding to target audio to which the approximate global audio feature belongs, and on the basis of the audio segment feature sequences respectively corresponding to the audio to be processed and the target audio, determining an audio similarity between the audio to be processed and the target audio; and if the audio similarity reaches a preset similarity or greater, determining the target audio as copy audio corresponding to the audio to be processed.

Description

Audio copy detection method and device, electronic device, computer storage medium and computer program product

[0001] Cross-reference to Related Applications

[0002] The present application is based on and claims priority to Chinese Patent Application No. 2024111429279, filed on August 20, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to an audio copy detection method and device, an electronic device, a computer storage medium, and a computer program product. BACKGROUND

[0004] Audio copy detection technology is a technology for identifying whether audio content is similar or identical to known audio. If the identification is yes, the known audio can be used as a copy of the audio to be identified. Audio copy detection technology is often applied to audio copyright protection, music recommendation and other application scenarios. Related technologies often rely on the matching of global audio features when detecting audio copies, and the detection dimension is single, which easily ignores the detailed differences of the audio, resulting in low accuracy of detection. How to improve the accuracy of audio copy detection is still a technical problem that needs to be researched and solved by those skilled in the art. SUMMARY

[0005] To solve the above technical problems, the embodiments of the present application provide an audio copy detection method, an audio copy detection device, an electronic device, a computer readable storage medium and a computer program product. The embodiments of the present application can improve the accuracy of audio copy detection.

[0006] In one aspect of the embodiments of the present application, an audio copy detection method is provided, the method is executed by an electronic device, and the method comprises: extracting a global audio feature corresponding to a to-be-processed audio, and extracting an audio segment feature sequence corresponding to the to-be-processed audio; retrieving an approximate global audio feature corresponding to the global audio feature from an audio retrieval library; obtaining an audio segment feature sequence corresponding to a target audio to which the approximate global audio feature belongs, and determining an audio similarity between the to-be-processed audio and the target audio according to the audio segment feature sequences respectively corresponding to the to-be-processed audio and the target audio; and if the audio similarity is above a preset similarity, determining the target audio as a copy audio corresponding to the to-be-processed audio.

[0007] In another aspect of the embodiments of the present application, an audio copy detection device is provided, which comprises: a feature extraction module configured to extract a global audio feature corresponding to a to-be-processed audio and extract an audio segment feature sequence corresponding to the to-be-processed audio; a feature retrieval module configured to retrieve an approximate global audio feature corresponding to the global audio feature from an audio retrieval library and acquire an audio segment feature sequence corresponding to a target audio to which the approximate global audio feature belongs; a similarity determination module configured to determine an audio similarity between the to-be-processed audio and the target audio according to the audio segment feature sequence corresponding to the to-be-processed audio and the audio segment feature sequence corresponding to the target audio; and a copy determination module configured to determine the target audio as a copy audio corresponding to the to-be-processed audio if the audio similarity reaches a preset similarity or above.

[0008] In another aspect of the embodiments of the present application, an electronic device is provided, which comprises: one or more processors; and a memory configured to store one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the audio copy detection method as described above.

[0009] In another aspect of the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program, which, when executed by a processor of an electronic device, causes the electronic device to perform the audio copy detection method as described above.

[0010] In another aspect of the embodiments of the present application, a computer program product is provided, which comprises a computer program, which, when executed by a processor of an electronic device, implements the audio copy detection method as described above.

[0011] The technical solution provided by the embodiments of the present application first retrieves an approximate global audio feature from an audio retrieval library according to a global audio feature of a to-be-processed audio, then determines an audio similarity between the to-be-processed audio and a target audio according to audio segment feature sequences corresponding to the to-be-processed audio and the target audio, which can be understood as first reflecting the overall properties of the audio through the global audio feature to achieve coarse-grained detection, quickly narrow the detection range in the audio retrieval library, and improve the detection efficiency, and then capturing the local details and timing association of the audio through the audio segment feature sequence to achieve fine-grained detection, so as to focus the detection on the potentially similar audios and enhance the pertinence of the detection. The detection of the global audio feature and the detection of the audio segment feature sequence are combined to reduce the false judgment caused by a single feature, thereby ensuring the accuracy of the audio copy detection. The target audio is determined as a copy audio of the to-be-processed audio according to whether the audio similarity reaches a preset similarity, so that the determination process has a clear quantitative standard, the subjective factor interference is reduced, and the consistency and reliability of the detection result are ensured.

[0012] It should be understood that the general description above and the detailed description below are only exemplary and explanatory, and are not limiting to the present application. BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1 is a schematic diagram of an implementation environment related to embodiments of the present application.

[0014] FIG. 2 is a first flowchart of an audio copy detection method according to an embodiment of the present application.

[0015] FIG. 3 is a second flowchart of an audio copy detection method according to an embodiment of the present application.

[0016] FIG. 4 is a flowchart of extracting global audio features corresponding to audio to be processed according to an embodiment of the present application.

[0017] FIG. 5 is a schematic diagram of audio classification distribution of a video published by a user on a video publishing platform according to an embodiment of the present application.

[0018] FIG. 6 is a flowchart of an audio identification processing method based on a video publishing platform according to an embodiment of the present application.

[0019] FIG. 7 is a schematic diagram of a principle architecture of adding audio identification to a video published on a video publishing platform according to an embodiment of the present application.

[0020] FIG. 8 is a flowchart of an audio identification processing method based on a video publishing platform according to an embodiment of the present application.

[0021] FIG. 9 is a flowchart of a video processing method according to an embodiment of the present application.

[0022] FIG. 10 is a schematic diagram of a front-end interface jump of a video publishing platform according to an embodiment of the present application.

[0023] FIG. 11 is a block diagram of an audio copy detection apparatus according to an embodiment of the present application.

[0024] FIG. 12 is a schematic diagram of a computer system of an electronic device suitable for implementing embodiments of the present application.

[0025] It should be noted that the above-mentioned "first" and "second" are only used to distinguish different schemes, and do not represent the degree of superiority or priority in the implementation process. DETAILED DESCRIPTION

[0026] Detailed description will be given to the exemplary embodiments with examples shown in the drawings. In the following description, same numbers in different drawings represent same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they only represent examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] The block diagrams shown in the drawings are only functional entities, and do not necessarily have to correspond to physically independent entities. That is, the functional entities can be implemented in software, or in one or more hardware modules or integrated circuits, or in at least one of different networks, processor devices, and microcontroller devices.

[0028] The flowcharts shown in the drawings are only exemplary illustrations, and do not necessarily include all contents, and at least one of operations and steps, and do not necessarily have to be executed in the described order. For example, at least one of the operations and steps can be further divided, and at least one of the operations and steps can be combined or partially combined, and thus the actually executed order can be changed according to the actual situation.

[0029] "Plural" mentioned in the present application means two or more. "At least one of" describes the association of the associated objects, and indicates that there can be three relationships, for example, at least one of A and B can indicate three cases of A alone, A and B together, and B alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0030] The terms "first", "second", "third", and "fourth" and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a particular order. The terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus including a series of steps or units is not limited to the listed steps or units, but can optionally further include steps or units not listed, or can optionally further include other steps or units inherent to the process, method, product, or apparatus.

[0031] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0032] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0033] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0034] 1) Global audio features refer to the features extracted through comprehensive and holistic analysis of the entire audio segment. They comprehensively reflect the overall attributes of the audio, including but not limited to the overall spectral distribution trend, long-term energy variation patterns, and macroscopic stylistic features. Global audio features can be obtained by processing the audio spectrogram sequence through a trained feature extraction model, and are an abstract representation of the overall content of the audio.

[0035] 2) Audio segment features refer to the features obtained by dividing the audio into several consecutive audio segments according to a preset unit duration, and then extracting features from each audio segment individually. Audio segment features can capture the local characteristics of audio within a specific short time period, such as detailed information such as frequency components and energy fluctuations within that time period. Each audio segment has its corresponding audio segment features.

[0036] 3) Audio segment feature sequence refers to a sequence formed by dividing audio into multiple ordered audio segments based on unit duration, extracting audio features from each segment, and then arranging and combining the extracted audio features in sequence according to the inherent temporal order of these audio segments in the original audio. The audio segment feature sequence completely preserves the local feature changes of the audio in the time dimension and can reflect the temporal correlation of the local characteristics of the audio.

[0037] 4) shallow features, which are features generated by a shallow feature extraction network in the feature extraction model, and the shallow feature extraction network outputs shallow features after preliminary processing of the input spectrogram or audio segment. The shallow features focus more on reflecting the basic and surface features of the audio, such as the local texture in the spectrogram, the short-term frequency distribution within the audio segment, and other more intuitive information, which is the basis for subsequent feature processing.

[0038] 5) deep features, which are features obtained by further integrating, abstracting, and processing the shallow features by a deep feature extraction network in the feature extraction model. The deep features can penetrate the surface information of the audio, reflect the more abstract and complex global correlation and internal regularity of the audio, and are a more in-depth representation of the overall characteristics of the audio.

[0039] 6) audio similar duration, which is a detailed comparison of the audio segment feature sequences of the to-be-processed audio and the target audio. By analyzing the feature matching in the two audio segment feature sequences, the similar parts in the two audios are determined, and the total duration corresponding to these similar parts in the time dimension is the audio similar duration, which can quantitatively reflect the similarity of the two audio contents.

[0040] First of all, as mentioned earlier, the audio copy detection technology is a technology for identifying whether the audio content is similar or identical to a known audio. In related technologies, audio copy detection is mainly achieved by the following methods:

[0041] 1. Detection method based on fingerprint features: audio fingerprint features need to be extracted to represent the fingerprint content, and by comparing the similarity between audio fingerprints, it can be determined whether they are the same or similar audio;

[0042] 2. Detection method based on time domain features: the time domain features (such as energy, zero-crossing rate, etc.) of the audio need to be analyzed, and by calculating the similarity of the time domain features between audios, it can be determined whether they are the same or similar audio;

[0043] 3. Detection method based on frequency domain features: the frequency domain features (such as spectrum, harmonic, etc.) of the audio need to be analyzed, and by calculating the similarity of the frequency domain features between audios, it can be determined whether they are the same or similar audio;

[0044] 4. Detection method based on time-frequency domain features: the time-frequency domain features (such as mel-frequency cepstral coefficients, pitch frequency, etc.) of the audio need to be analyzed, and by calculating the similarity of the time-frequency domain features between audios, it can be determined whether they are the same or similar audio;

[0045] 5. The detection method based on deep learning: using a deep learning model (such as a convolutional neural network, a recurrent neural network, a long short-term memory network, etc.) to encode the audio features, and comparing the encoded audio features to determine whether they are the same or similar.

[0046] In the deep learning-based detection method, the deep learning model can automatically learn the high-level feature representation of the audio, thereby improving the accuracy and robustness of the copy detection. Therefore, the audio copy detection scheme proposed in the embodiments of the present application also uses the deep learning-based detection method to determine whether the audio content is the same or similar.

[0047] The audio copy detection scheme proposed in the embodiments of the present application will be described in detail below.

[0048] First, refer to FIG. 1, which is a schematic diagram of an implementation environment related to the embodiments of the present application. The implementation environment includes a terminal 110 and a server 120, which communicate with each other through wired or wireless means.

[0049] The terminal 110 is a device that interacts with the user to receive the user's input information, such as audio, and returns information to the user. For example, the terminal 110 is configured with a graphical user interface (GUI), through which the user can interact with the terminal 110, such as triggering the graphical user interface to input information, and obtaining the required information from the graphical user interface. For another example, the terminal 110 can be configured with a keyboard, a mouse, etc. to assist in information interaction with the user.

[0050] The server 120 provides data services for the information interaction between the terminal 110 and the user. For example, the terminal 110 uploads the user's input audio to the server 120, so that the server 120 performs detection of the same or similar copy audio as the user's input audio, and returns the detection result to the terminal 110. The terminal 110 returns the information to the user after receiving the information transmitted by the server 120.

[0051] Of course, in the case where there is no data service requirement for the terminal 110, the terminal 110 can also not interact with the server 120. For example, after receiving the user's input audio, the terminal 110 can directly perform detection of the same or similar copy audio as the user's input audio, and return the detection result to the user.

[0052] It should be noted that the terminal 110 can be a smart phone, a tablet, a notebook computer, a computer, a smart home appliance, a smart terminal, or the like, and the server 120 can be a physical server, a server cluster composed of multiple physical servers, or a distributed system, or a cloud server providing basic cloud computing services, and the embodiments of the present application do not limit this.

[0053] Referring to FIG. 2, FIG. 2 is a first flowchart of an audio copy detection method according to an embodiment of the present application. The method can be applied to the implementation environment shown in FIG. 1, for example, can be specifically executed by the server 120, or can be specifically executed by the terminal 110, or can be executed by the terminal 110 and the server 120 together. Of course, the method can also be applied to other implementation environments, and executed by a terminal or a server in other implementation environments, or executed by a terminal and a server in other implementation environments together, and the embodiments of the present application do not limit this.

[0054] As shown in FIG. 2, in an exemplary embodiment, the audio copy detection method includes S210-S240, which are described in detail as follows.

[0055] S210, extracting global audio features corresponding to the audio to be processed, and extracting an audio segment feature sequence corresponding to the audio to be processed.

[0056] First of all, it should be noted that the global audio features disclosed in the embodiments are used to represent the overall properties of the audio, and the global audio features are usually extracted by a multi-layer neural network, so the global audio features can also be understood as deep features.

[0057] The audio segment feature sequence disclosed in the embodiments is formed by arranging a plurality of audio segment features in chronological order. For example, by dividing the audio into a plurality of audio segments, then extracting the audio features of each audio segment, and sorting the audio features in chronological order, the corresponding audio segment feature sequence is obtained.

[0058] Based on the global audio features corresponding to the audio to be processed, the embodiments can realize the recall of the target audio similar to the audio to be processed; based on the audio segment feature sequence corresponding to the audio to be processed, the embodiments can realize further verification of whether the recalled target audio is similar to the audio to be processed, thereby ensuring that the obtained detection result has high accuracy.

[0059] For example, if the audio to be processed is a song with a length of 3 minutes, the global audio feature extracted by the multi-layer neural network can be a 128-dimensional deep feature that comprehensively reflects the overall style, rhythm framework, and other global properties of the song. After dividing the song into segments of 0.5 seconds, 360 audio segments are obtained. The 64-dimensional audio segment features corresponding to each audio segment are extracted, and the multiple audio segment features are arranged into an audio segment feature sequence in the time order of the audio segments in the song. The audio segment feature sequence includes 360 64-dimensional vector audio segment features.

[0060] S220, retrieving the approximate global audio feature corresponding to the global audio feature from the audio retrieval library.

[0061] The audio retrieval library stores a large number of global audio features of known audios. The global audio features of the known audios are extracted in the same way as the global audio feature of the audio to be processed.

[0062] For example, by calculating the similarity between the global audio feature of the audio to be processed and the global audio feature of the known audio, the global audio feature of the known audio with a similarity greater than or equal to a preset threshold can be determined as an approximate global audio feature, and the known audio with a similarity greater than or equal to the preset threshold can be referred to as a target audio. Alternatively, by converting the global audio features of the audio to be processed and the known audio into hash values, it can be quickly determined whether the two global audio features are the same or similar by comparing the hash values.

[0063] The highest similarity corresponding known audio can also be taken as the target audio. The similarity between the global audio feature of the to-be-processed audio and the global audio feature of the known audio can be determined by using any one of the following algorithms: cosine similarity, Euclidean distance, Manhattan distance, and Pearson correlation coefficient. Taking the cosine similarity as an example, the vector product of the global audio feature of the to-be-processed audio and the global audio feature of the known audio is determined; the length product of the length of the global audio feature of the to-be-processed audio and the length of the global audio feature of the known audio is determined; and the ratio of the vector product and the length product is determined as the similarity. For example, the global audio feature of the to-be-processed audio is m, and the global audio feature of the known audio is n, the vector product is m*n, the length of the global audio feature of the to-be-processed audio is |m|, the length of the global audio feature of the known audio is |n|, the length product is |m|*|n|, and the ratio of the vector product and the length product is (m*n) / (|m|*|n|), which is taken as the similarity. Taking the comparison of the hash values as an example, the global audio feature of the to-be-processed audio is mapped into a first hash value by using a hash function, the global audio feature of the known audio is mapped into a second hash value by using the hash function, the Hamming distance between the first hash value and the second hash value is determined, if the Hamming distance is greater than zero and less than a Hamming distance threshold, it is determined that the global audio feature of the to-be-processed audio is similar to the global audio feature of the known audio, if the Hamming distance is equal to the Hamming distance threshold, it is determined that the global audio feature of the to-be-processed audio is identical to the global audio feature of the known audio, and if the Hamming distance is greater than or equal to the Hamming distance threshold, it is determined that the global audio feature of the to-be-processed audio is not identical to and not similar to the global audio feature of the known audio.

[0064] It should be noted that the number of the approximate global audio features in the audio retrieval library can be one, multiple, or none, and the embodiment does not limit the number of the approximate global audio features obtained by retrieval. In the case where there is no approximate global audio feature, it indicates that the to-be-processed audio is a new audio relative to the audio retrieval library.

[0065] In S230, the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs is obtained, and the audio similarity between the to-be-processed audio and the target audio is determined according to the audio segment feature sequences corresponding to the to-be-processed audio and the target audio respectively.

[0066] In the embodiment, the known audio also corresponds to a global audio feature and an audio segment feature sequence. The extraction manner of the global audio feature and the audio segment feature sequence corresponding to the known audio can be the same as the extraction manner of the global audio feature and the audio segment feature sequence corresponding to the to-be-processed audio.

[0067] In some example embodiments, both the global audio feature and the audio segment feature sequence of the known audio can be stored in the audio retrieval library, and can be stored in association, so that after the approximate global audio feature is retrieved from the audio retrieval library, the audio segment feature sequence associated with the approximate global audio feature can be found in the audio retrieval library, so as to obtain the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs.

[0068] For example, in the audio retrieval library, the global audio feature and the audio segment feature sequence of the known audio can be stored in key-value pairs, in which the global audio feature is the key, and each audio segment feature constituting the audio segment feature sequence is a different value.

[0069] Alternatively, the global audio feature and the audio segment feature sequence of the known audio can be stored separately in the audio retrieval library. For example, each global audio feature and each audio segment feature sequence stored in the audio retrieval library has an audio identifier, and the audio corresponding to the same audio identifier should also be the same or similar.

[0070] Therefore, in some example embodiments, after the copy audio corresponding to the to-be-processed video is detected, the audio identifier corresponding to the copy audio is also taken as the audio identifier corresponding to the to-be-processed audio, and the global audio feature and the audio segment feature sequence corresponding to the to-be-processed audio are stored in the audio retrieval library based on the audio identifier corresponding to the to-be-processed audio.

[0071] In other example embodiments, the audio segment feature sequence of the known audio is stored in a database other than the audio retrieval library. For example, each known audio has a respective audio identifier, the audio identifier of the known audio is stored in association with the global audio feature of the known audio in the audio retrieval library, and the audio identifier of the known audio is stored in association with the audio segment feature sequence of the known audio in the other database. Thus, the audio segment feature sequence associated with the audio identifier can be obtained from the other database according to the audio identifier associated with the approximate global audio feature, so as to obtain the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs.

[0072] For example, in the other database, the audio identifier and the audio segment feature sequence of the known audio can be stored in key-value pairs, in which the audio identifier is the key, and each audio segment feature constituting the audio segment feature sequence is a different value. Thus, the other database can be a key-value database.

[0073] It should be further noted that the storage of the global audio feature and the audio segment feature sequence in the database can have a specified storage time limit, that is, the database will only retain the global audio feature and the audio segment feature sequence for a specified time. Therefore, in some exemplary embodiments, when the global audio feature and the audio segment feature sequence are stored, the currently stored global audio feature and audio segment feature sequence automatically obtain a specified storage time limit, and the remaining storage time of the global audio feature and the audio segment feature sequence corresponding to the copy audio can also be updated to the specified time. Thus, the database is updated iteratively as new audio is added.

[0074] The embodiment is based on the audio segment feature sequences corresponding to the to-be-processed audio and the target audio, and the audio similarity between the to-be-processed audio and the target audio can be determined by comparing the audio segment feature sequences, that is, by comparing the audio segment features in the audio segment feature sequences to determine the same audio segments in the to-be-processed audio and the target audio.

[0075] For example, the audio similarity time between the to-be-processed audio and the target audio can be obtained by comparing the audio segment features of the audio segment feature sequence corresponding to the to-be-processed audio and the audio segment feature sequence corresponding to the target audio, and the audio similarity between the to-be-processed audio and the target audio can be determined according to the audio lengths corresponding to the to-be-processed audio and the target audio and the audio similarity time.

[0076] For example, the smaller length can be determined from the length of the to-be-processed audio and the length of the target audio, and then the ratio between the audio similarity time and the smaller length can be calculated to obtain the audio similarity between the to-be-processed audio and the target audio. For another example, the audio similarity between the to-be-processed audio and the target audio can also be calculated according to the larger length of the length of the to-be-processed audio and the length of the target audio, and the embodiment does not limit this.

[0077] For ease of understanding, for example, it is assumed that the length of the to-be-processed video is 110 seconds and the length of the target video is 120 seconds. Through comparison between the audio segment feature sequences corresponding to the two, it is determined that the 100 seconds of audio from the 10th second to the 110th second of the to-be-processed video and the 20th second to the 120th second of the target video are similar, and the audio similarity time is 100 seconds. The audio similarity is: 100 seconds / min(110 seconds, 120 seconds)=91%.

[0078] In some embodiments, the audio segment feature sequence of the audio to be processed and the audio segment feature sequence of the target audio are time axis aligned to ensure that the time granularity of the two audio segment feature sequences is consistent; a vector similarity measurement method is used to calculate the similarity of the audio segment features of the audio to be processed and the audio segment features of the target audio corresponding to the same time period one by one, a feature matching threshold is set, and if the similarity of the audio segment features of the audio to be processed and the audio segment features of the target audio is greater than or equal to the feature matching threshold, it is determined that the audio to be processed and the target audio are similar in the time period; the continuous similar audio segments are merged into an audio similar interval to avoid fragmented judgment caused by single segment feature fluctuation. The audio similar interval in the audio to be processed is converted into a first actual time interval, and the audio similar interval in the target audio is converted into a second actual time interval; the length of the overlapping part of the first actual time interval and the second actual time interval is taken as the audio similarity length.

[0079] For example, the audio segment feature sequence of the audio to be processed includes 220 feature vectors (corresponding to 110 seconds, one segment every 0.5 seconds), and the sequence of the target audio includes 240 feature vectors (corresponding to 120 seconds). After calculating the vector similarity segment by segment, it is found that the similarity of the audio segment features of each audio segment in the 20th to 220th segment (corresponding to 10 to 110 seconds) of the audio to be processed and the 40th to 240th segment (corresponding to 20 to 120 seconds) of the target audio is greater than or equal to 0.85, and these similar audio segments are continuous and uninterrupted, so these audio segments are merged into an audio similar interval. The first actual time interval of the audio to be processed is 10 to 110 seconds (length 100 seconds), the second actual time interval of the target audio is 20 to 120 seconds (length 100 seconds), and the overlapping interval of the two is 20 to 110 seconds, and the length of the overlapping interval is 100 seconds, that is, the audio similarity length is 100 seconds.

[0080] S240, if the audio similarity reaches a preset similarity, the target audio is determined as a copy audio corresponding to the audio to be processed.

[0081] The preset similarity can be set according to actual application requirements. For example, in some application scenarios with high requirements for copy detection accuracy, the preset similarity can be set to a larger value, such as 90%, 85%, 80%, etc. In some application scenarios with low requirements for copy detection accuracy, the preset similarity can be set to a slightly lower value, such as 70%, 75%, etc., which is not limited herein.

[0082] For example, it is assumed that two approximate global audio features are retrieved in S220, that is, two target audios are determined, and then further comparison is performed based on the audio segment feature sequence. If the audio similarity between the audio segment feature sequence of one of the target audios and the audio segment feature sequence of the audio to be processed is relatively high, for example, higher than a preset similarity, and the audio similarity between the audio segment feature sequence of the other target audio and the audio segment feature sequence of the audio to be processed is relatively low, for example, lower than the preset similarity, the target audio with the audio similarity higher than the preset similarity is determined as the copy audio corresponding to the audio to be processed. Thus, it can be seen that the present embodiment first performs coarse-grained detection through the global audio feature, and then performs fine-grained detection through the audio segment feature sequence, thereby ensuring the accuracy of the audio copy detection.

[0083] In some embodiments, sample data is collected, the sample data includes copy audio pairs (such as different versions of the same song, edited homologous audios) and non-copy audio pairs (such as different songs, irrelevant audio segments), the audio similarity of two audios in each audio pair is calculated, and based on the audio similarity of a plurality of audio pairs, a similarity distribution curve is drawn to observe the similarity threshold critical point of the copy audio pairs and the non-copy audio pairs. For example, a threshold point balancing the false positive rate (misjudging a non-copy as a copy) and the false negative rate (misjudging a copy as a non-copy) is found through a receiver operating characteristic curve (ROC), or a minimum threshold corresponding to a target index (such as an accuracy greater than or equal to 95%) is set as the preset similarity.

[0084] For example, 1000 copy audio pairs and 1000 non-copy audio pairs are collected, the similarity of each audio pair is calculated, 98% of the copy audio pairs have a similarity greater than or equal to 82%, 99% of the non-copy audio pairs have a similarity less than or equal to 78%, and when the preset similarity is set to 80%, the false positive rate and the false negative rate are both lower than 3% through ROC curve analysis, and therefore the preset similarity is determined to be 80%.

[0085] Please continue to refer to FIG. 3, which is a second flowchart of the audio copy detection method according to an embodiment of the present application.

[0086] As shown in FIG. 3, the process of extracting the audio segment feature sequence in S210 includes the following S310-S320:

[0087] S310, dividing the audio to be processed based on a preset first unit time length to obtain a plurality of first audio segments contained in the audio to be processed;

[0088] In some embodiments, the to-be-processed audio is continuously divided into first audio segments with a first unit time length starting from a starting time point of the to-be-processed audio. For example, the starting time point is taken as the starting point of the first first audio segment, and the first unit time length is taken as the ending point of the first first audio segment. Then, the ending point of the first first audio segment is taken as the starting point of the second first audio segment, and the division is continued until the entire time length of the to-be-processed audio is covered. If the total time length of the to-be-processed audio is not an integer multiple of the first unit time length, the time length of the last first audio segment is the remaining time length.

[0089] For example, the total time length of the to-be-processed audio is 10.3 seconds, and the preset first unit time length is 1 second. Starting from 0 second, 10 complete first audio segments (corresponding to 0-1 second, 1-2 second, …, 9-10 second, respectively) and the 11th first audio segment (corresponding to 10-10.3 second) are obtained, and there are a total of 11 first audio segments.

[0090] In S320, audio features are extracted from each first audio segment, respectively, and the audio features of each first audio segment are sorted in chronological order to obtain an audio segment feature sequence corresponding to the to-be-processed audio.

[0091] It can be understood that in S310, the first unit time length is usually a small value, such as 1 second. The to-be-processed audio is divided based on the first unit time length to obtain a plurality of first audio segments with the first unit time length.

[0092] In S320, audio features are extracted from each first audio segment, respectively, and the audio features of each first audio segment are sorted in chronological order to obtain an audio segment feature sequence corresponding to the to-be-processed audio. For example, assuming that the total time length of the to-be-processed audio is 100 seconds and the first unit time length is 1 second, the audio segment feature sequence corresponding to the to-be-processed audio is obtained by arranging the audio features of 100 first audio segments in chronological order. It can be seen that the audio segment feature sequence can represent more fine-grained audio content.

[0093] The embodiments of the present application divide the audio segments by a fixed first unit time length, and arrange the audio segment features of the audio segments in chronological order, so as to ensure that the audio segment feature sequence strictly corresponds to the time flow direction of the original audio, and provide an accurate time sequence basis for subsequent comparison with the target audio segment feature sequence. The to-be-processed audio is divided into a plurality of first audio segments, and the audio segment features are extracted, respectively, so as to finely capture the local audio characteristics of different time periods, avoid loss of local details caused by global feature smoothing, and improve the detection sensitivity of local audio. The division rule based on the preset first unit time length is fixed, so that different to-be-processed audio segment feature sequences have a uniform time granularity, and the fairness of subsequent cross-audio comparison and the repeatability of the results are ensured.

[0094] Continuing to refer to FIG. 3, the process of extracting the global audio features corresponding to the audio to be processed in S210 further includes the following contents shown in S330-S350:

[0095] S330, performing division processing on the audio to be processed based on a preset second unit time length, to obtain a plurality of second audio segments contained in the audio to be processed;

[0096] In some embodiments, the audio to be processed is continuously segmented according to the second unit time length starting from the starting time of the audio to be processed, for example, the starting time is the starting point of the first second audio segment, and the end point of the first second audio segment is obtained after the second unit time length; then the end point of the first second audio segment is taken as the starting point of the second second audio segment, and the segmentation is continued until the total time length of the audio to be processed is covered. If the total time length of the audio to be processed is not an integer multiple of the second unit time length, the time length of the last second audio segment is the remaining time length.

[0097] For example, the total time length of the audio to be processed is 10.3 seconds, the preset first unit time length is 1 second, and the preset second unit time length is 2 seconds (greater than the first unit time length). Starting from 0 seconds, 5 complete second audio segments (corresponding to 0-2 seconds, 2-4 seconds, 4-6 seconds, 6-8 seconds, and 8-10 seconds, respectively) and a sixth second audio segment (corresponding to 10-10.3 seconds) are obtained, a total of 6 second audio segments.

[0098] S340, performing spectrum graph conversion on each second audio segment respectively to obtain a spectrum graph sequence corresponding to the audio to be processed;

[0099] Here, if the second audio segment has a mute part, the mute part can be removed to retain the valid audio signal; a uniform sampling rate adjustment is performed on each second audio segment to ensure that the resolution of the time domain signals of different second audio segments in the time dimension is consistent; for each second audio segment after preprocessing, the following processing is performed: the time domain signal of the second audio segment is divided into a plurality of frames according to a fixed time window, and an overlap is set between adjacent frames to reduce the loss of information between frames; a window function (such as a Hanning window) is applied to each frame signal to reduce the spectral leakage of the frame edge; a short-time Fourier transform is performed on each frame after windowing to convert the time domain signal into a frequency domain signal, obtaining the spectrum of each frame; the spectra of all frames in the same second audio segment are arranged in time sequence to form a two-dimensional matrix (the horizontal axis is the time frame, the vertical axis is the frequency point, and the matrix element value is the energy of the corresponding frequency), which is the spectrum graph corresponding to the second audio segment.

[0100] S350, inputting the spectrum graph sequence into the trained feature extraction model to obtain the global audio features output by the trained feature extraction model.

[0101] It can be understood that, since the total length of the audio to be processed is usually large, if the extraction of the global feature is directly performed on the audio to be processed, the load of the deep learning network is too large and the deep learning network may collapse, therefore, in S330, the audio to be processed is divided into multiple second audio segments based on the second unit length, and the global audio feature is extracted based on the second audio segment, thereby reducing the load of the deep learning network to a certain extent.

[0102] Therefore, based on different starting points, the second unit length is usually greater than or equal to the first unit length. Usually, the second unit length is greater than the first unit length, for example, the second unit length can be 10 seconds. In the case that the length of the audio segment is less than the second unit length, the second audio segment conforming to the second unit length can be obtained by lengthening the audio segment.

[0103] In S340, the waveform of the audio time sequence is converted into a mel spectrum graph by performing spectrum graph conversion on each second audio segment, thereby obtaining a spectrum graph sequence corresponding to the audio to be processed. In S350, the spectrum graph sequence corresponding to the audio to be processed is input into the trained feature extraction model, and then the global audio feature output by the trained feature extraction model can be obtained.

[0104] Please refer to FIG. 4, which is a flow diagram of extracting the global audio feature corresponding to the audio to be processed according to an embodiment of the present application. As can be seen, the trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network, the spectrum graph sequence corresponding to the audio to be processed is input into the trained feature extraction network, the shallow feature corresponding to each frequency graph in the spectrum graph sequence is extracted by the shallow feature extraction network first, and the shallow feature sequence corresponding to the audio to be processed is obtained accordingly, then the deep feature corresponding to each shallow feature is extracted by the deep feature extraction network, and finally the average value of the features is calculated according to each deep feature, so that the calculated average value of the features is taken as the global audio feature corresponding to the audio to be processed. The order of the shallow feature extraction network in the trained feature extraction model is less than the order of the deep feature extraction network in the trained feature extraction model, for example, the shallow feature is output by the network layer (such as the initial convolution layer, the processing unit near the input layer) at the front of the feature extraction model, and the deep feature is output by the network layer (such as the multiple convolution layers, the fully connected layer or the output layer near the processing unit after the stacking of the pooling layer) at the back of the feature extraction model.

[0105] For ease of understanding, for example, it is assumed that the second unit time length is 10 seconds, and a 768-dimensional feature vector is extracted from each second audio segment via the trained feature extraction model, so that the audio features corresponding to each second audio segment can be represented as (batch, 768), where batch is the quotient between the time length of the audio to be processed and the second unit time length, and then the average value of the batch dimension is calculated to obtain the global audio features corresponding to the audio to be processed.

[0106] For example, the shallow feature extraction network can include a multi-layer CNN (Convolutional Neural Networks) to learn shallow features by using the translational invariance of the CNN; the deep feature extraction network can include a TransFormer network to learn deep features by using the long-term memory capability of the TransFormer network. It should be noted that the specific network structure of the shallow feature extraction network and the deep feature extraction network is not limited in the embodiment, and any combination of the shallow feature extraction network and the deep feature extraction network that can achieve the above functions can be used as the structure of the feature extraction model.

[0107] The embodiment of the present application focuses on the basic visual features of the spectrum graph through the shallow network, which is suitable for fine-grained audio segment features; the shallow features are further integrated and abstracted through the deep network to capture the global correlation across segments, and the generated deep features can represent the overall properties of the audio. This hierarchical extraction mechanism makes the global audio features and the audio segment feature sequence correspond to different abstraction levels, respectively, to meet the comparison requirements of different granularities. Taking the average value of the deep features as the global audio features can integrate the deep feature information of multiple second audio segments, avoid the one-sidedness of a single second audio segment feature, more comprehensively reflect the overall characteristics, and improve the representativeness of the global audio features.

[0108] Based on the network structure of the feature extraction model as in the above example, in an exemplary embodiment, S210 can input the first audio segment sequence corresponding to the audio to be processed into the trained feature extraction model, and output the shallow feature sequence output by the shallow feature extraction network as the audio segment feature sequence corresponding to the audio to be processed. That is, the audio segment features contained in the audio segment feature sequence corresponding to the audio to be processed are shallow features, which are semantically neutral and can more accurately express the audio framework. Therefore, after the recall of similar global audio features is realized using the global audio features, the feature similarity is further checked through the shallow feature sequence, which can further improve the accuracy of audio duplicate detection.

[0109] It should be further explained that the trained feature extraction model is obtained by pre-training an initial feature extraction model using a pre-collected training data set. The pre-training process is a self-supervised learning process. The basic idea of self-supervised learning is to transform the sample data in a certain way (for example, rotation, occlusion, color transformation, etc.), so that the feature extraction model can learn useful feature representations from the transformed data.

[0110] In some exemplary embodiments, at least one training task in the reconstruction learning task and the contrastive learning task can be generated respectively, and then the at least one training task is executed based on the training data set to pre-train the initial feature extraction model to obtain the trained feature extraction model.

[0111] The reconstruction learning task indicates that the model parameter optimization is performed by calculating a mean absolute error (MAE) loss value. The calculation formula of the loss function corresponding to the reconstruction learning task can be represented as formula (1):

[0112] Wherein, n represents the number of samples in a training batch, represents the global audio features output by the initial feature extraction model, y i represents the real global audio features, loss(MAE) represents the loss function corresponding to the reconstruction learning task, and ∑ is used for summing data.

[0113] The contrastive learning task indicates that the model parameter optimization is performed by calculating a noise contrast estimation loss value. The basic idea of the contrastive learning task is to optimize the model by comparing positive samples and negative samples. The goal of the model is to maximize the similarity between the positive samples and the target, and to minimize the similarity between the negative samples and the target. The loss function corresponding to the contrastive learning task can be represented as loss(InfoNCE), and the calculation formula is as formula (2):

[0114] Wherein, n represents the number of samples in a training batch, A + represents the similarity score of the positive sample pair, B - represents the sum of the similarity scores of all negative sample pairs, loss(InfoNCE) represents the loss function corresponding to the contrastive learning task, ∑ is used for summing data, and log is used for logarithmic operation.

[0115] During the training process, the training stability of the feature extraction model can be improved by at least one of gradient reduction and a feature penalty term. It can be understood that gradient reduction is to avoid extreme changes in model parameters during updating by limiting the size of the gradient, and the feature penalty term is to add a term related to the complexity of the model in the loss function, so that the model performs well on the training data while maintaining a certain generalization ability.

[0116] In addition, it should be noted that in some example embodiments, the training data can be collected by distributed data crawling, thereby obtaining a multi-element heterogeneous training data set, and then a large-scale training data set can be used to pre-train the initial feature extraction model. For example, the training data comes from different types of audio services, and the data related to these audio services is stored in a distributed manner. By distributed crawling of audio data of these audio services, a training data set composed of different types of audio data can be obtained.

[0117] The embodiments of the present application promote the feature extraction model to capture the detailed information of the spectrogram by the reconstruction learning task, improve the restoration ability of the local features of the audio, and promote the feature extraction model to distinguish the feature difference between similar and non-similar audios by the contrast learning task, and enhance the discrimination ability of the global semantics. The combination of the two tasks makes the feature extraction model maintain stable feature extraction performance in complex audio scenes and improve the generalization ability. Based on the training data set, the reconstruction learning task is performed, the accuracy of feature extraction is directly fed back by quantifying the mean absolute error, the sensitivity of the feature extraction model to key features is quickly strengthened by the noise contrast estimation of the contrast learning task, the parameter adjustment without target is avoided, the training period is shortened, and the model convergence quality is improved.

[0118] In an example application scenario, the audio copy detection method proposed by the embodiments of the present application is applied to a video publishing platform. The video publishing platform can be understood as a UGC (User Generated Content) platform, and users can publish their own videos on the video publishing platform, or browse videos published by other users on the video publishing platform.

[0119] When making a video work to be published, a user usually adds background music, sound effects, commentary, monologue, etc. to assist in presenting the video content, so as to give the video scene and drive the emotional atmosphere of the audience. FIG. 5 is a schematic diagram of audio classification distribution of a video published by a user on a video publishing platform according to an embodiment of the present application. As can be seen, the proportion of users who publish videos using pure music is not large, and most users are used to publishing videos using a combination of multiple audios. In this application scenario, if audio copy detection is to be performed, it is not accurate to only consider the background music contained in the video work. Therefore, the embodiment of the present application also proposes to realize audio copy detection based on the original sound of the video.

[0120] The global audio features corresponding to the audios contained in the videos published on the video publishing platform are stored in the audio retrieval library, and each global audio feature has an audio identifier in the audio retrieval library. This can also be understood as that the same audio corresponds to the same audio identifier, and different audios correspond to different audio identifiers.

[0121] Please refer to FIG. 6, which is a flow chart of audio identifier processing based on a video publishing platform according to an embodiment of the present application. As can be seen, for a video published on the video publishing platform, first, the audio information contained in the video is extracted as a to-be-processed audio, then the global audio feature and the audio segment feature sequence corresponding to the to-be-processed audio are extracted, and the approximate global audio feature corresponding to the global audio feature is retrieved based on the audio retrieval library. If the approximate global audio feature is retrieved, it means that the same audio feature is retrieved, and the audio identifier corresponding to the approximate global audio feature is added as the audio identifier of the to-be-processed audio. If the approximate global audio feature is not retrieved, the audio identifier corresponding to the to-be-processed audio is generated based on the latest audio identifier in the audio identifier list.

[0122] The audio identifier list records the audio identifiers corresponding to the audios contained in the videos published on the video publishing platform. Therefore, the embodiment generates the audio identifier corresponding to the to-be-processed audio based on the latest audio identifier in the audio identifier list, which can ensure that the generated audio identifier is a new audio identifier that does not exist in the audio identifier list. For example, the value of the latest audio identifier in the audio identifier list can be updated according to a preset value update rule, such as increasing the value by a preset value (e.g. 1) each time, and the obtained value is used as the audio identifier of the to-be-processed audio. It should be noted that the embodiment does not limit the specific update method of the audio identifier, for example, the value increased each time can be other values except 1, or a new audio identifier can be generated by performing value reduction each time.

[0123] For example, the numerical value updating rule further comprises: performing a hash operation on the latest audio identifier in the audio identifier list, taking the operation result as the updated numerical value, that is, the audio identifier corresponding to the audio to be processed, or taking the operation result from the head to the numerical value of the preset length as the audio identifier corresponding to the audio to be processed.

[0124] In the embodiment of the application, if the audio similarity is greater than the preset similarity, the audio identifier of the duplicate audio is assigned to the audio to be processed, so that audios with similar contents share the same audio identifier, thereby reducing data redundancy; if the audio similarity is less than the preset similarity, the audio identifier corresponding to the audio to be processed is generated based on the latest audio identifier in the audio identifier list, so that the audio identifier of the audio to be processed is distinguished from the existing identifier, and accurate correspondence between the audio identifier and the audio content is realized.

[0125] Referring to FIG. 7, FIG. 7 is a schematic diagram of a principle architecture for adding an audio identifier to a video published on a video publishing platform according to an embodiment of the application. For a video published on a video publishing platform, global audio features are extracted as video-level features, and audio segment feature sequences are extracted as frame-level features, then similar audios are recalled based on the video-level features by performing a similar recall based on an audio retrieval library, and duplicate audios are checked by performing a duplicate check based on a key-value database based on the frame-level features, so that an audio identifier is added to the video according to the result of the duplicate check. It should be understood that the audio retrieval library shown in FIG. 7 is used to store global audio features, and the key-value database is used to store audio segment feature sequences.

[0126] In addition, as shown in FIG. 6, the audio retrieval library is not only used to store global audio features and audio segment feature sequences, but also can store audio label information, such as genre, language, emotion, and the like, without limitation. These audio labels can distinguish the videos to which they belong to a certain extent, so as to meet certain video processing scenarios. For example, audios of a certain genre can be quickly filtered out by using the audio labels stored in the audio retrieval library.

[0127] In the application scenario of the video publishing platform, as shown in FIG. 8, the audio duplicate detection method can further comprise S810-S820, which are described in detail as follows:

[0128] S810, detecting a video to be recommended in a video publishing platform, and obtaining an audio identifier corresponding to an audio contained in the video to be recommended.

[0129] The technical scheme provided by the embodiment is applied to a video recommendation scene in a video publishing platform. The video to be recommended in the video publishing platform can be detected based on specific recommendation requirements. For example, the specific recommendation requirement is to perform video recommendation on a newly published video of a user, and the newly published video in the video publishing platform is detected as the video to be recommended. For another example, the specific recommendation requirement is to perform video recommendation on a video published by a user with a specific label, and the specific label indicates that the user has performed at least one of a payment operation and a recharge operation in the video publishing platform, and the video published by the user with the specific label in the video publishing platform is taken as the video to be recommended.

[0130] The audio contained in the video to be recommended corresponds to an audio identifier, that is, obtained according to the process shown in FIG. 6, and the process will not be repeated here.

[0131] In S820, a candidate video corresponding to the video to be recommended is determined based on the audio identifier corresponding to the video to be recommended, and a recommendation process of the video to be recommended is performed based on the candidate video; wherein the audio contained in the candidate video is a copy audio corresponding to the audio contained in the video to be recommended.

[0132] Since the same or similar audios have the same audio identifier, in the embodiment, the copy audio with the same audio identifier as the audio contained in the video to be recommended is first obtained, and then the video containing the copy audio is obtained as the candidate video, so that the video to be recommended is recommended based on the obtained candidate video.

[0133] For example, the audience of the candidate video can be obtained, and the video to be recommended can be recommended to the audience. Alternatively, based on the viewing data of the audience for the candidate video, such as viewing time, evaluation information, etc., the audience interested in the candidate video is further screened out, and the video to be recommended is recommended to the screened audience. In this way, accurate recommendation of the video to be recommended can be realized.

[0134] Alternatively, the video to be recommended and the candidate video can be packaged and recommended. For example, the video to be recommended and the candidate video are taken as a video package, and when any audience browses any video in the video package, other videos in the video package can be recommended to the audience according to a preset rule. The preset rule is, for example, to recommend to play other videos in the video package immediately after watching the current video, or to recommend to play other videos in the video package after a preset number of videos after watching the current video, which is not limited here.

[0135] The embodiment is not limited to a specific video recommendation manner, but it should be noted that the video recommendation performed in the embodiment relies on copy audio recognition on audio contained in a video. After extracting audio from the video, audio copy detection can be performed on the extracted audio. Since the audio extracted from the video is original audio, the original audio contains all audio information in the video, such as background music, commentary, monologue, etc. This enables the video recommendation performed in the embodiment to consider all audio information in the video, thereby ensuring good video recommendation effect.

[0136] The embodiment of the application obtains candidate videos containing copy audio of the same audio identifier based on the audio identifier corresponding to the video to be recommended, and then recommends the video to be recommended to audiences of the candidate videos, thereby ensuring the matching degree of the recommended object and the video to be recommended by using the association relationship of the same or similar audio, and improving the recommendation accuracy. The video to be recommended can be directly recommended to the audiences of the candidate videos, recommended after being filtered based on viewing data, packaged with the candidate videos, and recommended in multiple ways, and the preset rules of the packaged recommendation can be flexibly set to adapt to the recommendation requirements in different scenarios and enhance the diversity of the recommendation strategy.

[0137] In the application scenario of the video publishing platform, the audio copy detection scheme provided in the embodiment can also improve the work publishing rate in the video publishing platform. In another exemplary embodiment, as shown in FIG. 9, a video processing method performed by a user terminal includes the following steps:

[0138] S910, playing a video in a video playing page;

[0139] Here, the video playing page includes multiple videos, and in response to a selection operation on any video, the selected video is played in the video page. Alternatively, in response to an opening operation on the video playing page, a video recommended by the video platform is played. The video publishing platform loads the audio and video data of the video, initializes a player component in the video playing page, decodes and plays the video according to the original frame rate, resolution and other parameters of the video, simultaneously renders the video picture in real time and outputs the audio, thereby ensuring that the user can normally watch and listen to the video content.

[0140] S920, if it is detected that a copy audio search entry set in the video playing page is triggered, a candidate video display page is displayed; the candidate video display page displays multiple candidate videos, and the candidate videos contain copy audio corresponding to the audio contained in the video played in the video playing page;

[0141] S930, if it is detected that a video publishing entry set in the candidate video display page is triggered, a video editing page is displayed.

[0142] It should be noted that the video publishing platform has a front end and a back end. The front end refers to the part directly interacted with by the user, mainly responsible for displaying the data provided by the back end to the user in the form of a graphical interface, usually through a web browser, and processing the interaction between the user and the interface. The back end is responsible for processing the requests sent by the front end, executing the corresponding logic (such as accessing the database, processing data, etc.), and returning the results to the front end for display.

[0143] Here, the front-end interface transition diagram of the video publishing platform shown in FIG. 10 is taken as an example to explain the process shown in S910-S930.

[0144] The user terminal first displays a video playing page and plays a video on the video playing page. A duplicate audio search entry is set in the video playing page, such as the "find the same" button in the video playing page shown in FIG. 10, which is also a duplicate audio search entry. When the user terminal detects that the duplicate audio search entry is triggered, the candidate video display page is displayed.

[0145] The candidate video display page displays a plurality of candidate videos, and the audio contained in the candidate videos is the duplicate audio corresponding to the audio contained in the video played in the video playing page. It can also be understood that the audio identifier corresponding to the candidate video is the same as the audio identifier corresponding to the video played in the video playing page. The identification of the audio identifier corresponding to the audio contained in the video is realized through the processes recorded in the foregoing embodiments.

[0146] It can be understood that the candidate videos displayed in the candidate video display page use the same or similar audio, so the candidate videos may have common content attributes, such as being all funny videos or all movie commentary videos, etc. Therefore, by displaying the related candidate videos in the candidate video display page, the user can be guided to create and publish video works using the same audio to a certain extent, and therefore the work publishing rate in the video publishing platform can be improved.

[0147] In addition, it should be noted that, as shown in FIG. 10, the candidate video display page can also display a video cover, such as the plurality of candidate videos displayed on the candidate video display page being video covers. The video cover can be the cover content set by the user when the video was historically published, or it can be a cover content directly determined from the video content of the candidate video. The embodiment does not limit the way the video cover of the candidate video is obtained.

[0148] The candidate video display page is provided with a video publishing portal, such as the "video publishing" button shown in FIG. 10. When the user terminal detects that the video publishing portal is triggered, the video production page is displayed, and the user can perform a video production operation on the video production page. The specific process of video production is not limited in the embodiment. However, it should be noted that the same audio can be used as the background music by default in the video production page to facilitate the user to produce the video.

[0149] Since the audio copy detection technology can be obtained, the embodiment combines the page display in the front-end interface of the video publishing platform with the audio copy detection technology, which is very convenient to guide the user to produce and publish the video based on the same audio during the video watching process. Therefore, the work publishing rate in the video publishing platform can be improved, and the competitiveness of the video publishing platform in the same platform product can be improved.

[0150] The user can quickly find the candidate video containing similar audio through the audio copy search portal, expand the content browsing range, and reduce the threshold of the user to publish the associated content through the video publishing portal, and improve the platform interactivity. Based on the association relationship of the audio copy, the candidate video display page presents the content-related videos in a centralized manner, so that the videos with the same or similar audio form a content cluster, and the user can efficiently obtain the same information.

[0151] FIG. 11 is a block diagram of an audio copy detection device according to an embodiment of the present application. As shown in FIG. 11, the exemplary audio copy detection device includes:

[0152] The feature extraction module 1110 is configured to extract the global audio features corresponding to the to-be-processed audio, and extract the audio segment feature sequence corresponding to the to-be-processed audio;

[0153] The feature retrieval module 1120 is configured to retrieve the approximate global audio features corresponding to the global audio features from the audio retrieval library, and obtain the audio segment feature sequence corresponding to the target audio to which the approximate global audio features belong;

[0154] The similarity determination module 1130 is configured to determine the audio similarity between the to-be-processed audio and the target audio according to the audio segment feature sequence corresponding to the to-be-processed audio and the audio segment feature sequence corresponding to the target audio;

[0155] The copy determination module 1140 is configured to determine the target audio as the copy audio corresponding to the to-be-processed audio if the audio similarity is greater than or equal to the preset similarity.

[0156] In another example embodiment, the feature extraction module 1110 includes an audio segment feature extraction unit configured to: divide the to-be-processed audio based on a preset first unit time length to obtain a plurality of first audio segments contained in the to-be-processed audio; extract audio features from each first audio segment respectively, and sort the audio features of each first audio segment in chronological order to obtain an audio segment feature sequence corresponding to the to-be-processed audio.

[0157] In another example embodiment, the feature extraction module 1110 includes a global feature extraction unit configured to: divide the to-be-processed audio based on a preset second unit time length to obtain a plurality of second audio segments contained in the to-be-processed audio; the second unit time length is greater than or equal to the first unit time length; perform spectrum graph conversion on each second audio segment to obtain a spectrum graph sequence corresponding to the to-be-processed audio; and input the spectrum graph sequence into the trained feature extraction model to obtain global audio features output by the trained feature extraction model.

[0158] In another example embodiment, the trained feature extraction model is obtained by pre-training an initial feature extraction model using a training data set; the audio copy detection apparatus further includes a model training module configured to: generate at least one training task in a reconstruction learning task and a contrast learning task respectively; the reconstruction learning task indicates model parameter optimization by calculating an average absolute error loss value, and the contrast learning task indicates model parameter optimization by calculating a noise contrast estimation loss value; and pre-train the initial feature extraction model based on the training data set to obtain the trained feature extraction model.

[0159] In another example embodiment, the trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network; the global feature extraction unit is further configured to: extract shallow features corresponding to each spectrum graph in the spectrum graph sequence through the shallow feature extraction network; extract deep features corresponding to each shallow feature through the deep feature extraction network; and calculate a feature average value based on the deep features, and take the feature average value as the global audio features corresponding to the to-be-processed audio.

[0160] In another example embodiment, the audio segment feature extraction unit is further configured to: input each first audio segment in chronological order into the trained feature extraction model; and take a shallow feature sequence output by the shallow feature extraction network as the audio segment feature sequence corresponding to the to-be-processed audio.

[0161] In another example embodiment, the similarity determination module 1130 is further configured to: perform audio segment feature comparison on the audio segment feature sequence corresponding to the to-be-processed audio and the audio segment feature sequence corresponding to the target audio, to obtain an audio similar duration between the to-be-processed audio and the target audio; and determine an audio similarity between the to-be-processed audio and the target audio according to the audio duration corresponding to the to-be-processed audio and the target audio respectively and the audio similar duration.

[0162] In another example embodiment, each global audio feature stored in the audio retrieval library has an audio identifier; and the audio copy detection apparatus further comprises an audio storage module configured to: take the audio identifier corresponding to the copy audio as the audio identifier corresponding to the to-be-processed audio; and store the global audio feature and the audio segment feature sequence corresponding to the to-be-processed audio in the audio retrieval library based on the audio identifier corresponding to the to-be-processed audio.

[0163] In another example embodiment, the global audio feature and the audio segment feature sequence stored in the audio retrieval library have a specified storage time limit; and the audio storage module is further configured to: update the remaining storage time of the global audio feature and the audio segment feature sequence corresponding to the copy audio to the specified storage time limit.

[0164] In another example embodiment, the audio storage module is further configured to: if the audio similarity is lower than a preset similarity, generate the audio identifier corresponding to the to-be-processed audio based on the latest audio identifier in the audio identifier list.

[0165] In another example embodiment, the audio copy detection apparatus further comprises:

[0166] a video detection module configured to detect a to-be-recommended video in a video publishing platform and obtain an audio identifier corresponding to an audio contained in the to-be-recommended video;

[0167] a video recommendation module configured to determine a candidate video corresponding to the to-be-recommended video based on the audio identifier corresponding to the to-be-recommended video, and perform recommendation processing of the to-be-recommended video based on the candidate video; wherein the audio contained in the candidate video is a copy audio corresponding to the audio contained in the to-be-recommended video.

[0168] It should be noted that the apparatus provided in the above example embodiments and the method provided in the above example embodiments belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiments, which will not be described here. In actual application, the audio copy detection apparatus provided in the above example embodiments can allocate the above functions to different functional modules, i.e., divide the internal structure of the apparatus into different functional modules to complete all or part of the functions described above, and this is not limited herein.

[0169] Embodiments of the present application also provide an electronic device, comprising: one or more processors; a memory for storing one or more computer programs, which, when executed by the one or more processors, cause the electronic device to implement the audio copy detection method provided in each of the above embodiments.

[0170] FIG. 12 is a structural schematic diagram of a computer system of an electronic device suitable for implementing the embodiments of the present application. It should be noted that the electronic device can be the terminal 110 or the server 120 in the implementation environment shown in FIG. 1, or a terminal or a server in other implementation environments, which is not limited herein. It should also be noted that the computer system 1200 of the electronic device shown in FIG. 12 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0171] As shown in FIG. 12, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage portion 1208 into a random access memory (RAM) 1203, such as performing the methods described in the above embodiments. In the RAM 1203, various computer programs and data required for system operation are also stored. The CPU 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input or output (I / O) interface 1205 is also connected to the bus 1204.

[0172] The following components are connected to the I / O interface 1205: an input portion 1206 including a keyboard, a mouse, and the like; an output portion 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), and the like, and a speaker, and the like; a storage portion 1208 including a hard disk, and the like; and a communication portion 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication portion 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like is mounted on the drive 1210 as needed, so that a computer program read therefrom is installed in the storage portion 1208 as needed.

[0173] In particular, according to an embodiment of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product comprising a computer program tangibly embodied on a computer readable medium, the computer program containing instructions for executing the methods illustrated by the flowcharts. In such an embodiment, the computer program can be installed by being downloaded from a network through the communication part 1209 and installed from the detachable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.

[0174] It is noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination of the above. The computer program contained on the computer readable medium can be transmitted by any appropriate medium, including but not limited to wireless, wired, or the like, or any appropriate combination of the above.

[0175] The flowcharts and block diagrams in the drawings illustrate the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In the flowcharts or block diagrams, each block can represent a module, a program segment, or a part of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in different order than that shown in the drawings. For example, two blocks that are shown in succession can actually be executed substantially in parallel, and sometimes in reverse order, depending on the involved functions. It should also be noted that each block in the block diagrams or flowcharts, and the combination of blocks in the block diagrams or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0176] The units described in the embodiments of the present application can be implemented in the form of software, or can be implemented in the form of hardware, and the described units can also be arranged in a processor. In some cases, the names of the units do not constitute a limitation on the units themselves.

[0177] Another aspect of the present application also provides a computer readable storage medium having a computer program stored thereon, the computer program being executed by a processor of an electronic device to implement the audio copy detection method as described above. The computer readable storage medium can be included in the electronic device as described in the above embodiments, or can exist separately and not be assembled into the electronic device.

[0178] Another aspect of the present application also provides a computer program product including a computer program stored in a computer readable storage medium. The processor of an electronic device reads the computer program from the computer readable storage medium, and the processor executes the computer program to cause the electronic device to perform the audio copy detection method provided in each of the above embodiments.

[0179] The above is only a preferred exemplary embodiment of the present application, and is not intended to limit the implementation of the present application. Those skilled in the art can easily make corresponding modifications or variations according to the main idea and spirit of the present application, and the protection scope of the present application should be subject to the protection scope required by the claims.

[0180] It can be understood that in the specific embodiments of the present application, data related to video, audio, etc. is involved, and when the above embodiments of the present application are applied to specific products or technologies, the permission or consent of the user needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

Claims

1. An audio copy detection method, the method being performed by an electronic device, the method comprising: extracting a global audio feature corresponding to a to-be-processed audio, and extracting a sequence of audio segment features corresponding to the to-be-processed audio; retrieving, from an audio retrieval library, an approximate global audio feature corresponding to the global audio feature; obtaining a sequence of audio segment features corresponding to a target audio to which the approximate global audio feature belongs, and determining an audio similarity between the to-be-processed audio and the target audio according to the sequences of audio segment features respectively corresponding to the to-be-processed audio and the target audio; if the audio similarity is greater than or equal to a preset similarity, determining the target audio as a copy audio corresponding to the to-be-processed audio.

2. The method of claim 1, wherein, The extracting of the sequence of audio segment features corresponding to the to-be-processed audio comprises: dividing the to-be-processed audio based on a preset first unit time length to obtain a plurality of first audio segments contained in the to-be-processed audio; extracting audio features of each first audio segment respectively, and sorting the audio features of each first audio segment in chronological order to obtain the sequence of audio segment features corresponding to the to-be-processed audio.

3. The method of claim 2, wherein, The extracting of the global audio feature corresponding to the to-be-processed audio comprises: dividing the to-be-processed audio based on a preset second unit time length to obtain a plurality of second audio segments contained in the to-be-processed audio; the second unit time length is greater than or equal to the first unit time length; performing spectrum graph conversion on each second audio segment to obtain a spectrum graph sequence corresponding to the to-be-processed audio; inputting the spectrum graph sequence into a trained feature extraction model to obtain a global audio feature output by the trained feature extraction model.

4. The method of claim 3, wherein, The trained feature extraction model is obtained by pre-training an initial feature extraction model using a training data set; the method further comprises: generating at least one training task in a reconstruction learning task and a contrast learning task respectively; the reconstruction learning task indicates that model parameter optimization is performed by calculating an average absolute error loss value, and the contrast learning task indicates that model parameter optimization is performed by calculating a noise contrast estimation loss value; pre-training the initial feature extraction model based on the training data set by executing the at least one training task to obtain the trained feature extraction model.

5. The method of claim 3, wherein, The trained feature extraction model comprises a cascaded shallow feature extraction network and a deep feature extraction network; the global audio feature corresponding to the to-be-processed audio is obtained by executing the following steps: extracting a shallow feature corresponding to each spectrum graph in the spectrum graph sequence through the shallow feature extraction network; extracting a deep feature corresponding to each shallow feature through the deep feature extraction network; calculating a feature average value according to the deep features, and taking the feature average value as the global audio feature corresponding to the to-be-processed audio.

6. The method of claim 5, wherein, The extracting of the sequence of audio segment features corresponding to the to-be-processed audio comprises: Each first audio segment is input into the trained feature extraction model in chronological order. The shallow feature sequence output by the shallow feature extraction network is used as the audio segment feature sequence corresponding to the audio to be processed.

7. The method of claim 3, wherein, The step of performing spectrogram transformation on each second audio segment to obtain the spectrogram sequence corresponding to the audio to be processed includes: For each of the second audio segments, perform the following processing: The time-domain signal of the second audio segment is divided into multiple frames according to a preset time window; For each frame, a Fourier transform is performed on the frame to obtain the spectrum of the frame; The spectra corresponding to the multiple frames are combined in chronological order to form the spectrogram corresponding to the second audio segment. The spectrograms corresponding to the multiple second audio segments are combined into a spectrogram sequence corresponding to the audio to be processed.

8. The method of any one of claims 1-7, wherein, The step of determining the audio similarity between the audio to be processed and the target audio based on the audio segment feature sequences corresponding to the audio to be processed and the target audio respectively includes: The audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio are compared to obtain the audio similarity duration between the audio to be processed and the target audio. The audio similarity between the audio to be processed and the target audio is determined based on the audio durations corresponding to the audio to be processed and the target audio, as well as the audio similarity duration.

9. The method of any one of claims 1-8, wherein, Each global audio feature stored in the audio retrieval library has an audio identifier; the method further includes: Use the audio identifier corresponding to the copy audio as the audio identifier corresponding to the audio to be processed; Based on the audio identifier corresponding to the audio to be processed, the global audio features and audio segment feature sequences corresponding to the audio to be processed are stored in the audio retrieval library.

10. The method of claim 9, wherein, The global audio features and audio segment feature sequences stored in the audio retrieval library have a specified storage time limit; the method further includes: Update the remaining storage duration of the global audio features and audio segment feature sequences corresponding to the copied audio to the specified duration.

11. The method of claim 9, wherein, The method further includes: If the audio similarity is lower than the preset similarity, then an audio identifier corresponding to the audio to be processed is generated based on the latest audio identifier in the audio identifier list.

12. The method of claim 11, wherein, The step of generating the audio identifier corresponding to the audio to be processed based on the latest audio identifier in the audio identifier list includes: The latest audio identifier in the audio identifier list is updated according to a preset numerical update rule to obtain an updated value, and the updated value is used as the audio identifier corresponding to the audio to be processed. The numerical update rule includes any one of the following: Increase the latest audio identifier in the audio identifier list by a first preset value; Decrease the latest audio identifier in the audio identifier list by a second preset value; Perform a hash operation on the latest audio identifier in the audio identifier list.

13. The method of claim 9, wherein, The method further includes: Detect videos to be recommended in the video publishing platform and obtain the audio identifier corresponding to the audio contained in the videos to be recommended; The candidate video corresponding to the to-be-recommended video is determined based on the audio identifier corresponding to the to-be-recommended video, and the recommendation processing of the to-be-recommended video is performed based on the candidate video; wherein the candidate video contains a copy audio corresponding to the audio contained in the to-be-recommended video.

14. The method of claim 13, wherein, The to-be-recommended video in the video publishing platform is detected, including: If the video recommendation is performed for the newly published video of the user, the newly published video in the video publishing platform is taken as the to-be-recommended video; If the video recommendation is performed for the video published by the user with the specific label, the video published by the user with the specific label in the video publishing platform is taken as the to-be-recommended video, wherein the specific label represents that the user has at least one of a payment operation and a recharge operation in the video publishing platform.

15. An audio copy detection device, the device comprising: a feature extraction module configured to extract a global audio feature corresponding to a to-be-processed audio, and extract an audio segment feature sequence corresponding to the to-be-processed audio; a feature retrieval module configured to retrieve an approximate global audio feature corresponding to the global audio feature from an audio retrieval library, and obtain an audio segment feature sequence corresponding to a target audio to which the approximate global audio feature belongs; a similarity determination module configured to determine an audio similarity between the to-be-processed audio and the target audio according to the audio segment feature sequence corresponding to the to-be-processed audio and the audio segment feature sequence corresponding to the target audio; a copy determination module configured to determine the target audio as a copy audio corresponding to the to-be-processed audio if the audio similarity is above a preset similarity.

16. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, when the one or more computer programs are executed by the one or more processors, the electronic device implements the method of any one of claims 1-14.

17. A computer readable storage medium having stored thereon a computer program, when the computer program is executed by a processor of an electronic device, the electronic device executes the method of any one of claims 1-14.

18. A computer program product comprising a computer program, when the computer program is executed by a processor of an electronic device, the computer program implements the method of any one of claims 1-14.

Images