Audio copy detection method and device, equipment, storage medium, program product
By extracting the global and segment feature sequences of audio and combining with deep learning models for audio similarity search, the problem of insufficient accuracy of audio copy detection in the prior art is solved, and more efficient audio copy recognition and detection effects are achieved.
Patent Information
- Application Number
- CN202411142927.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-20
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-08-20
AI Technical Summary
The existing audio copy detection technology has shortcomings in terms of accuracy, and it is difficult to effectively identify whether the audio content is similar or the same as known audio, which affects the effect of application scenarios such as audio copyright protection and music recommendation.
By extracting the global audio features and audio band feature sequences of the to-process audio, using the audio search library for similarity search and comparison, and combining deep learning models (such as cascading shallow and deep feature extraction networks) to optimize feature extraction to achieve audio similarity Accurate calculation and copy determination of degrees.
It improves the accuracy and robustness of audio copy detection, ensures that the results of audio copy detection have high accuracy and reliability, and is suitable for scenes such as audio copyright protection and music recommendation.
Smart Images

Figure CN118747230B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to an audio copy detection method and device, electronic equipment, computer storage medium, and computer program product. Background Art
[0002] Audio copy detection technology is a technology used to identify whether audio content is similar or identical to known audio. If it is identified as such, the known audio can be used as a copy of the audio to be identified. Audio copy detection technology is usually used in application scenarios such as audio copyright protection and music recommendation. How to improve the accuracy of audio copy detection is still a technical problem that technicians in this field need to study and solve. Summary of the invention
[0003] To solve the above technical problems, the embodiments of the present application provide an audio duplicate detection method, an audio duplicate detection device, an electronic device, a computer-readable storage medium, and a computer program product. The embodiments of the present application can improve the accuracy of audio duplicate detection.
[0004] One aspect of an embodiment of the present application provides an audio copy detection method, the method comprising: extracting global audio features and audio segment feature sequences corresponding to audio to be processed; retrieving approximate global audio features corresponding to the global audio features from an audio retrieval library; obtaining an audio segment feature sequence corresponding to a target audio to which the approximate global audio features belong, and determining an audio similarity between the audio to be processed and the target audio based on the audio segment feature sequences corresponding to the audio to be processed and the target audio respectively; if the audio similarity reaches or exceeds a preset similarity, determining the target audio as a copy audio corresponding to the audio to be processed.
[0005] In another exemplary embodiment, global audio features and audio segment feature sequences corresponding to the audio to be processed are extracted, including: dividing the audio to be processed based on a preset first unit time length to obtain a plurality of first audio segments contained in the audio to be processed; extracting audio features from each first audio segment respectively, and sorting the audio features of each first audio segment in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed.
[0006] In another exemplary embodiment, extracting global audio features and audio segment feature sequences corresponding to the audio to be processed also includes: dividing the audio to be processed based on a preset second unit time length to obtain a plurality of second audio segments contained in the audio to be processed; the second unit time length is greater than or equal to the first unit time length; performing spectrogram conversion on each second audio segment to obtain a spectrogram sequence corresponding to the audio to be processed; and inputting the spectrogram sequence into a trained feature extraction model to obtain the global audio features output by the trained feature extraction model.
[0007] In another exemplary embodiment, the trained feature extraction model is obtained by pre-training the initial feature extraction model using a training data set; the method also includes: generating at least one training task of a reconstruction learning task and a contrast learning task respectively; the reconstruction learning task indicates that the model parameters are optimized by calculating the mean absolute error loss value, and the contrast learning task indicates that the model parameters are optimized by calculating the noise contrast estimation loss value; based on the training data set, the at least one training task is respectively executed to pre-train the initial feature extraction model to obtain the trained feature extraction model.
[0008] In another exemplary embodiment, the trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network; the global audio features corresponding to the audio to be processed are obtained by executing the following steps: extracting shallow features corresponding to each spectrogram in the spectrogram sequence through the shallow feature extraction network; extracting deep features corresponding to each shallow feature through the deep feature extraction network; calculating the feature average value based on the deep feature, and taking the feature average value as the global audio feature corresponding to the audio to be processed.
[0009] In another exemplary embodiment, audio features are extracted from each first audio segment respectively, and the audio features of each first audio segment are sorted in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed, including: inputting each first audio segment into the trained feature extraction model in chronological order; and using the shallow feature sequence output by the shallow feature extraction network as the audio segment feature sequence corresponding to the audio to be processed.
[0010] In another exemplary embodiment, determining the audio similarity between the audio to be processed and the target audio based on the audio segment feature sequences corresponding to the audio to be processed and the target audio respectively includes: performing audio segment feature comparison on the audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio to obtain the audio similarity duration between the audio to be processed and the target audio; determining the audio similarity between the audio to be processed and the target audio based on the audio durations corresponding to the audio to be processed and the target audio respectively, and the audio similarity duration.
[0011] In another exemplary embodiment, each global audio feature stored in the audio retrieval library has an audio identifier respectively; the method also includes: using the audio identifier corresponding to the copy audio as the audio identifier corresponding to the audio to be processed; based on the audio identifier corresponding to the audio to be processed, storing the global audio features and audio segment feature sequences corresponding to the audio to be processed in the audio retrieval library.
[0012] In another exemplary embodiment, the global audio features and audio segment feature sequences stored in the audio retrieval library have a storage time limit of a specified duration; the method further includes: updating the remaining storage time of the global audio features and audio segment feature sequences corresponding to the copy audio to the specified duration.
[0013] In another exemplary embodiment, the method further includes: if the audio similarity is lower than the preset similarity, generating an audio identifier corresponding to the audio to be processed based on the latest audio identifier in the audio identifier list.
[0014] In another exemplary embodiment, the method further includes: detecting the video to be recommended in the video publishing platform, and obtaining the audio identifier corresponding to the audio contained in the video to be recommended; based on the audio identifier corresponding to the video to be recommended, determining the candidate video corresponding to the video to be recommended, and performing recommendation processing of the video to be recommended based on the candidate video; wherein the audio contained in the candidate video is the copy audio corresponding to the audio contained in the video to be recommended.
[0015] In another aspect of an embodiment of the present application, an audio duplicate detection device is provided, which includes: a feature extraction module, configured to extract global audio features and audio segment feature sequences corresponding to the audio to be processed; a feature retrieval module, configured to retrieve approximate global audio features corresponding to the global audio features from an audio retrieval library, and obtain an audio segment feature sequence corresponding to the target audio to which the approximate global audio features belong; a similarity determination module, configured to determine the audio similarity between the audio to be processed and the target audio based on the audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio; and a duplicate determination module, configured to determine the target audio as the duplicate audio corresponding to the audio to be processed if the audio similarity reaches or exceeds a preset similarity.
[0016] Another aspect of an embodiment of the present application provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein when the one or more computer programs are executed by the one or more processors, the electronic device implements the audio copy detection method as described above.
[0017] Another aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor of an electronic device, the electronic device executes the audio copy detection method as described above.
[0018] Another aspect of an embodiment of the present application provides a computer program product, including a computer program, which implements the audio copy detection method as described above when executed by a processor of an electronic device.
[0019] The technical solution provided in the embodiments of the present application first retrieves approximate global audio features from an audio retrieval library based on the global audio features of the audio to be processed, and then determines the audio similarity between the audio to be processed and the target audio based on the audio segment feature sequences corresponding to the audio to be processed and the target audio. It can be understood that a coarse-grained detection is first achieved through global audio features, and then a finer-grained detection is achieved through audio segment feature sequences, thereby ensuring the accuracy of audio copy detection.
[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] Figure 1 It is a schematic diagram of the implementation environment involved in this application.
[0022] Figure 2 is a flow chart of an audio copy detection method shown in an exemplary embodiment of the present application.
[0023] Figure 3 is Figure 2 A flowchart of an audio copy detection method further proposed on the basis of the illustrated embodiment.
[0024] Figure 4 It is a schematic diagram of a flow chart of extracting global audio features corresponding to audio to be processed, shown in an exemplary embodiment of the present application.
[0025] Figure 5 The diagram illustrates a schematic diagram of the audio classification distribution contained in videos posted by users on an exemplary video publishing platform.
[0026] Figure 6 This is an exemplary audio identification processing flow chart based on a video publishing platform.
[0027] Figure 7 A schematic diagram of the principle architecture of adding audio tags to videos published on a video publishing platform is shown.
[0028] Figure 8 is another exemplary audio identification processing flow chart based on a video publishing platform.
[0029] Fig. 9 It is a flowchart of a video processing method shown in an exemplary embodiment of the present application.
[0030] Fig.10 This is a schematic diagram of the front-end interface jump of an exemplary video publishing platform.
[0031] Fig.11 It is a block diagram of an audio copy detection device shown in an exemplary embodiment of the present application.
[0032] Fig.12 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION
[0033] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.
[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0035] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.
[0036] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0037] The terms "first", "second", "third" and "fourth" etc. in the specification and claims of the present application and the drawings are used to distinguish different objects, rather than to describe a specific order. The terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.
[0038] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.
[0039] First, as mentioned above, audio duplication detection technology is a technology used to identify whether audio content is similar or identical to known audio. In existing technical implementations, audio duplication detection is mainly achieved through the following methods:
[0040] 1. Detection method based on fingerprint features: It is necessary to extract audio fingerprint features to represent the fingerprint content. By comparing the similarity between audio fingerprints, it can be determined whether they are the same or similar audio;
[0041] 2. Detection method based on time domain features: It is necessary to analyze the time domain features of the audio (such as energy, zero-crossing rate, etc.) and determine whether the audio is the same or similar by calculating the similarity of the time domain features between the audios;
[0042] 3. Detection method based on frequency domain features: It is necessary to analyze the frequency domain features of the audio (such as spectrum, harmonics, etc.) and determine whether the audio is the same or similar by calculating the similarity of the frequency domain features between the audios;
[0043] 4. Detection method based on time-frequency domain features: It is necessary to analyze the time-frequency domain features of the audio (such as Mel frequency cepstral coefficients, tone frequency, etc.), and determine whether the audio is the same or similar by calculating the similarity of the time-frequency domain features between the audios;
[0044] 5. Detection method based on deep learning: Use deep learning models (such as convolutional neural networks, recurrent neural networks, long short-term memory networks, etc.) to encode audio features, and compare the encoded audio features to determine whether they are the same or similar audio.
[0045] In the deep learning-based detection method, the deep learning model can automatically learn to obtain high-level feature representations of the audio, thereby improving the accuracy and robustness of duplicate detection. Therefore, the audio duplicate detection scheme proposed in the embodiment of the present application also adopts a deep learning-based detection method to determine whether the audio content is the same or similar.
[0046] The following is a detailed introduction to the audio copy detection solution proposed in the embodiments of the present application.
[0047] See first Figure 1 , Figure 1 1 is a schematic diagram of an implementation environment involved in the present application. The implementation environment includes a terminal 110 and a server 120, and the terminal 110 and the server 120 communicate with each other in a wired or wireless manner.
[0048] The terminal 110 is a device for information interaction with the user, and is used to receive information input by the user, such as audio, and return information to the user. For example, the terminal 110 is configured with a graphical user interface (GUI), and the user can interact with the terminal 110 through the graphical user interface, such as the user triggers the graphical user interface to input information, and can also obtain the required information from the graphical user interface. For another example, the terminal 110 can be configured with a keyboard, a mouse, and other devices to assist in information interaction with the user.
[0049] The server 120 is used to provide data services for information interaction between the terminal 110 and the user. For example, the terminal 110 uploads the audio input by the user to the server 120, so that the server 120 performs detection of duplicate audio that is identical or similar to the audio input by the user, and returns the detection result to the terminal 110. After receiving the information transmitted by the server 120, the terminal 110 returns the information to the user.
[0050] Of course, when the terminal 110 does not have a data service demand, the terminal 110 may not interact with the server 120. For example, after receiving the video input by the user, the terminal 110 may directly detect duplicate audio that is identical or similar to the audio input by the user, and return the detection result to the user.
[0051] It should be noted that the terminal 110 can be a smart phone, tablet, laptop, computer, smart home appliance, smart terminal or other device, and the server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services. The embodiments of the present application do not limit this.
[0052] See also Figure 2 , Figure 2 is a flowchart of an audio copy detection method shown in an exemplary embodiment of the present application. The method can be applied to Figure 1 The implementation environment shown, for example, can be specifically executed by the server 120, can also be specifically executed by the terminal 110, or can be jointly executed by the terminal 110 and the server 120. Of course, the method can also be applied to other implementation environments, and can be executed by a terminal or a server in other implementation environments, or can be jointly executed by a terminal and a server in other implementation environments, and this embodiment is not limited thereto.
[0053] like Figure 2 As shown, in an exemplary embodiment, the audio copy detection method includes S210-S240, which are described in detail as follows:
[0054] S210: extracting global audio features and audio segment feature sequences corresponding to the audio to be processed.
[0055] First of all, it should be noted that the global audio features disclosed in this embodiment are used to represent the overall properties of the audio. The global audio features usually require a multi-layer neural network to extract them, so the global audio features can also be understood as deep features.
[0056] The audio segment feature sequence disclosed in this embodiment is formed by arranging multiple audio segment features in chronological order. Exemplarily, the audio is divided into multiple audio segments, and then the audio features of each audio segment are extracted respectively, and the audio features are sorted in chronological order to obtain the corresponding audio segment feature sequence.
[0057] This embodiment can recall target audio that is similar to the audio to be processed based on the global audio features corresponding to the audio to be processed; and can further verify whether the recalled target audio is similar to the audio to be processed based on the audio segment feature sequence corresponding to the audio to be processed, thereby ensuring that the obtained detection result has a high accuracy.
[0058] S220, retrieving an approximate global audio feature corresponding to the global audio feature from an audio retrieval library.
[0059] The audio retrieval library stores a large number of global audio features of known audio. The global audio features of the known audio and the global audio features of the audio to be processed are extracted by the same feature extraction method.
[0060] Exemplarily, by calculating the similarity between the global audio features of the audio to be processed and the global audio features of the known audio, the global audio features of the known audio whose similarity is greater than or equal to a preset threshold can be determined as approximate global audio features, and the known audio whose similarity is greater than or equal to the preset threshold is called the target audio. Alternatively, by converting the global audio features of the audio to be processed and the known audio respectively into hash values, the hash values can be compared to quickly determine whether the two global audio features are the same or similar.
[0061] It should be noted that, in the audio retrieval library, the number of approximate global audio features may be one, multiple, or non-existent, and this embodiment does not limit the number of retrieved approximate global audio features. In the absence of approximate global audio features, it indicates that the audio to be processed is new audio relative to the audio retrieval library.
[0062] S230, obtaining an audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs, and determining the audio similarity between the audio to be processed and the target audio according to the audio segment feature sequences corresponding to the audio to be processed and the target audio respectively.
[0063] In this embodiment, the known audio also corresponds to global audio features and audio segment feature sequences. The global audio features and audio segment feature sequences corresponding to the known audio are extracted in the same manner as the global audio features and audio segment feature sequences corresponding to the audio to be processed.
[0064] In some exemplary embodiments, the global audio features and audio segment feature sequences of known audio can be stored in an audio retrieval library and can be stored in an associated manner. Therefore, after retrieving an approximate global audio feature from the audio retrieval library, the audio segment feature sequence associated with the approximate global audio feature can be searched in the audio retrieval library to obtain the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs.
[0065] Exemplarily, in the audio retrieval library, the global audio features and the audio segment feature sequences of the known audio can be stored as key-value pairs, wherein the global audio features serve as the keys and each audio segment feature constituting the audio segment feature sequence serves as a different value.
[0066] Alternatively, the global audio features and audio segment feature sequences of the known audio may be stored separately in the audio retrieval library. For example, each global audio feature and each audio segment feature sequence stored in the audio retrieval library has an audio identifier, and the audio corresponding to the same audio identifier should also be the same or similar.
[0067] Therefore, in some exemplary embodiments, after detecting the copy audio corresponding to the video to be processed, the audio identifier corresponding to the copy audio is also used as the audio identifier corresponding to the audio to be processed, and based on the audio identifier corresponding to the audio to be processed, the global audio features and audio segment feature sequences corresponding to the audio to be processed are stored in the audio retrieval library.
[0068] In some other exemplary embodiments, the audio segment feature sequence of the known audio is stored in other databases different from the audio retrieval library. For example, each known audio has its own audio identifier, and the audio identifier of the known audio is associated with the global audio feature of the known audio and stored in the audio retrieval library, and the audio identifier of the known audio is associated with the audio segment feature sequence of the known audio and stored in other databases. Thus, based on the audio identifier associated with the approximate global audio feature, the audio segment feature sequence associated with the audio identifier can be obtained from other databases, thereby obtaining the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs.
[0069] Exemplarily, in other databases, the audio identifier of the known audio and the audio segment feature sequence can be stored in key-value pairs, wherein the audio identifier is used as a key and each audio segment feature constituting the audio segment feature sequence is used as a different value. Thus, the other database can be a key-value database.
[0070] It should also be noted that the storage of global audio features and audio segment feature sequences in the database may have a storage time limit of a specified duration, that is, the database will only retain the global audio features and audio segment feature sequences for the specified duration. Therefore, in some exemplary embodiments, when storing global audio features and audio segment feature sequences, the currently stored global audio features and audio segment feature sequences automatically obtain a storage time limit of the specified duration, and the remaining storage duration of the global audio features and audio segment feature sequences corresponding to the copy audio may also be updated to the specified duration, thereby ensuring that the database is updated and iterated as the audio is added.
[0071] This embodiment is based on the audio segment feature sequences corresponding to the audio to be processed and the target audio respectively. By comparing the audio segment feature sequences, it can be understood that by comparing the audio segment features in the audio segment feature sequences, the same audio segments in the audio to be processed and the target audio are determined, thereby determining the audio similarity between the audio to be processed and the target audio.
[0072] Exemplarily, the audio segment features of the audio segment feature sequence corresponding to the processed audio and the audio segment feature sequence corresponding to the target audio can be compared to obtain the audio similarity duration between the processed audio and the target audio, and the audio similarity between the processed audio and the target audio can be determined based on the audio durations corresponding to the processed audio and the target audio, respectively, and the audio similarity duration.
[0073] For example, the shorter duration can be determined from the duration of the audio to be processed and the duration of the target audio, and then the ratio between the audio similarity duration and the shorter duration can be calculated to obtain the audio similarity between the audio to be processed and the target audio. For another example, the audio similarity between the audio to be processed and the target audio can be calculated based on the longer duration of the audio to be processed and the duration of the target audio, which is not limited in this embodiment.
[0074] For ease of understanding, for example, assuming that the length of the video to be processed is 110 seconds and the length of the target video is 120 seconds, by comparing the corresponding audio segment feature sequences of the two, it is determined that the 100 seconds of audio from the 10th to the 110th second of the video to be processed and the 20th to the 120th second of the target video are similar. Then, the audio similarity duration is 100 seconds, and the audio similarity is: 100 seconds / min (110 seconds, 120 seconds) = 91%.
[0075] S240: If the audio similarity reaches or exceeds a preset similarity, the target audio is determined as a duplicate audio corresponding to the audio to be processed.
[0076] The preset similarity can be set according to actual application requirements. For example, in some application scenarios that require high accuracy of duplicate detection, the preset similarity can be set to a larger value, such as 90%, 85%, 80%, etc. In some application scenarios that require low accuracy of duplicate detection, the preset similarity can be set to a slightly lower value, such as 70%, 75%, etc., which is not limited here.
[0077] For example, assuming that two approximate global audio features are retrieved in S220, that is, two target audios are determined accordingly, and then further comparison is performed based on the audio segment feature sequence. If the audio similarity between the audio segment feature sequence of one of the target audios and the audio segment feature sequence of the audio to be processed is high, for example, higher than the preset similarity, and the audio similarity between the audio segment feature sequence of the other target audio and the audio segment feature sequence of the audio to be processed is low, for example, lower than the preset similarity, then the target audio with an audio similarity higher than the preset similarity is determined to be the duplicate audio corresponding to the audio to be processed. It can be seen that this embodiment first implements coarse-grained detection through global audio features, and then implements finer-grained detection through audio segment feature sequences, thereby ensuring the accuracy of audio duplicate detection.
[0078] Please continue reading Figure 3 , Figure 3 is Figure 2 A flowchart of an audio copy detection method further proposed on the basis of the illustrated embodiment.
[0079] like Figure 3 As shown, the process of extracting the global audio features and audio segment feature sequences corresponding to the audio to be processed in S210 includes the following contents S310-S320:
[0080] S310, dividing the audio to be processed based on a preset first unit time length to obtain a plurality of first audio segments included in the audio to be processed;
[0081] S320: extract audio features from each first audio segment respectively, and sort the audio features of each first audio segment in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed.
[0082] It can be understood that in S310, the first unit time length is usually a smaller value, such as 1 second, and the audio to be processed is divided and processed based on the first unit time length to obtain a plurality of first audio segments of the first unit time length.
[0083] In S320, by extracting audio features from each first audio segment, and then sorting the audio features of each first audio segment in chronological order, an audio segment feature sequence corresponding to the audio to be processed is obtained. For example, assuming that the total duration of the audio to be processed is 100 seconds and the first unit duration is 1 second, the audio segment feature sequence corresponding to the audio to be processed is obtained by arranging the audio features of 100 first audio segments in chronological order. It can be seen that the audio segment feature sequence can represent more fine-grained audio content.
[0084] Continue to see Figure 3 The process of extracting the global audio features and audio segment feature sequences corresponding to the audio to be processed in S210 also includes the following contents shown in S330-S350:
[0085] S330, dividing the audio to be processed based on a preset second unit time length to obtain a plurality of second audio segments included in the audio to be processed;
[0086] S340, performing spectrogram conversion on each second audio segment to obtain a spectrogram sequence corresponding to the audio to be processed;
[0087] S350, inputting the spectrogram sequence into the trained feature extraction model to obtain the global audio features output by the trained feature extraction model.
[0088] It can be understood that since the total duration of the audio to be processed is usually large, if global features are extracted directly for the audio to be processed, it is easy to cause the deep learning network to be overloaded and crash. Therefore, in S330, the audio to be processed is divided and processed based on the second unit duration, and the audio to be processed is divided into multiple second audio segments. Subsequently, global audio features are extracted based on the second audio segments, thereby reducing the load of the deep learning network to a certain extent.
[0089] Therefore, based on different starting points, the second unit duration is usually greater than or equal to the first unit duration. Usually, the second unit duration is greater than the first unit duration, for example, the second unit duration may be 10 seconds. In the case where the duration of the audio segment is less than the second unit duration, a second audio segment that meets the second unit duration may be obtained by completing the duration of the audio segment.
[0090] In S340, the waveform of the audio time series is converted into a Mel-spectrogram by performing spectrogram conversion on each second audio segment, thereby obtaining a spectrogram sequence corresponding to the audio to be processed. In S350, the spectrogram sequence corresponding to the audio to be processed is input into a trained feature extraction model, and the global audio features output by the trained feature extraction model can be obtained.
[0091] See also Figure 4 , Figure 4 This is a flowchart of extracting global audio features corresponding to the audio to be processed, shown in an exemplary embodiment of the present application. It can be seen that the trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network, and the spectrum graph sequence corresponding to the audio to be processed is input into the trained feature extraction network. The shallow features corresponding to each frequency graph in the spectrum graph sequence are first extracted by the shallow feature extraction network, and the shallow feature sequence corresponding to the audio to be processed is obtained accordingly. Then, the deep features corresponding to each shallow feature are extracted by the deep feature extraction network, and finally, the feature average is calculated based on each deep feature, so that the calculated feature average is used as the global audio feature corresponding to the audio to be processed.
[0092] For ease of understanding, for example, assuming that the second unit duration is 10 seconds, each second audio segment extracts a 768-dimensional feature vector through the trained feature extraction model. Therefore, the audio features corresponding to each second audio segment can be expressed as (batch, 768), where batch is the quotient between the duration of the audio to be processed and the second unit duration. The average value of the batch dimension is then calculated to obtain the global audio features corresponding to the audio to be processed.
[0093] For example, a shallow feature extraction network may include a multi-layer CNN (Convolutional Neural Networks) to learn shallow features using the translation invariance of the convolutional neural network; a deep feature extraction network may include a TransFormer (a deep learning model architecture for sequence-to-sequence) network to learn deep features using the long-term memory capability of the TransFormer network. However, it should be noted that this embodiment does not limit the specific network structure of the shallow feature extraction network and the deep feature extraction network, and any combination of the shallow feature extraction network and the deep feature extraction network that can achieve the above functions can be used as a structural implementation of the feature extraction model.
[0094] Based on the network structure of the feature extraction model as exemplified above, in an exemplary embodiment, S210 can input the first audio segment sequence corresponding to the audio to be processed into the trained feature extraction model, and use the shallow feature sequence correspondingly output by the shallow feature extraction network as the audio segment feature sequence corresponding to the audio to be processed. That is, the audio segment features contained in the audio segment feature sequence corresponding to the audio to be processed are shallow features, which are not semantic and can more accurately express the audio framework. Therefore, after using global audio features to realize the recall of similar global audio features, further checking feature similarity through shallow feature sequences can further improve the accuracy of audio copy detection.
[0095] It should also be noted that the trained feature extraction model is obtained by pre-training the initial feature extraction model using the pre-collected training data set. The pre-training process is the process of self-supervised learning. The basic idea of self-supervised learning is to perform some transformation on the sample data (such as rotation, occlusion, color transformation, etc.) so that the feature extraction model can learn useful feature representations from these transformed data.
[0096] In some exemplary embodiments, at least one training task of a reconstruction learning task and a contrast learning task may be generated respectively, and then the at least one training task may be executed respectively based on the training data set to pre-train the initial feature extraction model to obtain a trained feature extraction model.
[0097] The reconstruction learning task indicates that the model parameters are optimized by calculating the mean absolute error (MAE) loss value. The calculation formula of the loss function corresponding to the reconstruction learning task can be expressed as follows:
[0098]
[0099] Where n represents the number of samples in a training batch. represents the global audio features output by the initial feature extraction model, Represents the true global audio features.
[0100] The contrastive learning task indicates that the model parameters are optimized by calculating the noise contrast estimation loss value. The basic idea of the contrastive learning task is to optimize the model by comparing positive samples and negative samples. The goal of the model is to maximize the similarity between the positive sample and the target, while minimizing the similarity between the negative sample and the target. The loss function corresponding to the contrastive learning task can be expressed as loss(InfoNCE), and the calculation formula is as follows:
[0101]
[0102] Where n represents the number of samples in a training batch. represents the similarity score of the positive sample pair, Represents the sum of similarity scores of all negative sample pairs.
[0103] During the training process, the training stability of the feature extraction model can be improved by at least one of gradient reduction and feature penalty. It can be understood that gradient reduction is to avoid extreme changes in model parameters during update by limiting the size of the gradient, and the feature penalty is to add a term related to the complexity of the model to the loss function so that the model performs well on the training data while maintaining a certain generalization ability.
[0104] It should also be noted that in some exemplary embodiments, training data can be collected by distributed data crawling, thereby obtaining a multivariate heterogeneous training data set, and then a large-scale training data set can be used to perform pre-training on the initial feature extraction model. For example, the training data comes from different types of audio services, and the data related to these audio services are distributedly stored. By distributed crawling of the audio data of these audio services, a training data set consisting of different types of audio data can be obtained.
[0105] In an exemplary application scenario, the audio copy detection method proposed in the embodiment of the present application is applied to a video publishing platform. The video publishing platform can be understood as a UGC (User Generated Content) platform, where users can publish their own videos and browse videos published by other users.
[0106] When users are creating video works to be released, they usually add background music, sound effects, commentary, monologues, etc. to assist in expressing the video content, so as to give the video a sense of scene and drive the audience's emotional atmosphere. Figure 5 The schematic diagram of the distribution of audio classifications contained in videos published by users on an exemplary video publishing platform shows that the proportion of users who publish videos with pure music is not large, and most users are accustomed to using a combination of multiple audios to publish videos. In this application scenario, if audio duplication detection is to be performed, it is inaccurate to only consider the background music contained in the video work. Therefore, the embodiment of the present application also proposes to implement audio duplication detection based on the original sound of the video.
[0107] The global audio features corresponding to the audio contained in the video released in the video publishing platform are stored in the audio retrieval library, and in the audio retrieval library, each global audio feature has an audio identifier. This can also be understood as the same audio corresponds to the same audio identifier, and different audio corresponds to different audio identifiers.
[0108] See also Figure 6 , Figure 6This is an exemplary audio identification processing flow chart based on a video publishing platform. It can be seen that for the video published on the video publishing platform, the audio information contained in the video is first extracted as the audio to be processed, and then the global audio features and audio segment feature sequences corresponding to the audio to be processed are extracted, and then the approximate global audio features corresponding to the global audio features are retrieved based on the audio retrieval library. If the approximate global audio feature is retrieved, it means that the same audio feature is retrieved, and the audio identification corresponding to this approximate global audio feature is added as the audio identification of the audio to be processed. If the approximate global audio feature is not retrieved, the audio identification corresponding to the audio to be processed is generated based on the latest audio identification in the audio identification list.
[0109] The audio identifier list records the audio identifiers corresponding to the audio contained in the videos published on the video publishing platform. Therefore, this embodiment generates the audio identifier corresponding to the audio to be processed based on the latest audio identifier in the audio identifier list, which can ensure that the generated audio identifier is a new audio identifier that does not exist in the audio identifier list. For example, according to a preset numerical update rule, such as increasing the numerical value by 1 each time, the numerical update processing can be performed on the latest audio identifier in the audio identifier list, and the obtained numerical value can be used as the audio identifier of the audio to be processed. It should be noted that this embodiment does not limit the specific update method of the audio identifier. For example, the numerical value increased each time can be other values except 1, and a new audio identifier can also be generated by reducing the numerical value each time.
[0110] See also Figure 7 , Figure 7 The schematic diagram of the principle architecture of adding audio tags to videos published on the video publishing platform is illustrated. For videos published on the video publishing platform, global audio features are extracted as video-level features, and audio segment feature sequences are extracted as frame-level features. Then, similarity recall is performed based on the audio retrieval library to recall similar audio based on video-level features, and duplicate verification is performed based on the key-value database to implement duplicate verification of duplicate audio based on frame-level features, thereby adding audio tags to the video based on the results of the duplicate verification. It should be understood that Figure 7 The illustrated audio retrieval library is used to store global audio features, and the key-value database is used to store audio segment feature sequences.
[0111] In addition, it should be noted that Figure 6 As shown in FIG. 1 , the audio retrieval library is not only used to store global audio features and audio segment feature sequences, but also to store audio tag information, such as the audio style, language, emotion and other tag types, which are not limited here. These audio tags can distinguish the videos to which they belong to to a certain extent to meet certain video processing scenarios. For example, the audio tags stored in the audio retrieval library can be used to quickly filter out videos with audio of a certain style.
[0112] In the application scenario of the video publishing platform, such as Figure 8 As shown, the audio copy detection method may further include S810-S820, which are described in detail as follows:
[0113] S810, detecting the video to be recommended in the video publishing platform, and obtaining the audio identifier corresponding to the audio contained in the video to be recommended.
[0114] The technical solution provided by this embodiment is applied to the video recommendation scenario in the video publishing platform. The videos to be recommended in the video publishing platform can be detected based on specific recommendation requirements. For example, if the specific recommendation requirement is to perform video recommendation for videos newly released by users, then the newly released videos in the video publishing platform are detected as videos to be recommended. For another example, if the specific recommendation requirement is to perform video recommendation for videos released by users with specific tags, and the specific tags, for example, indicate that the user has paid / recharged in the video publishing platform, then the videos released by users with this specific tag in the video publishing platform are detected as videos to be recommended.
[0115] The audio identifier corresponding to the audio contained in the video to be recommended is as follows Figure 6 The process is obtained by the example, and this process will not be described in detail here.
[0116] S820, based on the audio identifier corresponding to the video to be recommended, determine the candidate video corresponding to the video to be recommended, and perform recommendation processing of the video to be recommended based on the candidate video; wherein the audio included in the candidate video is the copy audio corresponding to the audio included in the video to be recommended.
[0117] Since the same or similar audio has the same audio identifier, in this embodiment, a copy audio having the same audio identifier as the audio contained in the video to be recommended is first obtained, and then the video containing the copy audio is obtained as a candidate video, so that the video to be recommended is recommended based on the obtained candidate video.
[0118] For example, the audiences of the candidate videos can be obtained and the videos to be recommended can be recommended to these audiences. Alternatively, based on the viewing data of the candidate videos by these audiences, such as viewing time, evaluation information, etc., the audiences who are interested in the candidate videos can be further screened out, and then the videos to be recommended can be recommended to the screened audiences. In this way, accurate recommendation of the videos to be recommended can be achieved.
[0119] Alternatively, the video to be recommended and the candidate videos may be packaged for recommendation. For example, the video to be recommended and the candidate videos are taken as a video package, and when any viewer browses any video in the video package, other videos in the video package may be recommended to the viewer according to a preset rule. The preset rule may be, for example, to recommend playing other videos in the video package immediately after watching the current video, or to recommend playing other videos in the video package after watching the current video and after a preset number of videos, which is not limited here.
[0120] This embodiment does not limit the specific video recommendation method, but it should be noted that the video recommendation performed in this embodiment relies on the duplicate audio recognition of the audio contained in the video. After the audio is extracted from the video, the extracted audio can be subjected to audio duplicate detection. Since the audio extracted from the video is the original audio, the original audio also contains all the audio information in the video, such as background music, commentary, monologue, etc., the video recommendation performed in this embodiment takes into account all the audio information in the video, thereby ensuring a better video recommendation effect.
[0121] In the application scenario of a video publishing platform, the audio copy detection solution provided by the embodiment of the present application can also improve the publication rate of works in the video publishing platform. Fig. 9 As shown, a video processing method performed by a user terminal includes the following steps:
[0122] S910, play the video on the video play page;
[0123] S920, if it is detected that the duplicate audio search entry set in the video playback page is triggered, then a candidate video display page is displayed; the candidate video display page displays multiple candidate videos, and the audio contained in the candidate video is the duplicate audio corresponding to the audio contained in the video played in the video playback page;
[0124] S930: If it is detected that the video publishing entry set in the candidate video display page is triggered, the video editing page is displayed.
[0125] It should be noted that the video publishing platform has a front-end and a back-end. The front-end refers to the part that users interact with directly, and is mainly responsible for displaying the data provided by the back-end to users in the form of a graphical interface, usually through a web browser, and handling the interaction between the user and the interface. The back-end is responsible for processing the requests sent by the front-end, executing the corresponding logic (such as accessing the database, performing data processing, etc.), and returning the results to the front-end for display.
[0126] Here with Fig.10 The front-end interface jump diagram of the video publishing platform shown in the figure is used as an example to explain the process shown in S910-S930.
[0127] The user terminal first displays the video playback page and plays the video on the video playback page. The video playback page is provided with a copy audio search entry, such as Fig.10 The "Find the Same" button in the video playback page shown is also the duplicate audio search entry. When the user terminal detects that the duplicate audio search entry is triggered, it jumps to display the candidate video display page.
[0128] The candidate video display page displays multiple candidate videos, and the audio contained in the candidate video is a copy of the audio contained in the video played on the video playback page. It can also be understood that the audio identifier corresponding to the candidate video is the same as the audio identifier corresponding to the video played on the video playback page. Among them, the identification of the audio identifier corresponding to the audio contained in the video is achieved through the relevant process recorded in the above embodiment.
[0129] It is understandable that since the candidate videos displayed in the candidate video display page use the same or similar audio, the candidate videos may have common content attributes, such as being all comedy videos, all movie commentary videos, etc. Therefore, by displaying related candidate videos in the candidate video display page, it can, to a certain extent, guide users to use the same audio to create and publish video works, thereby increasing the publication rate of works on the video publishing platform.
[0130] It should also be noted that if Fig.10 As shown, the candidate video display page can also display the video cover. For example, the multiple candidate videos displayed on the candidate video display page are video covers. The video cover can be the cover content set by the user when the video history is released, or it can be the cover content directly determined from the video content of the candidate video. This embodiment does not limit the method for obtaining the video cover of the candidate video.
[0131] The candidate video display page has a video publishing entrance, such as Fig.10 The "Send Video" button shown is the video publishing entrance. When the user terminal detects that the video publishing entrance is triggered, it jumps to display the video production page, and the user can make the video to be released on the video production page. This embodiment does not limit the specific process of video production. However, it should be noted that the same audio can be used as the added background music by default in the video production page to facilitate the user to make a video.
[0132] As can be seen, this embodiment combines the page display in the front-end interface of the video publishing platform with the audio copy detection technology, which is very convenient for guiding users to make and publish videos based on the same audio while watching videos, thereby improving the publication rate of works in the video publishing platform, thereby improving the competitiveness of the video publishing platform among similar platform products.
[0133] Fig.11 FIG. 1 is a block diagram of an audio copy detection device shown in an exemplary embodiment of the present application. Fig.11 As shown, the exemplary audio copy detection device includes:
[0134] A feature extraction module 1110 is configured to extract global audio features and audio segment feature sequences corresponding to the audio to be processed;
[0135] The feature retrieval module 1120 is configured to retrieve the approximate global audio feature corresponding to the global audio feature from the audio retrieval library, and obtain the audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs;
[0136] A similarity determination module 1130 is configured to determine the audio similarity between the audio to be processed and the target audio according to the audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio;
[0137] The copy determination module 1140 is configured to determine the target audio as the copy audio corresponding to the audio to be processed if the audio similarity reaches or exceeds a preset similarity.
[0138] In another exemplary embodiment, the feature extraction module 1110 includes an audio segment feature extraction unit, and the audio segment feature extraction unit is configured to:
[0139] Divide the audio to be processed based on a preset first unit time length to obtain a plurality of first audio segments included in the audio to be processed;
[0140] Audio features are extracted from each first audio segment respectively, and the audio features of each first audio segment are sorted in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed.
[0141] In another exemplary embodiment, the feature extraction module 1110 includes a global feature extraction unit, and the global feature extraction unit is configured as follows:
[0142] The audio to be processed is divided and processed based on a preset second unit time length to obtain a plurality of second audio segments included in the audio to be processed; the second unit time length is greater than or equal to the first unit time length;
[0143] Performing spectrogram conversion on each second audio segment to obtain a spectrogram sequence corresponding to the audio to be processed;
[0144] The spectrogram sequence is input into the trained feature extraction model to obtain the global audio features output by the trained feature extraction model.
[0145] In another exemplary embodiment, the trained feature extraction model is obtained by pre-training an initial feature extraction model using a training data set; the audio copy detection device further includes a model training module, and the model training module is configured as follows:
[0146] Generate at least one training task of a reconstruction learning task and a contrastive learning task respectively; the reconstruction learning task indicates to optimize the model parameters by calculating the mean absolute error loss value, and the contrastive learning task indicates to optimize the model parameters by calculating the noise contrast estimation loss value;
[0147] At least one training task is performed based on the training data set to pre-train the initial feature extraction model to obtain a trained feature extraction model.
[0148] In another exemplary embodiment, the trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network; the global feature extraction unit is further configured as follows:
[0149] The shallow features corresponding to each spectrum graph in the spectrum graph sequence are extracted through a shallow feature extraction network;
[0150] The deep features corresponding to each shallow feature are extracted through the deep feature extraction network;
[0151] The feature average is calculated based on the deep features, and the feature average is used as the global audio feature corresponding to the audio to be processed.
[0152] In another exemplary embodiment, the audio segment feature extraction unit is further configured to:
[0153] Input each first audio segment into the trained feature extraction model in chronological order;
[0154] The shallow feature sequence output by the shallow feature extraction network is used as the audio segment feature sequence corresponding to the audio to be processed.
[0155] In another exemplary embodiment, the similarity determination module 1130 is further configured to:
[0156] Performing audio segment feature comparison on the audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio to obtain the audio similarity duration between the audio to be processed and the target audio;
[0157] The audio similarity between the audio to be processed and the target audio is determined based on the audio durations corresponding to the audio to be processed and the target audio, respectively, and the audio similarity durations.
[0158] In another exemplary embodiment, each global audio feature stored in the audio retrieval library has an audio identifier; the audio copy detection device also includes an audio storage module, and the audio storage module is configured as follows:
[0159] Using the audio identifier corresponding to the duplicate audio as the audio identifier corresponding to the audio to be processed;
[0160] Based on the audio identifier corresponding to the audio to be processed, the global audio feature and the audio segment feature sequence corresponding to the audio to be processed are stored in the audio retrieval library.
[0161] In another exemplary embodiment, the global audio features and audio segment feature sequences stored in the audio retrieval library have a storage time limit of a specified duration; the audio storage module is further configured to update the remaining storage time of the global audio features and audio segment feature sequences corresponding to the copy audio to the specified duration.
[0162] In another exemplary embodiment, the audio storage module is further configured to: if the audio similarity is lower than a preset similarity, generate an audio identifier corresponding to the audio to be processed based on the latest audio identifier in the audio identifier list.
[0163] In another exemplary embodiment, the audio copy detection apparatus further includes:
[0164] The video detection module is configured to detect the video to be recommended in the video publishing platform and obtain the audio identifier corresponding to the audio contained in the video to be recommended;
[0165] The video recommendation module is configured to determine a candidate video corresponding to the video to be recommended based on the audio identifier corresponding to the video to be recommended, and perform recommendation processing of the video to be recommended based on the candidate video; wherein the audio contained in the candidate video is a copy audio corresponding to the audio contained in the video to be recommended.
[0166] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the audio copy detection device provided in the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.
[0167] An embodiment of the present application also provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs. When the one or more computer programs are executed by the one or more processors, the electronic device implements the audio copy detection method provided in the above-mentioned embodiments.
[0168] Fig.12 The structure diagram of the computer system of the electronic device suitable for implementing the embodiment of the present application is shown. It should be noted that the electronic device can be Figure 1 The terminal 110 or server 120 in the implementation environment shown may also be a terminal or server in other implementation environments, and is not limited here. It should also be noted that Fig.12 The computer system 1200 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0169] like Fig.12 As shown, the computer system 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1202 or a computer program loaded from a storage part 1208 to a random access memory (RAM) 1203, such as executing the method described in the above embodiment. Various computer programs and data required for system operation are also stored in RAM 1203. CPU 1201, ROM 1202 and RAM 1203 are connected to each other via a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.
[0170] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1210 as needed so that a computer program read therefrom is installed into the storage section 1208 as needed.
[0171] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 1209, and / or installed from the removable medium 1211. When the computer program is executed by the central processing unit (CPU) 1201, various functions defined in the system of the present application are executed.
[0172] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. A more specific example of a computer-readable storage medium may include, but is not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. The computer program contained in the computer-readable medium may be transmitted with any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.
[0173] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0174] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.
[0175] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the audio copy detection method as described above when the computer program is executed by a processor of an electronic device. The computer-readable storage medium may be included in the electronic device described in the above embodiment, or may exist independently without being assembled into the electronic device.
[0176] Another aspect of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer program from the computer-readable storage medium, and the processor executes the computer program, so that the electronic device performs the audio copy detection method provided in each of the above embodiments.
[0177] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.
[0178] It is understandable that in the specific implementation of this application, video, audio and other related data are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.
Claims
1. A video processing method, characterized in that: The method comprises: Play the video on the video playback page; If it is detected that the duplicate audio search entry set in the video playback page is triggered, a candidate video display page is displayed; the candidate video display page displays multiple candidate videos, and the audio contained in each candidate video is the duplicate audio corresponding to the audio contained in the video played in the video playback page; If it is detected that the video publishing entrance set in the candidate video display page is triggered, a video editing page is displayed, and the copy audio is used as the added background music in the video editing page; The duplicate audio corresponding to the audio contained in the video played in the video playback page is obtained based on the following process: Extracting global audio features and audio segment feature sequences corresponding to the audio to be processed; the audio to be processed is the audio contained in the video played in the video playback page; the audio segment features contained in the audio segment feature sequence are shallow features, and the shallow features are used to express the audio framework; Retrieving an approximate global audio feature corresponding to the global audio feature from an audio retrieval library; Acquire an audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs, and perform audio segment feature comparison on the audio segment feature sequence corresponding to the audio to be processed and the audio segment feature sequence corresponding to the target audio, to obtain an audio similarity duration between the audio to be processed and the target audio, so as to determine the audio similarity between the audio to be processed and the target audio according to the audio durations respectively corresponding to the audio to be processed and the target audio, and the audio similarity duration; If the audio similarity reaches or exceeds a preset similarity, the target audio is determined as the duplicate audio corresponding to the audio to be processed.
2. The method according to claim 1, characterized in that Extract the global audio features and audio segment feature sequences corresponding to the audio to be processed, including: Divide the audio to be processed based on a preset first unit time length to obtain a plurality of first audio segments included in the audio to be processed; Audio features are extracted from each first audio segment respectively, and the audio features of each first audio segment are sorted in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed.
3. The method according to claim 2, characterized in that Extracting the global audio features and audio segment feature sequences corresponding to the audio to be processed, further comprising: The audio to be processed is divided and processed based on a preset second unit time length to obtain a plurality of second audio segments included in the audio to be processed; the second unit time length is greater than or equal to the first unit time length; Performing spectrogram conversion on each second audio segment to obtain a spectrogram sequence corresponding to the audio to be processed; The spectrogram sequence is input into a trained feature extraction model to obtain global audio features output by the trained feature extraction model.
4. The method according to claim 3, characterized in that The trained feature extraction model is obtained by pre-training the initial feature extraction model using the training data set; the method further comprises: Generate at least one training task of a reconstruction learning task and a contrastive learning task respectively; the reconstruction learning task indicates to optimize the model parameters by calculating the mean absolute error loss value, and the contrastive learning task indicates to optimize the model parameters by calculating the noise contrast estimation loss value; The at least one training task is respectively performed based on the training data set to pre-train the initial feature extraction model to obtain the trained feature extraction model.
5. The method according to claim 3, characterized in that: The trained feature extraction model includes a cascaded shallow feature extraction network and a deep feature extraction network; the global audio features corresponding to the audio to be processed are obtained by executing the following steps: Extracting shallow features corresponding to each spectrogram in the spectrogram sequence through the shallow feature extraction network; Extracting deep features corresponding to each shallow feature through the deep feature extraction network; A feature average is calculated according to the deep features, and the feature average is used as a global audio feature corresponding to the audio to be processed.
6. The method according to claim 5, characterized in that Extracting audio features from each first audio segment respectively, and sorting the audio features of each first audio segment in chronological order to obtain an audio segment feature sequence corresponding to the audio to be processed, including: Inputting each first audio segment into the trained feature extraction model in chronological order; The shallow feature sequence output by the shallow feature extraction network is used as the audio segment feature sequence corresponding to the audio to be processed.
7. The method according to any one of claims 1 to 6, characterized in that: Each global audio feature stored in the audio retrieval library has an audio identifier; the method further includes: Using the audio identifier corresponding to the duplicate audio as the audio identifier corresponding to the audio to be processed; Based on the audio identifier corresponding to the audio to be processed, the global audio feature and the audio segment feature sequence corresponding to the audio to be processed are stored in the audio retrieval library.
8. The method according to claim 7, characterized in that The global audio features and audio segment feature sequences stored in the audio retrieval library have a storage time limit of a specified duration; the method further includes: The remaining storage duration of the global audio feature and the audio segment feature sequence corresponding to the copy audio is updated to the specified duration.
9. The method according to claim 7, characterized in that: The method further comprises: If the audio similarity is lower than the preset similarity, an audio identifier corresponding to the audio to be processed is generated based on the latest audio identifier in the audio identifier list.
10. The method according to claim 7, characterized in that The method further comprises: Detecting a video to be recommended in a video publishing platform, and obtaining an audio identifier corresponding to an audio contained in the video to be recommended; Based on the audio identifier corresponding to the video to be recommended, a candidate video corresponding to the video to be recommended is determined, and recommendation processing of the video to be recommended is performed based on the candidate video; wherein the audio contained in the candidate video is a copy audio corresponding to the audio contained in the video to be recommended.
11. A video processing device, characterized in that: The device comprises: A feature extraction module is configured to extract global audio features and an audio segment feature sequence corresponding to the audio to be processed; the audio to be processed is the audio contained in the video played in the video playback page; the audio segment features contained in the audio segment feature sequence are shallow features, and the shallow features are used to express the audio framework; A feature retrieval module is configured to retrieve an approximate global audio feature corresponding to the global audio feature from an audio retrieval library, and obtain an audio segment feature sequence corresponding to the target audio to which the approximate global audio feature belongs; a similarity determination module configured to compare audio segment features of an audio segment feature sequence corresponding to the audio to be processed and an audio segment feature sequence corresponding to the target audio, and obtain an audio similarity duration between the audio to be processed and the target audio, so as to determine the audio similarity between the audio to be processed and the target audio according to the audio durations respectively corresponding to the audio to be processed and the target audio, and the audio similarity duration; A copy determination module is configured to determine the target audio as a copy audio corresponding to the audio to be processed if the audio similarity reaches or exceeds a preset similarity; A playback module, configured to play the video on the video playback page; The first display module is configured to display a candidate video display page if it is detected that the duplicate audio search entry set in the video playback page is triggered; the candidate video display page displays multiple candidate videos, and the audio contained in each candidate video is the duplicate audio corresponding to the audio contained in the video played in the video playback page; The second display module is configured to display a video editing page if it is detected that the video publishing entrance set in the candidate video display page is triggered, and the copy audio is used as the added background music in the video editing page.
12. An electronic device, characterized in that: include: one or more processors; A memory for storing one or more computer programs, which, when executed by the one or more processors, enables the electronic device to implement the method according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of an electronic device, the electronic device is caused to execute the method according to any one of claims 1 to 10.
14. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor of an electronic device, the method according to any one of claims 1 to 10 is implemented.
