Split and merge framework for audio content processing

Abstract

A method and system are proposed to provide a framework for analyzing and classifying audio data using segmentation and merging methods. The audio data is segmented into multiple audio tracks corresponding to different characteristics. Each audio track is segmented, and features are extracted from each segment of that track. The features extracted from the audio segments of each audio track are analyzed. Based on a comparison of features extracted from audio segments of the first audio track with features extracted from audio segments of the second audio track, one or more correlations between the different audio tracks are determined. The audio data is then classified based on these one or more correlations.

Description

Technical Field

[0001] This specification generally relates to audio processing, and more specifically, to the classification of audio data according to various embodiments of this disclosure. Background Technology

[0002] Online service providers, such as online merchants, social media platforms, and other similar platforms, have enabled users to submit content to share with other users through sites hosted by the online service provider. For example, users of social media platforms can post user-generated text, audio clips, and / or video clips on social media websites to share with other users. In another example, customers of online merchants can post product reviews that include product videos. Because content is published and shared through sites hosted by service providers (e.g., websites, mobile applications, etc.), service providers may be required to ensure that user-submitted content complies with any local laws or regulations and / or the service provider's internal policies (e.g., whether the content includes offensive speech, such as hate speech, speech implying violence, speech involving pornography, speech including sensitive data such as health data, personally identifiable information (PII), etc.). Therefore, when a service provider receives user-submitted content, the service provider may first categorize the content (e.g., determine whether the user-submitted content complies with local laws or regulations and / or the service provider's internal policies), and may only publish the user-submitted content on the site if it is categorized as compliant with local laws or regulations and / or internal policies.

[0003] Due to the complexity of analyzing and processing audio data, classifying audio data (e.g., data containing audio content) is often more challenging than classifying textual data (or even visual data). Traditionally, audio data has been analyzed in a single dimension (as a single source of information). Because audio data often contains a wealth of information, this approach frequently produces inaccurate classification results, potentially leading to the distribution of illegal or useless content. Therefore, a more robust framework is needed for the comprehensive analysis and classification of audio data. Attached Figure Description

[0004] Figure 1 This is a block diagram illustrating an electronic trading system according to an embodiment of the present disclosure; Figure 2 This is a block diagram illustrating a content analysis module according to an embodiment of the present disclosure; Figure 3 An example data stream for segmenting and dividing audio content according to an embodiment of this disclosure is shown; Figure 4 An example data stream for extracting features from an audio segment according to an embodiment of this disclosure is shown; Figure 5 An example data stream for determining the correlation between different audio tracks is shown according to an embodiment of this disclosure; Figure 6 An example process for classifying audio data according to an embodiment of this disclosure is shown; Figure 7 An example neural network that can be used to implement a machine learning model according to embodiments of this disclosure is shown; and Figure 8 This is a block diagram of a system for implementing a device according to embodiments of the present disclosure.

[0005] The embodiments and advantages of this disclosure can be best understood by referring to the following detailed description. It should be understood that the same reference numerals are used to identify the same elements shown in one or more of the figures, wherein the illustrations are for illustrative purposes and not for limiting the embodiments of this disclosure. Detailed Implementation

[0006] This disclosure describes methods and systems for providing a framework for analyzing and classifying audio data using segmentation and merging methods. As discussed herein, many online service providers have enabled their users to submit content that can be shared with other users through sites associated with the online service provider in order to increase user interaction on their platforms and / or provide an improved user experience. Content may include single-source data (e.g., text data, image data, audio data) and / or multimedia data (e.g., including two or more types of data such as text data, audio data, image data, etc.). For example, social media platforms may enable their users to publish content to share with other users on social media websites. In another example, customers of an online merchant may post reviews of products recently purchased by consumers. Before publishing user-submitted content to share with other users, online service providers may analyze the user-submitted content to ensure that it does not violate any local laws or regulations and / or internal policies associated with the online service provider. For example, local laws or regulations and / or internal policies may prohibit the publication and / or sharing of content including offensive language, violence, pornography, sensitive data (such as health information, personally identifiable information (PII), etc.).

[0007] Therefore, online service providers can first categorize each user-submitted content (e.g., determine whether the user-submitted content includes prohibited content, etc.) and can determine whether to publish the user-submitted content based on this categorization. If the user-submitted content is categorized as compliant (e.g., it is determined that the user-submitted content does not include any prohibited content, etc.), the online service provider can publish the user-submitted content on its site. Once the user-submitted content is published, the online service provider's users can access (e.g., view, download, reshare, etc.) the user-submitted content through one or more of the online service provider's sites. On the other hand, if the user-submitted content is categorized as non-compliant, the online service provider may not publish the user-submitted content. In addition, the online service provider can also flag the content (or parts of the content that violate laws / policies) and have the user-submitted content undergo further review and / or further processing by a human reviewer (or another computer module). Flagging the content (or part of the content) may include selectively modifying one or more parts of the content (e.g., audio data) to indicate that the content (or part of the content) includes data that does not comply with local laws or regulations and / or policies associated with the online service provider.

[0008] Analyzing audio data can be challenging because, unlike text data, audio data contains a wealth of information. Specifically, audio clips or video clips may include multiple layers of sound (e.g., sounds produced by different sound sources such as different people or different objects), which may or may not overlap. For example, audio data may include speech data associated with one or more people speaking in the foreground (e.g., the main speaker, etc.). Audio clips or audio accompanying video clips may also include different background sounds (e.g., other people chatting in the background besides the main speaker, ambient sounds, sounds produced by other people or other objects near the main speaker, etc.).

[0009] Traditional methods for analyzing and classifying audio data treat it as a single source. That is, in traditional methods, audio data is analyzed as a whole, without considering the actual source of the sound or whether it corresponds to primary speech or background noise. While this approach allows for the derivation of different aspects of audio data, such as human speech data, environmental data, and emotional data, the analysis and resulting classifications lack precision and accuracy. This is because certain words or sounds can have different meanings depending on the context in which they are produced. For example, the sound of someone falling to the ground might indicate that a violent incident has occurred (e.g., someone is being physically assaulted), or it might simply mean that the person tripped and fell. In another example, a person reciting a series of numbers might indicate that the person is revealing sensitive information (e.g., credit card number, Social Security number, etc.), or it might simply mean that the person is reciting random numbers. By analyzing audio data as a single source of information, context may not be accurately detected due to noise in the audio data or a lack of data to verify the detected context (e.g., it may be impossible or inaccurate to determine whether a violent incident occurred based solely on the sound of someone falling to the ground).

[0010] Therefore, according to various embodiments of this disclosure, the audio analysis framework can use segmentation and merging methods to analyze audio data. Under the segmentation and merging method, audio data can be decomposed into multiple layers of sound. In some embodiments, each layer of sound can be considered a different audio track. For example, since the audio data may include speech data associated with one or more people speaking in the foreground (e.g., one or more main speakers, etc.), the audio analysis system can extract the speech of one or more speakers from the audio data as an audio track corresponding to the foreground speech of the content. In some embodiments, when there are multiple speakers in the foreground, the audio analysis system can treat the speech from each speaker as a different audio track. Since speakers may speak sequentially or sometimes simultaneously, their voices may overlap, so it may be beneficial to analyze the speech of different speakers as separate audio tracks. Similarly, the audio analysis system can separate background sounds from the foreground speech and treat the background sounds as a separate audio track. In some embodiments, the audio analysis system can also determine whether there are multiple background sound sources (e.g., music playing in the background, chatter of others other than the main speaker, sounds produced by background objects, etc.) and can treat sounds from different sound sources as different audio tracks.

[0011] An audio analysis system can first analyze different audio tracks individually (e.g., foreground speech track, background audio track, etc.). To analyze the audio tracks, the audio analysis system can divide the tracks into multiple segments. In some embodiments, the audio analysis system can determine the segment duration (e.g., 3 seconds, 5 seconds, etc.) and can divide each audio track into multiple segments based on that duration, such that each audio segment includes audio data of the determined duration (or shorter). After segmenting each audio track, the audio analysis system can extract features from each audio segment.

[0012] In some embodiments, the audio analysis system may use one or more machine learning models to extract features (e.g., embedding vectors, such as vectors in multidimensional space) from each audio segment based on characteristics associated with the corresponding audio track. For example, the audio analysis system may use one or more machine learning models and / or techniques (e.g., Mel spectrograms, Mel frequency-cephalic coefficients (MFCC), convolutional neural networks (CNN), etc.) to generate a time-frequency representation of the sound in each audio segment. The audio analysis system may also use one or more machine learning models (e.g., speech recognition models, such as wav2vec2 transformer models, wav2vec2 combiner models, etc.) to extract speech data from each audio segment. In some embodiments, the audio analysis system may use multiple models to extract different types of features from each audio segment. By using multiple models to extract different types of features from each audio segment, the audio analysis system can analyze different aspects of the sound within the audio segment. For example, the audio analysis system may use a speech recognition model to extract speech data from the audio segment. The audio analysis system may also use another machine learning model (e.g., Mel spectrogram model, MFCC model, CNN, etc.) to extract additional features corresponding to other aspects (e.g., emotional aspects, environmental aspects, etc.) from the same audio segment. Combining different types of features extracted from the same audio segment allows audio analysis systems to analyze the sound within an audio segment more comprehensively than using a single type of feature.

[0013] In some embodiments, the audio analysis system can also select different models to extract features from different audio segments based on the characteristics of the audio segments. For example, the audio analysis system can choose a Mel spectrogram model to extract features from audio segments that tend to include different types of sounds (e.g., audio segments corresponding to background audio tracks, etc.). Because the Mel spectrogram model is configured to transform the audio signal into a multidimensional visualization that captures a spectrogram based on the Mel scale of the audio signal, the features generated by the Mel spectrogram model provide information similar to what humans might perceive from audio data. This is useful for analyzing general sound, as these features provide comprehensive information about the sound, which can then be used to derive different types of data, such as speech data, emotion data, environmental data, motion data, etc. On the other hand, the audio analysis system can choose an MFCC model to extract features from audio segments that primarily include a single type of sound (e.g., human speech) (e.g., audio segments corresponding to foreground speech tracks, etc.). Similar to the Mel spectrogram model, the MFCC model is also a Mel-scale-based model. The MFCC model can be derived from the Mel spectrogram model and is configured to provide a more compressed representation of the audio signal than the Mel spectrogram model. By retaining only the most relevant and important information, MFCC models are generally more efficient than Mel spectrogram models, although the output of an MFCC model may not be as comprehensive. Therefore, audio analysis systems can choose MFCC models for audio segments that primarily consist of human speech because human speech has more structure and is less complex to analyze compared to other types of sound. In some embodiments, instead of using a Mel-based model to extract features from speech and / or background audio segments, audio analysis systems may also choose one or more other types of machine learning models (e.g., CNNs, etc.) to extract features from speech and / or background audio segments.

[0014] In some embodiments, the audio analysis system can analyze features generated by various machine learning models based on audio segments. Since audio segments from each audio track correspond to different time periods of the same content, it is conceivable that these audio segments may be related to each other (e.g., these audio segments may represent different parts of a speaker's speech). Specifically, each audio segment may be associated with a time element relative to the entire audio track (e.g., a specific timestamp, a specific time period within the entire audio track, etc.). Therefore, when analyzing a specific audio segment and / or a specific audio track, the audio analysis system of some embodiments may combine features of one or more audio segments associated with that specific audio segment along the time dimension (e.g., audio segments immediately before or after that specific audio segment in the audio track, etc.) to analyze features extracted from that specific audio segment. In some embodiments, the audio analysis system may use a recurrent neural network (e.g., a gated recurrent unit (GRU) or the like) to transform features extracted from audio segments associated with an audio track into time-dependent features. For example, the audio analysis system may feed features associated with audio segments to the recurrent neural network one audio segment at a time. The recurrent neural network may be configured to analyze these features based on the timestamps associated with the features. When the features associated with the first audio segment are analyzed by the recurrent neural network (RNN), the RNN can analyze the features associated with the first audio segment and generate a set of time-dependent features for the first audio segment. Then, the RNN can analyze the features associated with the second audio segment based on the timestamps associated with the features in the second audio segment (which is immediately after the first audio segment in the track). The RNN can combine the features associated with the first audio segment to analyze the features associated with the second audio segment, and can generate a set of time-dependent features for the second audio segment based on the features associated with both the first and second audio segments.

[0015] Recurrent neural networks (RNNs) can continue analyzing subsequent audio segments. An RNN can analyze each audio segment based on features associated with that segment and features associated with one or more previous audio segments (e.g., all previously provided audio segments for that track), generating a set of time-dependent features for that segment. Because the RNN considers features associated with previous audio segments when analyzing a particular segment, it can use the context derived from previous audio segments to generate the time-dependent features for that specific segment.

[0016] In some embodiments, the recurrent neural network can be configured to analyze features associated with different audio segments in a bidirectional manner. That is, when analyzing features associated with a particular audio segment, the recurrent neural network can consider features that appear before and after the features associated with that particular audio segment (e.g., features associated with audio segments that appear before that particular audio segment, and features associated with audio segments that appear after that particular audio segment).

[0017] In some embodiments, recurrent neural network mechanisms can also be applied to features within each segment. In this case, for example, for a first timestamp in a second segment, the time-dependent features generated for that first timestamp can depend on the input features for that timestamp and one or more feature vectors (e.g., time-dependent features) generated for previous timestamps (e.g., previous timestamps in the second segment, the last timestamp in the first segment, etc.), because those feature vectors (e.g., time-dependent features) summarize information from previous timestamps and / or previous segments. By doing so, the temporal relationships between segments can be taken into account.

[0018] Therefore, the time-dependent features generated for each specific audio segment can represent the characteristics of the audio signal within that segment, using the context of audio signals from one or more audio segments preceding and / or following it. Generating time-dependent features for audio segments is advantageous because, generally, utilizing information related to the sound preceding it (e.g., words or phrases preceding it, sounds preceding it, etc.) and / or sounds following it allows for a more accurate understanding and / or deciphering of sounds (e.g., words, phrases, noise, etc.). For example, an audio segment including gunshots might indicate a violent event or that an action movie is playing on a television in the background. Analyzing the audio segment in isolation might not yield definitive answers. However, sounds preceding the audio segment can provide additional context (e.g., many people screaming or no screaming, different speaking styles indicating a television channel change, etc.) to help recurrent neural networks and / or audio analysis systems understand the sounds within that segment.

[0019] In some embodiments, the audio analysis system may use a recurrent neural network to analyze audio segments of each audio track separately. For example, the audio analysis system may provide features corresponding to audio segments of the foreground speech track to generate time-dependent features of the audio segments of the foreground speech track. Then, the audio analysis system may provide features corresponding to audio segments of the background audio track to generate time-dependent features of the audio segments of the background audio track. In some embodiments, when different models (e.g., Mel-based models and speech recognition models, etc.) are used to extract features from audio segments of an audio track, the audio analysis system may also generate different sets of time-dependent features for the same audio track based on features generated using different models. Thus, the audio analysis system may use a recurrent neural network to generate first time-dependent features for an audio track based on features extracted from audio segments using a first model (e.g., Mel-based models, etc.), and may use a recurrent neural network (which may be the same as or different from the recurrent neural network used to generate the first time-dependent features) to generate second time-dependent features for the same audio track based on features extracted from audio segments using a second model (e.g., speech recognition models, etc.).

[0020] In some embodiments, once time-dependent features are generated for each audio track, the audio analysis system can determine the correlation between different audio tracks based on these features. For example, background sounds associated with people fighting (e.g., shoving, pushing, falling, etc.) are often accompanied by aggressive language in a scene where someone is being bullied. In another example, background sounds accompanying aggressive language and corresponding to someone falling to the ground may indicate a violent event (e.g., a physical altercation between two people). On the other hand, the same background sound accompanying sounds of surprise and corresponding to someone falling to the ground may indicate that the person may have fallen accidentally. Therefore, the correlation between different audio tracks (e.g., whether a sound from one track supports or contradicts a sound from another track) allows the audio analysis system to more accurately detect events occurring within the audio data and thus provide a more accurate classification.

[0021] In some embodiments, the audio analysis system may use a cross-attention module to determine the correlation (or lack thereof) between different audio tracks. The cross-attention module can be configured to accept time-dependent features associated with different audio tracks. For example, the audio analysis system may provide the cross-attention module with time-dependent features associated with the main audio track (e.g., background track) and time-dependent features associated with the secondary audio track (e.g., foreground speech track) as input data. The cross-attention module can be configured to analyze (and modify) the time-dependent features associated with the main audio track based on the time-dependent features associated with the secondary audio track.

[0022] For example, if the cross-attention module determines that a temporal dependency feature associated with a subtrack supports one or more temporal dependency features associated with a main track (e.g., the context determined based on audio segments from the subtrack is consistent with one or more temporal dependency features associated with the main track, such as when aggressive language included in the foreground speech track is consistent with features implying violence based on background sounds), the cross-attention module can emphasize and / or highlight that one or more temporal dependency feature. On the other hand, if the cross-attention module determines that a temporal dependency feature associated with a subtrack contradicts one or more temporal dependency features associated with the main track (e.g., the context determined based on audio segments from the subtrack is inconsistent with one or more temporal dependency features associated with the main track, such as when calm or surprised sounds and non-aggressive language included in the foreground speech track are inconsistent with features implying violence based on background sounds), the cross-attention module can weaken and / or remove that one or more temporal dependency feature.

[0023] The cross-attention module can output context-aware features associated with the main audio track based on modifications to the temporal-dependent features. Based on the relevance processing performed by the cross-attention module, the temporal-dependent features associated with the main audio track are modified to reflect the relevance detected between the main and secondary tracks, making the context-aware features associated with the main audio track more accurate in representing the events that occurred when the audio data was captured. In some embodiments, the audio analysis system can use the cross-attention module to repeat the same process by switching between the main and secondary tracks (e.g., performing a second iteration of the relevance process). Thus, the track used as the main audio track in the first iteration becomes the secondary track in the second iteration, and vice versa. By performing the relevance process using different tracks as the main audio track in multiple iterations, the audio analysis system can generate context-aware features for each track associated with the audio data.

[0024] For scenarios where more than two audio tracks are generated from audio data (e.g., multiple foreground speech tracks, multiple background audio tracks, etc.), the audio analysis system can designate one track as the primary track and use the remaining tracks as secondary tracks in each iteration. Then, the temporal dependency features associated with the primary track and the temporal dependency features associated with the multiple secondary tracks can be provided as input data to the cross-attention module, and the temporal dependency features associated with the primary track can be modified based on the correlation between the primary track and each secondary track.

[0025] In some embodiments, the audio analysis system can generate speech and non-speech classifications based on context-aware features associated with different audio tracks. For example, the audio analysis system can use context-aware features associated with one or more speech tracks (e.g., foreground speech tracks) generated by a cross-attention module to generate speech classifications, and can use context-aware features associated with one or more non-speech tracks (e.g., background audio tracks) generated by a cross-attention module to generate non-speech classifications. These classifications can indicate whether the speech or non-speech portions of the audio data comply with local laws / regulations and / or internal policies. The audio analysis system can then classify the audio data based on the speech and non-speech classifications.

[0026] If the audio analysis system determines that the audio data complies with local laws / regulations and / or internal policies, it may prompt the audio data to be published on sites associated with the online service provider (e.g., websites, mobile applications, etc.). Conversely, if the audio analysis system determines that the audio data does not comply with local laws / regulations and / or internal policies, it may choose not to publish the audio data on any site. The audio analysis system may also transfer the audio data to another computer module (or a human reviewer) for further processing (e.g., cleaning the audio data, removing sounds that do not comply with laws / regulations, etc.).

[0027] In some embodiments, when voice classification and / or non-voice classification indicate that one or more portions of audio data include non-compliant sounds, these classifications may also indicate which(one or more) portions (e.g., which audio segments) contain non-compliant sounds. The audio analysis system may tag that(one or more) portion(e.g., one or more audio segments) in the audio data. For example, the audio analysis system may modify that particular(one or more) audio segment(s) (e.g., add indicators such as specific alarms or beeps to the audio segment(s), add visual indicators to the corresponding video portion of a video clip, etc.), and then transmit the audio data (or video data) to another computer module for further processing. In some embodiments, modifications may be performed to avoid altering any pre-existing content in the audio data, such that the original audio content can be preserved in the audio data except for indicators (e.g., alarms or beeps, etc.). In some embodiments, modifications may alter pre-existing content, for example, by removing and / or editing audio content that has been classified into a specific category (e.g., offensive content, illegal content, content containing sensitive data, etc.). Therefore, using this segmentation and merging framework or method to process (classify) audio data can yield more accurate classifications, thereby preventing the release of unwanted audio data and ensuring the release of desired audio data.

[0028] In some embodiments, generative artificial intelligence (e.g., OpenAI®'s ChatGPT) can also be used in the process of classifying audio data. For example, the audio data provided to the segmentation and merging framework, the criteria set for classifying the audio data, and the classification results generated by the segmentation and merging framework can be used to generate training data to train a machine learning model (e.g., a Large Language Model (LLM)). This allows the LLM to be configured and trained to generate edited versions of various audio content based on any given criteria set.

[0029] Figure 1 An electronic transaction system 100 is illustrated, in which an audio analysis system can be implemented according to one embodiment of this disclosure. The electronic transaction system 100 includes a service provider server 130, a merchant server 120, and user devices 110, 180, and 190 associated with an online service provider, which can be communicatively coupled to each other via a network 160. The network 160 can be implemented as a single network or a combination of multiple networks. For example, the network 160 may include the Internet and / or one or more intranets, terrestrial networks, wireless networks, and / or other suitable types of communication networks. In another example, the network 160 may include a wireless telecommunications network (e.g., a cellular telephone network) adapted to communicate with other communication networks (e.g., the Internet).

[0030] User equipment 110 can be used by user 140 to interact with merchant server 120 and / or service provider server 130 via network 160. For example, user 140 can use user equipment 110 to conduct online transactions with merchant server 120 through a website hosted by merchant server 120 or a mobile application associated with merchant server 120. User 140 can also log in to a user account to access account services or conduct electronic transactions with service provider server 130 (e.g., data access, account transfers or payments, onboarding transactions, etc.). User equipment 110 can be implemented using any suitable combination of hardware and / or software configured for wired and / or wireless communication via network 160. In various implementations, user equipment 110 may include at least one of a wireless cellular phone, wearable computing device, PC, laptop, etc.

[0031] In one example, user device 110 includes a user interface (UI) application 112 (e.g., a web browser, mobile payment application, etc.) that user 140 can use to interact with merchant server 120 and / or service provider server 130 via network 160. In one implementation, the UI application 112 includes software programs (e.g., mobile applications) that provide a graphical user interface (GUI) for user 140 to interact and communicate with service provider server 130 and / or merchant server 120 via network 160. In another implementation, the UI application 112 includes a browser module that provides a network interface to browse information available via network 160. For example, the UI application 112 may be partially implemented as a web browser to view information available via network 160. Therefore, user 140 can use the UI application 112 to initiate electronic transactions with merchant server 120 and / or service provider server 130.

[0032] User equipment 110 may include additional applications 116, as may be expected in one or more embodiments of this disclosure, to provide additional features to user 140. In one example, such additional applications 116 may include security applications for implementing client security features, programmatic client applications for interacting with appropriate application programming interfaces (APIs) via network 160, and / or various other types of well-known programs and / or software applications. In other examples, additional applications 116 may interact with user interface application 112 to improve efficiency and convenience.

[0033] User equipment 110 may include at least one identifier 114, which may be implemented, for example, as an operating system registry entry, a cookie associated with user interface application 112, an identifier associated with the hardware of user equipment 110 (e.g., a media access control (MAC) address), or various other suitable identifiers. In various implementations, identifier 114 may be transmitted to service provider server 130 via network 160 along with a user login request, and identifier 114 may be used by service provider server 130 to associate a user with a specific user account (e.g., and a specific profile).

[0034] Each of user devices 180 and 190 may include hardware and software components similar to user device 110, such that each of user devices 180 and 190 can be operated by the corresponding user to interact with merchant server 120 and / or service provider server 130 in a manner similar to that of user device 110.

[0035] Merchant server 120 may be maintained by a business entity (or, in some cases, by a partner of the business entity that processes transactions on behalf of the business entity). Examples of business entities include merchants, resource information providers, utility providers, online retailers, real estate management providers, social networking platforms, cryptocurrency brokerage platforms, etc., which offer a variety of items for purchase and process payments for purchases. Merchant server 120 may include a merchant database 124 for identifying available items or services that can be viewed by user devices 110 and purchased by the corresponding users.

[0036] Merchant server 120 may include marketplace application 122, which may be configured to provide information to user interface application 112 of user device 110 via network 160. Marketplace application 122 may include a web server hosting merchant websites for merchants. For example, user 140 of user device 110 may interact with marketplace application 122 via network 160 through user interface application 112 to search and view various items or services available for purchase in merchant database 124. Furthermore, users (e.g., users 140 of user device 110, users of user devices 180 and 190, etc.) may submit additional content to the merchant website, such as reviews of products purchased from the merchant in the past. User-submitted content may include single-source data and / or multimedia data. In some embodiments, the merchant server may publish user-submitted content through a website.

[0037] Merchant server 120 may include at least one merchant identifier 126, which may be included as part of one or more items or services available for purchase, thereby associating a particular item and / or transaction with a particular merchant, for example. In one implementation, merchant identifier 126 may include one or more attributes and / or parameters associated with the merchant, such as business and banking information. Merchant identifier 126 may include attributes associated with merchant server 120, such as identification information (e.g., serial number, location address, GPS coordinates, network identifier, etc.).

[0038] Although Figure 1 Only one merchant server 120 is shown, but it is conceivable that multiple merchant servers, each associated with a different merchant, can be connected via network 160 to user devices 110, 180, and 190, as well as service provider server 130. Furthermore, although merchant server 120 is described in conjunction with a product, it can be broadly defined as including servers associated with any entity that publishes user-submitted content on its website or platform, such as, but not limited to, social networking sites, professional networking sites, content sharing sites, and commenting sites.

[0039] Service provider server 130 may be maintained by a transaction processing entity or online service provider, which can provide electronic transaction processing between users (e.g., between user 140 and users of other user devices) and / or between users and one or more merchants. Therefore, service provider server 130 may include service application 138, which may be adapted to interact with user device 110 and / or merchant server 120 via network 160 to facilitate electronic transactions (e.g., electronic payment transactions, data access transactions, content publishing, etc.) between users and merchants processed by service provider server 130. In one example, service provider server 130 may be provided by PayPal® Inc., located in San Jose, California, USA, and / or one or more service entities or corresponding intermediaries, which may provide multiple point-of-sale devices in various locations to facilitate transaction routing between merchants and, for example, service entities. In another example, service provider server 130 may be provided by any entity that publishes or posts content on a platform, application, or website (e.g., social networking sites, professional networking sites, content sharing sites, review sites, etc.).

[0040] Service application 138 may include a payment processing application (not shown) for processing payments for purchases and / or electronic transactions (including content publishing) between users and merchants or between any two entities (e.g., between two users, two merchants, etc.). In one implementation, the payment processing application assists in resolving electronic transactions through verification, delivery, and settlement. Thus, the payment processing application settles debts between users and merchants, where accounts can directly and / or automatically debit and / or credit monetary funds in a banking-acceptable manner.

[0041] Service provider server 130 may also include interface server 134, which is configured to provide content (e.g., web page content) to users and interact with them. For example, interface server 134 may include a web server configured to provide web page content in response to HTTP requests. In another example, interface server 134 may include an application server configured to interact with a corresponding application (e.g., a service provider mobile application) installed on user device 110 via one or more protocols (e.g., REST API, SOAP, etc.). Therefore, interface server 134 may include pre-generated electronic content ready to be provided to users. For example, interface server 134 may store login pages and be configured to provide login pages to users so that they can log in to their user accounts to access various services provided by service provider server 130. Interface server 134 may also include other electronic pages associated with different services (e.g., electronic transaction services, etc.) provided by service provider server 130. As a result, users (e.g., users 140, 180, and 190, or merchants associated with merchant server 120) can access user accounts associated with those users and access various services provided by service provider server 130 by generating HTTP requests to it. For example, users (e.g., users 140, 180, and 190) can submit content to share with other users on the service provider server. This content may relate to reviews or personal experiences of products or services offered by service provider server 130 and / or merchant server 120. The content can include various types of data, including single-source data or multimedia data. Interface server 134 can publish user-submitted content on sites (e.g., the website of service provider server 130, the mobile application of service provider server 130, etc.) so that other users of service provider server 130 can access and view the user-submitted content.

[0042] Service provider server 130 can be configured to maintain one or more user accounts and merchant accounts in account database 136. Each account can be associated with a profile and can include account information associated with one or more individual users (e.g., users 140, 180, and 190 associated with user device 110, etc.) and merchants. For example, account information may include users' and merchants' private financial information, such as one or more account numbers, passwords, credit card information, bank information, digital wallets used, or other types of financial information, transaction history, Internet Protocol (IP) addresses, and device information associated with user accounts. Account information may also include user purchase profile information, such as account funds options and payment options associated with the user, payment information, receipts, and other information collected in response to completed funds and / or payment transactions.

[0043] In one implementation, a user may have identity attributes stored in or accessible by the service provider server 130 (e.g., account database 136), and the user may have credentials for authenticating or verifying their identity with the service provider server 130. User attributes may include personal information such as a photograph, date of birth, Social Security number, home address, banking information, and / or source of funds. In various respects, user attributes may be passed to the service provider server 130 as part of login, search, selection, purchase, and / or payment requests, and may be used by the service provider server 130 to associate a user with one or more specific user accounts maintained by the service provider server 130, and to determine the authenticity of requests from user devices.

[0044] Service provider server 130 may also include content analysis module 132, which at least implements the audio analysis system discussed herein. In some embodiments, content analysis module 132 may be configured to classify content based on a set of criteria, such as whether the content complies with laws and regulations, whether the content complies with internal policies associated with the organization, etc. For example, when a user (e.g., user 140, users of user devices 180 and 190, users associated with merchant server 120, etc.) submits content to service provider server 130 through an interface provided by interface server 134, the content analysis module 132 may be requested to classify the content based on a set of criteria. Since this set of criteria may depend on the location of the submitted content, in some embodiments, interface server 134 may first determine the set of criteria used to classify the content. For example, if the content is submitted by a device within a geographic region (e.g., country, state, etc.), interface server 134 may retrieve laws and regulations associated with that geographic region and include these laws and regulations as part of the set of criteria. Interface server 134 may also retrieve internal policies of service provider server 130 (and / or merchant server 120) and include these internal policies as part of the set of criteria. Then, the interface server 134 can provide the user-submitted content along with the set of standards to the content analysis module 132. The content analysis module 132 can then use this specific set of standards to classify the content.

[0045] In some embodiments, when merchant server 120 receives content submitted by a user (e.g., user 140, user device 180, and 190, etc.) for sharing on merchant server 120's website, merchant server 120 may request content analysis module 132 to categorize the user-submitted content according to a set of criteria. In some embodiments, merchant server 120 may also provide the user-submitted content and the set of criteria to content analysis module 132. Based on the categorization of the user-submitted content, service provider server 130 and / or merchant server 120 may determine the action to be taken regarding the user-submitted content. For example, if the user-submitted content is categorized into a first category (e.g., indicating that the user-submitted content complies with laws and internal policies), service provider server 130 and / or merchant server 120 may publish the user-submitted content through the appropriate site (e.g., the appropriate website, the appropriate mobile application, etc.), allowing other users to access and view the user-submitted content through the appropriate site. On the other hand, if the content submitted by the user is classified as a second category (e.g., indicating that the content submitted by the user does not comply with the law or internal policies), the service provider server 130 and / or the merchant server 120 may withdraw the content submitted by the user so that it is not published, or require the content submitted by the user to be modified before publication.

[0046] Figure 2 This is a block diagram illustrating a content analysis module 132 according to various embodiments of the present disclosure. As shown, the content modification module 132 includes an analysis manager 202, a content segmentation module 204, a segmentation module 206, a feature extraction module 208, and a relevance module 210. As discussed herein, users (user 140 of user device 110, users of user devices 180 and 190, users associated with merchant server 120, etc.) can submit various types of content for sharing through an interface provided by service provider server 130 and / or merchant server 120. The types of content can include single-source data and / or multimedia data. Analyzing single-source data and / or multimedia data with audio data can be challenging because audio data contains rich information. For example, unlike text data, which is typically one-dimensional, audio clips or video clips that include audio data often represent sounds produced by multiple different sound sources, and sounds from different sound sources sometimes overlap with each other.

[0047] Therefore, in some embodiments, the content analysis module 132 is configured to use segmentation and merging methods when analyzing and classifying audio data. The content analysis module 132 can receive requests to classify audio data. For example, after receiving content submitted by a user, the interface server 134 can transmit a classification request to the content analysis module 132. The classification request may include the user-submitted content and a set of criteria, which the content analysis module 132 can use to classify the content. In another example, the merchant server 120 can receive user-submitted content and transmit a classification request to the content analysis module 132. The classification request may also include the user-submitted content and a set of criteria. Therefore, the content analysis module 132 can dynamically classify different content based on different sets of criteria.

[0048] As discussed herein, due to the richness of information contained in audio data, the content analysis module 132 can use segmentation and merging methods when analyzing and classifying audio data. Therefore, when the content analysis module 132 receives a classification request, it can first extract any audio data from the content (e.g., extract audio data from audio clips or video clips), and can use the content segmentation module 204 to divide the audio data into multiple audio tracks. Since user-submitted content is often generated by the user themselves in an uncontrolled environment (e.g., not in a soundproof recording studio), audio data typically includes sounds from multiple sound sources. For example, audio data may include the speech of one or more main speakers (e.g., describing a product or service). However, depending on the location where the audio data was captured, it may also include other sounds, such as ambient sounds (e.g., background music, the voices of other people in the background, the sounds made by the main speaker, the sounds made by other objects in the background, etc.). Therefore, the analysis manager 202 can use the content segmentation module 204 to divide the audio data into multiple audio tracks. In some embodiments, the content segmentation module 204 may include computer software programs (e.g., Spleeter®, Audacity). TM MP3DirectCut TM (etc.), the program is configured to analyze the frequencies (and / or spectral elements) of different sounds within audio data to identify multiple sound sources and segment the audio data into multiple tracks corresponding to the sounds produced by the multiple sound sources.

[0049] Then, the analysis manager 202 can use the segmentation module 206 to segment each audio track into multiple audio segments. In some embodiments, the segmentation module 206 can segment the audio track based on a predetermined duration (e.g., 3 seconds, 5 seconds, etc.) or a dynamic duration (depending on the volume, sound type, number of different sounds, frequency of sound feature changes, etc.). After dividing each audio track into multiple audio segments, the analysis manager 202 can use the feature extraction module 208 to extract features from each audio segment. In some embodiments, the feature extraction module 208 can also use a recurrent neural network to generate time-dependent features for each audio segment. The time-dependent features can represent the characteristics of the corresponding audio segment based on context derived from other related audio segments (e.g., one or more audio segments that appear before the corresponding audio segment in chronological order).

[0050] In some embodiments, the analysis manager 202 can use the correlation module 210 to determine the correlation between different audio tracks and modify the time-dependent features of audio segments based on these correlations. For example, if the correlation module 210 determines that the context derived from the first audio track supports one or more time-dependent features associated with an audio segment of the second audio track, the correlation module 210 can emphasize the one or more time-dependent features. On the other hand, if the correlation module 210 determines that the context derived from the first audio track contradicts one or more time-dependent features associated with an audio segment of the second audio track, the correlation module 210 can weaken the one or more time-dependent features. Based on the context derived from other audio tracks, the correlation module 210 can modify the time-dependent features of each audio segment to generate context-aware features. The analysis manager 202 can then classify the audio data based on the context-aware features generated by the correlation module 210. The following will explain... Figures 3 to 5 The operation of each of the content segmentation module 204, segmentation module 206, feature extraction module 208, and relevance module 210 is described in more detail.

[0051] Figure 3 Operations 300 for segmenting and dividing an audio file 302 according to various embodiments of the present disclosure are illustrated. A content analysis module 132 may receive classification requests from an interface server 134 and / or a merchant server 120. The classification request may include a multimedia file (e.g., a file containing multimedia data) and a set of criteria for classifying the multimedia file. The content analysis module 132 may extract audio content 302 (if any) from the multimedia file, allowing it to analyze the audio content 302 using segmentation and merging methods. The analysis manager 202 may first use a content segmentation module 204 to segment the audio content 302 into multiple audio tracks based on the sound sources of various sounds represented in the detected audio content 302. In this example, the content segmentation module 204 may identify the main speaker speaking in the foreground and background sounds (e.g., background music, background chatter, etc.).

[0052] Therefore, the content segmentation module 204 can divide the audio content 302 into multiple audio tracks, including a foreground speech track 312 corresponding to the main speaker speaking in the foreground, and a background audio track 322 corresponding to background sounds. In some embodiments, the different audio tracks are mutually exclusive. In other words, sounds included in one audio track will not appear in any other audio track generated from the same audio content 302. Although the audio content 302 is divided into only two audio tracks 312 and 322 in this example, in other examples, the content segmentation module 204 can divide the audio data into more audio tracks (three audio tracks, five audio tracks, etc.). For example, when multiple main speakers are detected speaking in the audio content 302, the content segmentation module 204 in some embodiments can generate multiple foreground speech tracks for different main speakers. Similarly, when the content segmentation module 204 identifies multiple background sound sources, the content segmentation module 204 can also generate multiple background audio tracks for background sounds generated by different sound sources. By segmenting the audio content 302 into different audio tracks corresponding to different sound sources in the audio content 302, the content analysis module 132 can analyze each audio track individually and then determine the correlation between different audio tracks.

[0053] In some embodiments, after the audio content 302 is segmented into different audio tracks (e.g., foreground speech track 312, background audio track 322, etc.), the analysis manager 202 may perform certain processing on the audio tracks 312 and 322 to enhance the sound quality within each track 312 and 322. For example, this processing may include volume gain, speed adjustment, noise reduction, etc., to enhance and / or clarify the sound within the audio track, allowing it to be analyzed more effectively by other modules within the content analysis module 132. For instance, the analysis manager 202 may generate an enhanced audio track 314 by processing the foreground speech track 312, and an enhanced audio track 324 by processing the background audio track 322.

[0054] Then, the analysis manager 202 can use the segmentation module 206 to divide each enhanced audio track 314 and 324 into multiple audio segments. In some embodiments, the analysis manager 202 can specify one or more criteria for segmenting the audio track. For example, the analysis manager 202 can specify the duration of each segment (e.g., 3 seconds, 5 seconds, 10 seconds, etc.) so that the segmentation module 206 can segment each audio track into different audio segments, each audio segment having a specified duration. In some embodiments, the analysis manager 202 can determine one or more criteria based on the content within the audio track. For example, the analysis manager 202 can analyze each audio track 314 and 324. Based on the analysis, the analysis manager 202 can determine several factors, such as speaking speed, sound density, sound variation, etc., and can determine one or more criteria based on these factors. When the speaking speed is slower than a threshold and / or the sound density and variation are lower, the analysis manager 202 can determine that each audio segment has a longer duration. When the speaking speed is faster than a threshold and / or the sound density and variation are higher, the analysis manager 202 can determine that each audio segment has a shorter duration.

[0055] Segmentation module 206 can divide each audio track 316 and 324 into multiple audio segments. For example, segmentation module 206 can divide enhanced audio track 314 into speech segments 316 (including segments 316a, 316b, 316c, etc.) according to one or more criteria, and can divide enhanced audio track 324 into background segments 326 (including segments 326a, 326b, 326c, etc.). In some embodiments, each segment 316a, 316b, 316c, 326a, 326b, 326c, etc., is different from each other and does not overlap. The segments extracted from each audio track can be used by other modules (e.g., analysis manager 202, feature extraction module 208, and correlation module 210) to perform further analysis on the audio tracks to classify the audio content 302.

[0056] Figure 4Operation 400 for extracting features from different audio segments according to various embodiments of the present disclosure is illustrated. For example, the analysis manager 202 may provide audio segments corresponding to different audio tracks (e.g., segment 316 corresponding to foreground speech track 314, segment 326 corresponding to background audio track 324, etc.) to the feature extraction module 208. The feature extraction module 208 may be configured to extract features from each audio segment using one or more machine learning models. In some embodiments, the feature extraction module 208 may include or have access to various machine learning models 402, 404, and 406. In some embodiments, machine learning models 402, 404, and 406 have different characteristics and are configured to extract different types of features from the audio segments. For example, machine learning models 402 and 406 may correspond to different Mel-based machine learning models configured to generate time-frequency representations of the sounds included in each audio segment. Machine learning model 402 may correspond to a Mel frequency-cephalic coefficient (MFCC) model, while machine learning model 406 may correspond to a Mel spectrogram model. The feature extraction module 208 may also include or have access to a speech recognition model. For example, machine learning model 404 can correspond to a speech recognition model (or a part of a speech recognition model), such as a Wav2Vec2 transformer model or a Wav2Vec2 conformer model.

[0057] In some embodiments, the feature extraction module 208 may select one or more machine learning models from machine learning models 402, 404, and 406 accessible to the feature extraction module 208 to extract features from each audio segment. In some embodiments, the feature extraction module 208 may determine to use a specific machine learning model (e.g., machine learning model 404) to extract features from all audio segments. Therefore, the feature extraction module 208 may provide each audio segment 316 and each audio segment 326 to the machine learning model 404. The machine learning model 404 may generate feature 414 for each audio segment 316 and feature 422 for each audio segment 326. In some embodiments, the machine learning model 404 may extract (or generate) multiple features from each audio segment. Since the machine learning model 404 corresponds to a speech recognition model, the features extracted by the machine learning model 404 may correspond to words spoken by one or more people within an audio segment.

[0058] In some embodiments, the feature extraction module 208 may select one or more machine learning models to extract features from certain audio segments but not from others. For example, the feature extraction module 208 may select machine learning model 406 (which may correspond to a Mel spectrogram model) to extract features from audio segment 326 corresponding to background audio track 322, but not from audio segment 316 corresponding to foreground speech track 312. Since the Mel spectrogram model is configured to transform the audio signal into a multidimensional visualization that captures a spectrogram based on the Mel scale of the audio signal, the features generated by machine learning model 406 provide information similar to what humans might perceive from audio data. This is useful for analyzing general sound because these features provide comprehensive information about the sound, which can then be used to deduce different data, such as speech data, emotion data, environmental data, motion data, etc. Therefore, machine learning model 406 may extract feature 424 from each audio segment 326. In some embodiments, machine learning model 406 may extract multiple features from each audio segment 326. The features extracted from each audio segment 326 can represent various characteristics of the sound included in that audio segment, such as speech data, emotion data, environmental data, motion data, etc.

[0059] On the other hand, the feature extraction module 208 may select a machine learning model 402 (which may correspond to an MFCC model) to extract features from audio segments 316 corresponding to the foreground speech track 312, but not from audio segments 326 corresponding to the background audio track 322. Similar to the Mel spectrogram model, the MFCC model is also a Mel-scale-based model. The MFCC model can be derived from the Mel spectrogram model and is configured to provide a more compressed audio signal representation than the Mel spectrogram model. By retaining only the most relevant and important information, the MFCC model is generally more efficient than the Mel spectrogram model, although the output of the MFCC model may not be as comprehensive. Since the audio segments from the foreground speech track 312 mainly consist of a single type of sound (e.g., human speech), which generally has more structure and is simpler to analyze than other types of sound, it is more efficient to extract features from audio segments 316 using the MFCC model. Therefore, the machine learning model 402 can extract feature 412 from each audio segment 316. In some embodiments, the machine learning model 402 can extract multiple features from each audio segment 316. The features extracted from each audio segment 316 can represent various characteristics of the sound included in that audio segment, such as speech data, emotion data, environmental data, motion data, etc.

[0060] Figure 5Operation 500 for analyzing audio segment features to classify audio data according to various embodiments of the present disclosure is illustrated. The analysis of audio segment features may include several steps, as described in more detail below. First, the analysis manager 202 may obtain various features, such as features 412, 414, 422, and 424, extracted from audio segments 316 and 326 by various machine learning models 402, 404, and 406 from the feature extraction module 208. Since the audio segments from each audio track correspond to different time periods of the same content, it is conceivable that these audio segments may be related to each other (e.g., these audio segments may represent different parts of a speaker's speech). Specifically, each audio segment may be associated with a time element relative to the entire audio track (e.g., a specific timestamp, etc.). For example, audio segment 316 may include segments 316a, 316b, 316c, etc., which may be arranged chronologically based on timestamps corresponding to these segments within audio track 312 (or enhanced audio track 314). Thus, audio segment 316a may correspond to the first segment (e.g., the first part) of audio track 312. Audio segment 316b may correspond to the second segment immediately following audio segment 316a, and audio segment 316c may correspond to the third segment immediately following audio segment 316b. Audio segment 326 may include segments 326a, 326b, 326c, etc., which may be arranged chronologically based on the timestamps of these segments corresponding to audio track 322 (or enhanced audio track 324). Therefore, audio segment 326a may correspond to the first segment (e.g., the first part) of audio track 322. Audio segment 326b may correspond to the second segment immediately following audio segment 326a, and audio segment 326c may correspond to the third segment immediately following audio segment 326b.

[0061] Therefore, when analyzing a specific audio segment and / or a specific audio track, the analysis manager 202 can combine features from one or more audio segments associated with that specific audio segment along the time dimension (e.g., audio segments immediately before or after that specific audio segment in the audio track) to analyze features extracted from that specific audio segment. In some embodiments, the analysis manager 202 can use a recurrent neural network (e.g., gated recurrent units (GRUs) 502a, 502b, 502c, 502d, etc.) to transform features extracted from audio segments associated with the audio track into time-dependent features. For example, the analysis manager 202 can provide features 414 associated with audio segment 316 to the recurrent neural network one audio segment at a time in chronological order. The analysis manager 202 can first provide features associated with audio segment 316a to GRU 502a, then provide features associated with audio segment 316b, then provide features associated with audio segment 316c, and so on.

[0062] When features associated with audio segment 316a are provided to GRU 502a, GRU 502a will analyze only the features associated with audio segment 316a and generate a set of time-dependent features for audio segment 316a. Then, Analysis Manager 202 can provide features associated with audio segment 316b to GRU 502a. GRU 502a can combine the features associated with audio segment 316a to analyze the features associated with audio segment 316b, and can generate a set of time-dependent features for audio segment 316b based on the features associated with both audio segment 316a and audio segment 316b. Then, Analysis Manager 202 can provide features associated with audio segment 316c to GRU 502a. GRU 502a can combine features associated with audio segment 316a and features associated with audio segment 316b to analyze features associated with audio segment 316c, and can generate a set of time-dependent features for audio segment 316c based on the features associated with audio segment 316a, audio segment 316b, and audio segment 316c. In some embodiments, when generating time-dependent features for a specific audio segment (e.g., audio segment 316c), GRU 502a can assign greater weight to features of other audio segments (e.g., audio segment 316b) that are temporally closer to the specific audio segment, and less weight to features of other audio segments (e.g., audio segment 316a) that are temporally farther away from the specific audio segment.

[0063] The audio analysis system can continue to provide features associated with subsequent audio segments of the audio segment 316 associated with the foreground speech track 312 to the GRU 502a. The GRU 502a can analyze each audio segment based on the features associated with the audio segment and features associated with one or more previous audio segments to generate a set of time-dependent features for that audio segment. Because the GRU 502a considers the features associated with previous (and / or subsequent) audio segments when analyzing a particular audio segment, the GRU 502a can use the context derived from the previous audio segments to generate the time-dependent features for that particular audio segment. Therefore, the time-dependent features generated for each particular audio segment can represent the characteristics of the audio signal within that audio segment, using the context from the audio signals of the audio signals from one or more audio segments that precede it.

[0064] Generating time-dependent features for audio segments is advantageous because, generally, utilizing information preceding the sound (e.g., preceding words or phrases, preceding sounds, etc.) allows for a more accurate understanding and / or deciphering of the sound (e.g., words, phrases, noise, etc.). For example, an audio segment including gunshots might indicate a violent event or that an action movie is playing on a television in the background. Analyzing the audio segment in isolation might not yield a definitive answer. However, sounds preceding the audio segment can provide additional context (e.g., whether there are many people screaming or not, different speaking styles indicating a television channel change, etc.) to help recurrent neural networks and / or audio analysis systems understand the sounds in this audio segment. By analyzing the features 414 extracted from audio segment 316 in chronological order, GRU 502a can generate time-dependent features 514.

[0065] Analysis manager 202 can use the same technique to generate time-dependent features 522 by sequentially providing features 422 extracted by machine learning model 404 from audio segment 326 corresponding to background audio track 322 to GRU 502b. Analysis manager 202 can also provide features 412 extracted by machine learning model 402 from audio segment 316 corresponding to foreground speech track 312 to GRU 502c to generate time-dependent features 512. Analysis manager 202 can also provide features 424 extracted by machine learning model 406 from audio segment 326 corresponding to background audio track 322 to GRU 502d to generate time-dependent features 524. In some embodiments, GRUs 502a, 502b, 502c, and 502d correspond to the same recurrent neural network.

[0066] In some embodiments, once time-dependent features have been generated for each audio track, the analysis manager 202 can use the correlation module 210 to determine the correlation between different audio tracks (e.g., foreground speech track 312, background audio track 322, etc.) based on the time-dependent features 514, 522, 512, and 524 generated by GRUs 502a, 502b, 502c, and 502d. For example, background sounds associated with people fighting (e.g., shoving, pushing, falling, etc.) are often accompanied by aggressive language in a scene where someone is being bullied. In another example, background sounds accompanying aggressive language corresponding to a person falling to the ground may indicate a possible violent event (e.g., a physical altercation between two people). On the other hand, the same background sound accompanying sounds of people making sounds of surprise, corresponding to a person falling to the ground, may indicate that the person may have fallen accidentally. Therefore, the correlation between different audio tracks 312 and 322 (e.g., whether the sound from one audio track supports the sound from another audio track or contradicts the sound from another audio track) enables the analysis manager 202 to more accurately detect events occurring within the audio content 302 and thus provide more accurate classification.

[0067] In some embodiments, the correlation module 210 may use cross-attention modules 504a, 504b, 504c, and 504d to determine the correlation (or lack thereof) between different audio tracks 312 and 322. In some embodiments, cross-attention modules 504a, 504b, 504c, and 504d correspond to the same cross-attention module. Each cross-attention module 504a, 504b, 504c, and 504d may be configured to accept temporal dependency features associated with two or more audio tracks. For example, the correlation module 210 may provide the cross-attention module (e.g., cross-attention module 504b) with temporal dependency features associated with the main audio track (e.g., temporal dependency feature 522 associated with the background audio track 322) and temporal dependency features associated with the secondary audio track (e.g., temporal dependency feature 514 associated with the foreground speech audio track 312) as input data. In some embodiments, the temporal dependency features provided to the cross-attention module are all extracted by the same type of machine learning model, allowing these temporal dependency features to be compared with each other. For example, the correlation module 210 can provide time-dependent features 514 and 522 as input data to the cross-attention modules 504a and / or 504b to determine the correlation between audio tracks 312 and 322, since the time-dependent features 514 and 522 are derived from features extracted by the speech recognition model. The correlation module 210 can also provide time-dependent features 512 and 524 as input data to the cross-attention modules 504c and / or 504d to determine the correlation between audio tracks 312 and 322, since the time-dependent features 512 and 524 are derived from features extracted by Mel-based models (e.g., Mel spectrogram models and MFCC models).

[0068] Upon receiving time-dependent features 522 (as the main audio track) associated with the background audio track 322 and time-dependent features 514 (as the secondary audio track) associated with the foreground audio track 312, the cross-attention module 504b can analyze (and modify) the time-dependent features 522 associated with the main audio track based on the time-dependent features 514 associated with the secondary audio track. In some embodiments, the cross-attention module 504b can compare the time-dependent features of each audio segment from the main audio track with the time-dependent features of each audio segment from the secondary audio track to determine whether there is a correlation between the main audio track and the secondary audio track.

[0069] If the cross-attention module 504b determines that the temporal dependency feature 514 associated with the subtrack supports one or more temporal dependency features 522 associated with the main track (e.g., the context determined based on the audio segment 316 from the subtrack is consistent with one or more temporal dependency features 522 associated with the main track, such as when aggressive language included in the foreground speech track 312 is consistent with features implying violence based on background sounds), the cross-attention module 504b may emphasize and / or highlight the one or more temporal dependency features 522. For example, the cross-attention module 504b may strengthen the one or more temporal dependency features (e.g., by increasing the value in the one or more temporal dependency features, etc.).

[0070] On the other hand, if the cross-attention module 504b determines that the time-dependent feature 514 associated with the sub-track contradicts one or more time-dependent features 522 associated with the main track (e.g., the context determined based on the audio segment 316 from the sub-track is inconsistent with one or more time-dependent features 522 associated with the main track, such as when calm or surprised sounds and non-aggressive language included in the foreground speech track are inconsistent with features implying violence based on background sounds), the cross-attention module 504b may weaken (e.g., by reducing the value of the feature, etc.) and / or remove the one or more time-dependent features.

[0071] The cross-attention module 504b can output a context-aware feature 534 associated with the main audio track based on modifications to the temporal dependency feature 522. Based on the relevance processing performed by the cross-attention module 504b, the temporal dependency feature 522 associated with the main audio track has been modified to reflect the relevance detected between the main and secondary audio tracks, making the context-aware feature 534 associated with the main audio track more accurate in representing the events that occurred when the audio content 302 was captured.

[0072] In some embodiments, the relevance module 210 repeats the same process on the same time-dependent features 514 and 522 by switching between the primary and secondary audio tracks (e.g., performing a second iteration of the relevance process). Thus, an audio track used as the primary track in the first iteration becomes a secondary track in the second iteration, and vice versa. By performing the relevance process using different audio tracks as the primary track in multiple iterations, the audio analysis system can generate context-aware features for each audio track associated with the audio data. Therefore, the relevance module 210 can provide the time-dependent feature 514 associated with the foreground speech track 312 as the primary track and the time-dependent feature 522 associated with the background audio track 322 as the secondary track to a cross-attention module (e.g., cross-attention module 504a). The cross-attention module 504a can use the techniques disclosed herein to analyze (and modify) the time-dependent feature 514 associated with the primary track based on the time-dependent feature 522 associated with the secondary track, and generate context-aware feature 532.

[0073] Similarly, the relevance module 210 can use the cross-attention modules 504c and 504d to modify the time-dependent features 512 and 524 by providing them to the cross-attention modules 504c and 504d in different iterations. During the first iteration, the relevance module 210 can provide the cross-attention module 504c with the time-dependent feature 512 as a feature associated with the main track and the time-dependent feature 524 as a feature associated with the secondary track. The cross-attention module 504c can modify the time-dependent feature 512 based on the time-dependent feature 524 to generate the context-aware feature 536. During the second iteration, the relevance module 210 can provide the cross-attention module 504d with the time-dependent feature 524 as a feature associated with the main track and the time-dependent feature 512 as a feature associated with the secondary track. The cross-attention module 504d can modify the time-dependent feature 524 based on the time-dependent feature 512 to generate the context-aware feature 538.

[0074] In some embodiments, the analysis manager 202 may combine context-aware features 532 and 534 to generate a hybrid feature 542, and may combine context-aware features 536 and 538 to generate a hybrid feature 544. The analysis manager 202 may provide the hybrid features 542 and 544 to each of the speech classification module 506 and the background classification module 508. The speech classification module 506 may be configured to generate a speech classification for the speech portion of the audio content 302 based on the hybrid features 542 and 544 and the set of criteria. The background classification module 508 may be configured to generate a background classification for the background portion of the audio content 302 based on the hybrid features 542 and 544 and the set of criteria. For example, when the set of criteria relates to local laws / regulations and / or internal policies, the speech classification and background classification may indicate whether the speech or non-speech portion of the audio data complies with local laws / regulations and / or internal policies.

[0075] If the analysis manager 202 determines that the audio content 302 complies with local laws / regulations and / or internal policies, the analysis manager 202 may prompt the audio content 302 to be published to a site (e.g., website, mobile application, etc.) associated with the service provider server 130 or merchant server 120. Conversely, if the analysis manager 202 determines that the audio content 302 does not comply with local laws / regulations and / or internal policies, the analysis manager 202 may withdraw or otherwise prevent the audio content 302 from being published on the site. The analysis manager 202 may also transfer the audio data to another computer module (or human reviewer) on the service provider server 130 or merchant server 120 for further processing of the audio data (e.g., cleaning the audio data, removing sounds that do not comply with laws / regulations, etc.).

[0076] In some embodiments, when voice classification and / or non-voice classification indicate that one or more portions of the audio data include non-compliant sounds, these classifications may also indicate which(one or more) portions (e.g., which audio segments) contain non-compliant sounds. Analysis manager 202 may tag that(one or more) portion(e.g., one or more audio segments) in the audio content 302. For example, analysis manager 202 may modify that particular(one or more) audio segment(s) (e.g., add indicators such as specific alarms or beeps to the audio segment(s), add visual indicators to the corresponding video portion of a video clip, etc.), and then transmit the audio data (or video data) to another computer module for further processing.

[0077] Figure 6A process 600 for performing audio data classification according to various embodiments of the present disclosure is illustrated. In some embodiments, at least a portion of process 600 may be performed by content analysis module 132. Process 600 begins by dividing audio content (in step 605) into speech portions and background portions, and segmenting these portions (in step 610) into speech segments and background segments. For example, analysis manager 202 may use content segmentation module 204 to segment audio content 302 into different audio tracks (e.g., foreground speech track 312 and background audio track 322). In some embodiments, analysis manager 202 may also perform one or more processes to enhance audio tracks 312 and 322 to generate automatically enhanced audio tracks 314 and 324, respectively. Analysis manager 202 may then segment each audio track 314 and 324 using segmentation module 206. For example, segmentation module 206 may segment audio track 314 into speech segments 316 and audio track 324 into background segments 326 based on duration criteria.

[0078] Then, process 600 extracts features from each segment (in step 615) using different models and incorporates temporal elements into these features (in step 620). For example, analysis manager 202 can use feature extraction module 208 to extract features from each speech segment 316 and from each background segment 326. In some embodiments, feature extraction module 208 can use different models to extract features from the same segment. For example, feature extraction module 208 can use machine learning model 402 (e.g., MFCC model) to extract the time-frequency representation of sound from each speech segment 316. Feature extraction module 208 can use machine learning model 404 (e.g., Wav2Vec2 model) to extract word features from each speech segment 316. Feature extraction module 208 can use machine learning model 406 (e.g., Mel spectrogram model) to extract the time-frequency representation of sound from each background segment 326. Feature extraction module 208 can use machine learning model 404 (e.g., Wav2Vec2 model) to extract word features from each background segment 326.

[0079] Process 600 incorporates temporal elements (in step 620) into the features. For example, analysis manager 202 may provide features associated with segments one at a time in chronological order to a recurrent neural network (e.g., GRU 502a, 502b, 502c, and 502d). Based on the features associated with the segments, the recurrent neural network may incorporate temporal context into the features of these segments based on the features of previous segments.

[0080] Then, process 600 determines (in step 625) one or more correlations between the speech segment and the background segment, and verifies (in step 630) the occurrence of an event based on the one or more correlations. For example, analysis manager 202 can use correlation module 210 to determine one or more correlations between different audio tracks 312 and 322. Specifically, correlation module 210 can provide features associated with different audio tracks to cross-attention modules (e.g., cross-attention modules 504a, 504b, 504c, and 504d). Correlation module 210 can designate different audio tracks as primary and secondary tracks. Cross-attention modules can use features associated with secondary tracks to modify features associated with primary tracks. For example, if cross-attention modules determine that features associated with secondary tracks are consistent with one or more features of primary tracks (e.g., support one or more features of primary tracks), cross-attention modules can enhance those features. On the other hand, if cross-attention modules determine that features associated with secondary tracks are inconsistent with one or more features of primary tracks (e.g., contradictory), cross-attention modules can weaken those features. Analysis Manager 202 can repeat the correlation process by switching between the main audio track and the sub-audio track.

[0081] In some embodiments, the analysis manager 202 may detect the occurrence of an event based on one or more features from an audio track (e.g., background audio track 322). If the correlation module 210 determines that there is one or more correlations between a feature from another audio track (e.g., foreground speech track 312) and the event, the analysis manager 202 may verify the occurrence of the event.

[0082] Then, process 600 categorizes the audio content based on the verified events (in step 635) and performs actions on the audio content based on the categorization (in step 640). For example, analysis manager 202 may categorize audio content 302 based on a set of criteria (e.g., determining whether audio content 302 includes prohibited audio content, etc.). If analysis manager 202 determines that audio content 302 meets the set of criteria, analysis manager 202 may publish audio content 302 to a site associated with service provider server 130 or merchant server 120. On the other hand, if analysis manager 202 determines that audio content 302 does not meet the set of criteria, analysis manager 202 may modify at least a portion of audio content 302 (e.g., removing one or more segments that do not meet the set of criteria, marking one or more segments that do not meet the set of criteria, etc.) and / or provide the modified audio content to another module for further processing.

[0083] Figure 7An example artificial neural network 700 that can be used to implement machine learning models is shown, such as machine learning models 402, 404, and 406, GRUs 502a, 502b, 502c, and 502d, cross-attention modules 504a, 504b, 504c, and 504d, and classification modules 506 and 508, etc. As shown, the artificial neural network 700 includes three layers—an input layer 702, a hidden layer 704, and an output layer 706. Each layer 702, 704, and 706 may include one or more nodes (also called “neurons”). For example, the input layer 702 includes nodes 732, 734, 736, 738, 740, and 742, the hidden layer 704 includes nodes 744, 746, and 748, and the output layer 706 includes node 750. In this example, each node in a layer is connected to each node in the adjacent layer via an edge, and each edge is typically associated with an adjustable weight. For example, node 732 in input layer 702 is connected to all nodes 744, 746, and 748 in hidden layer 704. Similarly, node 744 in the hidden layer is connected to all nodes 732, 734, 736, 738, 740, and 742 in input layer 702 and node 750 in output layer 706. While in this example each node in each layer is fully connected to nodes in one or more adjacent layers for illustrative purposes only, it is conceivable that nodes in different layers could be connected according to any other neural network topology required to perform the corresponding task.

[0084] Hidden layer 704 is an intermediate layer between the input layer 702 and the output layer 706 of the artificial neural network 700. While only one hidden layer is shown for illustrative purposes, it is conceivable that the artificial neural network 700 used to implement any computer-based model may include any number of hidden layers as desired. Hidden layer 704 is configured to extract and transform the input data received from the input layer 702 through a series of weighted computations and activation functions.

[0085] In this example, the artificial neural network 700 receives a set of inputs and produces an output. Each node in the input layer 702 can correspond to a different input. For example, when using the artificial neural network 700 to implement any of the machine learning models 402, 404, and 406, the nodes in the input layer 702 can correspond to different audio signals from an audio segment. When using the artificial neural network 700 to implement any of the GRUs 502a, 502b, 502c, and 502d, the nodes in the input layer 702 can correspond to different features extracted from the audio segment (e.g., different embedding vectors). When using the artificial neural network 700 to implement any of the cross-attention modules 504a, 504b, 504c, and 504d, the nodes in the input layer 702 can correspond to different features extracted from the audio segment (e.g., different embedding vectors). When using the artificial neural network 700 to implement any of the classification modules 506 and 508, the nodes in the input layer 702 can correspond to different mixed features (e.g., different embedding vectors).

[0086] In some examples, each node 744, 746, and 748 in hidden layer 704 generates a representation that may include mathematical computation (or algorithm) to produce values ​​based on input values ​​received from nodes 732, 734, 736, 738, 740, and 742. This mathematical computation may include assigning different weights (e.g., node weights, edge weights, etc.) to each data value received from nodes 732, 734, 736, 738, 740, and 742, performing a weighted summation of the input based on the weights assigned to each connection (e.g., each edge), and then applying the activation function associated with that corresponding node (or neuron) to the result. Nodes 744, 746, and 748 may include different algorithms (e.g., different activation functions) and / or different weights assigned to data variables from nodes 732, 734, 736, 738, 740, and 742, such that each of nodes 744, 746, and 748 can produce different values ​​based on the same input values ​​received from nodes 732, 734, 736, 738, 740, and 742. Activation functions may be the same or different across different layers. Example activation functions include, but are not limited to, the sigmoid function, the hyperbolic tangent function, the Rectified Linear Unit (ReLU), the Leaky ReLU, the Softmax transfer function, etc. In this way, after passing through multiple hidden layers, the input data received at input layer 702 is transformed into considerably different values ​​that indicate data characteristics corresponding to the task that the artificial neural network 700 is designed to perform.

[0087] In some examples, the weights of the input values ​​initially assigned to each of nodes 744, 746, and 748 can be randomly generated (e.g., using a computer randomizer). The values ​​generated by nodes 744, 746, and 748 can be used by node 750 in the output layer 706 to produce output values ​​for the artificial neural network 700 (e.g., responses to user queries, embedding vectors, classification predictions, etc.). The number of nodes in the output layer depends on the nature of the task being solved. For example, in a binary classification problem, the output layer may consist of a single node representing the probability of belonging to a class. In a multi-class classification problem, the output layer may have multiple nodes, each representing the probability of belonging to a specific class.

[0088] When using artificial neural network 700 to implement any of machine learning models 402, 404, and 406, output node 750 can be configured to generate features for different audio segments. When using artificial neural network 700 to implement any of GRU502a, 502b, 502c, and 502d, output node 750 can be configured to generate time-dependent features based on features extracted from sequential audio segments. When using artificial neural network 700 to implement any of cross-attention modules 504a, 504b, 504c, and 504d, output node 750 can be configured to generate context-aware features. When using artificial neural network 700 to implement any of classification modules 506 and 508, output node 750 can be configured to generate binary classification (or classification scores).

[0089] In some examples, the artificial neural network 700 can be implemented on one or more hardware processors, such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), FPGA (Field-Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), dedicated AI accelerators such as TPU (Tensor Processing Unit), and dedicated hardware accelerators specifically designed for the neural network computations described herein. Example-specific hardware for the neural network architecture may include, but is not limited to, Google Edge TPU, Deep Learning Accelerator (DLA), NVIDIA AI-specific GPUs, etc. The hardware used to implement the neural network architecture is specifically configured based on factors such as the complexity of the neural network, the scale of the task (e.g., training time, input data size, training dataset size, etc.), and the desired performance.

[0090] Artificial neural network 700 can be trained using training data based on one or more loss functions and one or more hyperparameters. By iteratively training the artificial neural network 700 using the training data via a feedback mechanism (e.g., comparing the output of the artificial neural network 700 with the expected output (also called the "true value" or "label")), the parameters of the artificial neural network 700 (e.g., weights, bias parameters, coefficients in the activation function, etc.) can be adjusted to achieve a goal based on one or more loss functions and one or more hyperparameters, thereby producing an optimal output in output layer 706 that minimizes the loss in the loss function. Given a loss, the negative gradient of the loss function is computed individually with respect to each weight in each layer. This negative gradient is computed iteratively layer by layer, from the last layer (e.g., output layer 706 of the artificial neural network 700) to input layer 702. These gradients quantify the sensitivity of the network output to changes in the parameters. Applying the chain rule, these gradients are computed efficiently by propagating the gradients backward from output layer 706 to input layer 702.

[0091] The parameters of the artificial neural network 700 are updated backward (through backpropagation) from the last layer to the input layer based on the negative gradient calculated using an optimization algorithm to minimize the loss. Backpropagation from the last layer (e.g., output layer 706) to the input layer 702 can be performed on multiple training samples over multiple training iterations. In this way, the parameters of the artificial neural network 700 can be gradually updated in directions that result in smaller or minimized loss, indicating that the artificial neural network 700 has been trained to generate predicted output values ​​that are closer to the target output value and have higher prediction accuracy. Training can continue until stopping criteria are met, such as reaching the maximum number of epochs or achieving satisfactory performance on validation data. At this point, the trained network can be used to make predictions on new, unseen data, such as classifying audio data. In some embodiments, the entire framework (e.g., content analysis module 132) can be collectively trained using previously classified audio data.

[0092] Figure 8 This is a block diagram of a computer system 800 suitable for implementing one or more embodiments of the present disclosure, including a service provider server 130, a merchant server 120, and user devices 110, 180, and 190. In various implementations, each of the user devices 110, 180, and 190 may include a mobile cellular phone, a personal computer (PC), a laptop computer, a wearable computing device, etc., adapted for wireless communication, and each of the service provider server 130 and the merchant server 120 may include a network computing device, such as a server. Therefore, it should be understood that devices 110, 120, 130, 180, and 190 can be implemented as computer system 800 in the following manner.

[0093] Computer system 800 includes a bus 812 or other communication mechanisms for transmitting information, data, signals, and information between various components of computer system 800. These components include an input / output (I / O) component 804 that processes user (i.e., sender, receiver, service provider) actions, such as selecting keys from a keypad / keyboard, selecting one or more buttons or links, and sending corresponding signals to bus 812. I / O component 804 may also include output components, such as a display 802 and a cursor controller 808 (e.g., keyboard, keypad, mouse, etc.). Display 802 may be configured to display a login page for logging into a user account or a checkout page for purchasing items from a merchant. Optional audio input / output component 806 may also be included to allow the user to input information using voice by converting audio signals. Audio I / O component 806 allows the user to hear audio. A transceiver or network interface 820 sends and receives signals between computer system 800 and other devices (e.g., another user device, merchant server, or service provider server) via network 822. In one embodiment, the transmission is wireless, but other transmission media and methods may also be suitable. Processor 814, which may be a microcontroller, digital signal processor (DSP), or other processing component, processes these various signals, for example, for display on computer system 800 or for transmission to other devices via communication link 824. Processor 814 can also control the transmission of information (such as cookies or IP addresses) to other devices.

[0094] The computer system 800 also includes a system memory component 810 (e.g., RAM), a static storage component 816 (e.g., ROM), and / or a disk drive 818 (e.g., a solid-state drive, hard disk drive). The computer system 800 performs specific operations by executing one or more sequences of instructions contained in the system memory component 810 through a processor 814 and other components. For example, the processor 814 may perform the audio data classification function described herein, for example, according to process 600.

[0095] The logic can be encoded in a computer-readable medium, which can refer to any medium that participates in providing instructions to the processor 814 for execution. Such a medium can take many forms, including but not limited to non-volatile media, volatile media, and transmission media. In various implementations, non-volatile media include optical discs or magnetic disks, volatile media include dynamic memory, such as system memory component 810, and transmission media include coaxial cables, copper wires, and optical fibers, including conductors forming bus 812. In one embodiment, the logic is encoded in a non-transitory computer-readable medium. In one example, the transmission medium can take the form of sound waves or light waves, such as those generated during radio, optical, and infrared data communications.

[0096] Some common forms of computer-readable media include, for example, floppy disks, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cassette tapes, or any other media that a computer is adapted to read from.

[0097] In various embodiments of this disclosure, the execution of a sequence of instructions for practicing this disclosure may be performed by a computer system 800. In various other embodiments of this disclosure, multiple computer systems 800 coupled to a network (e.g., such as a LAN, WLAN, PTSN, and / or various other wired or wireless networks, including telecommunications, mobile, and cellular telephone networks) via a communication link 824 may coordinate with each other to execute the sequence of instructions to practice this disclosure.

[0098] Where applicable, the various embodiments provided in this disclosure may be implemented using hardware, software, or a combination of hardware and software. Furthermore, where applicable, the various hardware and / or software components described herein may be combined into composite components comprising software, hardware, and / or both, without departing from the spirit of this disclosure. Where applicable, the various hardware and / or software components described herein may be separated into sub-components comprising software, hardware, or both, without departing from the scope of this disclosure. Furthermore, where applicable, it is contemplated that a software component may be implemented as a hardware component, and vice versa.

[0099] The software according to this disclosure, such as program code and / or data, may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general-purpose or special-purpose computers and / or computer systems (networked and / or otherwise). Where applicable, the order of the various steps described herein may be changed, combined into compound steps, and / or separated into sub-steps to provide the features described herein.

[0100] The various features and steps described herein can be implemented as a system including one or more memories storing the various information described herein and one or more processors coupled to one or more memories and a network, wherein the one or more processors are operable to perform the steps described herein; implemented as a non-transitory machine-readable medium including a plurality of machine-readable instructions, which, when executed by one or more processors, are adapted to cause one or more processors to perform a method including the steps described herein; and methods executed by one or more devices such as hardware processors, user equipment, servers, and other devices described herein.

Claims

1. A system comprising: Non-transitory memory; as well as One or more hardware processors, coupled to the non-transitory memory and configured to read instructions from the non-transitory memory to cause the system to perform operations, said operations including: The audio content is divided into a speech portion and a background portion; Speech features are extracted from the speech portion, and background features are extracted from the background portion; Based on the speech features and the background features, determine one or more correlations between the speech portion of the audio content and the background portion of the audio content; and The audio content is classified based on one or more of the aforementioned correlations.

2. The system according to claim 1, wherein, The classification includes classifying the audio content into a first audio type based on the one or more correlations, and wherein the operation further includes: The signal indicating the first audio type is incorporated into the audio content.

3. The system according to claim 1, wherein, The classification includes determining that a first segment of the audio content includes audio data corresponding to a first audio type, and the operation further includes: The first segment of the audio content is modified based on the first audio type.

4. The system according to claim 3, wherein, The modification includes removing the first segment from the audio content.

5. The system according to claim 1, wherein, The operation also includes: Enhance the speech portion and the background portion.

6. The system according to claim 1, wherein, The operation also includes: The speech portion is segmented into multiple speech segments; and The background portion is segmented into multiple background segments, wherein determining the one or more relevances includes determining a corresponding relevance score between a first speech segment among the multiple speech segments and each corresponding background segment among the multiple background segments.

7. The system according to claim 6, wherein, The operation also includes: Based on the corresponding correlation score, the correlation between the first speech segment and the first corresponding background segment from the plurality of background segments is determined.

8. A method comprising: The audio data associated with the digital content is divided into a first audio track and a second audio track; Extract a first plurality of audio features from the first audio track, and extract a second plurality of audio features from the second audio track; Based on the first plurality of audio features and the second plurality of audio features, determine one or more correlations between the first audio track and the second audio track; as well as The digital content is classified based on one or more of the aforementioned correlations.

9. The method according to claim 8, further comprising: The occurrence of an event is determined based on the second plurality of audio features extracted from the second audio track, wherein the one or more correlations indicate that one or more features from the first plurality of audio features are consistent with the occurrence of the event.

10. The method according to claim 9, wherein, The classification of the digital content is based on the occurrence of the event.

11. The method of claim 8, further comprising: Using a gated recursive unit (GRU), time information is incorporated into the first plurality of audio features.

12. The method according to claim 8, wherein, The first plurality of audio features include at least one of word features, emotion features, or tone features.

13. The method according to claim 8, wherein, Extracting the first plurality of audio features from the first portion of the audio data includes: Using a first machine learning model, a first portion of the first plurality of audio features is extracted from the first audio track; and A second part of the first plurality of audio features is extracted from the first audio track using a second machine learning model that is different from the first machine learning model.

14. The method of claim 8, further comprising: Divide the first audio track into a first plurality of audio segments; as well as The second audio track is segmented into a second plurality of audio segments, wherein determining the one or more correlations includes determining a corresponding correlation score between a first audio segment in the first plurality of audio segments and each corresponding audio segment in the second plurality of audio segments.

15. A non-transitory machine-readable medium storing machine-readable instructions thereon, the instructions being executable to cause a machine to perform operations, the operations including: The audio data is divided into a first part and a second part; Extract a first plurality of audio features from the first portion of the audio data, and extract a second plurality of audio features from the second portion of the audio data; Compare the first plurality of audio features with the second plurality of audio features; Based on the comparison, one or more correlations are determined between the first portion of the audio data and the second portion of the audio. as well as The audio data is classified based on one or more of the aforementioned correlations.

16. The non-transitory machine-readable medium according to claim 15, wherein, The operation also includes: The first portion of the audio data is segmented into a first plurality of audio segments; and The second portion of the audio data is segmented into a second plurality of audio segments, wherein determining the one or more correlations includes determining a corresponding correlation score between a first audio segment in the first plurality of audio segments and each corresponding audio segment in the second plurality of audio segments.

17. The non-transitory machine-readable medium according to claim 16, wherein, The operation also includes: Based on the corresponding correlation score, the correlation between the first audio segment and a specific corresponding audio segment from the second plurality of audio segments is determined.

18. The non-transitory machine-readable medium according to claim 17, wherein, The operation also includes: The occurrence of an event is detected based on one or more features extracted from the specific corresponding audio segment; and The event is classified based on the first audio segment and the correlation between the first audio segment and the specific corresponding segment, wherein the classification of the audio data is also based on the classification of the event.

19. The non-transitory machine-readable medium according to claim 15, wherein, The operation also includes: Based on one or more other audio features among the first plurality of audio features, the corresponding time information is merged into each of the first plurality of audio features.

20. The non-transitory machine-readable medium according to claim 15, wherein, The first plurality of audio features include at least one of text features, emotion features, or tone features.

Images