Audio detection method and device, nonvolatile storage medium and computer equipment

Abstract

The invention discloses an audio detection method and device, a nonvolatile storage medium and computer equipment. The method comprises the following steps: acquiring audio data; converting the audio data into a text sequence with a time identifier; based on the text sequence, determining a semantic triple; inputting the semantic triple into a target support vector machine model for first behavior prediction, and generating a first behavior corresponding to the audio data; the semantic triad is matched with multiple second behavior vectors in a preset second behavior vector library, second behaviors corresponding to the audio data are determined, and the multiple second behavior vectors represent feature vectors corresponding to the multiple preset second behaviors; and generating an audio detection result based on the time identifier, the first behavior and the second behavior. According to the invention, the technical problem of low detection efficiency caused by the fact that whether the behavior is compliant or not is generally judged by manually playing back the audio data during the detection of the business process based on the audio data at present is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio and video technology, in particular to an audio detection method and device, a non-volatile storage medium and a computer device. BACKGROUND

[0002] In the business operations of banks and financial institutions, in order to ensure the compliance and authenticity of the customer signing process, it is usually necessary to record the audio and video materials of the entire signing process. Currently, the review of these audio and video materials mainly relies on manual review, that is, the reviewer needs to play back the video segment by segment and judge whether the key behaviors in it meet the regulations, such as customer identity verification, risk notification, customer willingness confirmation, etc. However, this traditional manual review method has several significant shortcomings:

[0003] 1. Low efficiency: manual review of signing audio and video materials is time-consuming, especially in cases of high business volume, the reviewer needs to spend a lot of time and effort to watch the complete video, which greatly affects the speed of business processing.

[0004] 2. Subjectivity and inconsistency: the experience and judgment standard of the reviewer may vary from person to person, resulting in subjectivity and inconsistency of the review results. Such differences not only may cause compliance risks, but also affect customer experience and service quality.

[0005] 3. Difficulty in tracing and explaining: manual review is difficult to form a systematic and traceable review record, when there is a dispute or needs to be reviewed, it is difficult to provide a clear behavior chain and explanation, reducing the transparency and auditability of the business process.

[0006] In view of the above problems, no effective solution has been proposed so far. SUMMARY

[0007] The embodiments of the present application provide an audio detection method, device, non-volatile storage medium and computer device to at least solve the technical problem that the current detection of business process based on audio data usually judges whether the behavior is compliant by manually playing back the audio data, resulting in low detection efficiency.

[0008] According to one aspect of the present invention, an audio detection method is provided, comprising: acquiring audio data; converting the audio data into a text sequence with time stamps; determining semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; inputting the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine a second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent feature vectors corresponding to each of the preset multiple second behaviors; and generating an audio detection result based on the time stamps, the first behavior, and the second behavior.

[0009] Optionally, converting audio data into a text sequence with time stamps includes: performing noise reduction on the audio data to obtain noise-reduced audio data; segmenting the audio data based on a preset duration to obtain multiple audio segments; inputting the multiple audio segments into a preset speech recognition model to obtain the text corresponding to each of the multiple audio segments; and processing the text corresponding to each of the multiple audio segments based on a time classification alignment algorithm to obtain a text sequence with time stamps.

[0010] Optionally, based on the text sequence, determining semantic triples includes: performing word segmentation on the text sequence to obtain multiple words; inputting the multiple words into a preset semantic relation extraction model to determine semantic triples, wherein the semantic triples include a head entity, a relation, and a tail entity.

[0011] Optionally, multiple first-row samples are obtained; the multiple first-row samples are converted into sample vectors corresponding to each of the multiple first-row samples; the original support vector machine model is trained using the sample vectors corresponding to each of the multiple first-row samples to obtain the target support vector machine model.

[0012] Optionally, the semantic triples are matched with multiple second-behavior vectors in a preset second-behavior vector library to determine the second-behavior corresponding to the audio data, including: converting the semantic triples into corresponding vectors; calculating the similarity between the vectors corresponding to the semantic triples and multiple second-behavior vectors in the second-behavior vector library based on a weighted cosine similarity algorithm to obtain the similarity of each of the multiple second-behavior vectors; and determining the second-behavior corresponding to the audio data based on the similarity of each of the multiple second-behavior vectors.

[0013] Optionally, when the semantic triple includes a head entity, a relation, and a tail entity, the audio detection result is generated based on the time identifier, the first action, and the second action, including: generating structural detection text based on the time identifier, the first action, and the second action; determining the head entity and tail entity in the audio data based on the semantic triple, wherein the head entity represents the relation initiator of the event in the audio data, and the tail entity represents the relation receiver of the event in the audio data; removing duplicate head entities and tail entities from the audio data to obtain the target head entity and the target tail entity; determining the target node based on the target head entity and the target tail entity; generating the target edge based on the relation in the semantic triple; generating a knowledge graph based on the target node and the target edge; and determining the audio detection result based on the structural detection text and the knowledge graph.

[0014] According to another aspect of the present invention, an audio detection device is also provided, comprising: an acquisition module for acquiring audio data; a conversion module for converting the audio data into a text sequence with time stamps; a determination module for determining semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; an input module for inputting the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; a matching module for matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine a second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent feature vectors corresponding to each of the preset multiple second behaviors; and a generation module for generating an audio detection result based on the time stamps, the first behavior, and the second behavior.

[0015] According to another aspect of the present invention, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored program, wherein, when the program is running, it controls the device where the non-volatile storage medium is located to execute any of the above-described audio detection methods.

[0016] According to another aspect of the present invention, a computer device is also provided, the computer device including a processor, the processor being configured to run a program, wherein the program executes any of the above-described audio detection methods during runtime.

[0017] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements any of the above-described audio detection methods.

[0018] In this embodiment of the invention, an audio detection method is employed. This involves acquiring audio data; converting the audio data into a text sequence with time stamps; determining semantic triples based on the text sequence, where each semantic triple represents an event occurring in the audio data; inputting the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine the second behavior corresponding to the audio data, where each of the multiple second behavior vectors represents a feature vector corresponding to a preset second behavior; and generating an audio detection result based on the time stamp, the first behavior, and the second behavior. This achieves the goal of automatically identifying and detecting behaviors in audio data, thereby improving detection efficiency. Furthermore, it solves the current technical problem of low detection efficiency in business process detection based on audio data, which typically relies on manual playback of audio data to determine compliance. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0020] Figure 1 A hardware structure block diagram of a computer terminal for implementing an audio detection method is shown.

[0021] Figure 2 This is a flowchart illustrating the audio detection method provided according to an embodiment of the present invention;

[0022] Figure 3 This is a structural block diagram of an audio detection device provided according to an embodiment of the present invention. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0026] Signal-to-noise ratio (SNR): The ratio of the useful signal power to the background noise power in an audio signal, usually expressed in decibels (dB). A higher SNR indicates better audio quality and lower background noise interference.

[0027] According to an embodiment of the present invention, an embodiment of an audio detection method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing an audio detection method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1The different configurations shown.

[0029] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0030] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio detection method in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the audio detection method of the aforementioned application. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0031] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0032] Figure 2 This is a flowchart illustrating the audio detection method provided according to an embodiment of the present invention, as shown below. Figure 2 As shown, the method includes the following steps:

[0033] Step S202: Obtain audio data.

[0034] In this step, the audio data can be the bank's signing audio. The original signing audio signal can be received, and signal enhancement can be performed using techniques such as frequency domain noise spectrum estimation to output high-quality, noise-reduced audio data.

[0035] Step S204: Convert the audio data into a text sequence with time stamps.

[0036] In this step, audio data typically needs to be preprocessed before speech recognition to improve accuracy and robustness. Preprocessing steps may include noise reduction, gain control, and audio segmentation. Then, features are extracted from the audio data. These features reflect the essential information of the speech while ignoring irrelevant noise or other interference. Commonly used feature extraction methods include:

[0037] Mel frequency cepstral coefficients: Convert audio signals to the Mel frequency scale and extract coefficients that reflect the frequency characteristics of speech.

[0038] Spectrum extraction: Converts audio signals into a frequency domain representation, showing how the frequency distribution of the signal changes over time.

[0039] Then, end-to-end ASR models can be used, such as deep learning-based RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory Network), GRU (Gated Recurrent Unit), or Transformer architecture models. These models can learn directly from audio features to text output without requiring manual feature design. Connectionist Temporal Classification (CTC) or other sequence-to-sequence alignment methods can be used to ensure that the recognized text corresponds to the speech in the audio in time. After the ASR model recognizes the text, timestamp marking is a crucial step in accurately matching the recognition result with the time points in the audio. Typically, the ASR model provides the start and end times of each word or phoneme when outputting the text sequence. This generates a text sequence with temporal information.

[0040] Specifically, a noise reduction algorithm based on frequency domain noise spectrum estimation can be used to enhance the original contracted audio signal, eliminating environmental noise and device background noise, improving the audio signal-to-noise ratio, and ensuring speech recognition accuracy. Then, based on an end-to-end ASR model with the Transformer architecture, a CTC alignment strategy is used to achieve forced alignment between audio and text. The preprocessed audio signal is then feature-extracted and converted into a timestamped text sequence, providing high-precision structured input for subsequent semantic analysis.

[0041] Through the steps described above, audio data is effectively converted into a time-stamped text sequence. This format provides an easy-to-manage and understandable data form for subsequent applications such as natural language processing, audio analysis, and compliance checks. This technology has wide applications in areas such as meeting minutes, customer service conversation recordings, court audio transcription, and educational recording systems.

[0042] Step S206: Based on the text sequence, determine semantic triples, where semantic triples represent events occurring in the audio data.

[0043] In this step, semantic triples are typically expressed in the form of "(head entity, relation, tail entity)," but in audio data scenarios, time stamps can also be added to clarify the time point of the event. The text sequence can be decomposed into words or phrases. Each word is labeled with its part of speech in the sentence, such as noun, verb, adjective, etc. Entities in the text, such as names of people, places, and organizations, are identified and categorized. Keywords representing specific events in the text are found; these words can trigger the recognition of an event, such as "signing," "display," and "confirmation," which are particularly important in banking scenarios. The context of the triggering words is determined to understand the specific meaning of the event and the participants. Entities related to the event are identified and their roles are labeled, such as "customer" and "bank representative" in a signing event. Syntactic analysis tools can be used to understand the relationships between entities, and then the relationships between entities are extracted from the results of preprocessing and context analysis to form semantic triples. Time expression recognition technology can be used to extract the timestamps of events from the text sequence and add time elements to the triples.

[0044] For example, in a bank signing scenario, the extracted semantic triple could be ("customer", "display", "ID card"), which means that Mr. Zhang, the customer, displayed his ID card at a certain point in time.

[0045] Through the above steps, key events can be automatically identified and structurally represented from the text sequences transcribed from audio data in scenarios such as bank signing, including processes such as customers showing their ID cards, bank representatives confirming customer identities, and signing contracts. This is extremely important for applications such as automated compliance checks, event tracing, and quality control.

[0046] Step S208: Input the semantic triples into the target support vector machine model to predict the first action and generate the first action corresponding to the audio data.

[0047] In this step, semantic triples can be converted into vectors. This is often achieved using embedding models, such as pre-trained language models like BERT, Word2Vec, or GloVe, to convert "entity 1," "relation," and "entity 2" in the triple into vector representations. Then, the vectors of the three entities are combined in some form to form a vector representing the entire semantic triple. This can include simple vector concatenation or more advanced operations, such as weighted summation or fusion using neural networks, to better capture the semantic relationships within the triples. Next, the combined vector is fed as input into a pre-trained SVM model, i.e., a target support vector machine model. This model should be trained on a large amount of labeled semantic triple data and be able to predict the behavior category represented by the triple based on the features of the input vector.

[0048] The SVM model classifies the input vector and outputs a category label, i.e., the behavior type, thus determining the first behavior appearing in the audio data. This first behavior can be the most important behavior, and the specific type can be set according to the actual situation. For example, the model might predict that "a customer showing their ID card" is a compliance behavior, while "a bank representative asking about income" is another behavior category, and so on. Based on the SVM model's prediction results, combined with the actual business scenario and known behavior categories, the model's accuracy and reliability are evaluated.

[0049] For example, consider a semantic triple in a banking scenario: "(Zhang San, sign, loan contract)". Inputting the combined vector corresponding to this semantic triple into an SVM model, the model might predict that this is a "signing behavior" with a high degree of compliance. Subsequently, based on the prediction result, the system automatically generates a record with a specific timestamp, indicating that within a certain time period, customer Zhang San completed the loan contract signing process, complying with the bank's compliance requirements.

[0050] In this way, the SVM model can serve as a powerful tool for behavior classification and prediction, helping financial institutions and other entities to automate and standardize the signing process, greatly improving the efficiency and compliance of business operations.

[0051] Step S210: Match the semantic triple with multiple second behavior vectors in the preset second behavior vector library to determine the second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent the feature vectors corresponding to the preset multiple second behaviors.

[0052] In this step, the second behavior vector library contains vector representations of a predefined series of second behaviors, or auxiliary behaviors. These behaviors may be non-core but important details in the signing process, such as "the customer confirms understanding of the terms" or "the bank representative smiles." While not as critical as core behaviors (such as "signing the contract"), they have a significant impact on assessing the overall atmosphere of the conversation and the customer's level of understanding. Each second behavior vector is represented by the vector average or cluster center of related behavior instances.

[0053] For each semantic triple vector, its similarity to all vectors in the second row vector library can be calculated. This is typically done using metrics such as cosine similarity or Euclidean distance. Cosine similarity is a measure of the similarity between two vectors in terms of angles within a vector space, and it is particularly suitable for high-dimensional data. A matching threshold can be set, usually a value between 0 and 1. If the similarity between a semantic triple vector and any second row vector exceeds this threshold, it can be considered to have matched the corresponding second row.

[0054] Step S212: Generate audio detection results based on time markers, the first line, and the second line.

[0055] In this step, a timeline can be created that covers the entire length of the audio file. Based on this, the timestamps obtained from speech recognition and behavior analysis are used to accurately mark the occurrence time of each first and second behavior. Event lists are then created for the first and second behaviors, with each list containing elements such as the behavior type, a detailed description of the occurrence (e.g., the original text of the semantic triple), a timestamp, and related entity information. For example, the first behavior list might include events such as "customer showing ID card" or "bank representative explaining terms"; while the second behavior list might include more detailed behaviors such as "customer inquiring about additional fees" or "representative confirming customer understanding."

[0056] Integrate the first and second actions chronologically onto a single timeline to form a coherent sequence of behavioral events. This step can use data structures such as lists, dictionaries, or specially designed data models to store and represent the details of the behavioral events. Verify that each behavioral event meets the bank's or financial institution's compliance standards. For the first action, focus on whether it was executed completely and correctly; for the second action, assess whether it contributes to process flow and customer experience, and whether it is relevant to compliance. In addition to routine behavioral checks, identify any unexpected actions or process interruptions that may indicate anomalies in the recording, such as a customer suddenly ending the conversation or a representative failing to clearly explain the terms.

[0057] Based on the above analysis, behavior detection reports can be generated, enabling decision-makers to quickly obtain key information about behavioral events in audio, including compliance status and potential areas for improvement, providing a solid data foundation for subsequent approval or corrective actions.

[0058] Through the above steps, the goal of automatically identifying and detecting behaviors in audio data can be achieved, thereby improving the technical effect of detection efficiency. This solves the technical problem that the current detection of business processes based on audio data usually relies on manual playback of audio data to determine whether the behavior is compliant, resulting in low detection efficiency.

[0059] As an optional embodiment, converting audio data into a text sequence with time stamps includes: performing noise reduction processing on the audio data to obtain noise-reduced audio data; segmenting the audio data based on a preset duration to obtain multiple audio segments; inputting the multiple audio segments into a preset speech recognition model to obtain the text corresponding to each of the multiple audio segments; and processing the text corresponding to each of the multiple audio segments based on a time classification alignment algorithm to obtain a text sequence with time stamps.

[0060] Optionally, the purpose of audio denoising is to remove background noise, electronic noise, or other interference that may occur during the recording process, making subsequent speech recognition more accurate. Denoising can be achieved through various methods, commonly including frequency domain filtering, deep learning models (such as denoising networks based on RNNs or LSTMs), and noise threshold adjustment. The spectral characteristics of the audio can be analyzed to identify noise frequencies. Based on the identified noise frequencies, frequency domain filters are applied to remove noise components. A reasonable noise threshold is set; sounds below the threshold are considered noise and suppressed. Then, based on the audio characteristics and model performance, an appropriate segmentation duration is selected. The entire audio segment is divided into multiple audio segments according to a preset duration. The start and end times of each audio segment are recorded for subsequent text time alignment. A suitable speech recognition model is selected, and the trained model weights are loaded. Acoustic features, such as Mel-frequency cepstral coefficients (MFCCs), are extracted from each audio segment. The features are input into the model to obtain the text transcription result.

[0061] Temporal alignment algorithms are used to ensure that the text obtained from speech recognition corresponds precisely to the original audio in time. This step is crucial for subsequent behavior analysis because temporal information helps to accurately locate specific events. Timestamps can be added to the start and end times of each audio segment. The recognized text segments are then associated with their corresponding timestamps to form a time-stamped text sequence. Considering potential delays or errors in speech recognition, the temporal alignment results need to be fine-tuned to ensure the temporal localization of text segments is as accurate as possible.

[0062] The audio data undergoes noise reduction processing to obtain denoised audio data, ensuring a high signal-to-noise ratio. Based on a preset duration, the audio data is segmented into multiple audio segments for subsequent precise processing and analysis. These segments are then input into a pre-defined speech recognition model to obtain the corresponding text, achieving speech-to-text conversion. Finally, a time-based classification and alignment algorithm is used to process the text corresponding to each audio segment, resulting in a text sequence with time stamps. This ensures the temporal correspondence between the text and the original audio, providing accurate temporal location information for subsequent semantic analysis and behavior recognition. This series of steps aims to efficiently and accurately extract dialogue content from the original audio and video while preserving the temporal clues of the actions, providing a solid foundation for further intelligent quality inspection and ensuring the efficiency and accuracy of the review process.

[0063] As an optional embodiment, determining semantic triples based on a text sequence includes: performing word segmentation on the text sequence to obtain multiple words; inputting the multiple words into a preset semantic relation extraction model to determine semantic triples, wherein the semantic triples include a head entity, a relation, and a tail entity.

[0064] Optionally, the text sequence is segmented to obtain a series of words. These words are then used as input to a pre-defined semantic relation extraction model. This model automatically parses the logical relationships between texts, identifying the head entity, relation, and tail entity—forming structured semantic triples. This process not only improves the accuracy of understanding the signing dialogue content but also provides crucial data support for subsequent behavior recognition and knowledge graph construction. By converting the signing dialogue into standardized semantic triples, the system can more accurately identify key behaviors, achieving refined and automated review of the signing process, significantly improving review efficiency and compliance. Furthermore, this method can effectively identify auxiliary behaviors, further enriching the descriptive dimensions of the signing process and providing financial institutions with more comprehensive review criteria.

[0065] As an optional embodiment, multiple first rows of samples are obtained; the multiple first rows of samples are converted into sample vectors corresponding to each of the multiple first rows of samples; the original support vector machine model is trained using the sample vectors corresponding to each of the multiple first rows of samples to obtain the target support vector machine model.

[0066] Optionally, multiple first-behavior samples representing key behaviors are obtained, encompassing instances of various critical behaviors in real-world signing scenarios. Subsequently, text vectorization technology is applied to convert these first-behavior samples into their corresponding sample vectors. This step transforms the semantic information of the behaviors into numerical data that can be processed by a machine learning model. Finally, these sample vectors are used to iteratively train the original support vector machine model, resulting in a more accurate target support vector machine model. This improvement not only enhances the model's generalization ability, enabling it to more accurately identify diverse key behaviors, but also improves the accuracy and stability of behavior recognition by continuously optimizing model weights. Overall, this design approach effectively improves the efficiency of the intelligent quality inspection system for the signing process, ensuring the efficiency and compliance of the review process.

[0067] As an optional embodiment, the semantic triples are matched with multiple second-behavior vectors in a preset second-behavior vector library to determine the second behavior corresponding to the audio data, including: converting the semantic triples into corresponding vectors; calculating the similarity between the vectors corresponding to the semantic triples and multiple second-behavior vectors in the second-behavior vector library based on a weighted cosine similarity algorithm to obtain the similarity of each of the multiple second-behavior vectors; and determining the second behavior corresponding to the audio data based on the similarity of each of the multiple second-behavior vectors.

[0068] Optionally, by vectorizing the extracted semantic triples, a series of similarity values ​​are obtained by calculating their similarity to each vector in a pre-defined auxiliary behavior vector library using a weighted cosine similarity algorithm. Specifically, the weighted cosine similarity calculation method is used, and its formula derivation process is as follows:

[0069] The traditional cosine similarity formula is: ,in, and Let be two vectors whose similarity is to be calculated.

[0070] Let the triple vector be It consists of a head entity, a relation, and a tail entity. For a 3D matrix, the weighted cosine similarity formula is: ,in Weight matrix configured for 3x3; This represents the Frobenius norm of the matrix.

[0071] This series of similarity values ​​will be used as a criterion to determine whether the audio data contains auxiliary behaviors. This approach not only improves the accuracy and efficiency of behavior recognition but also captures more details, enhancing the understanding and description of the signing process, and playing a crucial role in comprehensively assessing contract compliance.

[0072] As an optional embodiment, when the semantic triple includes a head entity, a relation, and a tail entity, an audio detection result is generated based on a time identifier, a first action, and a second action. This includes: generating structure detection text based on the time identifier, the first action, and the second action; determining the head entity and tail entity in the audio data based on the semantic triple, wherein the head entity represents the relation initiator of an event in the audio data, and the tail entity represents the relation receiver of an event in the audio data; removing duplicate head entities and tail entities from the audio data to obtain target head entities and target tail entities; determining target nodes based on target head entities and target tail entities; generating target edges based on the relations in the semantic triple; generating a knowledge graph based on target nodes and target edges; and determining the audio detection result based on the structure detection text and the knowledge graph.

[0073] Optionally, a structured text report can be generated based on the time stamp of each audio segment and the identified first and second actions. This report should include not only a textual description of the action but also the specific time the action occurred, and whether the action was considered a compliant first action or a supplementary second action. From each semantic triple, head and tail entities are identified and extracted. The head entity typically represents the initiator of the action, while the tail entity is the object or receiver of the action. For example, in the triple "(customer, inquiry, interest rate details)", "customer" is the head entity, and "interest rate details" is the tail entity. Since the same entity may be mentioned multiple times in the audio, the extracted head and tail entities can be deduplicated to retain unique entities and reduce redundant information. The result of this step is a set of target head and tail entities. Based on the deduplicated entities, each head and tail entity is considered a node in the knowledge graph. Target nodes will include all entities involved in the signing process, such as "customer," "bank representative," "ID card," and "interest rate details." Edges are generated between target nodes in the knowledge graph based on the relationships in each semantic triple. The type of edge is determined by the relationship, such as "inquire", "show", "confirm", etc., which constructs an interaction network between entities.

[0074] By utilizing target nodes and target edges, a complete knowledge graph is constructed to display the interactions and relationships between all entities in the signing process. The knowledge graph should be visual and can be displayed through a graphical interface to facilitate understanding and analysis by reviewers.

[0075] Finally, the generated structured detection text is combined with the knowledge graph to form the final audio detection result.

[0076] Through the above steps, not only can key information be extracted from audio, but it can also be transformed into an easily understandable and analyzable format, providing strong support for intelligent quality inspection. The construction of a knowledge graph clearly reveals the logical relationships between behaviors, helping reviewers quickly grasp the overall situation and specific details of the signing process, thereby making accurate judgments.

[0077] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, because according to the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the present invention.

[0078] Through the above description of the embodiments, those skilled in the art can clearly understand that the audio detection method according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0079] According to embodiments of the present invention, an audio detection apparatus for implementing the above-described audio detection method is also provided. Figure 3 This is a structural block diagram of an audio detection device provided according to an embodiment of the present invention, such as... Figure 3 As shown, the audio detection device includes: an acquisition module 302, a conversion module 304, a determination module 306, an input module 308, a matching module 310, and a generation module 312. The audio detection device will be described below.

[0080] The acquisition module 302 is used to acquire audio data.

[0081] The conversion module 304, connected to the acquisition module 302, is used to convert audio data into a text sequence with time stamps.

[0082] The determination module 306, connected to the conversion module 304, is used to determine semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data.

[0083] The input module 308, connected to the determination module 306, is used to input the semantic triples into the target support vector machine model to predict the first behavior and generate the first behavior corresponding to the audio data.

[0084] The matching module 310, connected to the input module 308, is used to match the semantic triple with multiple second behavior vectors in a preset second behavior vector library to determine the second behavior corresponding to the audio data. The multiple second behavior vectors represent the feature vectors corresponding to the preset multiple second behaviors.

[0085] The generation module 312, connected to the matching module 310, is used to generate audio detection results based on time markers, a first line, and a second line.

[0086] It should be noted that the acquisition module 302, conversion module 304, determination module 306, input module 308, matching module 310, and generation module 312 mentioned above correspond to steps S202 to S212 in the embodiments. Multiple modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of the device, can run on the computer terminal 10 provided in the embodiments.

[0087] Embodiments of the present invention may provide a computer device. Optionally, in this embodiment, the computer device may be located in at least one of a plurality of network devices in a computer network. The computer device includes a memory and a processor.

[0088] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio detection method and apparatus in this embodiment of the invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned audio detection method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] The processor can invoke information and applications stored in the memory via a transmission device to perform the following steps: acquiring audio data; converting the audio data into a text sequence with time stamps; determining semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; inputting the semantic triples into a target support vector machine model for first-behavior prediction to generate a first-behavior corresponding to the audio data; matching the semantic triples with multiple second-behavior vectors in a preset second-behavior vector library to determine a second-behavior corresponding to the audio data, wherein the multiple second-behavior vectors represent feature vectors corresponding to each of the preset multiple second-behaviors; and generating an audio detection result based on the time stamps, the first-behavior, and the second-behavior.

[0090] Optionally, the processor may also execute program code for the following steps: converting audio data into a text sequence with time stamps, including: performing noise reduction processing on the audio data to obtain noise-reduced audio data; segmenting the audio data based on a preset duration to obtain multiple audio segments; inputting the multiple audio segments into a preset speech recognition model to obtain the text corresponding to each of the multiple audio segments; and processing the text corresponding to each of the multiple audio segments based on a time classification alignment algorithm to obtain a text sequence with time stamps.

[0091] Optionally, the processor may also execute program code for the following steps: determining semantic triples based on a text sequence, including: performing word segmentation on the text sequence to obtain multiple words; inputting the multiple words into a preset semantic relation extraction model to determine semantic triples, wherein the semantic triples include a head entity, a relation, and a tail entity.

[0092] Optionally, the processor may also execute program code that performs the following steps: acquiring multiple first-row samples; converting the multiple first-row samples into sample vectors corresponding to each of the multiple first-row samples; and training the original support vector machine model using the sample vectors corresponding to each of the multiple first-row samples to obtain the target support vector machine model.

[0093] Optionally, the processor may also execute program code that performs the following steps: matching semantic triples with multiple second-row vectors in a preset second-row vector library to determine the second-row corresponding to the audio data, including: converting semantic triples into corresponding vectors; calculating the similarity between the vectors corresponding to the semantic triples and multiple second-row vectors in the second-row vector library based on a weighted cosine similarity algorithm to obtain the similarity of each of the multiple second-row vectors; and determining the second-row corresponding to the audio data based on the similarity of each of the multiple second-row vectors.

[0094] Optionally, the processor may also execute program code with the following steps: When the semantic triple includes a head entity, a relation, and a tail entity, generate an audio detection result based on a time identifier, a first action, and a second action, including: generating structure detection text based on the time identifier, the first action, and the second action; determining the head entity and tail entity in the audio data based on the semantic triple, wherein the head entity represents the relation initiator of an event in the audio data, and the tail entity represents the relation receiver of an event in the audio data; removing duplicate head entities and tail entities from the audio data to obtain the target head entity and the target tail entity; determining the target node based on the target head entity and the target tail entity; generating the target edge based on the relation in the semantic triple; generating a knowledge graph based on the target node and the target edge; and determining the audio detection result based on the structure detection text and the knowledge graph.

[0095] This invention provides an audio detection method that involves: acquiring audio data; converting the audio data into a text sequence with time stamps; determining semantic triples based on the text sequence, where each semantic triple represents an event occurring in the audio data; inputting the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine a second behavior corresponding to the audio data, where each of the multiple second behavior vectors represents a feature vector corresponding to a preset second behavior; and generating an audio detection result based on the time stamps, the first behavior, and the second behavior. This method achieves the goal of automatically identifying and detecting behaviors in audio data, thereby improving detection efficiency and solving the problem of low detection efficiency in current audio-based business process detection, which typically relies on manual playback of audio data to determine compliance.

[0096] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a non-volatile storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0097] Embodiments of the present invention also provide a non-volatile storage medium. Optionally, in this embodiment, the non-volatile storage medium can be used to store the program code executed by the audio detection method provided in the above embodiments.

[0098] Optionally, in this embodiment, the non-volatile storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0099] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: acquiring audio data; converting the audio data into a text sequence with time stamps; determining semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; inputting the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine a second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent feature vectors corresponding to each of the preset multiple second behaviors; and generating an audio detection result based on the time stamps, the first behavior, and the second behavior.

[0100] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: converting audio data into a text sequence with time stamps, including: performing noise reduction processing on the audio data to obtain noise-reduced audio data; segmenting the audio data based on a preset duration to obtain multiple audio segments; inputting the multiple audio segments into a preset speech recognition model to obtain the text corresponding to each of the multiple audio segments; and processing the text corresponding to each of the multiple audio segments based on a time classification alignment algorithm to obtain a text sequence with time stamps.

[0101] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: determining semantic triples based on a text sequence, including: performing word segmentation on the text sequence to obtain multiple words; inputting the multiple words into a preset semantic relation extraction model to determine semantic triples, wherein the semantic triples include a head entity, a relation, and a tail entity.

[0102] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: obtaining multiple first-row samples; converting the multiple first-row samples into sample vectors corresponding to each of the multiple first-row samples; and training the original support vector machine model using the sample vectors corresponding to each of the multiple first-row samples to obtain the target support vector machine model.

[0103] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: matching semantic triples with multiple second-behavior vectors in a preset second-behavior vector library to determine the second behavior corresponding to the audio data, including: converting the semantic triples into corresponding vectors; calculating the similarity between the vectors corresponding to the semantic triples and the multiple second-behavior vectors in the second-behavior vector library based on a weighted cosine similarity algorithm to obtain the similarity of each of the multiple second-behavior vectors; and determining the second behavior corresponding to the audio data based on the similarity of each of the multiple second-behavior vectors.

[0104] Optionally, in this embodiment, the non-volatile storage medium is configured to store program code for performing the following steps: When the semantic triple includes a head entity, a relation, and a tail entity, generating an audio detection result based on a time identifier, a first action, and a second action includes: generating structure detection text based on the time identifier, the first action, and the second action; determining the head entity and tail entity in the audio data based on the semantic triple, wherein the head entity represents the relation initiator of an event in the audio data, and the tail entity represents the relation receiver of an event in the audio data; removing duplicate head entities and tail entities from the audio data to obtain target head entities and target tail entities; determining target nodes based on the target head entities and target tail entities; generating target edges based on the relations in the semantic triple; generating a knowledge graph based on the target nodes and target edges; and determining the audio detection result based on the structure detection text and the knowledge graph.

[0105] Embodiments of the present invention also provide a computer program product, including a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, it can: acquire audio data; convert the audio data into a text sequence with time stamps; determine semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; input the semantic triples into a target support vector machine model for first behavior prediction to generate a first behavior corresponding to the audio data; match the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine a second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent feature vectors corresponding to each of the preset multiple second behaviors; and generate an audio detection result based on the time stamps, the first behavior, and the second behavior.

[0106] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0107] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0108] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0109] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0110] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a non-volatile storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0112] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An audio detection method, characterized in that, include: Acquire audio data; The audio data is converted into a text sequence with time stamps; Based on the text sequence, semantic triples are determined, wherein the semantic triples represent events occurring in the audio data; The semantic triples are input into the target support vector machine model to predict the first action, thereby generating the first action corresponding to the audio data. The semantic triples are matched with multiple second behavior vectors in a preset second behavior vector library to determine the second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent the feature vectors corresponding to each of the preset multiple second behaviors; Based on the time identifier, the first action, and the second action, an audio detection result is generated.

2. The method according to claim 1, characterized in that, The step of converting the audio data into a text sequence with time stamps includes: The audio data is subjected to noise reduction processing to obtain noise-reduced audio data; Based on a preset duration, the audio data is segmented to obtain multiple audio segments; The multiple audio segments are input into a preset speech recognition model to obtain the text corresponding to each of the multiple audio segments; The text corresponding to each of the multiple audio segments is processed based on a time-based classification and alignment algorithm to obtain the text sequence with time identifiers.

3. The method according to claim 1, characterized in that, The determination of semantic triples based on the text sequence includes: The text sequence is segmented to obtain multiple words; The multiple words are input into a preset semantic relation extraction model to determine the semantic triples, wherein the semantic triples include a head entity, a relation, and a tail entity.

4. The method according to claim 1, characterized in that, Also includes: Obtain multiple first-row samples; Convert the plurality of first-row samples into sample vectors corresponding to each of the plurality of first-row samples; The original support vector machine model is trained using the sample vectors corresponding to the samples in the multiple first rows to obtain the target support vector machine model.

5. The method according to claim 1, characterized in that, The step of matching the semantic triples with multiple second behavior vectors in a preset second behavior vector library to determine the second behavior corresponding to the audio data includes: Transform the semantic triples into corresponding vectors; Based on the weighted cosine similarity algorithm, the similarity between the vector corresponding to the semantic triple and multiple second-row vectors in the second-row vector library is calculated to obtain the similarity between each of the multiple second-row vectors. The second behavior corresponding to the audio data is determined based on the similarity of each of the multiple second behavior vectors.

6. The method according to any one of claims 1 to 5, characterized in that, When the semantic triple includes a head entity, a relation, and a tail entity, an audio detection result is generated based on the time identifier, the first action, and the second action, including: Based on the time identifier, the first behavior, and the second behavior, generate structure detection text; Based on the semantic triples, the head entity and tail entity in the audio data are determined, wherein the head entity represents the relationship initiator of the event in the audio data, and the tail entity represents the relationship receiver of the event in the audio data; Remove duplicate head and tail entities from the audio data to obtain the target head entity and target tail entity; Based on the target head entity and the target tail entity, the target node is determined; Based on the relations in the semantic triples, generate the target edge; A knowledge graph is generated based on the target node and the target edge; The audio detection result is determined based on the structured text and the knowledge graph.

7. An audio detection device, characterized in that, include: The acquisition module is used to acquire audio data; A conversion module is used to convert the audio data into a text sequence with time stamps; The determining module is used to determine semantic triples based on the text sequence, wherein the semantic triples represent events occurring in the audio data; The input module is used to input the semantic triples into the target support vector machine model to predict the first behavior and generate the first behavior corresponding to the audio data; The matching module is used to match the semantic triple with multiple second behavior vectors in a preset second behavior vector library to determine the second behavior corresponding to the audio data, wherein the multiple second behavior vectors represent the feature vectors corresponding to the preset multiple second behaviors respectively; The generation module is used to generate audio detection results based on the time identifier, the first behavior, and the second behavior.

8. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the non-volatile storage medium to perform the audio detection method according to any one of claims 1 to 6.

9. A computer device, characterized in that, include: Memory and processor The memory stores computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor performs the audio detection method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the audio detection method according to any one of claims 1 to 6.

Images