Information monitoring and tracing method and system based on voiceprint and semantic analysis

Abstract

The present application relates to a kind of information monitoring and tracing method and system based on voiceprint and semantic analysis, by introducing the condition triggering mechanism of prior analysis of non-content features such as voiceprint and emotion, risk confidence threshold triggering deep semantic understanding, effectively solve the privacy invasion and waste of computing power problem brought by full-time full analysis mode. Only when the comprehensive risk assessment shows that there is real risk, the voice content is analyzed directionally, and the balance of privacy protection and effective monitoring is realized. At the same time, by fusing the multi-modal features such as voiceprint, emotion, environmental sound and risk assessment, and establishing an anonymous voiceprint behavior database for historical tracing, the accuracy and depth of risk identification are significantly improved, so that the system can not only alarm in real time, but also provide the behavior trajectory and pattern analysis of the subject involved, realize the leap from single event alarm to continuous risk research and tracing, and provide more comprehensive and intelligent decision support for managers.

Description

Technical Field

[0001] This invention relates to the field of information technology, specifically to an information monitoring and tracing method and system based on voiceprint and semantic analysis. Background Technology

[0002] In specific scenarios such as public safety, judicial supervision, and campus management, security monitoring of sensitive areas, such as prison cells and dormitories, is often required. Traditional solutions mainly rely on video surveillance, but this cannot be deployed in areas involving personal privacy. To address this, monitoring technologies based on audio analysis have emerged, such as triggering alarms by recognizing preset keywords (e.g., "help"). However, these technologies typically employ a "full-time, full-volume" speech recognition and content analysis mode, continuously processing audio content during non-risk periods, which poses a risk of infringing on personal privacy and consumes significant computing resources. Furthermore, their analysis dimensions are limited, mainly relying on isolated word matching, lacking the fusion analysis and comprehensive risk assessment of multi-dimensional information such as speaker characteristics, emotional state, and environmental sounds. This results in a high false alarm rate and an inability to effectively trace the source and conduct in-depth analysis of risk events. Summary of the Invention

[0003] In view of the above problems, the present invention provides an information monitoring and tracing method and system based on voiceprint and semantic analysis.

[0004] To achieve the above objectives, in a first aspect, the present invention provides an information monitoring and tracing method based on voiceprint and semantic analysis, comprising: Acquire raw audio stream data of the target monitoring area; Parallel feature extraction is performed on the raw audio stream data to generate acoustic feature vectors and audio event labels. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event labels include non-speech physical environment sound categories. Based on acoustic feature vectors and audio event labels, real-time risk confidence is calculated using a pre-defined risk assessment model. When the real-time risk confidence exceeds the preset risk threshold, speech recognition is triggered on the original audio stream data in the current time window and subsequent time windows to generate the corresponding transcribed text sequence. The transcribed text sequence is input into a domain-adaptive natural language understanding model to generate a structured risk event report, which includes the risk type, the anonymous voiceprint identifier involved, and a key semantic summary. Based on the anonymous voiceprint identifiers involved in the structured risk event report, perform a correlation query in the anonymous voiceprint behavior database to obtain the correlation historical behavior trajectory. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. By integrating structured risk event reports with related historical behavior trajectories, a comprehensive source tracing and analysis report is generated and an early warning is triggered.

[0005] In some embodiments, parallel feature extraction is performed on the raw audio stream data to generate acoustic feature vectors and audio event labels, including: Speech activity detection is performed on the raw audio stream data to separate speech segments containing human voices from non-speech audio segments; For the speech segment, an anonymized voiceprint feature vector is generated using the Mel frequency cepstral coefficient extraction algorithm and the fundamental frequency contour tracking algorithm; Meanwhile, the spectral features of the speech segment are analyzed by a pre-trained acoustic emotion classification model to generate an emotion feature vector. The voiceprint feature vector and the emotion feature vector together constitute the acoustic feature vector. For non-speech audio segments, an audio event detection model based on convolutional neural networks is used to identify them and generate audio event labels that characterize the categories of physical environment sounds. Audio event labels include, but are not limited to, the sound of glass breaking, the sound of heavy objects hitting, and the sound of running.

[0006] In some embodiments, for a speech segment, an anonymized voiceprint feature vector is generated using a Mel-frequency cepstral coefficient extraction algorithm and a fundamental frequency contour tracking algorithm, including: The speech segment is preprocessed by framing and windowing to obtain short-time speech frames; For each short-time speech frame, the short-time speech frame is converted to the frequency domain by fast Fourier transform, filtered by the Mel filter bank, and the logarithmic energy of the short-time speech frame is calculated to obtain the Mel spectrum; By applying discrete cosine transform to the Mel spectrum, multi-order Mel frequency cepstral coefficients are extracted to form the first feature vector characterizing the vocal tract characteristics. For a speech segment, a fundamental frequency extraction algorithm based on autocorrelation function is used to track the contour of the fundamental frequency of the speech segment changing over time, and the statistical characteristics of the fundamental frequency are calculated to form a second feature vector characterizing the characteristics of the sound source. The first feature vector and the second feature vector are concatenated and normalized to generate an anonymized voiceprint feature vector, which is not directly related to the speaker's real identity information.

[0007] In some embodiments, based on acoustic feature vectors and audio event labels, a pre-defined risk assessment model is used to calculate real-time risk confidence, including: The emotion feature vector in the acoustic feature vector is matched with the pre-set negative emotion feature library to generate an emotion risk score. The audio event tags are matched with a pre-defined high-risk audio event database to generate environmental event risk scores. An initial risk score is calculated based on emotional risk scores and environmental event risk scores using a weighted fusion algorithm. The initial risk score is input into a time series prediction model trained based on historical risk event data. The model combines the risk score sequences of the current time window and adjacent historical time windows to output the real-time risk confidence level.

[0008] In some embodiments, when the real-time risk confidence level exceeds a preset risk threshold, speech recognition is triggered on the original audio stream data within the current time window and subsequent time windows to generate a corresponding transcribed text sequence, including: When the real-time risk confidence level exceeds the preset risk threshold, a voice recognition trigger command is generated; Based on the speech recognition trigger command, the original audio stream data of the current time window and the preset duration in the subsequent time windows are extracted from the audio buffer to form the audio segment to be recognized. The audio segment to be recognized is input into a domain-optimized speech recognition engine, which is loaded with an acoustic model of high-frequency words and noisy environment for the monitoring scenario. The speech recognition engine decodes the audio segment to be recognized and generates a transcribed text sequence containing timestamps. The words in the transcribed text sequence are associated with the domain knowledge base of the monitoring scene.

[0009] In some embodiments, the acoustic model of the speech recognition engine adopts a hybrid architecture based on connection-time classification and attention mechanisms, and uses noisy speech data collected from the monitoring scene for adversarial training to enhance its robustness under complex environmental noise. Meanwhile, the language model of the speech recognition engine is a fusion model of N-gram and neural language model trained on domain text corpus based on the monitoring scenario. The domain text corpus contains high-frequency words and language expression patterns in specific scenarios. During the decoding process, the acoustic score output by the acoustic model and the language probability output by the language model are fused through a dynamic decoding network to generate a transcribed text sequence that best matches the semantic context of the monitored scene.

[0010] In some embodiments, the transcribed text sequence is input into a domain-adaptive natural language understanding model to generate a structured risk event report, including: The transcribed text sequence is input into the intent recognition module, which is built based on a bidirectional long short-term memory network and an attention mechanism, and the module identifies and outputs the risk intent category corresponding to the transcribed text sequence. The transcribed text sequence and the identified risk intent category are input into the named entity recognition module to extract entity information related to the risk intent. The entity information includes personnel references, time points, and location coded messages. Based on risk intent categories and extracted entity information, key semantic summaries are generated through preset rule templates and semantic role annotations. The risk intent category is mapped to the risk type, the anonymized voiceprint features in the acoustic feature vector are mapped to the involved anonymous voiceprint identifier, and combined with the key semantic summary, a structured risk event report is generated.

[0011] In some embodiments, the transcribed text sequence and the identified risk intent category are input together into the named entity recognition module to extract entity information related to the risk intent, including: The transcribed text sequence and risk intent category are input together into the named entity recognition module built on conditional random fields and pre-trained language models; The named entity recognition module dynamically activates the corresponding entity type extraction strategy based on the risk intent category. The entity type extraction strategy includes a nickname extraction strategy for personnel in violent conflict scenarios and a time and location cryptic extraction strategy for planning scenarios. The named entity recognition module identifies and labels entity fragments that conform to the entity type extraction strategy from the transcribed text sequence. It then utilizes the contextual representation capabilities of the pre-trained language model to disambiguate and normalize the entity fragments, generating standardized entity information.

[0012] In some embodiments, based on the anonymous voiceprint identifier involved in the structured risk event report, a correlation query is performed in the anonymous voiceprint behavior database to obtain the related historical behavior trajectory, including: Using the anonymous voiceprint identifier involved in the structured risk event report as the query key, retrieve all related historical records in the anonymous voiceprint behavior database; For the retrieved historical records, extract the corresponding historical timestamps, historical monitoring area locations, and historical acoustic feature vectors in chronological order. Based on historical timestamps and historical monitoring area locations, the spatiotemporal activity trajectory of the anonymous voiceprint identifier involved was constructed; Based on historical acoustic feature vectors, clustering analysis algorithms are used to identify risk behavior patterns that frequently appear in the historical records of the anonymous voiceprint identifiers involved in the case. By integrating spatiotemporal activity trajectories with identified risk behavior patterns, associated historical behavior trajectories are generated.

[0013] In a second aspect, the present invention also provides an information monitoring and tracing system based on voiceprint and semantic analysis, applicable to the method described in the first aspect. The system includes an audio acquisition module, a parallel feature extraction module, a risk assessment module, a conditionally triggered speech recognition module, a semantic understanding and report generation module, a behavior tracing query module, and a comprehensive judgment and early warning module. The audio acquisition module is configured to acquire raw audio stream data of the target monitoring area. The parallel feature extraction module is configured to perform parallel feature extraction on the raw audio stream data, generating acoustic feature vectors and audio event tags. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event tags include non-speech physical environment sound categories. The risk assessment module is configured to calculate real-time risk confidence based on the acoustic feature vectors and audio event tags using a preset risk assessment model. The conditionally triggered speech recognition module... The system is configured to trigger speech recognition of the original audio stream data within the current time window and subsequent time windows when the real-time risk confidence exceeds a preset risk threshold, generating a corresponding transcribed text sequence. The semantic understanding and report generation module is configured to input the transcribed text sequence into a domain-adaptive natural language understanding model to generate a structured risk event report, which includes the risk type, the involved anonymous voiceprint identifier, and key semantic summaries. The behavior tracing query module is configured to perform a correlation query in the anonymous voiceprint behavior database based on the involved anonymous voiceprint identifier in the structured risk event report, obtaining related historical behavior trajectories. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. The comprehensive analysis and early warning module is configured to fuse the structured risk event report and related historical behavior trajectories to generate a comprehensive tracing analysis report and trigger an early warning.

[0014] Unlike existing technologies, the above technical solution acquires raw audio stream data of the target monitoring area; performs parallel feature extraction on the raw audio stream data to generate acoustic feature vectors containing anonymized voiceprint features and emotional features, as well as audio event tags containing non-speech physical environment sound categories; calculates real-time risk confidence based on the acoustic feature vectors and audio event tags using a preset risk assessment model; only when the real-time risk confidence exceeds a preset risk threshold is speech recognition triggered on the raw audio stream data within the current time window and subsequent time windows to generate corresponding transcribed text sequences; inputs the transcribed text sequences into a domain-adaptive natural language understanding model to generate a structured risk event report containing risk type, involved anonymous voiceprint identifiers, and key semantic summaries; performs correlation queries in an anonymous voiceprint behavior database constructed based on long-term accumulated acoustic feature vectors based on the involved anonymous voiceprint identifiers to obtain related historical behavior trajectories; finally, integrates the structured risk event report and related historical behavior trajectories to generate a comprehensive source tracing and judgment report and trigger an early warning. By introducing a conditional triggering mechanism that first analyzes non-content features such as voiceprints and emotions, and then triggers deep semantic understanding only after the risk confidence level is reached, the privacy infringement and computational waste problems caused by the full-time, full-volume analysis mode are effectively solved. Voice content is only analyzed in a targeted manner when a comprehensive risk assessment indicates a real risk, achieving a balance between privacy protection and effective monitoring. Simultaneously, by integrating multimodal features such as voiceprints, emotions, and ambient sounds for risk assessment, and establishing an anonymous voiceprint behavior database for historical tracing, the accuracy and depth of risk identification are significantly improved. This enables the system not only to issue real-time alerts but also to provide behavioral trajectory and pattern analysis of the involved parties, achieving a leap from single-event alerts to continuous risk assessment and tracing, providing managers with more comprehensive and intelligent decision support.

[0015] The above description of the invention is merely an overview of the technical solution of this application. In order to enable those skilled in the art to better understand the technical solution of this application and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of this application easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of this application. Attached Figure Description

[0016] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on this application.

[0017] In the accompanying drawings of the instruction manual: Figure 1 This is a schematic diagram illustrating steps S101 to S107 of the specific implementation method. Figure 2This is a schematic diagram illustrating steps S201 to S203 as described in the specific implementation method. Figure 3 This is a schematic diagram of steps S301 to S305 as described in the specific implementation method. Detailed Implementation

[0018] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this application in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this application and are therefore intended to limit the scope of protection of this application.

[0019] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this application, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0020] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit this application.

[0021] In this application, terms such as “first” and “second” are used only to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy or order relationship between these entities or operations.

[0022] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this application is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0023] The processor described in the embodiments of this application can be implemented by hardware, firmware, software, or a combination thereof. It can be a circuit, one or more of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a central processing unit (CPU), a controller, a microcontroller, or a microprocessor. It also includes other physical, biological, or chemical structures that can implement the same or equivalent functions as the processors listed above, such as biological neurons, quantum computing units, DNA computing units, etc., so that the processor can execute some or all of the steps in the computer program or method involved in the various embodiments of this application, or any combination of the steps mentioned therein.

[0024] The computer program involved in the embodiments can be stored in a computer device readable storage medium, which includes, but is not limited to, disks, magnetic tapes, magnetic cards, floppy disks, flash memory, optical disks, optical cards, read-only memory (ROM), random access memory (RAM), erasable programmable ROM (EPROM), and electrically erasable programmable ROM (EEPROM), etc., and also includes other biological, physical, or chemical structures that can achieve the same or equivalent functions as the storage media listed above, such as DNA, RNA, proteins, and other units with information storage capabilities. In specific embodiments, the storage medium involved can be one of the above-mentioned media types, or a combination of the above-mentioned media types. In different embodiments, the computer program involved in the embodiments can be centrally stored in a single medium, or distributed and stored in multiple media. The memory containing the computer device readable storage medium can be non-volatile memory or random access memory. These computer device readable storage media can be built into the device, or can be connected to the device involved in the embodiments as an external device or part of an external device. In some embodiments, the memory having a computer device readable storage medium is deployed locally; in other embodiments, the memory may be deployed remotely from the processor, for example, as a network-attached memory accessed via RF circuitry or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, a local area network (LAN), a wide area network (WLAN), a storage area network (SAN), or a suitable combination thereof, as long as computer device access to the memory is enabled. Furthermore, the computer program involved in the embodiments may be stored in plaintext / ciphertext form, or it may be designed as training data, integrated and recombined through model training and implicitly stored in the parameter states of a deep neural network or other machine learning model.

[0025] Please see Figure 1 In a first aspect, this embodiment provides an information monitoring and tracing method based on voiceprint and semantic analysis, including: S101. Obtain the raw audio stream data of the target monitoring area; S102. Perform parallel feature extraction on the original audio stream data to generate acoustic feature vectors and audio event labels. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event labels include non-speech physical environment sound categories. S103. Based on acoustic feature vectors and audio event labels, calculate real-time risk confidence using a pre-defined risk assessment model; S104. When the real-time risk confidence exceeds the preset risk threshold, trigger speech recognition of the original audio stream data in the current time window and subsequent time windows to generate the corresponding transcribed text sequence. S105. Input the transcribed text sequence into a domain-adaptive natural language understanding model to generate a structured risk event report. The structured risk event report includes the risk type, the anonymous voiceprint identifier involved, and a key semantic summary. S106. Based on the anonymous voiceprint identifiers involved in the structured risk event report, perform a correlation query in the anonymous voiceprint behavior database to obtain the correlation historical behavior trajectory. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. S107. Integrate structured risk event reports with related historical behavior trajectories to generate a comprehensive source tracing and analysis report and trigger an early warning.

[0026] In step S101, the target monitoring area refers to a privacy-sensitive area that requires security monitoring but where video equipment is unsuitable or impossible to deploy, such as school dormitories, detention center cells, or hospital wards. The raw audio stream data refers to unprocessed digital audio signals continuously collected by audio acquisition devices (such as high-sensitivity microphone arrays or dedicated intelligent audio terminals) deployed within the target monitoring area. Parameters such as sampling rate and bit depth are configured based on the monitoring distance and ambient noise level; for example, a 16kHz sampling rate is preferred to balance sound quality with transmission and storage overhead.

[0027] In step S102, the core lies in parallel and non-content-based feature extraction. Specifically, the original audio stream undergoes speech activity detection, separating it into speech segments containing human voices and non-speech audio segments without human voices. For speech segments, a set of anonymized voiceprint features related to the speaker's vocal tract and sound source characteristics, but stripped of personal identity information, is generated using a Mel-frequency cepstral coefficient extraction algorithm and a fundamental frequency contour tracking algorithm. Simultaneously, a pre-trained acoustic emotion classification model (e.g., classifying spectrograms based on convolutional neural networks) analyzes the spectral characteristics of the speech segment, outputting emotion features representing the speaker's emotional state (e.g., calm, anger, fear). The voiceprint features and emotion features together constitute an acoustic feature vector, describing only "who" (anonymous identity) and "with what emotion" they are speaking, without involving the content of the speech. For non-speech audio segments, another audio event detection model based on convolutional neural networks is used for identification, generating audio event labels to represent the categories of physical environment sounds, such as "sound of breaking glass," "sound of a heavy object hitting," and "sound of rapid running." While protecting privacy (without touching the voice content), we extract multi-dimensional clues from the audio signal to the greatest extent possible that can be used for risk assessment, providing rich and comprehensive input for subsequent initial risk judgment.

[0028] In step S103, the risk assessment model is a trained machine learning model (e.g., a gradient boosting decision tree or a lightweight neural network). Its input is the fusion features extracted in step S102, and its output is a real-time risk confidence score between 0 and 1. The training data for the risk assessment model comes from risk events marked in historical monitoring and their acoustic features and audio events before and after their occurrence. During inference, the risk assessment model comprehensively evaluates whether the current emotional characteristics are biased towards negativity (e.g., anger, fear) and whether the audio event label belongs to a high-risk category (e.g., impact sound), and calculates the risk based on the intensity and combination pattern of these features. The preset risk threshold is a configurable parameter, typically determined through historical data verification based on the tolerance for false positives and false negatives in the actual application scenario. Transforming multimodal acoustic cues into a unified, quantifiable risk probability index achieves the first abstraction and quantification from raw signals to risk perception, providing a decision-making basis for whether to initiate more computationally expensive and privacy-intrusive deep semantic analysis.

[0029] In step S104, triggering refers to the system switching from a low-power monitoring and analysis mode to a high-precision content parsing mode. The current time window and subsequent time window refer to audio data that is backtracked (e.g., 2 seconds) from the moment the risk confidence exceeds the limit and extended forward by a preset duration (e.g., 10 seconds) to ensure the capture of the complete dialogue context before and after the risk event. This audio segment to be identified is sent to a domain-optimized speech recognition engine for transcription. The acoustic model of the speech recognition engine preferably adopts a model enhanced by noise enhancement training for the monitoring environment, while the language model of the speech recognition engine incorporates high-frequency words and expression habits of the monitoring scene to improve the recognition accuracy in specific scenarios. The final generated transcribed text sequence is a text stream with timestamps. Following the "minimum necessary" principle, the speech content is parsed only when the preliminary analysis indicates a high risk, thereby effectively obtaining key semantic information while minimizing unnecessary contact and processing of irrelevant privacy content, achieving a balance between privacy protection and security monitoring.

[0030] In step S105, the domain-adaptive natural language understanding model refers to an NLP model that has been fine-tuned or specially trained for specific domains such as public safety and regulation. The natural language understanding model uses an intent recognition module (e.g., a BiLSTM-ATT-based model) to determine the core intent of the transcribed text, such as "physical conflict," "verbal threat," or "plea for help," and maps it to risk types. Simultaneously, a named entity recognition module extracts entity information related to the risk intent, such as personal designations (nicknames), time points, and location coded messages, and generates a key semantic summary based on this information. The anonymous voiceprint identifier directly originates from the unique ID mapped by the anonymized voiceprint features corresponding to the current speech segment generated in step S102. Finally, this information is encapsulated into a structured risk event report. This process transforms unstructured natural language text into machine-readable, queryable, and associative structured intelligence, providing clear information elements for subsequent handling and tracing.

[0031] In step S106, the anonymous voiceprint behavior database is a long-running database that continuously stores and indexes all acoustic feature vectors generated in step S102, along with their corresponding anonymous IDs, timestamps, and location information. When a query request for an anonymous voiceprint identifier is received, the anonymous voiceprint behavior database retrieves all historical occurrence records for that ID and constructs its spatiotemporal activity trajectory (e.g., when and where it usually appears) and behavioral pattern profile (e.g., historical emotional feature distribution, frequently accompanying audio events) based on these records. The associated historical behavioral trajectory constitutes a summary of the ID's historical activities. By leveraging the uniqueness and stability of voiceprint features, continuous tracking and behavioral pattern analysis of specific actors across time and space can be achieved without involving their real identity. This places isolated events within the long-term behavioral context of the actor, significantly enhancing the depth of the analysis.

[0032] In step S107, fusion refers to integrating and cross-analyzing the specific details of the current event (risk type, semantic summary) with the historical behavioral patterns of the involved parties. For example, the report will not only indicate "a verbal threat has occurred," but also add a note that "the involved voiceprint ID has had three records of emotional agitation in the past week, and is often active in area A at night." Based on this more comprehensive integrated source tracing and judgment report, a tiered warning is triggered to the monitoring center. This completes the final closed loop from "perceiving the current event" to "understanding the event background and the risks to the parties involved," outputting a decision support report that includes real-time snapshots and historical depth, enabling security managers to make precise interventions and resource allocations based on richer information.

[0033] This embodiment provides a hierarchical, condition-triggered intelligent audio monitoring and tracing solution. Its core principle lies in performing real-time risk screening through non-content-based parallel analysis of voiceprints, emotions, and ambient sounds. Only when the risk probability reaches a threshold is targeted deep analysis of the voice content initiated, thus achieving a balance between the inherent contradiction of privacy protection and effective monitoring in the technical architecture. Utilizing anonymous voiceprint IDs as a bridge, the current event is linked to a long-accumulated behavioral database, achieving a leap from instantaneous alerts to continuous behavior tracking and tracing.

[0034] The conditional triggering mechanism in this embodiment greatly reduces unnecessary access to privacy data and consumption of computing resources, complying with ethical and regulatory requirements. Compared with single keyword matching, the risk assessment based on multimodal feature fusion significantly improves the accuracy of risk perception and early warning capabilities. The established anonymous voiceprint behavior database enables the system to create long-term profiles and pattern recognition of risky individuals, realizing a shift from reactive post-event response to proactive pre-event warning and precise in-event assessment, providing a feasible and sustainable intelligent solution for the security management of high-risk and sensitive areas.

[0035] Correspondingly, in the second aspect, this embodiment also provides an information monitoring and tracing system based on voiceprint and semantic analysis, applicable to the method described in the first aspect. The system includes an audio acquisition module, a parallel feature extraction module, a risk assessment module, a conditionally triggered speech recognition module, a semantic understanding and report generation module, a behavior tracing query module, and a comprehensive judgment and early warning module. The audio acquisition module is configured to acquire raw audio stream data of the target monitoring area; the parallel feature extraction module is configured to perform parallel feature extraction on the raw audio stream data to generate acoustic feature vectors and audio event tags. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event tags include non-speech physical environment sound categories; the risk assessment module is configured to calculate real-time risk confidence based on the acoustic feature vectors and audio event tags using a preset risk assessment model; the conditionally triggered speech recognition module... The risk identification module is configured to trigger speech recognition of the original audio stream data within the current time window and subsequent time windows when the real-time risk confidence exceeds a preset risk threshold, generating a corresponding transcribed text sequence. The semantic understanding and report generation module is configured to input the transcribed text sequence into a domain-adaptive natural language understanding model to generate a structured risk event report, which includes the risk type, the anonymous voiceprint identifier involved, and key semantic summaries. The behavior tracing query module is configured to perform a correlation query in the anonymous voiceprint behavior database based on the anonymous voiceprint identifier involved in the structured risk event report, obtaining the associated historical behavior trajectory. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. The comprehensive judgment and early warning module is configured to fuse the structured risk event report and the associated historical behavior trajectory to generate a comprehensive tracing judgment report and trigger an early warning.

[0036] The method steps mentioned in the above embodiments correspond to the method steps described in the first aspect. They can be understood by referring to the embodiments described in the first aspect. This embodiment will not be described in detail here.

[0037] Please see Figure 2 In some embodiments, parallel feature extraction is performed on the raw audio stream data to generate acoustic feature vectors and audio event labels, including: S201. Perform speech activity detection on the raw audio stream data to separate the speech segments containing human voices from the non-speech audio segments. S202. For the speech segment, an anonymized voiceprint feature vector is generated by the Mel frequency cepstral coefficient extraction algorithm and the fundamental frequency contour tracking algorithm. At the same time, the spectral features of the speech segment are analyzed by the pre-trained acoustic emotion classification model to generate an emotion feature vector. The voiceprint feature vector and the emotion feature vector together constitute the acoustic feature vector. S203. For non-speech audio segments, an audio event detection model based on a convolutional neural network is used to identify them and generate audio event labels that characterize the categories of physical environment sounds. Audio event labels include, but are not limited to, the sound of glass breaking, the sound of heavy objects hitting, and the sound of running.

[0038] In step S201, speech activity detection refers to the process of determining in real time whether the current frame of the audio stream contains valid human voices using signal processing algorithms (e.g., based on short-time energy and zero-crossing rate, or based on a pre-trained deep neural network model). The output is to divide the continuous audio stream into two categories of segments on the time axis: speech segments and non-speech audio segments. Non-speech audio segments may contain environmental noise, music, or other non-human sounds. By utilizing the statistical characteristics of human voices in the frequency and time domains and their differences from other sounds, preliminary audio content classification is achieved, ensuring that voiceprint and emotion analysis only apply to valid human voices, while audio event detection focuses on abnormal environmental sounds that are not human voices, thereby improving the accuracy and efficiency of their respective feature extraction.

[0039] In step S202, the Mel-frequency cepstral coefficient extraction algorithm is a feature extraction method that simulates the auditory characteristics of the human ear. By performing Mel-scale filtering and cepstral analysis on the spectrum of the speech signal, a set of coefficients representing the shape of the vocal tract are obtained. These coefficients have good speaker discrimination ability but are not sensitive to text content. The fundamental frequency contour tracking algorithm is used to extract the basic frequency and variation pattern of vocal cord vibration in speech, reflecting the characteristics of the sound source. By concatenating and fusing these two features, an anonymized voiceprint feature vector is formed. The voiceprint feature vector can uniquely represent the physiological characteristics of a speaker's pronunciation, but it cannot directly deduce the person's real identity, name, or other biometric information, thus meeting privacy protection requirements. Meanwhile, the pre-trained acoustic emotion classification model is a deep learning-based classifier (such as a convolutional neural network). Its input is the spectrogram of the speech segment or a series of acoustic features (such as MFCC, prosodic features), and its output is the probability distribution or classification label corresponding to different emotional states (such as neutral, angry, fearful, happy), i.e., the emotion feature vector. The training data of the acoustic emotion classification model comes from publicly available or self-built emotional speech databases. The acoustic feature vector is constructed by combining voiceprint feature vector and emotion feature vector. The principle is to extract and encapsulate two types of key information from the same human voice: an anonymous identity identifier representing "who is speaking" and state information representing "what emotion is being spoken". This provides a three-dimensional description of the speaker for subsequent risk assessment, without involving semantic content understanding.

[0040] In step S203, the audio event detection model based on convolutional neural networks is a deep learning model that performs end-to-end classification of audio signals (usually converted into Mel spectrograms or acoustic spectrograms as two-dimensional image inputs). During the training phase, the audio event detection model learns the spectral features of a large number of labeled environmental sound samples (such as glass breaking, impacts, running, door closing, alarms, etc.). During the inference phase, when a non-speech audio segment is input, the model outputs the probability of it belonging to each preset category, taking the one with the highest probability as the audio event label. "Sound of glass breaking," "sound of heavy objects hitting," and "sound of rapid running" are particularly crucial label examples in high-risk monitoring scenarios, as they are usually directly related to violence, vandalism, and emergency escape. Utilizing the powerful spatial feature extraction capabilities of convolutional neural networks, the model automatically learns and identifies physical event sounds with specific patterns from the time-frequency representation of sound, thereby transforming raw, meaningless noise segments into event labels with clear semantic meaning, supplementing the risk perception dimension (such as the sounds produced by actions) that pure speech analysis cannot cover.

[0041] This embodiment details the specific implementation path of parallel feature extraction. For the distinct information components in the audio stream (human voice vs. ambient sound), a specialized processing channel is used for feature extraction: for human voice, the focus is on non-content attributes such as the speaker's physiological characteristics and emotional state; for ambient sound, the focus is on identifying the category of physical event it represents. This parallel and decoupled processing approach allows the system to maximize the extraction of acoustic information usable for security risk assessment while protecting personal voice privacy (without transcribing content). Voiceprints and emotional features constitute an anonymized profile of the "performer," while audio event tags characterize the "behavior" itself or its environmental state. The combination of these two provides the risk assessment model with multi-dimensional, low-privacy-intrusive input features that include both the subject's state and the environmental context. This lays the foundation for the privacy compliance of the entire method, and at the same time, multi-cue fusion significantly improves the reliability of subsequent initial risk assessment and early warning capabilities, avoiding the shortcomings of single information sources (such as keywords alone) being easily interfered with and having a high false alarm rate.

[0042] Please see Figure 3 In some embodiments, for speech segments, an anonymized voiceprint feature vector is generated using a Mel frequency cepstral coefficient extraction algorithm and a fundamental frequency contour tracking algorithm, including: S301. Perform framing and windowing preprocessing on the speech segment to obtain short-time speech frames; S302. For each short-time speech frame, convert the short-time speech frame to the frequency domain through Fast Fourier Transform, filter it through the Mel filter bank, calculate the logarithmic energy of the short-time speech frame, and obtain the Mel spectrum. S303. Apply discrete cosine transform to the Mel spectrum to extract multi-order Mel frequency cepstral coefficients, which constitute the first feature vector characterizing the vocal tract characteristics. S304. For a speech segment, the fundamental frequency extraction algorithm based on the autocorrelation function is used to track the contour of the fundamental frequency of the speech segment changing over time, and the statistical characteristics of the fundamental frequency are calculated to form a second feature sub-vector representing the characteristics of the sound source. S305. The first feature vector and the second feature vector are concatenated and normalized to generate an anonymized voiceprint feature vector. The voiceprint feature vector is not directly related to the speaker's real identity information.

[0043] In step S301, since the speech signal is time-varying, it can be approximated as stationary within a sufficiently short time period (typically 10-30 milliseconds). Framing refers to dividing continuous speech segments into a series of overlapping short time periods with a fixed duration (i.e., frame length, such as 25 milliseconds) and a fixed step size (i.e., frame shift, such as 10 milliseconds). Windowing refers to multiplying each frame signal by a window function (such as a Hamming window) to reduce the discontinuity of the signal at frame edges caused by framing and reduce spectral leakage. The short-time speech frames obtained after this preprocessing are the basic processing units for subsequent frequency domain analysis and feature extraction. Transforming the non-stationary speech signal into a series of short-time segments that can be approximated as stationary signals lays the foundation for extracting stable spectral features.

[0044] In step S302, the Fast Fourier Transform (FFT) is an efficient algorithm used to convert the time-domain signal into a frequency-domain representation, obtaining the linear spectrum of the speech frame. The Mel filter bank is a set of triangular bandpass filters distributed according to the Mel scale (a non-linear scale that simulates human ear perception of frequency). Passing the linear spectrum through this filter bank smooths the spectrum and focuses it on the frequency region sensitive to human hearing, simulating the auditory characteristics of the human ear. Calculating the logarithm of the output energy of each filter yields the Mel spectrum. Through FFT and Mel filtering, the time-domain waveform of the speech is transformed into a time-spectrum representation that compresses the data volume, conforms to auditory perception, and is sensitive to speaker characteristics.

[0045] In step S303, DCT is performed on the Mel frequency spectrum to deconvolve the vocal tract response (represented by the low-time portion of the cepstral spectrum) and the sound source excitation (represented by the high-time portion of the cepstral spectrum). Typically, the first 12-16 coefficients after DCT are selected, i.e., the Mel frequency cepstral coefficients. These coefficients effectively characterize the shape and movement of the vocal tract during articulation, and vocal tract shape is one of the key physiological factors determining individual timbre. The set of these coefficients constitutes the first feature vector, which primarily describes the speaker's vocal tract characteristics. Cepstral analysis separates and extracts relatively stable and personally discriminative vocal tract information from the speech signal, while remaining to a certain extent insensitive to text content, thus meeting anonymization requirements.

[0046] In step S304, the fundamental frequency (FFF) refers to the basic frequency of vocal cord vibration, which determines the pitch of speech. The FFF extraction algorithm based on autocorrelation function estimates the FFF by calculating the correlation between the speech signal and its delayed version and finding periodic peaks. By estimating the FFF frame by frame for the entire speech segment, the FFF profile, i.e., the pitch change curve over time, can be obtained. Furthermore, a series of statistical features, such as the mean, standard deviation, maximum, minimum, and range of the FFF, are calculated from the FFF profile, forming a second feature vector that reflects the vibration characteristics of the speaker's vocal cords and their modulation patterns during articulation, and is an important manifestation of the sound source characteristics. The FFF and its dynamic change pattern are another important speaker feature, and they complement the vocal tract features represented by MFCC, together providing a more comprehensive acoustic profile.

[0047] In step S305, concatenation refers to connecting the first feature vector representing vocal tract characteristics and the second feature vector representing sound source characteristics dimensionally to form a longer, more informative composite feature vector. Normalization (e.g., using mean-variance normalization) is used to eliminate feature distribution differences caused by different recording sessions and devices, improving the robustness and comparability of the features. The final voiceprint feature vector generated after this step is a high-dimensional mathematical vector that comprehensively represents the physiological acoustic pattern of a specific speaker in a specific pronunciation. Crucially, the voiceprint feature vector itself does not contain real identity information such as name or ID number, nor can the original speech or identity be obtained through simple reverse calculation. It is used only as an anonymous and unique "acoustic fingerprint" identifier within the system. The implementation principle of this step is to fuse complementary acoustic features and eliminate irrelevant interference through standardization, ultimately generating a stable, distinguishable, and privacy-preserving anonymized speaker representation, providing core data units for subsequent speaker tracking and association analysis.

[0048] This embodiment reveals in detail the generation of anonymized voiceprint feature vectors. It extracts spectral envelope information (via MFCC) representing the shape of the vocal tract and pitch contour information (via fundamental frequency statistics) representing the excitation of the sound source from the speech signal, and then fuses these two physiologically based, relatively stable, and complementary features into a single comprehensive vector. The entire process is strictly performed at the signal processing and feature engineering level, without involving any binding or mapping of identity information. This ensures that the generated voiceprint feature vector has high individual distinguishability, sufficient to track and associate different speeches by the same speaker within the same monitoring scenario, while also meeting privacy protection regulations due to its decoupling from the real identity. It provides a technically feasible and ethically compliant core tool that enables continuous behavioral analysis that identifies the same person even without knowing their identity. This is a key technological foundation for building an anonymous voiceprint behavior database and achieving cross-event tracing.

[0049] In some embodiments, based on acoustic feature vectors and audio event labels, a pre-defined risk assessment model is used to calculate real-time risk confidence, including: The emotion feature vector in the acoustic feature vector is matched with the pre-set negative emotion feature library to generate an emotion risk score. The audio event tags are matched with a pre-defined high-risk audio event database to generate environmental event risk scores. An initial risk score is calculated based on emotional risk scores and environmental event risk scores using a weighted fusion algorithm. The initial risk score is input into a time series prediction model trained based on historical risk event data. The model combines the risk score sequences of the current time window and adjacent historical time windows to output the real-time risk confidence level.

[0050] In this step of the embodiment, the emotion feature vector is a mathematical vector representing the speaker's emotional state (such as anger, fear, neutrality, etc.) obtained through spectral analysis of speech segments. The pre-built negative emotion feature library is a pre-constructed reference set containing typical feature vector templates of negative emotions (such as anger, fear, extreme tension) labeled as high-risk or requiring attention. Similarity matching can be achieved by calculating metrics such as cosine similarity or Euclidean distance, and the output is the emotion risk score. The higher the emotion risk score, the closer the speaker's current emotional state is to the template in the negative emotion library, and the higher the risk in their emotional dimension. Quantifying the abstract emotion feature vector into a comparable risk indicator provides a risk basis from the speaker's subjective state for subsequent comprehensive assessment.

[0051] In this step of the embodiment, the audio event tag refers to the identified category of physical environment sound (such as "sound of breaking glass"). The preset high-risk audio event library is a list or set that enumerates sound categories highly related to violence, destruction, and emergencies (for example, a preferred library would include at least "sound of breaking glass," "sound of a heavy object hitting," and "sound of rapid running"). Matching is a direct lookup or mapping process: if the current audio event tag exists in the high-risk library, it is assigned a higher fixed score or a score pre-set according to the category; if it does not exist, it is assigned a lower score or zero. The resulting environmental event risk score directly reflects the probability of abnormal, high-risk action sounds occurring in the current physical environment. This transforms discrete audio event tags into a quantitative value characterizing the degree of risk of actions in the objective environment.

[0052] In this step shown in this embodiment, the weighted fusion algorithm is a linear or nonlinear information fusion method. In this embodiment, it can be simply expressed as: Initial risk score = w1 * emotional risk score + w2 * environmental event risk score. Here, the weights w1 and w2 are preset parameters that determine the relative importance of emotional risk and event risk in the comprehensive assessment. The weights can be set based on domain knowledge (for example, in a school anti-bullying scenario, emotionally charged words may be more indicative than a random impact sound), or they can be obtained through training with historical data. The initial risk score is a preliminary, single risk quantification indicator that integrates subjective emotions and objective environmental actions. By integrating risk evidence from different dimensions (personal state, environmental actions) through configurable weights, a more comprehensive and stable preliminary judgment of overall risk is obtained, avoiding the potential bias of a single indicator.

[0053] In this step of the embodiment, the time series prediction model can be a recurrent neural network (such as LSTM, GRU), a temporal convolutional network, or a simpler autoregressive moving average model. This model uses a sequence of initial risk scores from before and after a real risk event in historical records as training data to learn typical evolution patterns of risk scores before the event. In the application phase, the model not only considers the current initial risk score but also combines it with historical scores from several previous time windows (e.g., the past 30 seconds) to form a risk score sequence as input. The time series prediction model analyzes this sequence, and its output real-time risk confidence score reflects not only the static risk level at the current moment but also includes dynamic predictive components based on recent risk trends. For example, even if the current score is moderate, if the recent score shows a rapid upward trend, the time series prediction model may output a high confidence score. Introducing a time dimension for analysis upgrades risk assessment from "looking at a snapshot" to "looking at dynamic trends," enabling earlier detection of signs of risk accumulation and escalation, potentially achieving earlier warnings, and effectively filtering out isolated, transient false alarms (such as accidental shouts or dropped objects).

[0054] This embodiment details a hierarchical and dynamic risk assessment calculation strategy. It matches two types of features—emotion and audio events—with preset high-risk patterns to obtain basic risk values. Weighted fusion yields a comprehensive initial risk score, completing the static integration of multimodal information. This initial score is then placed within a time-series context, and a prediction model trained on historical data is used to assess the dynamic evolution trend of the risk, outputting the final real-time risk confidence level. The component matching and weighted fusion make the risk assessment process interpretable and configurable, facilitating adjustments to risk sensitivity based on different scenarios. The introduction of a time-series prediction model significantly enhances the system's intelligence, enabling it to "sense" the risk's incubation process, rather than simply responding to risks that have already reached their peak. This significantly improves the timeliness (earlier warnings) and accuracy (reducing false alarms to transient, isolated noise). This provides crucial technical support for transitioning from "passive alarm" to "proactive warning," making the entire monitoring system more forward-looking and practical.

[0055] In some embodiments, when the real-time risk confidence level exceeds a preset risk threshold, speech recognition is triggered on the original audio stream data within the current time window and subsequent time windows to generate a corresponding transcribed text sequence, including: When the real-time risk confidence level exceeds the preset risk threshold, a voice recognition trigger command is generated; Based on the speech recognition trigger command, the original audio stream data of the current time window and the preset duration in the subsequent time windows are extracted from the audio buffer to form the audio segment to be recognized. The audio segment to be recognized is input into a domain-optimized speech recognition engine, which is loaded with an acoustic model of high-frequency words and noisy environment for the monitoring scenario. The speech recognition engine decodes the audio segment to be recognized and generates a transcribed text sequence containing timestamps. The words in the transcribed text sequence are associated with the domain knowledge base of the monitoring scene.

[0056] In this step shown in this embodiment, the preset risk threshold is a pre-set critical value used to determine whether to initiate deep semantic analysis. The preset risk threshold can be determined based on the analysis of historical operational data, striking a balance between acceptable false positive and false negative rates, for example, by determining the optimal operating point through receiver operation characteristic curves. The voice recognition trigger command is an internal system control signal that marks the system's formal switch from a low-power, non-content-analysis-focused monitoring mode to a computationally-intensive, voice content parsing-related recognition mode. Transforming the quantified real-time risk confidence obtained from the aforementioned steps into a clear, executable system state switching command is the key decision point in achieving a balance between conditional triggering and privacy in the entire method.

[0057] In this step of the embodiment, the audio buffer refers to a memory or storage area used to temporarily store raw audio stream data for a recent period (e.g., 1-2 minutes), managed using a first-in-first-out (FIFO) queue mechanism. The current time window refers to a period of time (e.g., 2 seconds) preceding the trigger command generation time, used to capture dialogue that may have started before the risk confidence level exceeded the limit; the subsequent time window refers to a period of time (e.g., 10 seconds) continuing after the trigger, used to capture the subsequent development of the risk event. The preset duration is the sum of the lengths of these two windows, and its specific value can be configured according to the dialogue duration of typical risk events in the scenario. The continuous audio data blocks extracted in this way constitute the audio segment to be identified, ensuring that the system can obtain the complete speech context before and after the occurrence of the risk event, providing a coherent and meaningful information source for subsequent semantic understanding, and avoiding misjudgment due to incomplete analysis segments.

[0058] In this step of the embodiment, the domain-optimized speech recognition engine refers to an automatic speech recognition system that has been customized and trained for a specific monitoring scenario (such as a campus or detention center). The acoustic model of the speech recognition engine is trained or fine-tuned using speech data collected from the target scenario that includes typical environmental noise (such as dormitory fan noise or corridor echoes), thereby improving robustness and recognition accuracy in this noisy environment. The language model or decoding dictionary of the speech recognition engine is enhanced with high-frequency words related to the monitoring scenario (e.g., "fighting," "tell the teacher," and "dormitory number" in a campus scenario, and specific coded language and numbers in a detention center scenario). By using a scenario-customized recognition engine, the problems of out-of-vocabulary and noise interference faced by general ASR in professional domains and complex environments can be significantly overcome, thus providing a more accurate and relevant text foundation for subsequent semantic analysis. This is an important technical guarantee for ensuring the final effectiveness of the entire method.

[0059] In this step shown in this embodiment, decoding is the core process of the speech recognition engine. It combines the acoustic feature probabilities output by the acoustic model and the word sequence probabilities output by the language model to search for the most likely word sequence. The decoding output includes not only the text content but also the timestamp information corresponding to each word or speech segment, which is crucial for subsequent analysis of dialogue turns, speech rate changes, etc. The generated transcribed text sequence is a continuous text with timestamps. The domain knowledge base of the monitored scene is a structured information database containing a list of entities (such as names and locations), terms, common expression patterns, etc., within that scene. Association means that the words in the transcribed text can be indexed and understood by this knowledge base; for example, the identified "Room 305" can be mapped to the "dormitory location" entity in the knowledge base. This completes the conversion from audio signals to structured text information, and through timestamps and potential associations with the domain knowledge base, it provides sufficient data preparation for the next step of natural language understanding. This ensures that the text is no longer an isolated string but an object to be analyzed, carrying spatiotemporal information and domain semantics.

[0060] This embodiment details the execution process of the crucial condition-triggered speech recognition step, constructing an efficient and accurate response chain: an instruction is triggered based on an excessive risk confidence level; the instruction drives the system to accurately extract an audio segment containing the complete risk context from the buffer; this segment is fed into a speech recognition engine tailored to the current scenario for high-precision transcription; finally, a text sequence with time information and domain knowledge is output. Strict adherence to the principles of minimum necessity and conditional triggering ensures that most everyday conversations unrelated to privacy are not transcribed; content analysis is only initiated when the system comprehensively determines a high risk, achieving a substantial balance between privacy protection and security monitoring at the technical level. By using a scenario-optimized recognition engine and an associated domain knowledge base, the accuracy and semantic relevance of speech transcription in complex and professional scenarios are significantly improved, laying a reliable data foundation for the subsequent generation of high-quality structured risk reports.

[0061] In some embodiments, the acoustic model of the speech recognition engine adopts a hybrid architecture based on connection-time classification and attention mechanisms, and uses noisy speech data collected from the monitoring scene for adversarial training to enhance its robustness under complex environmental noise. Meanwhile, the language model of the speech recognition engine is a fusion model of N-gram and neural language model trained on domain text corpus based on the monitoring scenario. The domain text corpus contains high-frequency words and language expression patterns in specific scenarios. During the decoding process, the acoustic score output by the acoustic model and the language probability output by the language model are fused through a dynamic decoding network to generate a transcribed text sequence that best matches the semantic context of the monitored scene.

[0062] In this step illustrated in this embodiment, the acoustic model is the core component of the speech recognition system responsible for mapping audio frame sequences to phoneme or sub-word unit probability distributions. The hybrid architecture based on connection-temporal classification and attention mechanisms is an advanced acoustic modeling step: the connection-temporal classification layer allows the model to directly handle the problem of inconsistent input-output sequence lengths and optimize the likelihood probability of the entire sequence; while the attention mechanism enables the model to dynamically focus on the parts of the input sequence that are more critical to the current output. The hybrid architecture combines the training stability of CTC with the alignment flexibility of the attention mechanism. The noisy speech data collected from the monitoring scene refers to speech samples recorded in actual deployment environments (such as dormitory corridors, common areas of prison cells) that contain real background noise (such as echoes, fan noise, and distant human voices). Using this data for adversarial training refers to introducing adversarial samples or using an adversarial loss function during model training, forcing the model to learn features that are not robust to noise perturbations, thereby significantly improving the model's recognition stability in complex and variable noise environments. By employing an advanced model architecture and using real noise data from the target scene for enhanced training, the acoustic model's adaptability to the complex acoustic environment of the monitoring site is fundamentally improved. This is the underlying foundation for ensuring the accuracy of the speech recognition process.

[0063] In this step illustrated in this embodiment, the language model is used to evaluate the probability of a word sequence occurring, providing linguistic constraints for decoding. The N-gram language model is a classic statistical model, computationally efficient, and adept at capturing local word order patterns; while neural language models (such as those based on recurrent neural networks or Transformers) can model more complex long-term dependencies. Fusion models (e.g., through linear interpolation or neural network-based adapters) combine the advantages of both, maintaining strong memorization of high-frequency domain vocabulary while also possessing better generalization capabilities. The domain text corpus is specifically collected and organized from textual materials related to the target monitoring scenario, such as historical event reports, conversation logs, and scenario-specific terminology lists. This corpus inevitably contains high-frequency vocabulary (such as "room check," "report," and "lights out") and specific language expression patterns (such as short, directive language and dialogues with coded language). The language model is trained using this corpus to ensure its probability distribution is highly aligned with the language habits of the monitoring scenario. By constructing and applying highly contextualized language models, the recognition system is more likely to output word sequences that conform to the language habits of the context, thereby effectively correcting acoustic recognition errors caused by accents and noise, and improving the recognition accuracy of domain-specific vocabulary and expressions.

[0064] In this step illustrated in this embodiment, the decoding process is a search for the optimal word sequence. The dynamic decoding network (typically a weighted finite state converter or an end-to-end decoder based on an attention mechanism) is the mechanism for fusing acoustic and linguistic information. Instead of simply adding acoustic scores to linguistic probabilities, it dynamically adjusts the weights of both in the decision-making process based on the current decoding state, historical outputs, and the confidence levels of the acoustic and linguistic information. Dynamic fusion allows the system to rely more on the acoustic model when the acoustic signal is clear, and more on the language model when the acoustic signal is ambiguous but the context is clear. The ultimate goal is to generate a transcribed text sequence that best matches the semantic context of the monitoring scene; that is, the sequence is not only acoustically sound and linguistically fluent, but also semantically consistent with the most likely dialogue content in the current monitoring scene. Through an intelligent dynamic fusion strategy, the advantages of the enhanced acoustic model and the highly contextualized language model are deeply combined, making globally optimal decisions at each step of decoding, thereby outputting the text result with the highest accuracy, fluency, and semantic relevance in the specific monitoring environment.

[0065] In this embodiment, at the input end, the acoustic model, through a hybrid architecture and adversarial training, specifically overcomes the challenge of complex environmental noise. At the language level, the language model, by fusing classical and neural methods and utilizing domain-specific corpora for training, accurately models the vocabulary and expression patterns unique to the monitoring scenario. At the decision end, the dynamic decoding network intelligently integrates the outputs of the first two methods to achieve optimal sequence search within the contextualized semantic context. This improves the performance of the speech recognition process in practical applications, ensuring that even in noisy and suboptimal real-world monitoring environments, the system can accurately convert risk-related key dialogues into text, providing high-quality and reliable input for subsequent semantic understanding and risk assessment.

[0066] In some embodiments, the transcribed text sequence is input into a domain-adaptive natural language understanding model to generate a structured risk event report, including: The transcribed text sequence is input into the intent recognition module, which is built based on a bidirectional long short-term memory network and an attention mechanism, and the module identifies and outputs the risk intent category corresponding to the transcribed text sequence. The transcribed text sequence and the identified risk intent category are input into the named entity recognition module to extract entity information related to the risk intent. The entity information includes personnel references, time points, and location coded messages. Based on risk intent categories and extracted entity information, key semantic summaries are generated through preset rule templates and semantic role annotations. The risk intent category is mapped to the risk type, the anonymized voiceprint features in the acoustic feature vector are mapped to the involved anonymous voiceprint identifier, and combined with the key semantic summary, a structured risk event report is generated.

[0067] In this step illustrated in this embodiment, the intent recognition module, built upon a bidirectional long short-term memory network and an attention mechanism, is a deep learning model. The bidirectional long short-term memory network captures the contextual dependencies of text sequences from both forward and backward directions, while the attention mechanism allows the model to focus on the words (such as verbs and specific nouns) most crucial for determining intent. After training, this module can classify the input transcribed text sequence into predefined risk intent categories, such as "physical conflict threat," "verbal abuse," "plea for help," and "planning illegal activities." These categories are predefined based on common risky behaviors in target monitoring scenarios. Understanding the core purpose or action direction of a dialogue text through a deep neural network model and categorizing it into a standardized risk category is the first step in achieving high-level semantic understanding of unstructured text, providing a clear thematic framework.

[0068] In this step illustrated in this embodiment, the named entity recognition module identifies and categorizes entity fragments with specific meanings within the text. The extraction process of this module is correlated with the risk intent category. The module dynamically prioritizes extracting the most relevant entity types based on the current risk intent (e.g., "planning illegal activities"). Person references include, but are not limited to, names, nicknames, and pronouns (e.g., "he," "that guy"); time points include explicit times (e.g., "after lights out") or relative times (e.g., "tomorrow"); and location slang refers to specific, potentially informal, location descriptions within the scene (e.g., "the usual place," "behind the water room"). Using intent categories as contextual input significantly improves the accuracy and relevance of entity extraction. For example, under the intent of "seeking help," "help" itself is not an entity, while under the intent of "planning," "water room" might be a key location entity. Guided by a clear dialogue intent, the module accurately locates and extracts specific information constituting the core elements of a risk event, providing factual elements for constructing an event description.

[0069] In this step shown in this embodiment, the preset rule template is a set of predefined sentence structures or filling frameworks based on different risk intent categories. Semantic role labeling analyzes the semantic relationships (such as agent, patient, time, and location) between predicates (usually related to intent) and various entities in the sentence. For example, for the intent of "physical conflict threat," the template might be "[Person A] threatens to take action against [Person B] at [time] [location euphemism]." The extracted entity information is filled into the corresponding template slots according to the relationships identified in the semantic role labeling, thereby automatically generating a fluent and complete key semantic summary. By automatically combining discrete intent categories and entity information into a highly condensed, easily understood natural language description through rule and semantic analysis, the transformation from machine-readable structured data to human-friendly report content is achieved.

[0070] In this step shown in this embodiment, the risk type can be the same as or a more general category of risk intent (e.g., classifying both "verbal abuse" and "threats" as "verbal conflict"). The involved anonymous voiceprint identifier is obtained by querying or calculating the anonymized voiceprint features corresponding to the audio segment that triggered this speech recognition, converting them into a unique and consistent anonymous ID within the system. Finally, the three core elements—risk type, involved anonymous voiceprint identifier (potentially multiple), and key semantic summary—along with necessary metadata such as timestamps and location information, are encapsulated according to a predetermined data format (e.g., JSON, XML) to generate a structured risk event report. A standardized digital event record containing key elements such as "what (risk type and summary), who (anonymous ID), where, and when" is created as the output of this risk assessment and also as a standardized input for subsequent behavior tracing and comprehensive analysis.

[0071] This embodiment details the transformation process from raw dialogue text to a structured intelligence report. A deep learning model grasps the main theme of the dialogue (risk intent), and then, guided by this intent, accurately extracts relevant entities such as people, times, and locations. Rules and semantic analysis are used to organize these elements into a coherent text summary, which is then standardized and encapsulated with risk types and anonymous voiceprint IDs from other modules. This achieves automated and intelligent understanding and archiving of unstructured speech content. The generated structured report is not only machine-readable and searchable, facilitating database entry and correlation analysis, but its key semantic summaries also allow humans to quickly grasp the full picture of the event. More importantly, by binding voiceprint IDs with semantic content in the report, precise linking points are provided for subsequent cross-event behavior tracing. The entire process decomposes the complex natural language understanding task into manageable and optimizable sub-steps, ensuring the accuracy, consistency, and usability of the final output information.

[0072] In some embodiments, the transcribed text sequence and the identified risk intent category are input together into the named entity recognition module to extract entity information related to the risk intent, including: The transcribed text sequence and risk intent category are input together into the named entity recognition module built on conditional random fields and pre-trained language models; The named entity recognition module dynamically activates the corresponding entity type extraction strategy based on the risk intent category. The entity type extraction strategy includes a nickname extraction strategy for personnel in violent conflict scenarios and a time and location cryptic extraction strategy for planning scenarios. The named entity recognition module identifies and labels entity fragments that conform to the entity type extraction strategy from the transcribed text sequence. It then utilizes the contextual representation capabilities of the pre-trained language model to disambiguate and normalize the entity fragments, generating standardized entity information.

[0073] In this step of the embodiment, the named entity recognition module is a component in natural language processing used to identify specific types of named entities in text. This embodiment adopts an architecture based on Conditional Random Fields (CRFs) and pre-trained language models. CRFs are discriminative probabilistic graphical models, well-suited for labeling sequential data, effectively modeling dependencies between labels (e.g., after a word is labeled as the beginning of a "person's name," subsequent words are more likely to be labeled as the middle or end of a "person's name"). Pre-trained language models (such as BERT, RoBERTa, or their lightweight versions) provide powerful context-sensitive word vector representations. The model uses the pre-trained language model to obtain deep contextual semantic representations of the input text, and then inputs these representations into the CRF layer for sequence labeling decisions. The powerful semantic understanding capabilities of the pre-trained model are utilized to improve the accuracy of entity boundary recognition and type judgment, while the sequence modeling advantages of CRFs ensure the overall consistency of the labeling results.

[0074] In this step illustrated in this embodiment, the entity type extraction strategy is not a fixed, all-encompassing set of entity types, but rather a set of pre-configured, targeted extraction rules or model focus areas based on different risk intent categories. For example, when the risk intent is identified as "violent conflict" or "physical threat," the module dynamically activates a strategy focused on extracting nicknames, concentrating on identifying informal terms referring to specific individuals, such as "bald," "fat," or "third brother," while potentially reducing attention to entities like "time point" or "tool." Conversely, when the intent is "planning illegal activities," a strategy focused on extracting time and location coded language is activated, prioritizing the capture of spatiotemporal information related to the plan's implementation, such as "after lights out," "tomorrow afternoon," "behind the warehouse," and "grove." By injecting the upstream intent understanding results as prior knowledge into the entity extraction process, the model can focus on the information dimensions most relevant to the current risk, thereby significantly improving the recall and accuracy of key entity extraction and reducing mislabeling of irrelevant entities, making the extraction results more targeted and practical.

[0075] In this step shown in this embodiment, the module locates and marks entity fragments belonging to the target type in the transcribed text sequence according to the activation strategy (e.g., marking "after lights out" as...). <time>Subsequently, deep processing is performed using the contextual representation capabilities of the pre-trained language model: first, disambiguation, which determines the exact referent of an entity based on the context, such as determining which nickname of a person mentioned earlier is being referred to by "he"; second, normalization, which transforms non-standard expressions into unified expressions within the system, such as normalizing "after lights out" and "when the lights go out" to "lights out time", and normalizing "behind the water room" and "behind the restroom" to "area behind the water room". After these processes, the final standardized entity information is a set of data with a clear structure and explicit meaning, which can be directly used for subsequent summary generation and report encapsulation. After completing the initial identification, a deep semantic model is further used to address the issues of referential ambiguity and expression diversity in natural language, ensuring that the extracted entity information is accurate, consistent, and directly usable for subsequent logical processing, thereby improving the reliability and output quality of the entire semantic understanding process.

[0076] This embodiment employs a hybrid model architecture (CRF + pre-trained model) to achieve powerful basic extraction capabilities. It introduces risk intent categories as high-level semantic guidance, dynamically adjusting the extraction focus to achieve a shift from "aimless full extraction" to "targeted and precise extraction." Utilizing the deep semantic understanding capabilities of the same pre-trained model, it disambiguates and normalizes the initial extraction results, producing high-quality standard information. This significantly improves the effectiveness and efficiency of entity extraction in specific security scenarios. The system no longer attempts to identify all types of entities but concentrates computing power to accurately capture core elements most relevant to the current risk, such as people, time, and location, making the generated structured report more focused and with higher value density. Simultaneously, through disambiguation and normalization, it ensures that the same entity under different events and different expressions can be correctly associated and understood, providing a solid and reliable data foundation for subsequent cross-event behavior pattern analysis and precise source tracing.

[0077] In some embodiments, based on the anonymous voiceprint identifier involved in the structured risk event report, a correlation query is performed in the anonymous voiceprint behavior database to obtain the related historical behavior trajectory, including: Using the anonymous voiceprint identifier involved in the structured risk event report as the query key, retrieve all related historical records in the anonymous voiceprint behavior database; For the retrieved historical records, extract the corresponding historical timestamps, historical monitoring area locations, and historical acoustic feature vectors in chronological order. Based on historical timestamps and historical monitoring area locations, the spatiotemporal activity trajectory of the anonymous voiceprint identifier involved was constructed; Based on historical acoustic feature vectors, clustering analysis algorithms are used to identify risk behavior patterns that frequently appear in the historical records of the anonymous voiceprint identifiers involved in the case. By integrating spatiotemporal activity trajectories with identified risk behavior patterns, associated historical behavior trajectories are generated.

[0078] In this step of the embodiment, the anonymous voiceprint identifier involved is the anonymized voiceprint ID associated with the current risk event report. The anonymous voiceprint behavior database is a continuously running time-series database, whose core data unit is a "record." Each record contains at least: an anonymous voiceprint identifier, the timestamp at which the identifier was detected, its location in the monitoring area, and the original acoustic feature vector (or derived features extracted from it) that generated the identifier. By using the identifier in the current report as the query key and performing an exact match search in the database, the set of historical records captured by the system for that ID at all past moments can be obtained. Utilizing the uniqueness and stability of voiceprint identifiers, rapid retrieval and aggregation of all traceable past activities of a specific anonymous individual can be achieved, providing a raw data foundation for behavioral analysis.

[0079] In this step shown in this embodiment, the historical timestamp records the specific time each time the ID appears; the historical monitoring area location can be a specific room number, corridor number, or area partition code; and the historical acoustic feature vector contains the voiceprint features and emotional feature vector at the time of the appearance. Arranging these records in chronological order constructs an activity log for the ID organized chronologically. The original record entries in the database are converted into a structured data sequence arranged in a time series, containing spatiotemporal and status information.

[0080] In this step illustrated in this embodiment, the spatiotemporal activity trajectory is a visualized or structured description of an individual's movement and behavior across time and space. Specifically, each historical record is considered a "point," with its horizontal axis representing a historical timestamp and its vertical axis (or, on a two-dimensional map, its corresponding historical monitoring area location). Connecting these points chronologically, or rendering them in a geographic information system, forms the spatiotemporal activity trajectory for that ID. This trajectory can reveal the individual's activity patterns, such as when and where they typically appear, the size of their activity range, whether they exhibit unusual loitering, or nighttime activity. Through simple spatiotemporal data association and visualization techniques, discrete observation points are transformed into continuous, meaningful behavioral paths, thus intuitively showcasing the individual's macroscopic activity patterns.

[0081] In this step shown in this embodiment, the historical acoustic feature vector not only contains voiceprint features (used to confirm it's the same person), but more importantly, it contains emotional features at each occurrence. Clustering analysis algorithms (such as K-means, DBSCAN) are unsupervised machine learning methods used to group similar data points. In this step, the algorithm performs cluster analysis on the emotional feature vectors in all historical records. If the emotional feature vectors of multiple historical records of this ID (especially records unrelated to the labeled risk events) are found to be clustered into categories such as "anger," "fear," or "excitement," and this clustering shows a certain pattern in time (such as always at night), then a frequently occurring risky behavior pattern is identified, for example, "this ID often exhibits emotional agitation at night." By using machine learning methods, recurring, potentially high-risk behavioral tendencies or state patterns can be automatically discovered from an individual's long-term emotional state data, achieving a sublimation from single events to long-term behavioral characteristics.

[0082] In this step shown in this embodiment, integration refers to the correlation and comprehensive description of the macro-level activity patterns (spatiotemporal trajectories) and micro-level behavioral tendencies (risk patterns) obtained in the preceding steps. For example, the final correlation historical behavior trajectory report may state: "Identifier ID-007's activity trajectory in the past month was mainly concentrated in the corridor between Area A and Area B, often active between 22:00 and 24:00; cluster analysis shows that three independently recorded emotional characteristics during this period belong to the 'high conflict tendency' cluster. Combined with its spatiotemporal trajectory, it is judged that it has a potential behavioral pattern of causing trouble in specific areas at night." This generates a comprehensive behavioral profile that combines qualitative and quantitative analysis, not only describing "where and when the individual went," but also revealing "what kind of risk state they were often in at those times and places," thus providing extremely valuable background information and trend judgment for assessing current events.

[0083] This embodiment uses identifiers to link all historical data; it analyzes this data from both spatiotemporal and emotional state dimensions to construct macro-level activity trajectories and uncover micro-level risk patterns; integrating the two creates a comprehensive historical behavior trajectory report. This elevates the safety management system from merely responding to immediate events to understanding an individual's long-term behavioral background and potential risk tendencies. Managers receive not just an alarm, but a "behavioral profile" of the involved party. This allows for more precise handling of current events (e.g., more decisive intervention for parties with a history of misconduct) and enables true early warning (proactively monitoring parties exhibiting specific risk patterns but not yet triggering serious incidents).

[0084] Unlike existing technologies, the above technical solution acquires raw audio stream data of the target monitoring area; performs parallel feature extraction on the raw audio stream data to generate acoustic feature vectors containing anonymized voiceprint features and emotional features, as well as audio event tags containing non-speech physical environment sound categories; calculates real-time risk confidence based on the acoustic feature vectors and audio event tags using a preset risk assessment model; only when the real-time risk confidence exceeds a preset risk threshold is speech recognition triggered on the raw audio stream data within the current time window and subsequent time windows to generate corresponding transcribed text sequences; inputs the transcribed text sequences into a domain-adaptive natural language understanding model to generate a structured risk event report containing risk type, involved anonymous voiceprint identifiers, and key semantic summaries; performs correlation queries in an anonymous voiceprint behavior database constructed based on long-term accumulated acoustic feature vectors based on the involved anonymous voiceprint identifiers to obtain related historical behavior trajectories; finally, integrates the structured risk event report and related historical behavior trajectories to generate a comprehensive source tracing and judgment report and trigger an early warning. By introducing a conditional triggering mechanism that first analyzes non-content features such as voiceprints and emotions, and then triggers deep semantic understanding only after the risk confidence level is reached, the privacy infringement and computational waste problems caused by the full-time, full-volume analysis mode are effectively solved. Voice content is only analyzed in a targeted manner when a comprehensive risk assessment indicates a real risk, achieving a balance between privacy protection and effective monitoring. Simultaneously, by integrating multimodal features such as voiceprints, emotions, and ambient sounds for risk assessment, and establishing an anonymous voiceprint behavior database for historical tracing, the accuracy and depth of risk identification are significantly improved. This enables the system not only to issue real-time alerts but also to provide behavioral trajectory and pattern analysis of the involved parties, achieving a leap from single-event alerts to continuous risk assessment and tracing, providing managers with more comprehensive and intelligent decision support.

[0085] Finally, it should be noted that although the above embodiments have been described in the text and drawings of this application, this should not limit the scope of patent protection of this application. Any technical solutions that are based on the essential concept of this application and utilize the content described in the text and drawings of this application, resulting in equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this application.< / time>

Claims

1. A method for information monitoring and tracing based on voiceprint and semantic analysis, characterized in that, include: Acquire raw audio stream data of the target monitoring area; Parallel feature extraction is performed on the original audio stream data to generate acoustic feature vectors and audio event tags. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event tags include non-speech physical environment sound categories. Based on the acoustic feature vector and the audio event label, the real-time risk confidence level is calculated using a preset risk assessment model; When the real-time risk confidence exceeds the preset risk threshold, speech recognition is triggered on the original audio stream data in the current time window and subsequent time windows to generate the corresponding transcribed text sequence. The transcribed text sequence is input into a domain-adaptive natural language understanding model to generate a structured risk event report, which includes the risk type, the anonymous voiceprint identifier involved, and a key semantic summary. Based on the anonymous voiceprint identifier involved in the structured risk event report, perform a correlation query in the anonymous voiceprint behavior database to obtain the correlation historical behavior trajectory. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. By integrating the structured risk event reports with the associated historical behavioral trajectories, a comprehensive source tracing and analysis report is generated and an early warning is triggered. The process includes parallel feature extraction of the original audio stream data to generate acoustic feature vectors and audio event labels, including: Speech activity detection is performed on the raw audio stream data to separate speech segments containing human voices from non-speech audio segments; For the aforementioned speech segment, an anonymized voiceprint feature vector is generated using the Mel frequency cepstral coefficient extraction algorithm and the fundamental frequency contour tracking algorithm. Simultaneously, the spectral features of the speech segment are analyzed using a pre-trained acoustic emotion classification model to generate an emotion feature vector. The voiceprint feature vector and the emotion feature vector together constitute the acoustic feature vector. For the non-speech audio segments, an audio event detection model based on a convolutional neural network is used to identify them and generate audio event tags that characterize the categories of physical environment sounds. The audio event tags include, but are not limited to, the sound of glass breaking, the sound of heavy objects hitting, and the sound of running.

2. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 1, characterized in that, For the aforementioned speech segment, an anonymized voiceprint feature vector is generated using a Mel-frequency cepstral coefficient extraction algorithm and a fundamental frequency contour tracking algorithm, including: The speech segment is preprocessed by framing and windowing to obtain short-time speech frames; For each short-time speech frame, the short-time speech frame is converted to the frequency domain by fast Fourier transform, filtered by the Mel filter bank, and the logarithmic energy of the short-time speech frame is calculated to obtain the Mel spectrum; Applying discrete cosine transform to the Mel spectrum, multi-order Mel frequency cepstral coefficients are extracted to form the first feature vector characterizing the vocal tract characteristics; For the speech segment, the contour of the fundamental frequency of the speech segment changing over time is tracked by a fundamental frequency extraction algorithm based on autocorrelation function, and the statistical characteristics of the fundamental frequency are calculated to form a second feature sub-vector characterizing the characteristics of the sound source. The first feature vector and the second feature vector are concatenated and normalized to generate the anonymized voiceprint feature vector, which is not directly related to the speaker's real identity information.

3. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 1, characterized in that, Based on the acoustic feature vector and the audio event label, a real-time risk confidence level is calculated using a preset risk assessment model, including: The emotion feature vector in the acoustic feature vector is matched with a preset negative emotion feature library to generate an emotion risk score. The audio event tags are matched with a preset high-risk audio event database to generate an environmental event risk score; Based on the emotional risk score and the environmental event risk score, an initial risk score is calculated using a weighted fusion algorithm; The initial risk score is input into a time series prediction model trained based on historical risk event data. The model combines the risk score sequences of the current time window and adjacent historical time windows to output the real-time risk confidence level.

4. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 1, characterized in that, When the real-time risk confidence level exceeds a preset risk threshold, speech recognition is triggered on the original audio stream data within the current time window and subsequent time windows to generate a corresponding transcribed text sequence, including: When the real-time risk confidence exceeds the preset risk threshold, a voice recognition trigger command is generated; According to the speech recognition trigger command, the original audio stream data of the current time window and the preset duration in the subsequent time windows are extracted from the audio buffer to form the audio segment to be recognized; The audio segment to be identified is input into a domain-optimized speech recognition engine, which is loaded with an acoustic model of high-frequency words and noisy environment for the monitoring scenario. The speech recognition engine decodes the audio segment to be recognized, generating a transcribed text sequence containing timestamps. The words in the transcribed text sequence are associated with the domain knowledge base of the monitoring scene.

5. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 4, characterized in that, The acoustic model of the speech recognition engine adopts a hybrid architecture based on connection-time classification and attention mechanism, and uses noisy speech data collected from the monitoring scene for adversarial training to enhance its robustness under complex environmental noise. Meanwhile, the language model of the speech recognition engine is a fusion model of N-gram and neural language model trained on the domain text corpus of the monitoring scenario. The domain text corpus includes the high-frequency words and language expression patterns in specific scenarios. During the decoding process, the acoustic score output by the acoustic model and the language probability output by the language model are fused through a dynamic decoding network to generate a transcribed text sequence that best matches the semantic context of the monitored scene.

6. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 1, characterized in that, The transcribed text sequence is input into a domain-adaptive natural language understanding model to generate a structured risk event report, including: The transcribed text sequence is input into an intent recognition module built on a bidirectional long short-term memory network and an attention mechanism to identify and output the risk intent category corresponding to the transcribed text sequence. The transcribed text sequence and the identified risk intent category are input into the named entity recognition module to extract entity information related to the risk intent. The entity information includes personnel references, time points, and location cryptic messages. Based on the risk intent category and the extracted entity information, the key semantic summary is generated through a preset rule template and semantic role labeling. The risk intent category is mapped to the risk type, the anonymized voiceprint features in the acoustic feature vector are mapped to the involved anonymous voiceprint identifier, and combined with the key semantic summary, the structured risk event report is encapsulated and generated.

7. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 6, characterized in that, The transcribed text sequence and the identified risk intent category are input into the named entity recognition module to extract entity information related to the risk intent, including: The transcribed text sequence and the risk intent category are input together into the named entity recognition module built based on conditional random fields and pre-trained language models; The named entity recognition module dynamically activates the corresponding entity type extraction strategy according to the risk intent category. The entity type extraction strategy includes a nickname extraction strategy for violent conflict scenarios and a time and location cryptic extraction strategy for planning scenarios. The named entity recognition module identifies and labels entity fragments that conform to the entity type extraction strategy from the transcribed text sequence, and uses the context representation capability of the pre-trained language model to disambiguate and normalize the entity fragments to generate standardized entity information.

8. The information monitoring and tracing method based on voiceprint and semantic analysis according to claim 1, characterized in that, Based on the anonymous voiceprint identifier involved in the structured risk event report, a correlation query is performed in the anonymous voiceprint behavior database to obtain the related historical behavior trajectory, including: Using the anonymous voiceprint identifier involved in the structured risk event report as the query key, retrieve all related historical records in the anonymous voiceprint behavior database; For the retrieved historical records, extract the corresponding historical timestamps, historical monitoring area locations, and historical acoustic feature vectors in chronological order. Based on the historical timestamps and historical monitoring area locations, the spatiotemporal activity trajectory of the anonymous voiceprint identifier involved in the incident is constructed; Based on the historical acoustic feature vectors, a clustering analysis algorithm is used to identify risk behavior patterns that frequently appear in the historical records of the anonymous voiceprint identifiers involved in the case. The spatiotemporal activity trajectory is integrated with the identified risk behavior patterns to generate the associated historical behavior trajectory.

9. An information monitoring and tracing system based on voiceprint and semantic analysis, characterized in that, The system applicable to the method of any one of claims 1 to 8, the system comprising: The audio acquisition module is configured to acquire raw audio stream data of the target monitoring area; A parallel feature extraction module is configured to perform parallel feature extraction on the original audio stream data to generate acoustic feature vectors and audio event labels. The acoustic feature vectors contain anonymized voiceprint features and emotion features, and the audio event labels include non-speech physical environment sound categories. The risk assessment module is configured to calculate real-time risk confidence based on the acoustic feature vector and the audio event label using a preset risk assessment model. The condition-triggered speech recognition module is configured to trigger speech recognition of the original audio stream data in the current time window and subsequent time windows and generate the corresponding transcribed text sequence when the real-time risk confidence exceeds the preset risk threshold. The semantic understanding and report generation module is configured to input the transcribed text sequence into a domain-adaptive natural language understanding model to generate a structured risk event report, which includes the risk type, the anonymous voiceprint identifier involved, and key semantic summaries. The behavior tracing query module is configured to perform a correlation query in the anonymous voiceprint behavior database based on the anonymous voiceprint identifier involved in the structured risk event report, and obtain the correlation historical behavior trajectory. The anonymous voiceprint behavior database is constructed based on long-term accumulated acoustic feature vectors. The comprehensive analysis and early warning module is configured to integrate the structured risk event report with the associated historical behavior trajectory, generate a comprehensive source tracing analysis report, and trigger an early warning.

Images