State determination method and device based on audio information, equipment and medium

Abstract

The invention relates to the technical field of voice processing, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a state determination method, device, equipment and medium based on audio information. The method comprises the following steps: uniformly performing voice-to-text processing after a preset fixed time length to obtain text information, extracting acoustic features and linguistic features, generating a multi-dimensional feature vector after fusion, inputting a pre-trained analysis model, generating a state probability value, and determining a target state corresponding to an audio signal based on the state probability value. According to the method, the multi-dimensional feature vectors are fused on the basis of acoustic features and linguistic features, and the pre-trained analysis model is introduced to judge the state probability value, so that the problems of incomplete feature extraction, insufficient feature fusion and poor judgment result accuracy and generalization ability are effectively solved; and the accuracy and the stability of audio signal state judgment are improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a method, apparatus, device, and storage medium for determining the state based on audio information. Background Technology

[0002] In the healthcare field, Parkinson's disease, a common neurodegenerative disease, is crucial for early diagnosis in order to slow disease progression and improve patients' quality of life. Current diagnostic methods primarily rely on the clinical experience and subjective judgment of neurologists, combined with auxiliary examinations such as neuroimaging and motor function assessments. These methods demand a high level of expertise from physicians, are highly subjective, and have low diagnostic efficiency, especially in the early stages of the disease when symptoms are subtle and accurate identification is even more difficult. While some attempts at speech analysis-based assisted diagnosis have been made, utilizing features in speech data that reflect neurological dysfunction, the extraction of acoustic and linguistic features in current technologies is often insufficient. This fails to fully capture the multidimensional changes in a patient's speech, resulting in limited accuracy, insufficient generalization ability, and poor adaptability to diverse populations and different contexts.

[0003] In the fintech sector, existing voice processing technologies have been gradually applied to scenarios such as identity authentication, customer service, and risk monitoring. However, in these applications, voice analysis algorithms are typically geared towards static, structured, single-feature inputs, lacking the ability to analyze voice data comprehensively and multidimensionally. This deficiency results in limited feature extraction coverage when dealing with complex customer interaction voice data in fintech scenarios. It makes it difficult to support a deep understanding and accurate judgment of implicit risk states or user states within voice information, thus reducing the value of voice data in risk control and service optimization. Summary of the Invention

[0004] The main objective of this invention is to provide a method, apparatus, device, and storage medium for determining the state of audio information, aiming to solve the technical problem that the existing technology is insufficient in the comprehensive extraction and fusion processing of acoustic and linguistic features, resulting in poor accuracy and generalization ability in determining the state of audio signals.

[0005] To achieve the above objectives, the present invention provides a state determination method based on audio information, comprising:

[0006] Acquire an audio signal, and perform noise reduction and endpoint detection on the audio signal to obtain an initial audio segment;

[0007] The initial audio segments are unified to a preset fixed duration to obtain target audio segments, and the target audio segments are processed for speech-to-text conversion to obtain text information;

[0008] Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic features and the linguistic features are then fused to generate a multi-dimensional feature vector.

[0009] The multi-dimensional feature vector is input into a pre-trained analysis model, and the analysis model generates state probability values.

[0010] The target state corresponding to the audio signal is determined based on the state probability value.

[0011] Furthermore, to achieve the above objectives, the present invention provides a state determination device based on audio information, comprising:

[0012] An audio preprocessing module is used to acquire audio signals and perform noise reduction and endpoint detection on the audio signals to obtain initial audio segments;

[0013] The audio standardization module is used to standardize the initial audio segment to a preset fixed duration to obtain the target audio segment, and to perform speech-to-text processing on the target audio segment to obtain text information;

[0014] The feature fusion module is used to extract acoustic features from the target audio segment and linguistic features from the text information, and to fuse the acoustic features and the linguistic features to generate a multi-dimensional feature vector;

[0015] The model inference module is used to input the multi-dimensional feature vector into a pre-trained analysis model and generate state probability values ​​through the analysis model.

[0016] The state determination module is used to determine the target state corresponding to the audio signal based on the state probability value.

[0017] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an audio information-based state determination program stored in the memory and executable on the processor, wherein when the audio information-based state determination program is executed by the processor, it implements the steps of the audio information-based state determination method as described above.

[0018] Furthermore, to achieve the above objectives, the present invention also provides a computer-readable storage medium storing a state determination program based on audio information, wherein when the state determination program based on audio information is executed by a processor, it implements the steps of the state determination method based on audio information as described above.

[0019] Beneficial Effects: This invention relates to the field of speech processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for determining the state of an audio signal, including: acquiring an audio signal and performing noise reduction and endpoint detection to obtain an initial audio segment; unifying the initial audio segment to a preset fixed duration to obtain a target audio segment and performing speech-to-text processing to obtain text information; extracting acoustic features from the target audio segment; extracting linguistic features from the text information; fusing the acoustic and linguistic features to generate a multi-dimensional feature vector; inputting the multi-dimensional feature vector into a pre-trained analysis model to generate a state probability value; and determining the target state corresponding to the audio signal based on the state probability value. This invention effectively solves the problems of incomplete feature extraction, insufficient feature fusion, and poor accuracy and generalization ability in existing technologies by fusing multi-dimensional feature vectors based on acoustic and linguistic features and introducing a pre-trained analysis model to determine the state probability value, thereby improving the accuracy and stability of audio signal state determination. Attached Figure Description

[0020] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0021] Figure 1 This is a schematic diagram of an application environment for a state determination method based on audio information according to an embodiment of the present invention;

[0022] Figure 2 This is a flowchart illustrating an embodiment of the state determination method based on audio information according to the present invention;

[0023] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the state determination device based on audio information of the present invention;

[0024] Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0025] Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0026] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0027] The state determination method based on audio information provided in this invention can be applied to, for example... Figure 1In this application environment, the user terminal communicates with the server via a network. The server can acquire audio signals from the user terminal, perform noise reduction and endpoint detection to obtain an initial audio segment. This initial audio segment is then standardized to a preset fixed duration to obtain a target audio segment, which is then processed for speech-to-text conversion to obtain text information. Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic and linguistic features are fused to generate a multi-dimensional feature vector. This multi-dimensional feature vector is input into a pre-trained analysis model to generate state probability values. Based on these state probability values, the target state corresponding to the audio signal is determined. This invention effectively solves the problems of incomplete feature extraction, insufficient feature fusion, and poor accuracy and generalization ability in existing technologies by fusing multi-dimensional feature vectors based on acoustic and linguistic features and introducing a pre-trained analysis model to determine the state probability values, thereby improving the accuracy and stability of audio signal state determination. The user terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The invention will be described in detail below through specific embodiments.

[0028] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the state determination method based on audio information provided by the present invention. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0029] like Figure 2 As shown, the state determination method based on audio information proposed in this invention includes the following steps:

[0030] S10, acquire the audio signal, and perform noise reduction processing and endpoint detection on the audio signal to obtain an initial audio segment;

[0031] In this embodiment, the first step is to acquire audio signals. Acquisition is not limited to recording devices; it can also involve acquiring speech-related vibration signals through other sensors or receiving remotely transmitted audio data through communication devices. The acquired audio signals are human voice data, and their sources can include, but are not limited to, face-to-face acquisition, remote video conference recordings, or telephone call recordings. In practical implementation, the acquisition device can use a multi-channel microphone array to improve signal acquisition sensitivity and spatial resolution, or a directional microphone to reduce environmental interference. The acquired audio signals will contain environmental noise and background interference, requiring noise reduction processing. Noise reduction typically employs digital signal processing algorithms to convert the acquired audio data into a digital sequence, followed by filtering to eliminate high-frequency noise, low-frequency interference, or impulse noise. In digital filtering, adaptive filtering algorithms can be used to dynamically adjust filter parameters based on environmental noise characteristics, or spectral subtraction algorithms can be used to calculate the difference between the noise spectrum and the speech spectrum in the frequency domain. After filtering, the signal-to-noise ratio is improved, resulting in higher robustness for subsequent analysis. After noise reduction, endpoint detection is performed to identify the start and end positions of the speech, thereby extracting effective segments containing the actual spoken content. Endpoint detection is implemented by performing short-time energy calculations on the audio data and using a combination of preset energy thresholds and zero-crossing rate thresholds to determine the location of speech segments in the audio. Specifically, the audio data can be segmented into frames, with each frame length selectable from 20 to 30 milliseconds and a frame shift length selectable from 10 to 20 milliseconds. Then, the short-time energy and zero-crossing rate are calculated for each frame, and a dual-threshold strategy is used to determine the start and end points of the speech segment. Finally, by extracting continuous frames of audio data from the start point to the end point, a valid segment, i.e., the initial audio segment, is obtained, which can be used for subsequent processing. The order of the above operation steps cannot be changed. Noise reduction processing must be performed after acquiring the audio signal, and endpoint detection must be performed after noise reduction processing; otherwise, the energy feature calculation will be inaccurate, affecting the positioning accuracy of the start and end points.

[0032] Audio signals can be acquired using a single microphone or a multi-microphone array for enhanced anti-interference capabilities and spatial selectivity. For noise reduction, a spectral subtraction algorithm can be used. This involves pre-collecting background noise samples from non-speech segments to estimate the noise power spectral density, then subtracting the noise estimate from the speech segment to suppress noise interference. Alternatively, adaptive filtering algorithms, such as the LMS algorithm based on the minimum mean square error criterion, can be used to adaptively adjust filter weights to minimize output error and further improve speech quality. In endpoint detection, in addition to short-time energy and zero-crossing rate, spectral entropy can be introduced as an auxiliary criterion to enhance detection accuracy in low-energy speech segments. The energy threshold can be adjusted for different scenarios. For example, in environments with low background noise, the threshold can be set lower to avoid missing speech segments; in environments with high background noise, the threshold can be set higher to reduce false detections. Frame length and frame shift parameters can also be adjusted according to the sampling rate and speech characteristics. For example, at higher sampling rates, shorter frame lengths and frame shifts can be used to improve temporal resolution.

[0033] Example: In the healthcare field, patients' voice data can be acquired through dedicated acquisition devices. Even in the presence of noise from other equipment in the examination room, noise reduction and endpoint detection operations can ensure that clear segments containing only the patient's voice are obtained, providing a reliable data foundation for subsequent extraction of pronunciation abnormalities.

[0034] In the fintech business, it can be used for remote identity verification. Customers submit voice data through a voice interaction system. After noise reduction and endpoint detection, the input voice data is ensured to be clear, complete and authentic, which helps to improve the security and reliability of the remote verification process.

[0035] This embodiment acquires audio signals, performs noise reduction processing and endpoint detection, enabling subsequent analysis to be based on clear, accurate and well-positioned effective speech segments. This effectively reduces the impact of environmental interference, improves the overall accuracy of state determination, and enhances the system's adaptability to different noise environments.

[0036] S20, unify the initial audio segments to a preset fixed duration to obtain target audio segments, and perform speech-to-text processing on the target audio segments to obtain text information;

[0037] In this embodiment, the initial audio segments first need to be length-uniformed. Length uniformity means ensuring that all audio segments to be processed have the same duration. This process helps with subsequent algorithm alignment, batch computation, and model input format requirements. The preset fixed duration is an adjustable parameter, typically set according to the target application requirements, such as 1 second, 2 seconds, or 3 seconds. This duration needs to balance the integrity of speech content for user groups with different speaking speeds with model capacity. The duration of the initial audio segment is calculated by dividing the number of sampling points of the digital audio data by the sampling rate. If the actual duration of the initial audio segment exceeds the preset fixed duration, redundant audio data at both ends needs to be trimmed to retain the core speech content in the middle. The specific implementation of the trimming operation is to calculate the midpoint of the time axis and cut off an interval of the preset fixed duration length to both sides. If the actual duration is less than the preset fixed duration, zero-value audio data needs to be padded at the end of the initial audio segment. Padded zero values ​​are essentially adding silence sampling points to make up the length, ensuring that no additional speech content is introduced after length alignment, thereby maintaining data consistency. Encapsulation refers to converting trimmed or padded audio data into a target audio segment format consistent with a fixed format. Commonly used encapsulation formats include standard audio encapsulation formats such as PCM or WAV. After length unification, the target audio segment needs to undergo speech-to-text processing. Speech-to-text uses a trained acoustic model, language model, and decoder to map the input speech signal into text information. In specific implementations, commonly used acoustic models include the CTC (Connectionist Temporal Classification) model based on deep neural networks or an end-to-end model based on the Transformer architecture. After receiving the target audio segment, the acoustic model first segments it into frames, extracts short-time spectral features at the frame level as input features, then performs pattern matching and time alignment through the network, outputting a sequence of text symbols. Finally, context constraints are applied based on the language model to obtain the final text information output. During this process, it is necessary to ensure that the input target audio segment is consistent with the sampling rate, frame length, frame shift, and other parameters during model pre-training to obtain optimal recognition performance. Length unification and speech-to-text are closely related; length unification ensures the consistency of the input format, and format consistency is a necessary prerequisite for accurate speech-to-text recognition.

[0038] Unified length can be implemented in different ways. For example, in real-time systems, streaming length control can be used, where audio buffers are truncated in real time until a preset fixed duration is reached before direct output, suitable for low-latency applications. For non-real-time batch processing tasks, audio files can be globally analyzed before trimming or padding to ensure data quality. Trimming operations can also use dynamic energy distribution analysis to determine the retained segments, maximizing the proportion of effective speech information, rather than simply symmetrically truncating at the midpoint of time. When padding silence, the padding position can be adjusted, such as padding all the way to the end, or distributing padding to the beginning and end, to meet the temporal structure requirements of certain models. The encapsulation format can be adjusted according to the specific hardware platform. For example, a compact audio encoding format can be used in embedded systems to reduce storage overhead, while a high-fidelity lossless encapsulation format can be used in server systems to ensure data quality. The speech-to-text processing module can support loading speech recognition models in different languages, and can also enhance specialized speech recognition capabilities by fine-tuning model parameters in specific domains. For example, in the medical field, a language model containing medical terminology can be loaded, and in the financial field, a language model containing financial terminology can be loaded to improve industry adaptability.

[0039] Example: In the healthcare field, patients may have different speaking speeds during voice sampling. By standardizing the duration to a fixed length, the subsequent recognition system can work normally regardless of whether the input is a long or short sentence. This is especially suitable for remote health monitoring systems to collect patient voice data and analyze abnormal voice patterns.

[0040] In the fintech business, when users authenticate their identities via voice, audio segments of uniform length facilitate batch processing and model inference, enabling the system to provide a consistent input standard when faced with differences in the length of speech from different users, thereby ensuring the fairness and accuracy of voice verification.

[0041] This embodiment ensures that the subsequent speech-to-text module receives standardized input data through length uniformity processing, avoiding recognition errors or calculation interruptions caused by differences in input length. At the same time, by pruning or padding, effective speech information is preserved or supplemented, which helps to improve the accuracy of speech-to-text conversion, thereby improving the standardization and robustness of data in the entire processing flow.

[0042] S30, extract acoustic features from the target audio segment and extract linguistic features from the text information, fuse the acoustic features and the linguistic features to generate a multi-dimensional feature vector;

[0043] In this embodiment, the multi-dimensional data analysis first extracts acoustic features from the target audio segment. Acoustic features are a set of parameters reflecting the physical properties of speech, including but not limited to fundamental frequency variation, speech rate, pause duration, vocal cord vibration stability, spectral quality, spectral envelope, and dynamic spectral changes. The specific extraction process includes frame segmentation of the target audio segment, typically using a 25-millisecond frame length and a 10-millisecond frame shift to ensure temporal continuity. In each frame, short-time energy and fundamental frequency trajectory are calculated. The fundamental frequency variation parameters are determined using the autocorrelation function or the Cepstrum method. The average fundamental frequency variation and standard deviation across the entire audio segment are statistically analyzed to characterize fundamental frequency stability. Speech rate is estimated using syllable density, i.e., the number of syllables per unit time. Pause duration is extracted through silence interval analysis of the speech signal, and the length of all silence segments is statistically analyzed. Vocal cord vibration stability is measured using fundamental frequency perturbation and amplitude perturbation indices, calculating the degree of variation in fundamental frequency and amplitude between consecutive periods. Spectral quality is measured using the harmonic-to-noise ratio, which measures the proportion of harmonic components to noise components in the speech. The spectral envelope is described using Mel-frequency cepstral coefficients (MFCC) or linearly predicted cepstral coefficients (LPCC) to depict the spectral shape. Spectral dynamics are captured by spectral flux and spectral entropy to determine the time-varying characteristics of the spectrum. The extracted parameters form a set of acoustic features.

[0044] After acquiring text information, linguistic features are extracted. These features reflect the content and structural attributes of language expression, including semantic vector representation, lexical diversity measurement, sentence structure complexity, and text similarity. Semantic vector representation encodes the text using a pre-trained semantic model (e.g., BERT), mapping text information to fixed-dimensional vectors. Lexical diversity is measured by calculating the ratio of the number of distinct words to the total number of words or by using entropy-based distribution metrics. Sentence structure complexity is assessed through parse tree depth analysis, statistically analyzing the average nesting depth of clauses and the length of syntactic dependency paths within sentences. Text similarity is calculated by traversing the text using a sliding window, employing cosine similarity to measure the semantic closeness of adjacent sentences. The extracted parameters form a set of linguistic features.

[0045] Acoustic and linguistic feature sets are fused after dimensional alignment. Fusion methods include feature concatenation or attention-based weighted fusion to generate a multi-dimensional feature vector. This vector serves as input to the subsequent model, ensuring that the feature dimensions are consistent with those used during model training, and that the sub-features are arranged in a predefined order to guarantee data consistency and interpretability. The fusion of acoustic and linguistic features ensures the complementarity of the physical properties of speech and the information in the text content, enabling the generated multi-dimensional feature vector to simultaneously possess multimodal expressive capabilities of both speech and language, providing a rich foundation of input information for state determination.

[0046] Different feature combinations can be employed, such as concatenating Mel-frequency cepstral coefficients with semantic vector representations, suitable for edge device scenarios with lower computational resource requirements. Dimensionality reduction can also be performed before feature fusion, for example, using principal component analysis (PCA) to reduce the dimensionality of acoustic feature sets and decrease the model input size to adapt to mobile devices. For environments requiring high-precision analysis, such as cloud server applications, a cross-feature generation step can be added during the fusion process, such as the product or ratio of acoustic features and semantic vectors, to enhance the model's ability to recognize cross-modal association patterns. In specific domain applications, such as fintech, domain dictionaries can be introduced to annotate text information, improving the lexical diversity measurement and the professionalism of semantic vector representations. In the healthcare field, specific linguistic features, such as the structural complexity of medical inquiry statements, can be used as a supplementary dimension in the fusion process to improve adaptability to patient speech in medical scenarios.

[0047] Example: In the medical and health field, when applied to the early detection of Parkinson's disease, acoustic features such as fundamental frequency variation parameters and vocal cord vibration stability parameters can reflect vocal tremor, while linguistic features such as sentence structure complexity can reflect language organization ability. Combining the two can more comprehensively depict the patient's condition.

[0048] In the fintech business, combining acoustic features of emotional tone indicators with linguistic features of text content features can be used for multimodal user authentication or risk interaction analysis, helping to improve the security and accuracy of customer interactions in financial service scenarios.

[0049] This embodiment extracts acoustic features from the target audio segment and linguistic features from the text information, and then fuses the two to generate a multi-dimensional feature vector. This fully integrates the physical features of speech and the content features of language, enhances the diversity and expressiveness of feature input, provides richer input information for subsequent state analysis, and thus improves the accuracy and adaptability of overall state determination.

[0050] S40, input the multi-dimensional feature vector into the pre-trained analysis model, and generate state probability values ​​through the analysis model;

[0051] In this embodiment, the operation of inputting multi-dimensional feature vectors into a pre-trained analysis model includes three main steps: preprocessing of the multi-dimensional feature vectors, data input, and model inference. First, the multi-dimensional feature vectors, as numerical vectors generated by concatenating acoustic and linguistic feature groups, have a fixed dimensional arrangement and normalized feature distribution. Before inputting them into the analysis model, the multi-dimensional feature vectors typically need to be normalized or standardized, such as zero-mean unit variance normalization or min-max scaling, to ensure that the distribution of the input data is consistent with that during model training, avoiding performance degradation due to feature distribution deviations. Subsequently, the normalized multi-dimensional feature vectors are sequentially input into the analysis model's input interface. The analysis model is a pre-trained parametric model capable of accepting feature vectors in a fixed format and performing forward propagation calculations.

[0052] The process of generating state probability values ​​through analysis models involves deep computation of input features. These models typically contain multiple sub-modules. Analysis models may include multi-head attention modules to capture correlations between different feature dimensions, calculate the association weights between features, and form attention-enhanced features through weighted intermediate results. They may also include bidirectional temporal modeling modules, such as Bidirectional Long Short-Term Memory (BiLSTM) networks, which capture potential order dependencies in multi-dimensional feature vectors through forward and backward units, obtaining positive and negative contextual information, and then combining them to generate temporal dependency features. Attention-enhanced features and temporal dependency features can be concatenated to form a fused feature representation, which serves as input to the subsequent classification output layer. The classification output layer generates a continuous probability distribution at the output nodes through fully connected computation and the weight matrix learned during model training. Finally, activation functions such as Sigmoid or Softmax normalize the output values ​​to state probability values, representing the degree of association between the current audio information and the target state.

[0053] Different model structures can be adopted to adapt to different application scenarios. For edge computing devices with high response speed requirements, a lightweight deep feedforward network can be used to replace the multi-head attention module and the bidirectional temporal modeling module, reducing computation and improving inference speed. For scenarios with high accuracy requirements, such as cloud services in healthcare, the Transformer encoder can be combined with the BiLSTM network. By leveraging the global feature modeling capability of the Transformer module and the local temporal feature capture capability of the BiLSTM module, multi-level and multi-scale feature understanding can be achieved, improving the analysis model's ability to learn complex speech patterns. In the fintech business field, by adding a specially trained industry feature embedding layer, features related to user behavior in the financial field can be encoded in advance and jointly input with multi-dimensional feature vectors, improving the model's adaptability to state judgment of financial interactive speech.

[0054] Example: In the healthcare field, inputting multi-dimensional feature vectors into an analysis model for determining the state of a patient's speech data can effectively identify acoustic and language abnormalities related to Parkinson's disease, improving the sensitivity of early detection.

[0055] In the fintech business, inputting multi-dimensional feature vectors into the analysis model can be applied to the intelligent recognition of user behavior states in financial customer service voice interaction scenarios, such as detecting emotional features such as anxiety and hesitation in voice and the standardization of dialogue content, providing data support for risk control and customer service quality assessment.

[0056] This embodiment inputs multi-dimensional feature vectors into a trained analysis model and generates state probability values, enabling the state determination results to be based on in-depth acoustic and linguistic information to improve the accuracy, robustness, and universality of state determination, and ensuring that the model has strong generalization ability under different data distributions.

[0057] S50, determine the target state corresponding to the audio signal based on the state probability value.

[0058] In this embodiment, determining the target state corresponding to an audio signal based on its state probability value involves two main steps: threshold comparison and classification. The state probability value, as the output of the analysis model, is a numerical value representing the relationship between audio information and a predefined target state, typically normalized to the range of 0 to 1. The threshold comparison step first extracts a set of preset judgment thresholds from the system configuration. These thresholds can be empirically set values, such as 0.5, or optimal boundary values ​​obtained from historical data statistics, used to divide different state categories in the probability space. After extracting the judgment thresholds, the state probability values ​​are compared numerically with them. When the state probability value is greater than or equal to the judgment threshold, the audio signal is considered to exhibit characteristics highly correlated with the target state, and is therefore classified as a positive state. When the state probability value is less than the judgment threshold, the audio signal is considered to exhibit characteristics with low correlation to the target state, and is therefore classified as a negative state. The final classification result needs to be output through standardized encapsulation to ensure that subsequent processing or system calls can use it in a fixed result interface, guaranteeing system consistency.

[0059] The judgment threshold can be adjusted according to actual application needs. In the healthcare field, the optimal threshold for different stages of disease can be obtained through clinical data statistics. A lower threshold is used in early screening to increase sensitivity, while a higher threshold is used in the diagnosis stage to improve specificity. In the fintech field, different threshold parameters can be configured for business modules with different risk preferences. For example, the threshold can be lowered in risk identification scenarios to reduce false negatives, while the threshold can be raised in customer compliance reviews to reduce false positives. Adaptive threshold mechanisms can also be used to dynamically adjust the judgment threshold, for example, by dynamically setting the threshold range based on the overall distribution of current business data, thereby improving the flexibility of decision-making.

[0060] Example: In the healthcare business, for scenarios requiring automated preliminary screening, the system can automatically mark whether a patient's voice exhibits features related to a disease state by comparing the probability value of the state with a set threshold, thereby achieving automated health early warning.

[0061] In the fintech business, when the goal is to identify whether there are abnormalities in voice interaction, the classification results of the customer's voice status can be output in real time by comparing the state probability value with the dynamic threshold, which can be used for subsequent processing in the risk monitoring system.

[0062] This embodiment achieves accurate binary classification of audio signal states by comparing state probability values ​​with preset judgment thresholds and standardizing the output of positive and negative states. This improves the flexibility, accuracy, and controllability of the state judgment process, while also adapting to different business needs and supporting flexible parameter configuration and result interpretation.

[0063] This invention relates to the field of speech processing technology and can be applied to business scenarios such as fintech and healthcare. It discloses a method, apparatus, device, and medium for determining the state of an audio signal, comprising: acquiring an audio signal and performing noise reduction and endpoint detection to obtain an initial audio segment; unifying the initial audio segment to a preset fixed duration to obtain a target audio segment and performing speech-to-text processing to obtain text information; extracting acoustic features from the target audio segment; extracting linguistic features from the text information; fusing the acoustic and linguistic features to generate a multi-dimensional feature vector; inputting the multi-dimensional feature vector into a pre-trained analysis model to generate a state probability value; and determining the target state corresponding to the audio signal based on the state probability value. This invention effectively solves the problems of incomplete feature extraction, insufficient feature fusion, and poor accuracy and generalization ability of the determination results in existing technologies by fusing multi-dimensional feature vectors based on acoustic and linguistic features and introducing a pre-trained analysis model to determine the state probability value, thereby improving the accuracy and stability of audio signal state determination.

[0064] In one embodiment, step S10 includes:

[0065] S101, in an environment where the background noise is below a set threshold, the original audio signal is acquired by a recording device;

[0066] S102, Perform noise reduction processing on the original audio signal to generate a noise-reduced audio signal;

[0067] S103, perform energy feature analysis on the noise-reduced audio signal, locate the speech start point and speech end point through a dual-threshold decision mechanism, and generate endpoint position information;

[0068] S104, based on the endpoint position information, extract the effective segment of the noise-reduced audio signal to generate an initial audio segment.

[0069] In this embodiment, raw audio signals are acquired using a recording device in an environment where background noise is below a set threshold. This operation involves the dynamic evaluation and selection of the external recording environment. First, a noise threshold parameter needs to be preset, which serves as the basis for determining whether acquisition is suitable. The noise threshold can be derived from historical statistical data, standardized acquisition specifications, or signal-to-noise ratio requirements in the scenario. The acquisition device should have environmental noise monitoring capabilities. In actual operation, the noise energy level can be monitored in real time. If the noise energy is below the threshold, the recording device is triggered to start recording, ensuring that the acquired data has high signal purity and providing favorable conditions for subsequent processing. The recording device can be various types of audio acquisition hardware, including but not limited to digital microphone arrays, built-in microphones in mobile terminals, or high-fidelity recording devices. The selection should be based on the application scenario and the desired frequency response and sensitivity indicators.

[0070] The original audio signal undergoes noise reduction processing to generate a denoised audio signal. Noise reduction requires distinguishing between the target speech signal and background noise components, suppressing background noise energy through methods such as spectral subtraction, Wiener filtering, or deep learning adaptive filtering. This step can be implemented using time-domain or frequency-domain noise reduction algorithms. For example, short-time Fourier transform combined with spectral thresholding can be used for frequency domain processing, or time-series feature modeling can be used to extract noise statistical parameters for filtering. Performance tuning of the noise reduction algorithm includes setting parameters such as the window function length, overlapping window ratio, and noise estimation update rate. These parameters directly affect the quality of the denoised signal and speech intelligibility. In multi-device or multi-scene acquisition scenarios, the noise reduction algorithm parameters can be dynamically adjusted to match the actual noise characteristics of the input signal, ensuring consistent noise reduction results.

[0071] Energy feature analysis is performed on the denoised audio signal. A dual-threshold decision mechanism is used to locate the start and end points of speech, generating endpoint location information. The energy feature analysis process requires framing and windowing the denoised audio signal to extract short-time energy values ​​within each frame as analysis indicators. Short-time energy reflects the signal strength within a speech frame, facilitating the distinction between speech and silence regions. The dual-threshold decision mechanism includes two levels of comparison: a high threshold and a low threshold. The high threshold is used to identify clearly defined speech regions, while the low threshold tolerates weak start or attenuation portions, improving the stability and robustness of endpoint detection. In the specific implementation, the frame sequence is traversed. If the frame energy exceeds the high threshold, it is marked as a speech region. If the frame energy is between the low and high thresholds, it is classified as a speech region based on continuity and contextual information. The introduction of dual thresholds effectively reduces false positives and false negatives in speech boundary detection, especially ensuring the accuracy of endpoint detection in scenarios with changing background noise. Endpoint location information is represented by frame indexes or timestamps, identifying the start and end of the valid speech interval.

[0072] Based on the endpoint location information, effective segments of the denoised audio signal are extracted to generate initial audio segments. The extraction operation is based on the sampling point indices corresponding to the start and end positions of the endpoints in the denoised audio signal. Samples between the start and end points are extracted from the sampling point sequence to form a continuous audio data segment of variable length as the initial audio segment. This process requires ensuring boundary alignment accuracy, which can be achieved by fine-tuning the boundary frames after endpoint detection, such as smoothing or extending them by a few milliseconds, to capture the complete speech context. The extracted initial audio segment must maintain the temporal continuity and data consistency of the original signal to provide reliable input for subsequent duration normalization, acoustic feature extraction, and speech-to-text processing. In the overall process, the tight coupling between endpoint detection and extraction ensures efficient extraction of effective information from the speech signal and minimizes redundant data.

[0073] This embodiment automatically acquires the original audio signal in an environment with background noise below a threshold, uses adaptive noise reduction algorithms such as spectral subtraction to suppress noise interference, and combines a dual-threshold decision mechanism to achieve stable speech endpoint detection. It also accurately extracts effective speech segments to generate initial audio fragments, which can systematically improve the quality and effectiveness of input audio data. This lays a high-quality data foundation for subsequent feature extraction, model inference, and state determination, effectively reducing the interference of environmental noise and non-speech segments on the overall processing link, and ensuring the accuracy and robustness of the determination results.

[0074] In one embodiment, step S20 above includes:

[0075] S201, Detect the actual duration parameter of the initial audio segment;

[0076] S202, compare the actual duration parameter with the preset fixed duration;

[0077] S203, when the actual duration parameter exceeds the preset fixed duration, the redundant audio segments at both ends of the initial audio segment are trimmed and the core speech segment is retained to generate trimmed audio data;

[0078] S204, when the actual duration parameter does not reach the preset fixed duration, zero-value audio data is filled at the end of the initial audio segment to generate filled audio data;

[0079] S205, the cropped or padded audio data is encapsulated into equal-length audio data to generate the target audio segment;

[0080] S206, the target audio segment is converted into a sequence of text symbols using a pre-trained speech recognition model;

[0081] S207, Generate text information based on the sequence of text symbols.

[0082] In this embodiment, detecting the actual duration parameter of the initial audio segment requires analyzing the sampling rate and sample size of the audio segment. The duration is obtained by calculating the ratio of the sample size to the sampling rate. For example, for audio with a sampling rate of 16000Hz, if the sample size is 160000, the actual duration is 10 seconds. This detection operation relies on the metadata of the audio signal or directly statistically calculates the signal length for subsequent duration comparison and processing. Comparing the actual duration parameter with a preset fixed duration is usually implemented as a single-value comparison. By determining whether the actual duration is greater than, equal to, or less than the fixed duration, the process of pruning or padding is initiated. The preset fixed duration originates from model training requirements or input constraints of downstream processing modules. For example, setting the fixed duration to 10 seconds is to unify the input format and simplify the complexity of feature extraction and model inference.

[0083] When the actual duration exceeds a preset fixed duration, redundant audio segments at both ends of the initial audio clip need to be trimmed, while retaining the core speech segments. The trimming operation relies on endpoint detection information or energy distribution analysis, prioritizing the retention of high-energy, semantically dense segments in the middle. For example, in a continuous speech stream, sliding window analysis of the energy distribution can locate the core part of the speech, while deleting silent or noisy segments at the beginning and end. Generating the trimmed audio data requires indexing and slicing the original audio data to ensure data continuity is not disrupted, and outputting it in a format consistent with the sampling rate.

[0084] When the actual duration parameter does not reach the preset fixed duration, zero-value audio data is padded to the end of the initial audio segment to generate padded audio data. The padding operation is typically achieved by appending a sequence of zero values ​​(i.e., silent segments) with the same format as the input to the end of the audio data. The length of this sequence is the fixed duration minus the current duration, ensuring that the output data reaches the required length. The purpose of padding with zero values ​​is to maintain the consistency of the input's temporal dimension, without introducing new audio information or interfering with subsequent feature extraction and semantic recognition.

[0085] The cropped or padded audio data is encapsulated into equal-length audio data to generate the target audio segment. This process requires specifying the storage format of the output data, such as WAV, PCM, or other standardized audio formats, to ensure that subsequent modules can read it without loss of quality. The encapsulation process may involve format conversion, sampling rate reshaping, or channel number adjustment to ensure complete consistency with the model input requirements.

[0086] Converting target audio segments into sequences of written symbols using a pre-trained speech recognition model requires calling acoustic and language models based on deep neural networks. Common implementations include CTC (Connection-Temporal Classification) models and end-to-end ASR (Automatic Speech Recognition) models. The model first segments the audio signal into frames and extracts low-level acoustic features (such as Mel-frequency cepstral coefficients (MFCC), spectrograms, etc.). Then, it combines linguistic context to infer the corresponding characters, phonemes, or word-level units, outputting a continuous sequence of written symbols.

[0087] Generating text information based on sequences of alphanumeric symbols requires further parsing of the intermediate symbol sequences output by the model into readable text. This includes operations such as removing duplicate characters, filling in missing words, correcting spelling, and aligning syntactic units. For example, CTC decoding requires removing whitespace and concatenating consecutive repeating characters, while an end-to-end model may directly output a word sequence, but the compatibility between the encoding and language rules still needs to be verified.

[0088] This embodiment ensures standardized audio input length by detecting the actual duration parameter of the initial audio segment and comparing it with a fixed duration. Regardless of whether redundant parts are trimmed or zero-value segments are filled, the final result is a uniform audio segment of equal length. This process improves the controllability and consistency of the data flow, providing a stable and reliable input foundation for subsequent feature extraction, speech recognition, and text generation. It effectively reduces the model's sensitivity to variable-length inputs, improving recognition accuracy and overall system robustness.

[0089] In one embodiment, step S30 above includes:

[0090] S301, extract the prosodic feature parameters of the target audio segment, including fundamental frequency variation parameters, speech rate parameters, and pause duration parameters;

[0091] S302, determine the vocal cord vibration stability parameters and spectral quality index of the target audio segment;

[0092] S303, extract the spectral envelope description features and spectral dynamic change indicators of the target audio segment;

[0093] S304, combine the prosodic feature parameters, vocal cord vibration stability parameters, spectral quality index, spectral envelope description features and spectral dynamic change index to generate an acoustic feature group;

[0094] S305, The text information is processed by a pre-trained semantic model to generate a semantic vector representation;

[0095] S306, determine the lexical diversity measure of the text information and analyze the sentence structure complexity features of the text information;

[0096] S307, determine the text similarity parameter of adjacent sentences in the text information;

[0097] S308, combine the semantic vector representation, lexical diversity measure, sentence structure complexity feature and text similarity parameter to generate a linguistic feature set;

[0098] S309, the acoustic feature group and the linguistic feature group are connected in series to generate a multi-dimensional feature vector.

[0099] In this embodiment, extracting prosodic feature parameters from the target audio segment, including fundamental frequency variation parameters, speech rate parameters, and pause duration parameters, requires first dividing the audio segment into frames and applying window weighting. Then, fundamental frequency detection is performed on each frame, for example, by measuring the fundamental frequency using an autocorrelation function or the YIN algorithm, and calculating the standard deviation of the fundamental frequency over time to measure fundamental frequency variation. Speech rate parameters are calculated by counting the number of syllables or words per unit time; a common approach is to combine this with automatic phoneme boundary detection, measuring the ratio of time length to the number of speech units. Pause duration parameters are measured by identifying silent segments and accumulating their total duration to measure the proportion of pauses in speech. These prosodic parameters reflect the speaker's behavioral characteristics in terms of tone, rhythm, and pauses, and are sensitive to speech abnormalities associated with neurodegenerative diseases.

[0100] The vocal cord vibration stability parameters and spectral quality indices of the target audio segment are determined. Vocal cord vibration stability parameters include fundamental frequency jitter and amplitude jitter, which are calculated by the rate of change of the fundamental frequency period and amplitude over consecutive frames to describe the periodic stability of the sound production. Spectral quality indices, such as harmonic noise ratio (HNR), measure the energy ratio of harmonic components to noise components, reflecting the purity of the sound source, and are often obtained through Fourier analysis and source-channel separation algorithms.

[0101] Spectral envelope descriptive features and spectral dynamic change indices are extracted. Spectral envelope descriptive features are commonly represented by Mel-frequency cepstral coefficients (MFCC) or linear predictive cepstral coefficients (LPCC), describing the short-time power spectrum envelope of sound and used to capture the resonant characteristics of the vocal organs. Spectral dynamic change indices include spectral flux and spectral entropy. Spectral flux is calculated through inter-frame spectral difference and measures rapid changes in the spectrum. Spectral entropy measures the uniformity of spectral energy distribution, reflecting the complexity and modulation degree of the sound production process.

[0102] By combining prosodic feature parameters, vocal cord vibration stability parameters, spectral quality indicators, spectral envelope description features, and spectral dynamic change indicators, an acoustic feature set is generated. All feature vectors are required to be aligned dimensionally and form a unified data structure to facilitate input into subsequent calculation modules and ensure that the multidimensional description of acoustic information is complete and consistent.

[0103] Text information is processed by a pre-trained semantic model to generate semantic vector representations. This requires inputting text information into a deep semantic encoder based on a Transformer architecture or a BERT-like model, and outputting a fixed-dimensional context-sensitive vector. This semantic vector captures the overall semantic features of the text and possesses the ability to correlate context and word sequence dependencies.

[0104] To determine the lexical diversity measure of textual information, we can measure the richness of the text's vocabulary by statistically analyzing the ratio of the number of independent words to the total number of words or by calculating lexical entropy. The sentence structure complexity features of textual information can be calculated using syntactic parse tree depth or clause nesting levels to reflect the complexity of language organization and assess the grammatical level of sentence structure.

[0105] Determine the text similarity parameters of adjacent sentences in the text information, and calculate the cosine similarity or other distance measures of adjacent sentence pairs through word embedding or sentence vectors to reflect semantic coherence and contextual consistency.

[0106] Combining semantic vector representations, lexical diversity measures, sentence structure complexity features, and text similarity parameters to generate linguistic feature sets requires standardization and concatenation of feature dimensions to form a unified linguistic feature vector expression.

[0107] By concatenating acoustic and linguistic feature sets, a multi-dimensional feature vector is generated. This requires strict dimensional concatenation of acoustic and linguistic features to form a composite feature representation that is input into the subsequent analysis model, possessing complete multi-modal information coverage capabilities.

[0108] This embodiment extracts and fuses fine-grained acoustic and linguistic features from the target audio segment and text information, respectively, to comprehensively capture the speaker's vocal behavior, speech dynamics, semantic content, and language organization features, generating a multi-dimensional feature vector. This feature vector not only contains low-level acoustic information but also integrates high-level linguistic expression characteristics, providing rich and robust multimodal input for subsequent analysis models, effectively improving the accuracy, generalization, and adaptability to speech diversity in state determination.

[0109] In one embodiment, step S40 above includes:

[0110] S401, The multi-dimensional feature vector is input into the multi-head attention module of the analysis model, and the positional association weights between features are determined by the multi-head attention module;

[0111] S402, Generate attention-enhanced features based on the location association weights;

[0112] S403, the multi-dimensional feature vector is input into the bidirectional temporal modeling module of the analysis model, and the forward temporal unit of the bidirectional temporal modeling module processes the positive time dependency of the multi-dimensional feature vector to generate the forward temporal unit output;

[0113] S404, The backward time dependency relationship of the multi-dimensional feature vector is processed by the backward time unit of the bidirectional time series modeling module to generate the backward time series unit output;

[0114] S405, the outputs of the forward timing unit and the backward timing unit are fused to generate timing-dependent features;

[0115] S406, Concatenate the attention enhancement feature and the temporal dependency feature to generate a fused feature representation;

[0116] S407, the fused feature representation is input into the classification output layer of the analysis model, and the result of the classification output layer is processed by the activation function of the analysis model to generate state probability values.

[0117] In this embodiment, the multi-dimensional feature vector is a high-dimensional data representation derived from the fusion of acoustic and linguistic feature groups. The acoustic feature group includes prosodic feature parameters, vocal cord vibration stability parameters, spectral quality indices, spectral envelope description features, and spectral dynamic change indices. The linguistic feature group includes semantic vector representation, lexical diversity measurement, sentence structure complexity features, and text similarity parameters. This multi-dimensional feature vector, as input to the analysis model, carries the global statistical and dynamic characteristics contained in the audio signal and corresponding text information, reflecting the overall expressive power of cross-modal features.

[0118] When multi-dimensional feature vectors are input into the multi-head attention module of the analysis model, the module first performs a linear mapping on the input vectors, converting them into three tensor representations: query vector, key vector, and value vector. The multi-head design instructs the model to learn multiple sets of queries, keys, and values ​​on the same input using different weight matrices, allowing the model to capture the relationships between features in different attention spaces in parallel. The multi-head attention module determines the positional association weights between features by calculating the dot product of the query vector and the key vector, scaling it, and applying Softmax normalization to obtain the weight matrix, explicitly expressing the correlation between different feature components in the sequence. Based on these weights, a weighted sum is calculated using the value vectors to form attention-enhanced features, strengthening feature interactions at different locations and focusing on the most relevant parts of the input features.

[0119] Simultaneously, multi-dimensional feature vectors are input into the bidirectional temporal modeling module of the analysis model. This module includes forward temporal units and backward temporal units. The forward unit inputs data sequentially from the start to the end of the sequence step by step, capturing forward temporal dependencies and generating the forward temporal unit output. The backward unit uses the same structure, but inputs data in reverse order from the end to the start of the sequence, capturing backward temporal dependencies and generating the backward temporal unit output. This bidirectional structure performs symmetrical processing on the sequence data, enabling the model to consider both historical and future contexts simultaneously, making it suitable for time series scenarios such as speech and text. When fusing the outputs of the forward and backward temporal units, the bidirectional temporal dependency features are integrated through concatenation or linear transformation to generate temporal dependency features, giving the model bidirectional context-awareness capabilities on the time axis.

[0120] Attention-enhancing features are concatenated with temporal-dependent features to generate a fused feature representation. This requires alignment along the feature dimension, concatenating them into a higher-dimensional feature vector to fuse spatial correlation and temporal dependence in the representation space. The fused feature representation is input to the classification output layer of the analysis model. This layer typically consists of a set of fully connected weight matrices and bias vectors, mapping the high-dimensional feature representation to the state classification space and outputting a real-valued vector. The results from the classification output layer are processed by the analysis model's activation function to generate state probability values. The activation function normalizes the output to meet probability distribution requirements. For binary classification, the Sigmoid activation function is often used, restricting the output to the interval between 0 and 1, reflecting the probability of a predicted sample belonging to a certain state. For multi-class classification, the Softmax activation function is often used, summing all components of the output vector to 1, corresponding to the probability distribution of different states.

[0121] This processing flow implements spatial correlation computation (multi-head attention module), temporal dependency modeling (bidirectional temporal modeling module), comprehensive feature fusion (splitting and fusing feature representation), classification mapping (classification output layer), and normalized probability output (activation function) in different stages. These components are coupled together to form an end-to-end analysis path, ensuring that multi-dimensional features are fully integrated at the spatial, temporal, and semantic levels, thereby improving the reliability and accuracy of the final state probability value.

[0122] This embodiment uses multi-dimensional feature vectors as a unified input, and then uses a multi-head attention module to fine-grainedly mine the correlation between features. Combined with a bidirectional temporal modeling module to model the forward and backward dependencies of time series, and then through classification output layer and activation function mapping and normalization, the final state probability value is generated. This enables the system to make full use of the spatial structure, temporal dependencies and cross-modal interaction features hidden in the input data, and solves the problems of existing technologies in temporal scenarios such as separation of spatial and temporal information, inability to finely integrate, and insufficient reliability of model judgment. This improves the accuracy, stability and generalization ability of the analysis model in state prediction under complex speech and text data.

[0123] In one embodiment, step S50 above includes:

[0124] S501, Obtain the preset state determination threshold;

[0125] S502, compare the state probability value with the state determination threshold to generate a probability comparison result;

[0126] S503, when the probability comparison result indicates that the state probability value exceeds the state determination threshold, the target state is marked as a positive state;

[0127] S504, when the probability comparison result indicates that the state probability value does not exceed the state determination threshold, the target state is marked as a negative state.

[0128] In this embodiment, when using the state probability value as input to determine the target state corresponding to the audio signal, the first step is to obtain a preset state determination threshold. This parameter is not merely a fixed value, but a key decision baseline that the system configures or dynamically adjusts based on specific application scenarios. The sources of the state determination threshold include, but are not limited to, determining it through historical data analysis of the optimal boundary point in the training set. For example, using the Youden Index to optimize the trade-off between sensitivity and specificity, or iteratively correcting it through user feedback and performance evaluation during actual use. Obtaining this threshold requires loading it from a configuration file, database, or external management service during the system initialization phase to ensure that subsequent state determination operations are performed according to a stable and consistent standard.

[0129] The comparison of state probability values ​​is the logical core of the entire decision-making process. The comparison is not merely a simple mathematical greater than or less than condition; it also requires consideration of precision issues in floating-point arithmetic, such as using machine precision upper and lower limits to avoid boundary misjudgments. The comparison operation can be implemented in various computing environments, including high-performance CPUs, parallel GPUs, and even FPGA acceleration modules, to adapt to different scales of real-time processing needs. In parallel processing environments, it is necessary to ensure that the comparison operation has stable latency for batch input state probability values, supporting pipelined data flow.

[0130] The comparison output forms a probability comparison result. This intermediate result can be a Boolean value or a binary classification label such as "high risk" and "low risk". This result is not only used to directly drive the subsequent status labeling process, but can also be used as a statistical monitoring indicator input to other business modules, providing visualization support for the overall system operation status.

[0131] When the comparison results show that the probability value of a state exceeds a threshold, the target state is marked as a positive state. The labeling of a positive state is not a simple string output; it can be mapped to internal encoding, a tag object, or a database record for subsequent statistics, reporting, and tracking. Here, "positive state" is a domain abstraction, applicable to marking "high-risk state" or "requiring further investigation" in healthcare business scenarios, and also applicable to determining "suspicious transaction state" or "high-risk user" in fintech businesses.

[0132] When the comparison result shows that the state probability value does not exceed the threshold, the target state is marked as a negative state, and the logic is consistent with the marking process for positive states. To ensure the scalability of the system, the negative state marking process should support multiple output formats, such as structured data JSON, message queue data packets, or visual API interface responses, to facilitate integration with external application systems.

[0133] The entire process not only generates state classification results but also acts as a bridge between probability prediction and usable judgment labels. This is of great significance for patient state classification in the healthcare field and user risk screening in the fintech field. The modular design of this judgment process allows for independent tuning of thresholds and judgment logic, supporting flexible adjustments to standards for different business needs. For example, higher sensitivity may be required in medical scenarios (lowering the threshold), while specificity may be emphasized in financial risk control scenarios (raising the threshold), thereby optimizing applicability to different business scenarios.

[0134] This embodiment obtains a preset state determination threshold and compares the state probability value with the threshold. The system can establish a stable and clear conversion mechanism between continuous probability prediction and discrete classification labels, which solves the problem that the probability output results cannot be directly used for decision-making in the prior art. This makes the output results easier to interpret and can be directly used as input for downstream business logic, improving the accuracy and efficiency of state classification in different application scenarios and ensuring the system's wide applicability and operability in the fields of medical and health business and financial technology business.

[0135] In one embodiment, after step S50 above, the method further includes:

[0136] S601, determine the acoustic feature index that has the highest correlation with the state probability value;

[0137] S602, Identify the linguistic feature indicators that contribute most to determining the target state;

[0138] S603, Based on the acoustic feature index with the highest correlation and the linguistic feature index with the greatest contribution, generate a feature importance report;

[0139] S604, the feature importance report is associated with the target state and output.

[0140] In this embodiment, after determining the target state corresponding to the audio signal based on the state probability value, further interpretive analysis is conducted on the decision-making basis of the analysis model to improve the transparency and usability of the model's decision. The first key operation in this process is to determine the acoustic feature index with the highest correlation to the state probability value. Here, "acoustic feature index" specifically refers to the multi-dimensional acoustic descriptive quantities previously extracted from the target audio segment, including but not limited to fundamental frequency variation parameters, speech rate parameters, pause duration parameters, vocal cord vibration stability parameters, spectral quality indicators, spectral envelope descriptive features, and spectral dynamic change indicators. Each type of acoustic feature index reflects the changes in speech production mechanism, vocal tract characteristics, vocal cord vibration patterns, and spectral energy distribution in the audio signal, and therefore has a direct contribution to determining the state represented by the audio signal. In specific implementation, the intrinsic parameters of the analysis model (such as multi-head attention weights, model gradient sensitivity matrix, or post-training feature contribution coefficients) are used to quantitatively rank the relationship between each individual index in the acoustic feature group and the final state probability value, and the index or group of indices with the largest contribution value is selected as the acoustic feature index with the highest correlation. The accuracy requirement for this operation cannot be based solely on traditional correlation analysis, but must be measured by combining the interpretation of acoustic feature inputs with the actual inference path of the model.

[0141] Next, the analysis focuses on identifying the linguistic features that contribute most to the determination of the target state. These linguistic features are derived from semantic vector representations, lexical diversity measures, sentence structure complexity features, and text similarity parameters previously extracted from the text information. Since the content, lexical structure, and syntactic organization expressed by the text information are crucial foundations for the model's analysis of semantic patterns, their roles in the analysis model need to be evaluated separately. This can be achieved by backtracking the model's training weights, examining the attention distribution in the semantic embedding space, or using feature ablation experiments to evaluate the marginal contribution of each linguistic feature to the change in state probability values, thereby identifying the linguistic features that contribute most to the determination of the target state.

[0142] After acoustic and linguistic feature indicators are identified, a feature importance report is generated by combining the two. This report goes beyond a simple list of indicators; it must include the name, definition, calculation source, units of measurement, corresponding numerical range, and percentage of explanatory contribution to the state probability value for each indicator. The implementation can utilize a programmatic template generation method, supporting multiple output formats to meet the needs of different fields in healthcare and fintech. For example, it should support more readable document formats (such as PDF) or data formats that can be exchanged between systems (such as JSON and XML). Furthermore, to facilitate subsequent decision-making chain tracking, the feature importance report must include a timestamp, a unique identifier for the audio signal, and the analysis batch number corresponding to the current target state.

[0143] Finally, the feature importance report is correlated with the target state in the output. This operation requires that the output not only provide the classification result of the target state (e.g., positive or negative), but also simultaneously provide the feature importance report as an explanation of the decision-making basis. This correlated output requires the data structure to have inherent consistency. For example, there must be a traceable mapping relationship between the target state and the feature importance index, and the output interface design must support the reuse of structured data so that it can be directly called and displayed by medical decision support systems, financial risk assessment systems, etc. This process is achieved through data encapsulation and metadata binding, ensuring that the output can meet the needs of automated analysis while also being directly understandable and usable by human decision-makers.

[0144] In practical implementation, determining the acoustic feature index with the highest correlation to the state probability value can be accomplished as follows: First, after the analysis model outputs the state probability value, the model's internal weight sensitivity evaluation method, such as algorithms based on Integrated Gradients or SHAP (SHapley Additive exPlanations), is used to calculate the marginal contribution of each input acoustic feature group to the change in the state probability value. For each acoustic feature, such as fundamental frequency variation parameter, speech rate parameter, pause duration parameter, vocal cord vibration stability parameter, spectral quality index, spectral envelope description feature, and spectral dynamic change index, the degree of change of its value to the state probability value under the current sample input condition is calculated, and this degree of change is quantified as a contribution score. By ranking these scores, the acoustic feature with the largest contribution score is selected as the acoustic feature index with the highest correlation.

[0145] The identification of the most contributing linguistic features can also employ model interpretation techniques. Specifically, for each input linguistic feature group—including semantic vector representations, lexical diversity measures, sentence structure complexity features, and text similarity parameters—feature ablation is applied. This involves setting each linguistic feature to zero or removing it, recalculating the output state probability value of the analysis model, and observing the magnitude of the change. Linguistic features with larger changes indicate a greater impact on the state probability value and are therefore identified as the most contributing linguistic features. This ablation analysis can be optimized by combining batch computation to ensure high efficiency and accuracy in large-scale data processing. Furthermore, comparing the results of multiple rounds of ablation tests can further verify the robustness of the identified linguistic features across different input samples, improving the reliability and practicality of the interpretation results.

[0146] Example Description: In the healthcare field, a dedicated mobile health application combined with a portable recording device provides patients with a self-service voice testing service to assist doctors in the early screening and diagnosis of Parkinson's disease. First, patients use the recording device to collect voice samples in a quiet room with background noise below a preset threshold. Recording tasks include continuous pronunciation of "ah," reading fixed short phrases (such as "Today is a nice day"), and brief free descriptions (such as a brief account of morning activities). The collected raw audio signals are uploaded to a cloud server in real time.

[0147] On the server side, the received raw audio signal is processed by the digital signal preprocessing module, and a noise suppression filtering algorithm is executed to remove ambient background noise, resulting in a denoised audio signal. Subsequently, energy feature analysis and dual-threshold decision logic are used to accurately identify the start and end points of the speech, generating endpoint position information to trim out the effective speech segments, obtaining initial audio fragments as the basis for subsequent analysis.

[0148] After processing the initial audio segment, the system detects its actual duration parameter and compares it with a fixed duration configured by the system (e.g., 10 seconds). When the actual duration exceeds 10 seconds, a pruning operation is performed at both ends, and the core speech segment to be retained is located through signal energy distribution analysis. When the actual duration is less than 10 seconds, zero-value audio is added to the end to achieve a uniform duration requirement. The pruned or padded audio data is packaged into a target audio segment of standard length to ensure consistency in subsequent model processing. This target audio segment is input into a pre-trained automatic speech recognition model, converted into a sequence of text symbols, and further parsed into complete text information.

[0149] For the target audio segment and corresponding text information, the system extracts multi-dimensional features in parallel. On the audio side, this includes calculating prosodic features (such as fundamental frequency variation parameters reflecting pitch stability, speech rate parameters calculating the number of sounds per unit time, and pause duration parameters calculating the proportion of silence), analyzing vocal cord vibration stability (quantified by fundamental frequency perturbation and amplitude perturbation indices) and spectral quality (such as harmonic noise ratio), and calculating spectral envelope and dynamic changes (such as MFCC coefficients and spectral flux changes). These features are combined to generate an acoustic feature set. On the text side, semantic vectors are generated using the BERT semantic model, combined with lexical diversity analysis (such as type-tag ratio, TTR), syntactic complexity (such as average clause depth), and semantic similarity between adjacent sentences (such as cosine similarity) to generate a linguistic feature set. The two sets of features are concatenated to form a complete multi-dimensional feature vector, which is used as model input.

[0150] The multi-dimensional feature vector is fed into a pre-trained analysis model. Internally, the model first calculates the positional correlation weights between features using a multi-head attention module to generate attention-enhanced features. Then, a bidirectional temporal modeling module processes the forward and reverse temporal dependencies of the feature sequence separately, finally fusing the outputs from both directions to form a temporally dependent feature. The system concatenates the attention-enhanced features and the temporally dependent features to generate a fused feature representation, which is then input into the model's classification output layer. An activation function (such as sigmoid) normalizes the output to a state probability value. This state probability value reflects the degree of matching between the current patient's speech features and Parkinson's disease feature patterns.

[0151] The system compares the probability value of the status with a preset status judgment threshold (e.g., 0.5). If the probability value exceeds the threshold, the target status is judged as positive, indicating a suspected positive screening result for Parkinson's disease; if it is below the threshold, it is judged as negative. Ultimately, the positive or negative status is used as a standardized target status for subsequent electronic health record recording and auxiliary prompts on the doctor's end.

[0152] After diagnosis, the system further generates a feature interpretation report. Using feature contribution analysis algorithms from the model, such as the SHAP interpretation framework, it identifies the acoustic features most strongly correlated with the probability value of that state (e.g., in the current sample, the pause duration parameter has the highest explanatory strength) and the linguistic features that contribute the most (e.g., the sentence structure complexity index has the greatest impact on the probability value). Based on these feature indicators, a feature importance report is generated and output along with the current target state result for doctors to reference during clinical interpretation, improving the transparency and trustworthiness of decision support.

[0153] Through the above process, the system not only realizes the end-to-end processing link from audio acquisition, signal processing, feature extraction to model reasoning and result interpretation, but also meets the high standards of diagnostic consistency and interpretability requirements in medical and health application scenarios. It effectively alleviates the problem of traditional Parkinson's disease screening relying on doctors' subjective experience and provides efficient voice intelligent analysis capabilities for primary medical institutions and mobile health services.

[0154] In the fintech field, intelligent customer service systems used by banks or financial service institutions automatically analyze compliance and risk signals during user voice interactions. Users interact with the system through voice channels (such as intelligent voice customer service or telephone agents). The system first collects raw voice data through a front-end recording module in an environment with background noise below a set threshold, ensuring clear recordings that are resistant to background noise interference from telephone networks. The collected raw voice data is uploaded in real time to the financial institution's data center, where a voice processing engine executes digital noise reduction algorithms to remove background noise and improve the voice signal-to-noise ratio. Subsequently, the system uses energy feature detection combined with a dual-threshold decision method to locate the start and end points of the user's actual speech, forming clean and effective speech segments, thus obtaining the initial audio segment.

[0155] The system detects the duration of the initial audio segment and compares it with a fixed analysis duration set for financial transactions (e.g., 30 seconds). If it exceeds this duration, redundant data at both ends is trimmed, retaining only the core audio segments, such as the main part of the customer's description of their risk tolerance. If it is insufficient, silent zero-value audio is added to standardize the length. This standardized target audio segment is then input into the financial speech-to-text service module, where a pre-trained speech recognition model converts it into a sequence of text symbols, generating accurate text information for subsequent text analysis.

[0156] For target audio segments and text information, the system extracts acoustic and linguistic features in parallel. Acoustically, it extracts parameters including fundamental frequency variation (e.g., pitch fluctuations reflecting emotional fluctuations), speech rate (measuring fluency), and pause duration (indicating hesitation or concealment). It also calculates vocal cord vibration stability parameters and spectral quality indices (e.g., harmonic noise ratio measuring speech intelligibility), and extracts spectral envelope descriptions and dynamic change indices, forming an acoustic feature set. Linguistically, it processes text information based on semantic models such as BERT, generating semantic vector representations. It also calculates lexical diversity measures (e.g., vocabulary coverage), analyzes sentence structure complexity (e.g., clause nesting depth), and text similarity parameters between adjacent sentences (e.g., dialogue consistency indices), forming a linguistic feature set. The two sets of features are concatenated to form a multi-dimensional feature vector, which is then uniformly input into a pre-trained financial risk analysis model.

[0157] In the analysis model, the multi-dimensional feature vectors first calculate the relative importance weights between financial speech and language features through a multi-head attention module, generating attention-enhanced features. Simultaneously, a bidirectional temporal modeling module processes the positive and negative temporal dependencies of customer speech sequences, capturing the dynamic relationship between utterance order and historical-future context, forming temporal-dependent features. These two types of features are concatenated into a fused feature representation, fed into the model's classification layer, and converted into risk state probability values ​​via a sigmoid activation function.

[0158] The system compares the probability value of this status with a preset risk threshold (e.g., 0.7). If the probability exceeds the threshold, the system determines that there is a potential compliance risk in the user's voice interaction and marks it as a positive status; if the probability does not exceed the threshold, it is marked as a negative status. This status, as a standardized output, can be directly transmitted to the compliance monitoring system of financial institutions to trigger real-time risk alerts or further manual review.

[0159] While outputting the judgment result, the system generates an interpretability analysis report. Using models and algorithms such as SHAP, it identifies the acoustic features (e.g., outliers in pause duration) that have the highest correlation with the probability value of the risk status in the current speech data, and the linguistic features that contribute the most (e.g., excessively high syntactic complexity indicating abnormal statement structures). Finally, the feature importance report is output in conjunction with the status judgment result, providing highly interpretable analytical support for financial compliance personnel, facilitating manual intervention and subsequent investigations.

[0160] Through the above process, fintech institutions can use this system to achieve efficient monitoring and early warning of risk compliance in customer voice interaction scenarios, improve the automation and intelligence level of regulatory review, reduce omissions or misjudgments caused by human subjective judgment, and enhance the risk controllability and data compliance of the overall service process.

[0161] This embodiment identifies the acoustic and linguistic features that contribute most to the state probability values ​​and generates a feature importance report based on these key indicators. Finally, the report is correlated with the target state, achieving transparency and interpretability in the model's decision-making process. This process not only overcomes the lack of explicit explanation in existing technologies but also provides a clear traceability of the model's judgment criteria. This enables clinicians in the healthcare field to gain a deeper understanding of the reasons for patient state determination based on acoustic and linguistic indicators, and allows risk control personnel in the fintech field to explain the source of high-risk judgments through feature importance reports. This enhances user trust in the model and improves the practical value and compliance of the model results.

[0162] In one embodiment, a state determination device based on audio information is provided, which corresponds one-to-one with the state determination method based on audio information described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the audio information-based state determination device of the present invention. The modules include an audio preprocessing module 10, an audio standardization module 20, a feature fusion module 30, a model inference module 40, and a state determination module 50. Detailed descriptions of each functional module are as follows:

[0163] The audio preprocessing module 10 is used to acquire audio signals and perform noise reduction and endpoint detection on the audio signals to obtain initial audio segments;

[0164] The audio standardization module 20 is used to standardize the initial audio segment to a preset fixed duration to obtain a target audio segment, and to perform speech-to-text processing on the target audio segment to obtain text information;

[0165] The feature fusion module 30 is used to extract acoustic features from the target audio segment and linguistic features from the text information, and fuse the acoustic features and the linguistic features to generate a multi-dimensional feature vector;

[0166] The model inference module 40 is used to input the multi-dimensional feature vector into a pre-trained analysis model and generate state probability values ​​through the analysis model.

[0167] The state determination module 50 is used to determine the target state corresponding to the audio signal based on the state probability value.

[0168] In one embodiment, the audio preprocessing module 10 is specifically used for:

[0169] The original audio signal is acquired using a recording device in an environment where the background noise is below a set threshold.

[0170] The original audio signal is subjected to noise reduction processing to generate a noise-reduced audio signal;

[0171] Energy feature analysis is performed on the denoised audio signal, and the start and end points of speech are located through a dual-threshold decision mechanism to generate endpoint position information;

[0172] The effective segment of the denoised audio signal is extracted based on the endpoint position information to generate an initial audio segment.

[0173] In one embodiment, the audio normalization module 20 is specifically used for:

[0174] Detect the actual duration parameter of the initial audio segment;

[0175] Compare the actual duration parameter with the preset fixed duration.

[0176] When the actual duration parameter exceeds the preset fixed duration, the redundant audio segments at both ends of the initial audio segment are trimmed while the core speech segment is retained to generate trimmed audio data.

[0177] When the actual duration parameter does not reach the preset fixed duration, zero-value audio data is filled at the end of the initial audio segment to generate the filled audio data;

[0178] The cropped or padded audio data is encapsulated into equal-length audio data to generate the target audio segment.

[0179] The target audio segment is converted into a sequence of text symbols using a pre-trained speech recognition model;

[0180] Text information is generated based on the sequence of text symbols.

[0181] In one embodiment, the feature fusion module 30 is specifically used for:

[0182] Extract prosodic feature parameters from the target audio segment, including fundamental frequency variation parameters, speech rate parameters, and pause duration parameters;

[0183] Determine the vocal cord vibration stability parameters and spectral quality index of the target audio segment;

[0184] Extract the spectral envelope description features and spectral dynamic change indicators of the target audio segment;

[0185] By combining the prosodic feature parameters, vocal cord vibration stability parameters, spectral quality indicators, spectral envelope description features, and spectral dynamic change indicators, an acoustic feature set is generated.

[0186] The text information is processed by a pre-trained semantic model to generate a semantic vector representation;

[0187] Determine the lexical diversity measure of the text information and analyze the sentence structure complexity features of the text information;

[0188] Determine the text similarity parameters of adjacent sentences in the text information;

[0189] By combining the semantic vector representation, lexical diversity measure, sentence structure complexity feature, and text similarity parameter, a linguistic feature set is generated;

[0190] The acoustic feature set and the linguistic feature set are concatenated to generate a multi-dimensional feature vector.

[0191] In one embodiment, the model inference module 40 is specifically used for:

[0192] The multi-dimensional feature vector is input into the multi-head attention module of the analysis model, and the positional association weights between features are determined by the multi-head attention module.

[0193] Attention-enhanced features are generated based on the location association weights;

[0194] The multi-dimensional feature vectors are input into the bidirectional temporal modeling module of the analysis model. The forward temporal unit of the bidirectional temporal modeling module processes the positive time dependency of the multi-dimensional feature vectors and generates the output of the forward temporal unit.

[0195] The backward temporal dependencies of multi-dimensional feature vectors are processed by the backward temporal unit of the bidirectional temporal modeling module to generate the backward temporal unit output.

[0196] By fusing the outputs of the forward timing unit and the backward timing unit, a timing-dependent feature is generated;

[0197] The attention-enhanced features and the temporal dependency features are concatenated to generate a fused feature representation;

[0198] The fused feature representation is input into the classification output layer of the analysis model, and the result of the classification output layer is processed by the activation function of the analysis model to generate state probability values.

[0199] In one embodiment, the state determination module 50 is specifically used for:

[0200] Obtain the preset state determination threshold;

[0201] Compare the state probability value with the state determination threshold to generate a probability comparison result;

[0202] When the probability comparison result indicates that the state probability value exceeds the state determination threshold, the target state is marked as a positive state.

[0203] When the probability comparison result indicates that the state probability value does not exceed the state determination threshold, the target state is marked as a negative state.

[0204] In one embodiment, the state determination module 50 is specifically used for:

[0205] Determine the acoustic feature index that has the highest correlation with the state probability value;

[0206] Identify the linguistic feature indicators that contribute most to determining the target state;

[0207] Based on the acoustic feature index with the highest correlation and the linguistic feature index with the greatest contribution, a feature importance report is generated;

[0208] The feature importance report is then associated with the target state and output.

[0209] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used for communication with external user terminals via a network connection. When executed by the processor, the computer program implements the functions or steps of a state determination method based on audio information on the server side.

[0210] In one embodiment, a computer device is provided, which may be a user terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When executed by the processor, the computer program implements the functions or steps of a state determination method based on audio information on the user side.

[0211] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0212] Acquire an audio signal, and perform noise reduction and endpoint detection on the audio signal to obtain an initial audio segment;

[0213] The initial audio segments are unified to a preset fixed duration to obtain target audio segments, and the target audio segments are processed for speech-to-text conversion to obtain text information;

[0214] Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic features and the linguistic features are then fused to generate a multi-dimensional feature vector.

[0215] The multi-dimensional feature vector is input into a pre-trained analysis model, and the analysis model generates state probability values.

[0216] The target state corresponding to the audio signal is determined based on the state probability value.

[0217] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0218] Acquire an audio signal, and perform noise reduction and endpoint detection on the audio signal to obtain an initial audio segment;

[0219] The initial audio segments are unified to a preset fixed duration to obtain target audio segments, and the target audio segments are processed for speech-to-text conversion to obtain text information;

[0220] Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic features and the linguistic features are then fused to generate a multi-dimensional feature vector.

[0221] The multi-dimensional feature vector is input into a pre-trained analysis model, and the analysis model generates state probability values.

[0222] The target state corresponding to the audio signal is determined based on the state probability value.

[0223] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and user side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0224] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0225] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0226] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A state determination method based on audio information, characterized in that, Includes the following steps: Acquire an audio signal, and perform noise reduction and endpoint detection on the audio signal to obtain an initial audio segment; The initial audio segments are unified to a preset fixed duration to obtain target audio segments, and the target audio segments are processed for speech-to-text conversion to obtain text information; Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic features and the linguistic features are then fused to generate a multi-dimensional feature vector. The multi-dimensional feature vector is input into a pre-trained analysis model, and the analysis model generates state probability values. The target state corresponding to the audio signal is determined based on the state probability value.

2. The state determination method based on audio information as described in claim 1, characterized in that, Acquire an audio signal, and perform noise reduction and endpoint detection on the audio signal to obtain an initial audio segment, including: The original audio signal is acquired using a recording device in an environment where the background noise is below a set threshold. The original audio signal is subjected to noise reduction processing to generate a noise-reduced audio signal; Energy feature analysis is performed on the denoised audio signal, and the start and end points of speech are located through a dual-threshold decision mechanism to generate endpoint position information; The effective segment of the denoised audio signal is extracted based on the endpoint position information to generate an initial audio segment.

3. The state determination method based on audio information as described in claim 1, characterized in that, The initial audio segments are standardized to a preset fixed duration to obtain target audio segments. These target audio segments are then subjected to speech-to-text processing to obtain text information, including: Detect the actual duration parameter of the initial audio segment; Compare the actual duration parameter with the preset fixed duration. When the actual duration parameter exceeds the preset fixed duration, the redundant audio segments at both ends of the initial audio segment are trimmed while the core speech segment is retained to generate trimmed audio data. When the actual duration parameter does not reach the preset fixed duration, zero-value audio data is filled at the end of the initial audio segment to generate the filled audio data; The cropped or padded audio data is encapsulated into equal-length audio data to generate the target audio segment. The target audio segment is converted into a sequence of text symbols using a pre-trained speech recognition model; Text information is generated based on the sequence of text symbols.

4. The state determination method based on audio information as described in claim 1, characterized in that, Acoustic features are extracted from the target audio segment, and linguistic features are extracted from the text information. The acoustic features and the linguistic features are then fused to generate a multi-dimensional feature vector, including: Extract prosodic feature parameters from the target audio segment, including fundamental frequency variation parameters, speech rate parameters, and pause duration parameters; Determine the vocal cord vibration stability parameters and spectral quality index of the target audio segment; Extract the spectral envelope description features and spectral dynamic change indicators of the target audio segment; By combining the prosodic feature parameters, vocal cord vibration stability parameters, spectral quality indicators, spectral envelope description features, and spectral dynamic change indicators, an acoustic feature set is generated. The text information is processed by a pre-trained semantic model to generate a semantic vector representation; Determine the lexical diversity measure of the text information and analyze the sentence structure complexity features of the text information; Determine the text similarity parameters of adjacent sentences in the text information; By combining the semantic vector representation, lexical diversity measure, sentence structure complexity feature, and text similarity parameter, a linguistic feature set is generated; The acoustic feature set and the linguistic feature set are concatenated to generate a multi-dimensional feature vector.

5. The state determination method based on audio information as described in claim 1, characterized in that, The multi-dimensional feature vector is input into a pre-trained analysis model, and the analysis model generates state probability values, including: The multi-dimensional feature vector is input into the multi-head attention module of the analysis model, and the positional association weights between features are determined by the multi-head attention module. Attention-enhanced features are generated based on the location association weights; The multi-dimensional feature vectors are input into the bidirectional temporal modeling module of the analysis model. The forward temporal unit of the bidirectional temporal modeling module processes the positive time dependency of the multi-dimensional feature vectors and generates the output of the forward temporal unit. The backward temporal dependencies of multi-dimensional feature vectors are processed by the backward temporal unit of the bidirectional temporal modeling module to generate the backward temporal unit output. By fusing the outputs of the forward timing unit and the backward timing unit, a timing-dependent feature is generated; The attention-enhanced features and the temporal dependency features are concatenated to generate a fused feature representation; The fused feature representation is input into the classification output layer of the analysis model, and the result of the classification output layer is processed by the activation function of the analysis model to generate state probability values.

6. The state determination method based on audio information as described in claim 1, characterized in that, Determining the target state corresponding to the audio signal based on the state probability value includes: Obtain the preset state determination threshold; Compare the state probability value with the state determination threshold to generate a probability comparison result; When the probability comparison result indicates that the state probability value exceeds the state determination threshold, the target state is marked as a positive state. When the probability comparison result indicates that the state probability value does not exceed the state determination threshold, the target state is marked as a negative state.

7. The state determination method based on audio information as described in claim 1, characterized in that, After determining the target state corresponding to the audio signal based on the state probability value, the method further includes: Determine the acoustic feature index that has the highest correlation with the state probability value; Identify the linguistic feature indicators that contribute most to determining the target state; Based on the acoustic feature index with the highest correlation and the linguistic feature index with the greatest contribution, a feature importance report is generated; The feature importance report is then associated with the target state and output.

8. A state determination device based on audio information, characterized in that, The state determination device based on audio information includes: An audio preprocessing module is used to acquire audio signals and perform noise reduction and endpoint detection on the audio signals to obtain initial audio segments; The audio standardization module is used to standardize the initial audio segment to a preset fixed duration to obtain the target audio segment, and to perform speech-to-text processing on the target audio segment to obtain text information; The feature fusion module is used to extract acoustic features from the target audio segment and linguistic features from the text information, and to fuse the acoustic features and the linguistic features to generate a multi-dimensional feature vector; The model inference module is used to input the multi-dimensional feature vector into a pre-trained analysis model and generate state probability values ​​through the analysis model. The state determination module is used to determine the target state corresponding to the audio signal based on the state probability value.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and an audio-information-based state determination program stored in the memory and executable on the processor, wherein the audio-information-based state determination program, when executed by the processor, implements the steps of the audio-information-based state determination method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a state determination program based on audio information, which, when executed by a processor, implements the steps of the state determination method based on audio information as described in any one of claims 1-7.

Images