An intelligent speech and text processing method and system for structured interviews

Abstract

An intelligent speech and text processing method and system for structured interviews is disclosed. The method includes: preprocessing the acquired raw interview audio data and extracting acoustic feature sequences; inputting the acoustic feature sequences into a locally deployed end-to-end speech recognition model for recognition and transcription to obtain transcribed text containing timestamp information; standardizing the transcribed text based on the timestamps and aligning it with preset interview questions to obtain a set of answer texts aligned with the interview questions; encoding each answer text in the answer text set using a Transformer encoder to obtain a corresponding sentence vector; and based on all sentence vectors of the same test subject, performing semantic association modeling through a cross-question attention aggregation model based on a multi-head self-attention mechanism, and outputting the test subject's scores on various personality dimensions through a regression layer.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and natural language processing technology, and in particular to an intelligent speech and text processing method and system for structured interviews. Background Technology

[0002] Personality computation is a technology that aims to automatically infer an individual's inherent personality traits by analyzing observable signals (such as voice and language content) generated during communication. With the rapid development of applications such as online interviews, distance education, and intelligent customer service, automated personality recognition based on voice and text content has become an important research direction of common interest to both academia and industry.

[0003] Currently, the mainstream technological approaches to achieving personality computation can be divided into the following two categories: The first category is multimodal joint modeling schemes. These schemes attempt to integrate information from multiple modalities, such as speech, facial images, and gestures, to improve the accuracy of personality recognition and the model's generalization ability through the complementarity of cross-modal information. However, this approach faces significant challenges in practical applications: its hardware and environmental requirements for data acquisition are extremely stringent. Factors such as changes in lighting, facial occlusion, and network latency can easily lead to data quality degradation or misalignment between different modalities, significantly increasing the complexity of data preprocessing. Furthermore, in industries with strict data privacy and compliance requirements, such as finance, healthcare, and education, the collection, transmission, and storage of video-based biometric information are subject to stringent regulations, fundamentally hindering the widespread application and implementation of multimodal solutions.

[0004] The second category is lightweight solutions centered on text. These solutions first transcribe speech content into text, then use natural language processing techniques to extract features from the text, ultimately completing personality trait analysis. Due to their low data acquisition threshold, simple system flow, and interpretable results, they are more favored in engineering practice. However, when facing business scenarios such as structured interviews that involve multiple rounds of questioning, existing text-based solutions still face a series of interconnected and urgently need to be addressed technical bottlenecks: 1. Technical shortcomings of context modeling: Existing solutions typically treat the answers to different questions from the test subjects as independent units, process them separately, and then simply summarize them. This modeling approach lacks the technical means to effectively model the long-distance semantic dependencies in these questions and answers, resulting in fragmented semantic information and an inability to form a coherent contextual representation.

[0005] 2. Technical risks of error propagation and amplification: The technology, which generally relies on third-party speech recognition services, is a "black box" with uncontrollable output errors and a lack of confidence information. Transcription errors act as noise input, which is amplified step by step in subsequent natural language processing (NLP) stages, seriously impairing the overall robustness of the system. Furthermore, this technical chain is difficult to analyze and optimize end-to-end.

[0006] 3. Feature extraction and downstream task mismatch: Directly using general pre-trained language models to extract features results in a difference between the representation space and the semantic distribution of a specific domain (such as interview dialogues). This makes it impossible to effectively capture key language patterns in the context of that domain, leading to insufficient feature discrimination power.

[0007] 4. Mismatch between output format and model architecture: To simplify model design, a discrete classification output format is adopted, which fails to fully utilize the ability of deep regression models to fit continuous distributions, resulting in low output information density and failing to meet the measurement technology requirements for fine characterization of individual differences.

[0008] In summary, existing technical solutions suffer from systemic technical problems when processing structured, serialized speech-text data streams, including weak context modeling capabilities, uncontrollable errors, poor domain adaptability, and incomplete model output information. Summary of the Invention

[0009] To address the shortcomings of existing technologies, this application aims to provide an intelligent speech and text processing method and system for structured interviews, which achieves intelligent processing of structured speech and text data through localized and domain-adaptive technical links.

[0010] To achieve the above objectives, this application provides an intelligent speech and text processing method for structured interviews, comprising: The acquired raw interview audio data is preprocessed and acoustic feature sequences are extracted. The acoustic feature sequence is input into a locally deployed end-to-end speech recognition model for recognition and transcription, resulting in transcribed text containing timestamp information. Based on the timestamp, the transcribed text is standardized and segmented into sentences to obtain a set of answer texts aligned with the interview questions; For each answer text in the set of answer texts, use a Transformer encoder to encode it to obtain the corresponding sentence vector; Based on all sentence vectors of the same test subject, semantic association modeling is performed through a cross-question attention aggregation model based on a multi-head self-attention mechanism, and a regression layer is used to output the continuous evaluation scores of the test subject on each personality dimension.

[0011] Furthermore, the step of preprocessing the acquired raw interview audio data and extracting acoustic feature sequences further includes: The original audio data is uniformly resampled into a mono format with a specified sampling rate; The short-time Fourier transform of the resampled audio data is performed to obtain the spectrum; The Mel filter bank is applied to the spectrum and the logarithm is taken to obtain the logarithmic Mel spectrum as the acoustic feature sequence.

[0012] Furthermore, the step of inputting the acoustic feature sequence into a locally deployed end-to-end speech recognition model for recognition and transcription further includes: the end-to-end speech recognition model is trained using a joint objective of connection temporal classification loss and attention decoder loss; during the inference phase, a beam search algorithm is used for decoding, and optionally, probabilistic fusion is performed with an external language model; the output includes a transcription result containing a word sequence, timestamp, and word or word-level confidence.

[0013] Furthermore, the step of standardizing the transcribed text based on the timestamp and aligning it with preset interview questions includes at least one of the following: recovering punctuation marks in the text using a sequence labeling model; cleaning up colloquial filler words and correcting erroneous word segmentation based on a preset rule base; standardizing named entities according to a built-in dictionary; and aligning the processed text with preset interview questions based on the audio timestamp.

[0014] Furthermore, the step of encoding each answer text in the answer text set using a Transformer encoder to obtain the corresponding sentence vector further includes: segmenting the answer text using a sub-vocabulary consistent with the end-to-end speech recognition model; inputting the segmented tag sequence into a multi-layer Transformer encoder to obtain a hidden state representation; and performing a pooling operation on the hidden state representation and then linearly projecting it to obtain a fixed-dimensional sentence vector.

[0015] Furthermore, the step of semantic association modeling based on all sentence vectors of the same test subject through a cross-question attention aggregation model based on a multi-head self-attention mechanism, and outputting the test subject's scores on each personality dimension through a regression layer, further includes: calculating an enhanced sequence from the sentence vector sequence arranged in question order through a multi-head self-attention mechanism; performing learnable attention pooling on the enhanced sequence to aggregate and obtain a global feature vector; inputting the global feature vector into a regression layer, and outputting continuous scores for each personality dimension after mapping by an activation function.

[0016] Furthermore, after the step of outputting the scores of the test subjects on each personality dimension through a regression layer, the method further includes a step of calibrating the original output of the regression layer using a calibration dataset to eliminate model bias.

[0017] To achieve the above objectives, this application also provides an intelligent speech and text processing system for structured interviews, employing the intelligent speech and text processing method for structured interviews as described above. The system includes: an audio preprocessing module for preprocessing raw audio data and extracting acoustic feature sequences; a locally deployed speech recognition module for transing the acoustic feature sequences into text containing timestamp information; a text standardization and alignment module for cleaning, standardizing, and aligning the text with the interview questions based on the timestamps to obtain aligned answer text; a sentence vector encoding module for encoding the answer text into sentence vectors; and a cross-question modeling and prediction module, which includes an attention aggregation submodule and a regression submodule based on a multi-head self-attention mechanism, for performing semantic association modeling based on the sentence vector sequences and outputting scores for personality dimensions.

[0018] To achieve the above objectives, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to execute the computer program stored in the memory to implement the intelligent voice and text processing method for structured interviews as described above.

[0019] To achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the intelligent speech and text processing method for structured interviews as described above.

[0020] The intelligent speech and text processing method for structured interviews provided in this application is based on a locally deployed end-to-end speech recognition model and a cross-question attention aggregation model based on a multi-head self-attention mechanism. This method enables intelligent processing of speech and text data in structured interviews to obtain personality trait scores of interviewees.

[0021] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0022] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings: Figure 1The flowchart below shows an intelligent speech and text processing method for structured interviews according to Embodiment 1 of this application. Figure 2 The flowchart below shows an intelligent speech and text processing method for structured interviews according to Embodiment 2 of this application. Figure 3 This is a schematic diagram of the electronic device structure according to Embodiment 3 of this application. Detailed Implementation

[0023] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application.

[0024] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0025] The term "comprising" and its variations as used in this application are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0026] It should be noted that the terms "first" and "second" may be used in this application only to distinguish different devices, components or parts, and are not used to define the order of functions performed by these devices, components or parts or their interdependence.

[0027] It should be noted that the terms "one" and "more" used in this application are illustrative rather than restrictive, and those skilled in the art should understand that, unless explicitly stated otherwise in the context, they should be understood as "one or more". "More" should be understood as two or more.

[0028] Example 1 Figure 1 The flowchart below illustrates the intelligent speech and text processing method for structured interviews according to Embodiment 1 of this application. Figure 1 The implementation methods of this application are described in detail.

[0029] Step 101: Preprocess the audio data and extract acoustic features.

[0030] This step includes: acquiring the raw audio data of the interviewee (if the input is a video file, its audio track is extracted first); standardizing the audio data, including uniform resampling to 16 kHz, mono format; converting the time-domain signal to a frequency-domain signal using a short-time Fourier transform (STFT), with specific parameters: window length 25 milliseconds, frame shift 10 milliseconds; applying a Mel filter bank to the spectrum, calculating the logarithmic Mel spectrum energy, and thus obtaining an 80-dimensional frame-level acoustic feature sequence f. n To enhance model robustness, data augmentation operations such as temporal occlusion, frequency occlusion, or speech rate perturbation can be applied to the feature sequences during the training phase.

[0031] In this embodiment, the spectrum of the nth frame and the kth frequency point is: ;in, The input is a time-domain speech signal. Here, H is the Hamming window function, H is the frame shift, and N is the number of FFT points.

[0032] The spectrum is passed through a Mel filter bank to obtain the energy, which is expressed as follows: ; Where i is the index of the Mel filter; Based on the energy, the logarithm is taken to form an 80-dimensional log-Mel spectrum (i.e., an acoustic feature sequence): ; Where, m n Let be the Mel energy vector of the nth frame. As a smoothing term, it is a small constant that prevents numerical calculation errors.

[0033] In step 102: Perform local end-to-end speech recognition (ASR) and text transcription.

[0034] This step will use the acoustic feature sequence f obtained in step 101. n The input is fed into a locally deployed end-to-end speech recognition model, which outputs transcribed text containing timestamp information. This model employs an encoder-decoder architecture, where the encoder consists of stacked convolutional and Transformer layers to learn high-level temporal representations from acoustic features. , This represents the acoustic hidden layer feature sequence output by the encoder.

[0035] The model can be trained using two objective functions: 1) Connection-Time Classification (CTC) Loss: ; Where y is the subword sequence and B is the compression mapping (a mapping that removes duplicates and whitespace). Indicates the alignment path π at time π The output labels, This represents the hidden layer feature sequence of a given acoustic feature. Predicting labels under the given conditions The posterior probability.

[0036] 2) Joint loss of CTC and attention decoder: ; in, For cross-entropy based on autoregressive attention decoder, The weighted coefficients are the combined weights of CTC loss and attention loss. .

[0037] During the inference phase, a beam search algorithm is used for decoding, with the beam width configurable between 5 and 20. Beam search, in speech recognition decoding, retains multiple possible candidate sequences (beam widths) for each step, progressively expanding and selecting the optimal path to improve recognition accuracy.

[0038] To further improve transcription accuracy, especially for specialized terminology in interview scenarios, this step also includes a shallow language model fusion process. This involves weighting and combining the probabilities of an N-gram language model trained on domain-specific text with the output probabilities of the ASR model. This leverages the prior knowledge of the language model to further correct the transcription results, effectively reducing errors when handling specialized terminology or complex sentences. The final output of the ASR model includes: the transcribed word sequence, the timestamp for each word, and confidence scores at the word or word level.

[0039] Step 103: Standardize the transcribed text and segment it into sentences to obtain the answer text aligned with the question.

[0040] This step post-processes the transcribed text output from step 102 to generate clean answer text aligned with the interview questions. Where q = 1, 2, ..., Q, and Q is the total number of questions asked in this round of interviews. The post-processing includes: Sentence segmentation and punctuation recovery: Automatically add sentence-end punctuation using rule-based or lightweight sequence labeling models (such as BiLSTM-CRF); Filler word cleanup: Remove colloquial filler words such as "uh," "um," and "that," and merge or correct obvious repetitions and incorrect word segmentation; Named entity normalization: Based on a pre-defined dictionary, names of people, companies, and professional terms are standardized to a uniform spelling. Question alignment: Based on the timestamp information obtained from the speech recognition model and the preset question time interval, the processed text is segmented and mapped to each specific question, thereby obtaining the question-aligned answer text.

[0041] Step 104: Encode the answer text for each question into a fixed-dimensional sentence vector.

[0042] This step specifically includes: 1) Use a sub-vocabulary consistent with the ASR module to list each answer. Word segmentation is a sequence of sub-word tags. 2) Input the labeled sequence into a multi-layer Transformer encoder (Transformer is a neural network architecture based on self-attention mechanism) to obtain the hidden state matrix at the label level. , Represents the set of real numbers. Indicates the sequence length. This represents the dimension of the Transformer hidden layer. 3) Perform pooling operations (such as average pooling) on ​​O to obtain the intermediate sentence vector. The expression is as follows: ; in, For average pooling function, This is the middle sentence vector after pooling. d) It is then processed through a linear projection layer... Mapped to the final sentence vector The expression is as follows: ; in, and They are respectively The weight matrix and bias vector of the linear projection layer of the layer. The dimension obtained by the final mapping The sentence vector (d is the preset sentence vector dimension).

[0043] To enable the sentence vector encoder to better understand interview language style and capture personality-related semantic features, this step employs a domain-oriented contrastive learning fine-tuning strategy during the training phase. Specifically, using the InfoNCE loss function, two vectors generated from the same sentence after being masked with different random Dropout (a regularization technique in neural network training that prevents overfitting by randomly discarding some neurons) are considered positive sample pairs, while other sentence vectors in the same batch are considered negative samples. This encourages the model to narrow the distance between positive sample pairs and widen the distance between negative sample pairs. The loss function is as follows: ; in, To standardize the cross-entropy loss by scaling it to temperature, and This represents the sentence vector representation of two mutually positive samples generated when the same input sentence is processed by the encoder and influenced by different random Dropout masks. The cosine similarity function is used. This refers to the temperature parameter.

[0044] Step 105: Cross-problem attention convergence and personality trait regression.

[0045] This step aims to model the semantic relationships between the same test subject's answers to different questions using a cross-question attention aggregation model based on a multi-head self-attention mechanism, and output the test subject's scores on various personality dimensions through a regression layer. This step includes: 1) Construct a sequence of sentence vectors and perform self-attention calculation. This involves generating the sentence vectors for Q questions (e.g., Q=6) from a single interview with the same candidate. Stacking the questions in order, the resulting sequence S is as follows: ; The query, key, and value matrices are obtained through learnable linear transformations, and then... Q u , K , V This indicates the calculation of multi-head self-attention: ; in, These represent the transformed query matrix, key matrix, and value matrix, respectively. , , , These are the corresponding learnable linear projection weight matrices. Let be the dimension of the key matrix, used as a scaling factor to prevent the inner product from becoming too large and causing gradient vanishing. This represents the final attention-weighted context feature vector. The enhanced sequence is obtained after multi-head concatenation, feedforward, and residual processing. .

[0046] 2) Perform attention pooling. This is for processing the sequence... O Global features are aggregated and a learnable attention weight vector is used. u Perform a weighted summation: , ; in, Indicates the first Feature vector at time 1 Normalized attention weights; and These are all time step indices of the sequence. This is the aggregated global representation vector.

[0047] 3) Continuous score regression. This involves retrieving the global feature vector. Input a feedforward regression layer, pass the sigmoid function (an activation function that maps any real number to the (0,1) interval) and a linear transformation, and output continuous scores for each personality dimension (such as the five dimensions of the Big Five personality traits): ; in, This represents the Sigmoid function, which maps values ​​to the (0,1) interval and then linearly scales them to the [1,5] interval, consistent with the scale (a standard personality psychometric scale). and The first The transpose and bias terms of the linear projection weight vector of an individual personality dimension.

[0048] The model training uses the mean squared error (MSE) between the predicted score and the true labeled score as the loss function: ; in, The total number of personality dimensions. For the index of the current dimension, For the first The true label value of the dimension, This is the corresponding model prediction value.

[0049] Step 106: Model training and inference.

[0050] In the embodiments of this application, model training uses publicly available or proprietary datasets containing interview audio and personality tags (such as First Impressions V2).

[0051] The training, validation, and test sets must be strictly divided according to the principle of subject identity segregation to ensure that data from the same subject does not appear in different sets, in order to evaluate the model's generalization ability. For example, this dataset contains 10,000 high-resolution 1280×720 individual audio-visual clips from YouTube, which have been preprocessed into short video segments with an average length of 15 seconds. The labels are quantitative labels for each sample, representing the "Big Five" personality traits (openness, conscientiousness, extraversion, agreeableness, neuroticism) and overall interview performance, with all dimension scores normalized to the [0,1] interval. The dataset split follows the principle of subject identity mutual exclusion, strictly divided into a 6:2:2 ratio for a training set of 6000 examples, a validation set of 2000 examples, and a test set of 2000 examples.

[0052] The ASR module, sentence vector encoder, and cross-question attention and regression module can be jointly trained in stages or end-to-end. The optimizer uses AdamW, combined with learning rate warm-up and cosine annealing strategies. Dropout and early stopping are used to prevent overfitting.

[0053] During the inference phase, an interview audio clip goes through steps 101 to 105 sequentially, ultimately outputting continuous scores for the test subject across various personality dimensions. To ensure consistency of scores across different model versions or business scenarios, the regression output is calibrated before inference, for example, using temperature scaling. This involves using a preserved calibration set to learn a scaling parameter to correct for systematic biases in the model.

[0054] It should be noted that the audio data of the interviewees (test subjects) involved in this embodiment were all obtained in a compliant manner, such as by collecting and using their voice information with their consent.

[0055] The intelligent speech and text processing method for structured interviews provided in this application has the following significant advantages: (1) Achieved deep semantic understanding and coherent capture of personality cues in structured interview scenarios: By treating the test subject's answers to multiple questions as a whole sequence and using a cross-question attention aggregation model for modeling, the semantic dependencies and relative importance between different answers can be explicitly learned. This overcomes the problem of fragmented contextual information caused by processing each answer independently, and can extract a coherent and stable personality profile from the entire interview dialogue, resulting in a more comprehensive and accurate evaluation.

[0056] (2) A highly controllable and low-error-propagation localized technical link was constructed, enhancing the robustness and compliance of the system: an end-to-end speech recognition (ASR) model that can be trained and deployed locally was adopted to replace the uncontrollable third-party "black box" service. This not only improves the transcription accuracy in specific business scenarios (such as professional terminology and accents), but also provides a reliable basis for downstream processing with its synchronously output word-level confidence, effectively suppressing the gradual amplification of transcription errors in subsequent stages. At the same time, localized processing avoids the risk of sensitive audio data being transmitted outside the system, which is more in line with the compliance requirements for data privacy and security in fields such as finance and healthcare.

[0057] (3) It provides refined and interpretable assessment outputs that conform to psychometric practice: It innovatively adopts a continuous score regression model and calibrates the output to the psychologically common [1,5] continuous interval, replacing the traditional coarse-grained discrete classification. This method can more accurately characterize individual differences, and the output results can be directly compared with standard personality scales. It is more practical and interpretable, and is easy for business departments such as human resources to understand and apply directly.

[0058] (4) A complete, reproducible, and engineering-friendly end-to-end solution has been formed: From audio input to personality score output, this method designs a seamless and highly collaborative technical link. Through the design of a unified sub-vocabulary and standardized data interfaces, the consistency of the process is ensured. At the same time, this method provides a clear model structure, loss function, and training strategy, which is highly reproducible. The entire system supports flexible deployment from single machine to distributed cluster and can adapt to various application modes such as offline batch processing and online real-time evaluation, with strong engineering feasibility.

[0059] Example 2 This embodiment provides another intelligent voice and text processing method for structured interviews. From an engineering implementation perspective, it details how to deploy the method described in Embodiment 1 in a real-world hardware and software environment and build a complete, maintainable, and scalable evaluation system. This embodiment focuses on the implementation process, technology stack selection, and system architecture.

[0060] Figure 2 The flowchart below illustrates the intelligent speech and text processing method for structured interviews according to Embodiment 2 of this application. Figure 2 This embodiment will be described in detail.

[0061] Step 201: Set up the deployment environment.

[0062] In the embodiments of this application, the hardware employs a server equipped with a high-performance graphics processing unit (GPU). As an optional configuration, a single server can be configured with at least one NVIDIA A10 or A100 GPU, at least 64GB of system memory, and sufficient disk space reserved for storing models and data (such as model weights, vocabulary, configuration files, and intermediate data). For software, a Linux distribution (such as Ubuntu 20.04 LTS) is recommended. To ensure deployment flexibility and scalability, the entire system adopts containerization technology, using Docker to encapsulate each functional component into images, and Kubernetes for container orchestration to achieve automatic scaling and high availability of services.

[0063] Step 202: Audio data acquisition and standardization preprocessing.

[0064] In the embodiments of this application, the system receives raw audio or video streams from the interview terminal. If the input is video, the audio track is first extracted using ffmpeg and uniformly resampled to 16kHz, mono WAV format. Subsequently, acoustic feature extraction is performed using the torchaudio or librosa library, specifically including short-time Fourier transform (STFT), Mel filter bank mapping, and logarithmic operations, ultimately generating an 80-dimensional log-Mel spectral feature sequence. During the model training phase, to improve robustness, data augmentation is applied to this feature sequence, specifically including random masking (SpecAugment) in the time and frequency domains and speech rate perturbation of the original audio. This step generates a unique test object identifier (session_id) and timestamp for each processed data segment.

[0065] Step 203: Localized speech recognition (ASR) processing.

[0066] In the embodiments of this application, the acoustic features preprocessed in step 202 are fed into a locally deployed end-to-end ASR module. This module is implemented based on the PyTorch framework, and its model architecture can refer to integrated designs such as U2++, integrating convolution and self-attention mechanisms. The model's sub-vocabulary is generated using the SentencePiece tool, trained on business-related interview corpora, with a vocabulary size typically set to 2000 to 10000. Training is optimized using a joint loss of CTC and attention decoders. During inference, beam search decoding is used, with a configurable beam width (e.g., 5-20). Optionally, the transcription accuracy can be further improved by fusing an N-gram language model trained on domain text (e.g., built using the KenLM tool). Finally, the transcribed text with timestamps and confidence scores is output.

[0067] Step 204: Text post-processing and structured alignment.

[0068] In the embodiments of this application, the text output by the ASR module is cleaned and structured to correspond one-to-one with the interview questions. This step mainly relies on rules and a lightweight model to achieve the following process: Punctuation recovery: Automatically predict and add sentence-ending punctuation using a small sequence model finely tuned on local data (such as BiLSTM-CRF or Transformer).

[0069] Text cleaning: Based on a custom rule base, remove colloquial filler words such as "uh" and "that", and merge obvious erroneous word segmentation fragments.

[0070] Terminology standardization: Based on the built-in dictionary, non-standard company names, job titles, etc. are standardized.

[0071] Question Alignment: Based on the audio timestamp and the preset interview question timeline, the processed text is segmented and associated with specific question IDs to form a structured list.

[0072] Step 205: Domain-Adaptive Sentence Vector Extraction. In the embodiments of this application, the answer text for each question is encoded into a fixed-dimensional semantic vector. First, word segmentation is performed using the SentencePiece tokenizer shared with the ASR module. Then, a multi-layer Transformer encoder is input. To make the encoder more relevant to interview scenarios, a two-stage training strategy is adopted: first, pre-training is performed on a large amount of unlabeled interview text (such as a masked language model task); subsequently, fine-tuning is performed on personality-related labeled data using the contrastive learning loss (InfoNCE) or supervised triple loss described in Embodiment 1. During inference, the labeled vector output by the encoder is average-pooled and then passed through a linear projection layer to obtain the final sentence vector.

[0073] Step 206: Cross-problem relationship modeling and scoring calibration.

[0074] In the embodiments of this application, sentence vectors of all questions from the same interview are collected and stacked sequentially. Semantic dependencies between questions are modeled using a lightweight multi-head self-attention network. The sequence of attention outputs is aggregated into a global feature vector through a learnable attention pooling layer. Finally, an initial score for each personality dimension is output through a feedforward neural network regression layer. Crucially, a calibration step is introduced before outputting the final result: using a preserved calibration dataset, temperature scaling or isotonic regression methods are employed to calibrate the original output of the regression model to eliminate model bias and ensure consistency of scores across versions and scenarios.

[0075] Step 207: System training, deployment and operation and maintenance management.

[0076] In the embodiments of this application, the training and deployment of the model follow the standardized process as follows: 1) Training Management: Hydra is used for hyperparameter configuration management, and Mlflow is used to track the experimental process, record metrics, and save model versions. Training data must be strictly partitioned according to the test object ID to prevent data leakage.

[0077] 2) Model Deployment: The trained model and its associated files (vocabulary, configuration) are stored in versioned object storage. The inference service loads the model from the specified path and can perform health checks and shadow testing.

[0078] 3) Application Modes: The system supports multiple application modes. In batch processing mode, it can asynchronously process a large amount of historical audio and video; in real-time streaming mode, ASR can use streaming decoding to achieve real-time feedback with a latency of up to a second; in high-security intranet mode, the entire system can be deployed offline and run in a pure CPU environment using ONNX Runtime and model quantization technology.

[0079] This embodiment optimizes the text encoder specifically by performing contrastive learning fine-tuning or domain-adaptive pre-training on interview corpora. This makes the model more sensitive to specific language styles, expression habits, and subtle semantic cues related to personality in interview scenarios, enabling it to extract more discriminative deep semantic features. It solves the core bottleneck of feature mismatch in general pre-trained models on specific tasks, directly improving the accuracy of personality recognition.

[0080] Example 3 This embodiment provides an intelligent speech and text processing system for structured interviews, employing the intelligent speech and text processing method steps for structured interviews as described in the previous embodiment. This system is a complete, deployable integrated hardware and software solution, designed to concretize each step of the process into independently operable, maintainable, and expandable components through modular and service-oriented design, ensuring end-to-end process coherence, controllability, and high availability in engineering practice.

[0081] The system consists of seven modules that communicate asynchronously via high-performance RPC calls based on the gRPC protocol or via the Apache Kafka message queue, with data serialized and transmitted in Protocol Buffers format.

[0082] Data acquisition and preprocessing module: Uses ffmpeg to extract the audio track from the video and resamples it. Utilizes torchaudio to extract STFT and EL features, generating logs and verification information.

[0083] The locally deployed Automatic Speech Recognition (ASR) module implements an end-to-end model based on an open-source framework (such as WeNet). It trains a SentencePiece word model (vocabulary size 2000-10000) using business-relevant corpora. During inference, it loads an N-gram language model via KenLM for shallow fusion. This module outputs structured data including text, timestamps, and confidence scores.

[0084] Text normalization and alignment module: Developed in Python, it integrates word segmentation tools such as jieba / spaCy and a custom rule library. It uses a lightweight sequence model (such as Transformer or BiLSTM-CRF) for punctuation recovery. This module ensures that each output response text is clean and structured.

[0085] Sentence Vector Encoding Module: Implements a PyTorch Transformer encoder. This module provides two training interfaces: unsupervised contrastive learning based on InfoNCE, and supervised learning based on triplet loss. To obtain better domain representations, a masked language model (MLM) can be pre-trained on a large-scale interview text, followed by fine-tuning through contrastive learning. This module receives text and outputs fixed-dimensional sentence vectors.

[0086] Cross-problem modeling and prediction module: Implements lightweight multi-head self-attention layers and regression layers. Integrated calibration functionality supports temperature scaling or isotonic regression using scikit-learn. This module receives all sentence vectors from the same test subject and outputs continuous scores for personality dimensions.

[0087] Training and Inference Management Module: Hydra (a distributed task processing system) is used for configuration management, and Mlflow (an open-source machine learning lifecycle management platform) is used to track experiments, record parameters and metrics, and manage model versions. All model artifacts (weights, vocabulary, configuration) are stored in object storage, and version consistency is ensured through hash values. Shadow deployment and automatic rollback mechanisms are supported. Specifically, the configuration of all training tasks, including random seeds, data splitting schemes, hyperparameter combinations, and software dependency versions, is fixed by Hydra once when the job starts. Various metrics, logs, and visualization results during the training process are uniformly recorded and tracked through Mlflow. All model-related artifacts (weights, vocabulary, configuration files, etc.) are uploaded to object storage, and their SHA hash values ​​are recorded to completely eliminate the risk of version drift. When the model goes live, the inference service pulls the model from object storage according to the specified version number, performs a cold start and health check in shadow traffic, and only releases it in a canary manner after confirming that everything is correct. If online monitoring metrics show abnormalities, the management mechanism can trigger an automatic rollback to switch to the previous stable model version.

[0088] The aforementioned system is deployed on at least one server. As a more specific example, this server is configured with: an Intel Xeon Gold 6348 processor with an x86_64 CPU architecture, at least two NVIDIA A100 80GB GPUs, 512GB of DDR4 memory, and 10TB of NVMe SSDs for high-speed data caching and 2PB of distributed object storage (such as MinIO or an S3-compatible system) for persistent storage. The operating system is Ubuntu 20.04 LTS. The entire system is packaged and managed using containerization technology, with all core service modules packaged as Docker images and orchestrated using a Kubernetes 1.24 cluster to achieve automatic resource scheduling, elastic scaling of services, and high-availability deployment.

[0089] In the embodiments of this application, each module is encapsulated as an independent microservice module, providing a unified interface to the outside world through an API gateway, and performing asynchronous communication and data exchange through message queues or centralized storage (such as Redis, databases). This allows modules to be deployed, scaled, and updated independently, improving the maintainability and scalability of the system.

[0090] It's understandable that while each module is encapsulated as an independent microservice, the output of the upstream module is a necessary input for the downstream module, forming an irreversible dependency chain. Simultaneously, deep collaboration is achieved through shared key components and unified scheduling management (the training and management platform uniformly manages the training, version release, and updates of all module models), jointly completing end-to-end automatic evaluation from raw interview audio / video to personality quantification scoring. For example, the ASR module and the sentence vector encoding module share the same SentencePiece sub-word model, ensuring consistency in lexical representation throughout the entire process from sound to text to semantic vectors, avoiding information loss due to vocabulary mismatch. The text standardization and alignment module relies on the word-level timestamps and confidence scores output by the ASR module for more accurate problem alignment and error correction. The confidence score information output by the ASR module is passed downstream along with the text. This provides data reliability indicators for subsequent modules (such as aggregation models), allowing the system to weight or specially process low-confidence portions, achieving robust control over recognition errors. Before outputting the final score, the cross-problem modeling module (which includes an attention aggregation submodule and a regression submodule based on a multi-head self-attention mechanism) calls independent calibration parameters (provided by the management platform when the model is released) to standardize and calibrate the results, ensuring cross-version consistency of the evaluation results.

[0091] The system provided in this application has the following application scenarios: Offline batch processing: Suitable for scenarios such as campus recruitment, the system can process a large number of interview audios at night. ASR uses high-precision beamwidth and is executed asynchronously throughout the entire process.

[0092] Online real-time assessment: Suitable for online mock interviews or coaching. ASR switches to streaming decoding mode, and subsequent modules perform incremental processing, providing feedback within a second-level delay.

[0093] High-security intranet deployment: The entire system can be packaged as an offline installation package and run on a server without an external network. In CPU-only environments, models can be deployed using the ONNX Runtime framework and INT8 quantization technology to achieve a balance between performance and resources.

[0094] Example 4 In embodiments of this application, an electronic device is also provided. Figure 3 This is a schematic diagram of the electronic device structure according to Embodiment 3 of this application, as shown below. Figure 3 As shown, the electronic device of this application includes a processor 301 and a memory 302, wherein, The memory 302 stores a computer program, which, when read and executed by the processor 301, performs the steps described above in the embodiment of the intelligent speech and text processing method for structured interviews.

[0095] Example 5 In the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein the computer program is configured to execute the steps in the embodiments of the intelligent speech and text processing method for structured interviews as described above when it runs.

[0096] In this embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0097] It will be understood by those skilled in the art that the above descriptions are merely preferred embodiments of this application and are not intended to limit this application. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An intelligent speech and text processing method for structured interviews, characterized in that, Includes the following steps: The acquired raw interview audio data is preprocessed and acoustic feature sequences are extracted. The acoustic feature sequence is input into a locally deployed end-to-end speech recognition model for recognition and transcription, resulting in transcribed text containing timestamp information. Based on the timestamp, the transcribed text is standardized and aligned with preset interview questions to obtain a set of answer texts aligned with the interview questions; For each answer text in the set of answer texts, use a Transformer encoder to encode it to obtain the corresponding sentence vector; Based on all sentence vectors of the same test subject, semantic association modeling is performed through a cross-question attention aggregation model based on a multi-head self-attention mechanism, and the test subject's scores on each personality dimension are output through a regression layer.

2. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, The step of preprocessing the acquired raw interview audio data and extracting acoustic feature sequences further includes: The original audio data is uniformly resampled into a mono format with a specified sampling rate; The short-time Fourier transform of the resampled audio data is performed to obtain the spectrum; The Mel filter bank is applied to the spectrum and the logarithm is taken to obtain the logarithmic Mel spectrum as the acoustic feature sequence.

3. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, The step of inputting the acoustic feature sequence into a locally deployed end-to-end speech recognition model for recognition and transcription further includes: the end-to-end speech recognition model is trained using a joint objective of connection temporal classification loss and attention decoder loss; during the inference phase, a beam search algorithm is used for decoding, and optionally, probabilistic fusion is performed with an external language model; the output includes a transcription result containing a word sequence, timestamp, and word or word-level confidence.

4. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, The step of standardizing the transcribed text based on the timestamp and aligning it with preset interview questions includes at least one of the following: restoring punctuation marks in the text using a sequence labeling model; cleaning up colloquial filler words and correcting erroneous word segmentation based on a preset rule base; standardizing named entities according to a built-in dictionary; and aligning the processed text with preset interview questions based on the audio timestamp.

5. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, The step of encoding each answer text in the answer text set using a Transformer encoder to obtain the corresponding sentence vector further includes: segmenting the answer text using a sub-vocabulary consistent with the end-to-end speech recognition model; inputting the segmented tag sequence into a multi-layer Transformer encoder to obtain a hidden state representation; and performing a pooling operation on the hidden state representation and then linearly projecting it to obtain a fixed-dimensional sentence vector.

6. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, The step of semantic association modeling based on all sentence vectors of the same test subject through a cross-question attention aggregation model based on a multi-head self-attention mechanism, and outputting the test subject's scores on each personality dimension through a regression layer, further includes: calculating an enhanced sequence from the sentence vector sequence arranged in question order through a multi-head self-attention mechanism; performing learnable attention pooling on the enhanced sequence to aggregate and obtain a global feature vector; inputting the global feature vector into a regression layer, and outputting continuous scores for each personality dimension after mapping by an activation function.

7. The intelligent speech and text processing method for structured interviews according to claim 1, characterized in that, After the step of outputting the scores of the test subjects on each personality dimension through a regression layer, the method further includes a step of calibrating the output of the regression layer using a calibration dataset to eliminate model bias.

8. An intelligent voice and text processing system for structured interviews, characterized in that, The intelligent speech and text processing method for structured interviews according to any one of claims 1 to 7, the system comprising: an audio preprocessing module for preprocessing raw audio data and extracting acoustic feature sequences; a locally deployed speech recognition module for transing the acoustic feature sequences into text containing timestamp information; a text standardization and alignment module for cleaning, standardizing, and aligning the text with the interview questions based on the timestamps to obtain aligned answer text; a sentence vector encoding module for encoding the answer text into sentence vectors; and a cross-question modeling and prediction module, comprising an attention aggregation submodule and a regression submodule based on a multi-head self-attention mechanism, for performing semantic association modeling based on the sentence vector sequences and outputting scores for the personality dimension.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor is configured to execute the computer program stored in the memory to implement the intelligent voice and text processing method for structured interviews as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which is loaded and executed by a processor to implement the intelligent voice and text processing method for structured interviews as described in any one of claims 1 to 7.

Images