Speech prompt cheating behavior detection method based on deep learning and electronic device

Abstract

The application discloses a kind of voice prompt cheating behavior detection methods based on deep learning, comprising: the video data in car obtained is carried out sliding slice, extract the audio part in the sub-video after slicing, and the audio data with same time dimension and uniform format is output after format conversion;The corresponding time domain signal is generated by pre-processing to audio data, is converted into spectrogram with amplitude and frequency information, each spectrogram is filtered using mel filter bank, and mel spectrum is generated;Mel spectrum is input to feature extraction grid module to output one-dimensional feature vector of audio data, and then input to multi-label classifier network module to obtain the sound class contained in audio data;If the sound class contains the voice prompt class of suspected cheating behavior, it is preliminarily judged as suspected cheating.The application is combined with speech recognition and deep learning algorithm, and can be suitable for driving test to assist in judging whether safety officer exists cheating behavior.

Description

Technical Field

[0001] This invention belongs to the field of driving test detection technology, specifically relating to a method and electronic device for detecting cheating behavior using voice prompts based on deep learning. Background Technology

[0002] With the continuous development of the driving test field, regulating examination discipline is becoming increasingly crucial. During the driving test, the system checks whether the safety officer inside the vehicle provides verbal prompts to the examinee. If the safety officer coughs, taps on the vehicle, or speaks, it will be considered a suspected cheating offense, and the suspected cheating data will be recommended to the management for review. The management's review only needs to focus on the suspected cheating voice segments and combine them with the corresponding actions in the video.

[0003] Detecting whether safety officers use voice prompts during driving tests is a crucial part of driving test regulations. It's key to ensuring fairness and impartiality, and can, to some extent, regulate test discipline and reduce cheating. Currently, detecting voice prompt cheating in driving tests mainly relies on manual listening and watching by back-end administrators. These administrators need to review all test data throughout the entire process, which is time-consuming and inefficient, and urgently needs improvement. Summary of the Invention

[0004] To address the aforementioned problems, this invention provides a deep learning-based method and electronic device for detecting cheating behavior via voice prompts. This method combines speech recognition with deep learning algorithms and can be applied to assist in determining whether a safety officer is engaging in cheating during driving tests. Furthermore, it can recommend target data indicating suspected cheating behavior to management personnel for review, ultimately confirming whether cheating has indeed occurred.

[0005] The first aspect of this invention discloses a method for detecting cheating behavior through voice prompts based on deep learning, which mainly includes the following steps:

[0006] Step 1: Slide the acquired in-vehicle video data to obtain several sub-videos, extract the audio portion from the sub-videos, and output several audio data with the same time dimension and uniform format after format conversion;

[0007] Step 2: The audio data is preprocessed to generate corresponding time-domain signals. Then, the time-domain signals are converted into spectrograms with amplitude and frequency information by short-time Fourier transform. Finally, Mel filter banks are used to filter each spectrogram to generate the corresponding Mel spectrum.

[0008] Step 3: Input the Mel spectrum into the trained target deep neural network model. The deep neural network model includes a feature extraction network module and a multi-label classifier network module. The Mel spectrum outputs a one-dimensional feature vector of the audio data through the feature extraction network module, and the one-dimensional feature vector is input into the multi-label classifier network module to output the sound categories contained in the audio data. If the sound category contains a voice prompt category that is suspected of cheating, it is initially determined to be suspected cheating.

[0009] As an optional approach, step 1 specifically includes:

[0010] Step 1.1: Acquire in-vehicle video data using a camera fixed inside the vehicle, and convert the video data into MP4 format;

[0011] Step 1.2: Slice the video data into multiple sub-videos of equal duration by using a time window of fixed length t;

[0012] Step 1.3: Extract the PCM format audio stream from the sub-video to obtain audio data of equal duration, and then convert the PCM format audio stream to WAV format;

[0013] Step 1.4: Perform end padding on audio data with a duration of less than t to ensure that the output audio data has the same time dimension.

[0014] As an optional approach, in step 2, preprocessing refers to using pre-emphasis, framing, and windowing techniques to manipulate the speech signal in the audio data to generate a corresponding time-domain signal; the pre-emphasis is used to compensate for the attenuation of the high-frequency part in the audio data; the framing is used to obtain the short-time stationarity of the speech signal; the windowing is to multiply the framed speech signal with a window function so that the speech signal exhibits some characteristics of a periodic function.

[0015] As an optional approach, the pre-emphasis uses a first-order high-pass filter; the framing employs an overlapping segmentation method; and the window function is a Hamming window.

[0016] As an optional approach, in step 3, the feature extraction network module adopts the CNN14 network architecture; the backbone network of the CNN14 network architecture consists of 6 CNN blocks, each CNN block has an internal structure of 2*(Conv+BN+ReLU)+avg pooling network layers, each CNN block is connected to a Dropout layer, and the network ends by combining max pooling and avg pooling and the flatten operation of the fully connected layer to output a one-dimensional feature vector after feature fusion;

[0017] The multi-label classifier network module includes a fully connected network layer with a neuron structure and N binary sub-classifiers. The input of the fully connected network layer is connected to the output features of the feature extraction network module, and the output is connected to the binary sub-classifiers, converting the one-dimensional feature vector of the audio data into N output branches; each output branch corresponds to a sound category.

[0018] The binary subclassifier inputs the output values ​​of the neurons in the output branch into the sigmoid function to obtain the probability value of each sound category. The probability value ranges from (0, 1). The probability value represents the confidence level of each output category. If the confidence level is greater than a preset threshold, the corresponding sound category is output; otherwise, no output is made.

[0019] As an optional approach, the training process of the deep neural network model includes:

[0020] The training data and its corresponding training labels are input into the initially constructed deep neural network model. The deep neural network model is iteratively updated through backpropagation. The training ends when the total loss value tends to converge.

[0021] The total loss function of the deep neural network model consists of the losses of N binary sub-classifiers, with each binary sub-classifier having a loss value of Loss. cls Represented as:

[0022]

[0023] The total loss value is the loss value of all sub-classifiers. cls The average value is expressed as:

[0024]

[0025] In the formula, p is the output probability value of each binary subclassifier, with a value range of (0, 1); y is the numerical label value of each branch, y = 0 or 1; α is the positive and negative sample balance factor; γ is the hard sample weight adjustment factor; N is the number of categories in the output branch;

[0026] The training data consists of several Mel spectra obtained by processing the acquired in-vehicle video data through steps 1 and 2.

[0027] The training labels are numerical sound category labels corresponding to each video data point, which are manually labeled.

[0028] As an alternative approach, for the positive and negative sample balancing factor α, based on the overall distribution of the training dataset, a solution following the long-tail distribution assigns different weight coefficients to each category, where the weight of the category with fewer samples is greater than the weight of the category with more samples. For the hard sample weight adjustment factor γ, it is determined through different iteration cycles in the training process, with different values ​​assigned in the early, middle, and late stages of training. As the number of training iterations increases, the hard sample weight adjustment factor continuously increases.

[0029] As an optional solution, the sound categories include: coughing, knocking on the vehicle body, speaking, exam voice broadcasting, opening and closing the door, inserting and removing the seat belt, gear shifting, handbrake and foot brake operation, and light operation. Among them, the voice prompt categories that include suspected cheating behavior are: coughing, knocking, and speaking.

[0030] As an alternative, the detection method also includes:

[0031] Step 4: After determining that the sound category contains voice prompts that suggest cheating behavior, the corresponding audio data is recommended to the administrator for review to determine whether cheating behavior actually exists.

[0032] A second aspect of the present invention discloses an electronic device comprising a processor and a memory, wherein the processor is configured to invoke a computer program stored in the memory to execute the deep learning-based voice prompt cheating detection method described in the first aspect and any alternative embodiment thereof.

[0033] The present invention has the following beneficial effects:

[0034] (1) This invention combines speech recognition with deep learning algorithms, which can automatically identify suspected cheating behavior. In this way, target data that is initially judged to be suspected cheating behavior can be selected from a large amount of video data and recommended to the management personnel for review. This can greatly reduce the workload of the management personnel, improve work efficiency, and play a role in standardizing driving test discipline and improving the supervision ability of safety officers.

[0035] (2) This invention can effectively improve and optimize the focal loss function for multi-label speech classification problems, while taking into account the overall situation of its own data distribution, thereby further improving the overall classification performance of the model.

[0036] (3) Based on intelligent audio and video technology, this invention can be applied to any complex scenario. The hardware devices involved only include cameras, microphones and electronic devices such as computers and tablets. No vehicle body modification is required, which greatly reduces hardware costs and operating power consumption. It is also highly robust to different drivers and vehicle models, and has high practical value and broad application prospects. Attached Figure Description

[0037] Figure 1 is a schematic diagram of time-frequency analysis of speech signals, in which: Figure 1a For time-domain signals, that is, audio waveforms of speech signals after sampling at a certain sampling rate, the horizontal axis represents time and the vertical axis represents amplitude. For example, a 32kHz sampling rate means that there are 32,000 samples per second. Figure 1b The spectrum is the Fourier transform result, which is the spectrum generated after the audio waveform is subjected to a short-time Fourier transform. The horizontal axis represents time and the vertical axis represents frequency. Figure 1c This is a Mel spectrogram, which is the Mel spectrogram generated after filtering the spectrogram using a Mel filter bank. The horizontal axis represents time, and the vertical axis represents frequency.

[0038] Figure 2 This is a schematic diagram of the structure of a deep neural network model. Detailed Implementation

[0039] To further understand the present invention, the technical solution of the present invention will be further explained and described below with reference to specific embodiments and accompanying drawings.

[0040] This embodiment discloses a method for detecting cheating behavior using voice prompts based on deep learning, which mainly includes the following four steps, as detailed below:

[0041] Step 1: Process the acquired in-vehicle video data by sliding slicing, audio extraction, and format conversion to obtain several audio data with the same time dimension and uniform format.

[0042] The specific process of step 1 is as follows:

[0043] Step 1.1: Capture different types of video data, such as mkv and mp4, using cameras fixed in different driving test vehicles. If the video is not in mp4 format, use the ffmpeg program to convert it to a uniform mp4 format to ensure that a uniform mp4 format video is generated.

[0044] Step 1.2: The variable-length video data is sliced ​​using a fixed-length time window of t. Starting from the beginning of the video, sub-videos are extracted at a frequency of s, until the video ends. All extracted video segments are then obtained, which can be called "sub-videos". For example, for a video data with a total duration of 58s, a fixed-length time window of 3s is used for sliding slicing. Starting from the beginning of the video, sub-videos are extracted at a frequency of 2, resulting in 29 sub-videos: 0–2s, 2–4s, and 4–6s.

[0045] Typically, adjacent sub-videos will have a certain duration of overlap, allowing segments of the same target behavior to be distributed across different locations in multiple adjacent sub-videos. For example, a gear-shifting action segment might occur at 2 seconds in one sub-video and at 0 seconds in another, thus enhancing the diversity of the data sample.

[0046] Understandably, if a training dataset needs to be constructed, the obtained sub-videos need to be manually labeled, all sound categories contained therein need to be listened to, and the labeling information needs to be recorded in JSON file format.

[0047] Step 1.3: Extract the audio portion from the sub-video obtained by slicing, that is, extract the PCM format audio stream from the mp4 format file to obtain audio data of the same duration as the sub-video, and convert the PCM format audio stream into WAV audio data of a unified format.

[0048] In MP4 format videos, the audio stream is encoded in AAC format. AAC is an audio encoding format resulting from PCM compression, which introduces some data distortion compared to PCM. Converting PCM audio streams to a unified WAV format is to better adapt to the network model input, while WAV format exhibits almost no compression distortion compared to PCM. Therefore, this invention, to reduce data distortion and achieve standardization, first converts the audio data to PCM and then to WAV format before sending it to the network.

[0049] Step 1.4: During the slicing process, for sub-videos whose duration is less than the time window t at the end, their corresponding audio is uniformly aligned. Specifically, zero-value padding is performed at the end to ensure that the audio data input to the model has the same time dimension. For example, if the last video segment is 2 seconds long, zero values ​​are padded at the end of its corresponding audio to make its duration a uniform 3 seconds.

[0050] It should be noted that if the time window at the end of the video is exactly equal to a fixed length t, then no padding operation is needed.

[0051] Step 2: Perform time-frequency analysis on the audio data obtained after processing in Step 1, convert it into a spectrogram through short-time Fourier transform, and obtain the Mel spectrum after filtering.

[0052] The specific process of step 2 is as follows:

[0053] Step 2.1: Preprocess the audio data obtained after Step 1, that is, use techniques such as pre-emphasis, framing and windowing to manipulate the speech signal in the audio data to generate the corresponding time domain signal.

[0054] Pre-emphasis aims to maintain the low-frequency components (vowels) of the speech signal while boosting the high-frequency components (consonants). This boosting of high-frequency components is primarily because they contain more feature information, while low-frequency components generally contain less. Since the power spectrum of a speech signal decreases with increasing frequency, with most of its energy concentrated in the low-frequency range, the signal-to-noise ratio (SNR) of the high-frequency components can drop to an unacceptable level. Pre-emphasis can compensate for the excessive loss of high-frequency components in the audio data. In this invention, pre-emphasis can specifically employ a first-order high-pass filter to compensate for the high-frequency components of the audio signal.

[0055] Since the Fourier transform requires the input signal to be nearly stationary, short-time stationarity of the speech signal can be obtained through frame segmentation. To ensure smooth transitions between frames and maintain continuity, overlapping segmentation can be used for framing.

[0056] Windowing involves multiplying the framed speech signal by a window function, causing the originally non-periodic speech signal to exhibit some characteristics of a periodic function, thus facilitating better Fourier expansion later. For example, a Hamming window can be chosen as the window function, and its expression is as follows:

[0057]

[0058] In the formula, N represents the window width and t represents time.

[0059] Step 2.2: Using the short-time Fourier transform algorithm, perform Fourier operations on each frame of the preprocessed audio data to convert the time-domain signal into a frequency-domain signal and obtain the corresponding spectrogram.

[0060] Each speech signal can be viewed as a synthesis of sine or cosine signal units of different frequencies. Fourier transform can decompose the time-domain signal into multiple frequency-domain signals of different frequencies. By summarizing the Fourier operation results for each frame of each audio data, a spectrogram containing amplitude and frequency information is obtained for each audio data point.

[0061] Step 2.3: Use the Mel filter bank to filter each spectrogram to generate the corresponding Mel spectrum.

[0062] Humans are more sensitive to low-frequency signals. The human ear's perception of frequencies measured in Hz (Hz) is not linear. We can easily distinguish a doubling of the frequency difference between 500Hz and 1000Hz, but for sounds of 5000Hz and 10000Hz, our ears only perceive a slight increase in frequency, not a doubling. However, if we convert the ordinary frequency scale to the Mel-scale, the human ear's perception of frequency becomes linear. The mapping relationship is shown in the following equation:

[0063] mel(f) = 2595 * log 10 (1+f / 700)

[0064] In other words, under the Mel scale, if the Mel frequencies of two speech segments differ by a factor of two, the pitches that the human ear can perceive will also differ by a factor of two.

[0065] To convert frequency signals into signals with linear sensing capabilities, a Mel filter bank (containing multiple Mel filters) can be used to filter each spectrogram. For the high-frequency part, a nonlinear mapping is applied, that is, mapping the Hz frequency scale unit to the Mel frequency scale unit, and finally generating the Mel spectrogram.

[0066] Step 3: Use the Mel spectrum generated in Step 2 as input to the trained deep neural network model for inference and prediction, and output the category results corresponding to each audio data. If the category results include voice prompts that indicate suspected cheating behavior, it is preliminarily determined to be suspected cheating.

[0067] The specific process of step 3 is as follows:

[0068] Step 3.1: Construct the initial deep neural network model.

[0069] Understandably, an initial deep neural network model needs to be built before inference and prediction can be performed. After training, a target deep neural network model can be obtained. The target deep neural network model, after training, is used to perform inference and prediction on the input Mel spectrum super.

[0070] Combination Figure 2 As shown, the deep neural network model constructed in this invention mainly comprises two parts: a feature extraction network module and a multi-label classifier network module. The feature extraction network module is used for feature extraction of samples. The Mel spectrum obtained in step 2 is used as input to the feature extraction network module to extract the deep semantic feature vector of the corresponding audio data. The one-dimensional feature vector output by the feature extraction network module, i.e., the deep semantic features of the Mel spectrum, is used as input to the multi-label classifier network module to obtain the final audio data category output.

[0071] The feature extraction network module here adopts the CNN14 network architecture. The backbone network consists of 6 CNN blocks, each with an internal structure of 2*(Conv+BN+ReLU)+average pooling layers. Each CNN block is followed by a Dropout layer to prevent overfitting during training and improve model generalization. By stacking multiple CNN blocks, the receptive field of the convolutional neural network is continuously increased, and the number of convolutional kernels within each block is continuously increased to achieve full extraction of spectrogram features. At the end of the network, the output features are fused to form a one-dimensional feature vector by combining max pooling and average pooling, as well as the Flatten operation of the fully connected layer.

[0072] The specific network structure parameters of the feature extraction network module are as follows:

[0073] Table 1

[0074]

[0075]

[0076] In this embodiment, the multi-label classifier outputs a total of 9 categories. Internally, the multi-label classifier contains a fully connected network layer with a neuron structure, which connects to the output features of the CNN14 feature extraction network module. This converts the one-dimensional feature vector of the audio data into the output values ​​of 9 neurons, i.e., 9 output branches. It should be noted that the output values ​​refer to the network output values, which have no defined range. These output values ​​only become probability values, or confidence scores, after passing through the sigmoid function, and their range is (0,1).

[0077] The multi-label classifier network module also includes nine binary sub-classifiers, each containing a sigmoid function. The specific processing is as follows: the input of each binary sub-classifier is connected to a fully connected network layer, and the output values ​​of the neurons in the corresponding branch are input into the sigmoid function to obtain the probability output value for each category. This probability value ranges from (0,1], and represents the confidence level of each output category. When the confidence level is high and greater than a specified threshold, a value of 1 is output, indicating that the sound of that category exists in the current sample, and its category result is output; otherwise, a value of 0 is output, indicating that the sound does not exist in the sample, and no result is output. Here, the threshold value can be 0.5.

[0078] Each output branch of the classifier corresponds to a sound category, which in this case corresponds to various sound signals in the driving test vehicle environment. The nine categories are: coughing, knocking on the vehicle body, speaking, test voice broadcast, opening and closing doors, inserting and removing seat belts, gear shifting, handbrake and foot brake operation, light operation, and others.

[0079] Step 3.2 Train the initial deep neural network model to obtain the target deep neural network model.

[0080] After constructing the initial deep neural network model, the model is iteratively trained using pre-trained weights until the convergence condition is met, resulting in the target deep neural network model. The training process specifically includes:

[0081] The acquired in-vehicle video training data is processed sequentially using steps 1 and 2 to obtain the Mel spectrum corresponding to the training data. It is important to note that in step 1.2, each sub-video participating in the training needs to be manually labeled with a corresponding sound tag, indicating which sound category(s) it belongs to. Specifically, the category and source of the sound signal can be determined by combining the corresponding sub-video actions. If multiple sound categories exist in the same sample, all of them are labeled. For example, if a sub-video contains both the sound and action of pulling the handbrake and speaking, both sound categories are labeled. The labeling information can be saved and used by writing it to a JSON file.

[0082] To map the label information to the classifier output values, the audio labels for each sub-video also need to be quantified. Specifically, this involves assigning a 1 to the corresponding bit in the nine audio categories of the classifier if one or more categories exist in the current sample, otherwise assigning a 0. For example, if the current sample contains both "pulling the handbrake" (bit 7) and "speaking" (bit 3), the quantified label would be 001000100.

[0083] The numerically processed sound labels corresponding to each training data point, along with their corresponding Mel-frequency spectra, are used as training samples input into the pre-built initial deep neural network model. The initial deep neural network model is then trained using a loss function, along with the training data and labels, based on the network's pre-trained weights.

[0084] During training, training samples are fed from the input to the output of the deep neural network model via forward computation. In the classifier module at the output, the current loss value is calculated by combining the loss function and numerically encoded labels. The model then uses backpropagation to calculate the gradient of each parameter to be updated based on the loss value, thereby updating the network parameter values. After repeating the above training steps a certain number of times, the network parameter learning process tends to converge, at which point the total loss value tends to converge (for example, the total loss value stabilizes around 0.01), and training ends.

[0085] The total loss function of a deep neural network model consists of the losses of nine binary sub-classifiers, specifically including:

[0086] The loss function for each binary sub-classifier is the focal loss function, and the loss value for each binary sub-classifier is Loss. cls Represented as:

[0087]

[0088] The total loss function is the final result of merging the average losses of all binary sub-classifiers. The formula for calculating the total loss value (Loss) is as follows:

[0089]

[0090] In the formula, p is the output probability value of each binary subclassifier, with a value range of (0,1]; y is the numerical coding label value of each branch, y = 0 or 1; α and γ are the positive and negative sample balancing factors and the hard sample weight adjustment factors, respectively; N is the number of categories in the output branch, which is 9 here.

[0091] The focal loss function is typically used in binary or multi-class classification problems. Here, it is flexibly applied to multi-label classification scenarios and performs well. Furthermore, to address the imbalance between positive and negative samples and between easy and difficult samples, this invention introduces a positive-negative sample balancing factor α and a difficult sample weight adjustment factor γ, with fixed values ​​of 0.25 and 2, respectively.

[0092] Furthermore, considering our multi-label speech classification problem scenario and the overall distribution of our data, we effectively improved and optimized the focal loss function, thereby further enhancing the overall classification performance of the model.

[0093] The positive and negative sample balance factor α is flexibly determined by calculating the overall distribution of the training dataset. For example, some categories may have only about 200 data points, while others may have about 5000. Following a long-tail distribution approach, different weight coefficients are assigned to categories with different sample sizes: categories with fewer samples receive larger weights, and vice versa.

[0094] The adjustment factor γ for the weights of difficult samples is determined through different iteration cycles in the training process, with different values ​​assigned in the early, middle, and late stages of training. Specifically, these are set to 0.5, 1, and 2, respectively. As the number of training iterations increases, the adjustment factor for the weights of difficult samples continuously increases. This is because in the early stages of model training, the network has not learned enough from the entire training set, resulting in a poor ability to distinguish between easy and difficult samples; at this time, the weight distribution of easy and difficult samples is relatively even. As the number of learning iterations increases, the model's ability to distinguish between easy and difficult samples becomes stronger. Furthermore, in the later stages of training, the model's ability to distinguish between easy and difficult samples approaches an ideal level, and the learning effort for difficult samples should be increased. Therefore, the weight value for difficult samples should be increased at this point.

[0095] Step 3.3 Based on the trained target deep neural network model, the unlabeled audio data in the in-vehicle video data obtained after processing in Step 1 and Step 2 is used as test samples. The trained deep neural network model is used to perform inference and prediction, and the corresponding prediction label results are output. If the predicted label results contain voice prompts that indicate suspected cheating behavior, it is judged as suspected cheating behavior.

[0096] The inference and prediction process of the network model specifically includes:

[0097] Test samples are input into the network model. The samples are fed from the input to the output via forward inference. The classifier module at the output predicts the output probability for each category. If the probability is greater than a threshold of 0.5, that category is output. If multiple categories with high confidence are predicted simultaneously, all of them are output. For example, if the model predicts the following for a given data point: cough 0.8, inserting / removing seatbelt 0.9, and speaking 0.3, then the categories of cough and inserting / removing seatbelt will be output, while the category of speaking will not be output.

[0098] It's important to note that the model outputs nine sound categories, while the voice prompts for suspected cheating behavior mainly include coughing, tapping on the vehicle body, and speaking. From a business perspective, we only need to care whether the model output includes one or more of these three voice prompt types. However, from a model perspective, the output category design should not only include the target voice prompt type but also set corresponding categories for other sounds inside the vehicle. These categories, during loss function calculation and network parameter learning, will have strong distinguishability from the overall features of the target voice prompt type, thus effectively reducing model misjudgments.

[0099] Understandably, to ensure the accuracy of the results, when the system detects a target category from the voice data, it usually does not directly determine it as cheating, but gives a conclusion of suspected cheating. The final result is determined by the administrator after review.

[0100] Furthermore, the embodiments also disclose an electronic device, which mainly includes a processor and a memory. The processor is mainly used to call the computer program stored in the memory and execute the deep learning-based voice prompt cheating behavior detection method provided in the above embodiments, which will not be described in detail here.

[0101] Finally, it should be noted that although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the specific embodiments and application fields described above. The specific embodiments described above are merely illustrative and instructive, and not restrictive. Those skilled in the art, guided by this specification, can make many other forms without departing from the scope of protection of the claims of the present invention, and all of these are within the scope of protection of the present invention.

Claims

1. A method for detecting cheating behavior using voice prompts based on deep learning, characterized in that, Includes the following steps: Step 1: Slide the acquired in-vehicle video data to obtain several sub-videos, extract the audio portion from the sub-videos, and output several audio data with the same time dimension and uniform format after format conversion; Step 2: The audio data is preprocessed to generate corresponding time-domain signals. Then, the time-domain signals are converted into spectrograms with amplitude and frequency information by short-time Fourier transform. Finally, Mel filter banks are used to filter each spectrogram to generate the corresponding Mel spectrum. Step 3: Input the Mel spectrum into the trained target deep neural network model. The deep neural network model includes a feature extraction network module and a multi-label classifier network module. The Mel spectrum outputs a one-dimensional feature vector of the audio data through the feature extraction network module, and the one-dimensional feature vector is input into the multi-label classifier network module to output the sound categories contained in the audio data. If the sound category contains a voice prompt category that is suspected of cheating, it is initially determined to be suspected cheating. The multi-label classifier network module includes a fully connected network layer with a neuron structure and N binary sub-classifiers. The input of the fully connected network layer is connected to the output features of the feature extraction network module, and the output is connected to the binary sub-classifiers, converting the one-dimensional feature vector of the audio data into N output branches; each output branch corresponds to a sound category. The training process of the deep neural network model includes: The training data and its corresponding training labels are input into the initially constructed deep neural network model. The deep neural network model is iteratively updated through backpropagation. The training ends when the total loss value tends to converge. The total loss function of the deep neural network model consists of the losses of N binary sub-classifiers, where the loss value of each binary sub-classifier is... Represented as: Total loss value The loss values ​​for all subclassifiers The average value is expressed as: In the formula, The output probability value of each binary sub-classifier takes a value in the range of (0,1); Numerical label values ​​for each branch, ; This serves as a balance factor between positive and negative samples. For difficult samples, a weighting adjustment factor is used. This represents the number of categories in the output branches; The training data consists of several Mel spectra obtained by processing the acquired in-vehicle video data through steps 1 and 2. The training labels are numerical sound category labels corresponding to each video data point, which are manually labeled. Among them, the balance factor for positive and negative samples Based on the overall distribution of the training dataset, a solution following the long-tail distribution assigns different weight coefficients to each category, where the weight of the category with fewer samples is greater than the weight of the category with more samples. Weight adjustment factor for difficult samples The weight is determined by different iteration cycles in the training process. Different values ​​are given in the early, middle and late stages of training. As the number of training iterations increases, the weight adjustment factor for difficult samples continues to increase.

2. The detection method as described in claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Acquire in-vehicle video data using a camera fixed inside the vehicle, and convert the video data into MP4 format; Step 1.2: Slice the video data into multiple sub-videos of equal duration by using a time window of fixed length t; Step 1.3: Extract the PCM format audio stream from the sub-video to obtain audio data of equal duration, and then convert the PCM format audio stream to WAV format; Step 1.4: Perform end padding on audio data with a duration of less than t to ensure that the output audio data has the same time dimension.

3. The detection method as described in claim 1, characterized in that, In step 2, preprocessing refers to using pre-emphasis, framing, and windowing techniques to manipulate the speech signal in the audio data to generate a corresponding time-domain signal. The pre-emphasis is used to compensate for the attenuation of the high-frequency part in the audio data. The framing is used to obtain the short-time stationarity of the speech signal. The windowing is to multiply the framed speech signal with a window function so that the speech signal exhibits some characteristics of a periodic function.

4. The detection method as described in claim 3, characterized in that, The pre-emphasis uses a first-order high-pass filter; the framing uses an overlapping segmentation method; and the window function is a Hamming window.

5. The detection method as described in claim 1, characterized in that, In step 3, the feature extraction network module adopts the CNN14 network architecture. The backbone network of the CNN14 network architecture consists of 6 CNN blocks. The internal structure of each CNN block is a 2*(Conv+BN+ReLU)+avg pooling network layer. Each CNN block is connected to a Dropout layer. At the end of the network, by combining max pooling and avg pooling and the flatten operation of the fully connected layer, a one-dimensional feature vector after feature fusion is output. The binary subclassifier inputs the output values ​​of the neurons in the output branch into the sigmoid function to obtain the probability value of each sound category. The probability value ranges from (0,1). The probability value represents the confidence level of each output category. If the confidence level is greater than a preset threshold, the corresponding sound category is output; otherwise, no output is made.

6. The detection method as described in claim 1, characterized in that, The sound categories include: coughing, knocking on the vehicle body, speaking, exam voice broadcast, opening and closing the car door, inserting and removing the seat belt, gear shifting, handbrake and foot brake operation, and light operation. Among them, the voice prompt categories that are suspected of cheating are: coughing, knocking on the vehicle body, and speaking.

7. The detection method according to any one of claims 1 to 6, characterized in that, Also includes: Step 4: After determining that the sound category contains voice prompts that suggest cheating behavior, the corresponding audio data is recommended to the administrator for review to determine whether cheating behavior actually exists.

8. An electronic device, characterized in that, It includes a processor and a memory, wherein the processor is used to call a computer program stored in the memory to execute the deep learning-based voice prompt cheating detection method according to any one of claims 1 to 7.

Images