Drunk driving detection method and device based on audio data, equipment and medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a drunk driving detection method based on audio data, which comprises the following steps: acquiring at least one voice sequence in report audio data, and determining target audio data; extracting a voice feature vector of the target audio data by using a voice recognition module of the target drunk driving recognition model; recognizing a target key text of the target audio data according to the voice feature vector and a preset key text recognition method; performing alignment processing on the target key text and the voice feature vector by using a modal adapter of a target drunk driving recognition model to obtain a multi-modal feature vector; and analyzing the multi-modal feature vector by using a large language model of the target drunk driving identification model to obtain a label indicating whether the case reporting audio data is drunk driving or not. The invention further provides a drunk driving detection device and equipment based on the audio data and a medium. In addition, the invention also relates to a block chain technology, and case report audio data and the like can be stored in a block chain. The accuracy of drunk driving detection can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a drunk driving detection method based on audio data. Background Art

[0002] In the insurance field, drunk driving detection through audio is becoming increasingly important as a key link in maintaining public safety and protecting the interests of insurance companies. It can provide auxiliary judgment for drunk driving and prevent drunk driving users from taking special measures (such as drinking a lot of water or chewing gum) to try to cover up or reduce the alcohol content after reporting the case by phone.

[0003] At present, the mainstream detection method for drunk driving detection through report recordings is to analyze the report recordings through traditional NLP models, identify the keywords in the report recordings through the NLP model, and judge whether the user is drunk driving based on the keywords.

[0004] However, traditional NLP models can only process superficial text data and cannot process voice data, etc. The characteristics of voice data happen to be the key to drunk driving detection, resulting in low accuracy of drunk driving detection using report recordings in related technologies. Summary of the invention

[0005] The purpose of the embodiments of the present application is to propose a drunk driving detection method, device, equipment and medium based on audio data, the main purpose of which is to improve the accuracy of drunk driving detection.

[0006] In order to solve the above technical problems, the embodiment of the present application provides a drunk driving detection method based on audio data, which adopts the following technical solutions:

[0007] Acquire audio data of a case report, extract at least one voice sequence of the audio data of the case report, and determine target audio data according to the at least one voice sequence;

[0008] Extracting a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model;

[0009] Identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method;

[0010] Using the modality adapter of the target drunk driving recognition model, the target key text and the speech feature vector are aligned to obtain a multimodal feature vector;

[0011] The multimodal feature vector is analyzed using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving

[0012] Further, the extracting at least one voice sequence of the police report audio data and determining the target audio data according to the at least one voice sequence includes:

[0013] Extracting a speech sequence of at least one tone from the police report audio by a preset speech sequence recognition algorithm;

[0014] Calculating the timbre similarity between the speech sequence of the at least one timbre and the preset speech features in the speech feature library;

[0015] The target audio data is determined according to the timbre similarity.

[0016] Further, after determining the target audio data according to the at least one speech sequence, the method further includes:

[0017] Determine whether the duration of the target audio data is greater than a preset duration threshold;

[0018] If the duration of the target audio data is greater than the preset duration threshold, a frame operation is performed on the target audio data according to a predetermined frame length and frame shift to obtain a target framed audio data set.

[0019] Furthermore, before the speech recognition module of the target drunk driving recognition model is used to extract the speech feature vector of the target audio data, the method further includes:

[0020] Build an initial drunk driving recognition model using speech recognition modules, modality adapters, and large language models;

[0021] Acquire annotated speech training data, fix the parameters of the speech recognition module and the large language model of the initial drunk driving recognition model, and train the modality adapter of the initial drunk driving recognition model using the annotated speech training data to obtain an intermediate drunk driving recognition model;

[0022] The parameters of the speech recognition module of the intermediate drunk driving recognition model are fixed, and the modal adapter and the large language model of the intermediate drunk driving recognition model are trained using the annotated speech training data to obtain a target drunk driving recognition model.

[0023] Further, the step of identifying the target key text of the target audio data according to the speech feature vector and a preset key text recognition method includes:

[0024] Convert the target audio data into text to obtain target text data;

[0025] Extract key text from the target text data according to the change feature vector in the speech feature vector to obtain a first key text;

[0026] Extracting key text from the target text data using a preset key text recognition method to obtain a second key text;

[0027] The first key text and the second key text are collected to generate a target key text of the target audio data.

[0028] Furthermore, the modal adapter of the target drunk driving recognition model is used to align the target key text with the speech feature vector to obtain a multimodal feature vector, including:

[0029] Convert the target key text into a vector to obtain a target key text vector;

[0030] Calculating the alignment similarity between the speech feature vector and the target key text vector;

[0031] Performing an alignment operation on the speech feature vector and the target key text vector according to the alignment similarity;

[0032] The aligned speech feature vector is fused with the target key text to obtain a multimodal feature vector.

[0033] Furthermore, the large language model of the target drunk driving recognition model is used to analyze the multimodal feature vector to obtain a label of whether the reporting audio is drunk driving, including:

[0034] Analyze at least one of the multimodal feature vectors according to a predefined drunk driving key text library to obtain a drunk driving text score;

[0035] Performing speech feature analysis on at least one of the multimodal feature vectors to obtain a speech feature score;

[0036] Calculate the drunk driving text score and the voice feature score according to the predefined text vector weight and voice feature weight to obtain the drunk driving score corresponding to each multimodal feature vector;

[0037] Take all the drunk driving scores to get the average drunk driving score, and output the label of whether the reported audio data is drunk driving based on the average drunk driving score and the preset drunk driving score threshold

[0038] In order to solve the above technical problems, the embodiment of the present application also provides a vehicle drunk driving detection device based on audio data, which adopts the following technical solution:

[0039] An acquisition module, used to acquire the case reporting audio data, extract at least one voice sequence from the case reporting audio data, and determine the target audio data according to the at least one voice sequence;

[0040] An extraction module, used to extract a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model; and identify a target key text of the target audio data according to the speech feature vector and a preset key text recognition method;

[0041] An alignment module, used to align the target key text with the speech feature vector using the modality adapter of the target drunk driving recognition model to obtain a multimodal feature vector;

[0042] The detection module is used to analyze the multimodal feature vector using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a device, which adopts the following technical solution:

[0044] Acquire audio data of a case report, extract at least one voice sequence of the audio data of the case report, and determine target audio data according to the at least one voice sequence;

[0045] Extracting a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model;

[0046] Identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method;

[0047] Using the modality adapter of the target drunk driving recognition model, the target key text and the speech feature vector are aligned to obtain a multimodal feature vector;

[0048] The multimodal feature vector is analyzed using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

[0049] In order to solve the above technical problems, the embodiment of the present application further provides a medium, which adopts the following technical solution:

[0050] Acquire audio data of a case report, extract at least one voice sequence of the audio data of the case report, and determine target audio data according to the at least one voice sequence;

[0051] Extracting a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model;

[0052] Identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method;

[0053] Using the modality adapter of the target drunk driving recognition model, the target key text and the speech feature vector are aligned to obtain a multimodal feature vector;

[0054] The multimodal feature vector is analyzed using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

[0055] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0056] In this embodiment, by extracting at least one voice sequence from the reported audio data and determining the target audio data according to the at least one voice sequence, the calculation amount of the drunk driving test can be reduced, the efficiency of the drunk driving test can be improved, and the accuracy of the drunk driving test can be improved;

[0057] The speech feature vector of the target audio data is extracted by the speech recognition module of the target drunk driving recognition model, which overcomes the low accuracy of drunk driving detection caused by the traditional drunk driving detection method that can only be detected based on text, and improves the accuracy of drunk driving detection;

[0058] By combining the speech feature vector with a preset key text recognition method to identify the target key text of the target audio data, the key text can be more accurately recognized from the target audio data, overcoming the problem that the traditional model can only process the text, thereby improving the accuracy of subsequent drunk driving recognition;

[0059] By aligning the target key text with the speech feature vector through the modal adapter of the target drunk driving recognition model, a multimodal feature vector is obtained, which can obtain more comprehensive data and improve the accuracy of subsequent models for drunk driving detection, overcoming the problem of insufficient accuracy caused by traditional drunk driving methods that can only use text for analysis; on the other hand, the robustness of the model can also be improved;

[0060] The multimodal feature vector is analyzed through the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving. Multi-source information can be comprehensively analyzed to improve the accuracy of drunk driving detection and overcome the problem of low accuracy of drunk driving detection caused by traditional models only targeting single feature detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0062] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0063] Figure 2 A flowchart of an embodiment of a drunk driving detection method based on audio data according to the present application;

[0064] Figure 3 is a structural schematic diagram of an embodiment of a drunk driving detection device based on audio data according to the present application;

[0065] Figure 4 It is a structural schematic diagram of an embodiment of a device according to the present application. DETAILED DESCRIPTION

[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0069] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0070] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0071] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0072] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0073] It should be noted that the drunk driving detection method based on audio data provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the drunk driving detection device based on audio data is generally arranged in the server / terminal device.

[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of a drunk driving detection method based on audio data according to the present application. The drunk driving detection method based on audio data comprises the following steps:

[0076] S201. Acquire case reporting audio data, extract at least one voice sequence from the case reporting audio data, and determine target audio data according to the at least one voice sequence.

[0077] In this embodiment, the reporting audio data refers to the audio data recorded from the insurance claim call initiated by the reporter to the insurance company staff when an accident occurs while driving a vehicle, or the reporting call initiated when requesting the traffic police department to dispatch the police.

[0078] When the reporter and the corresponding relevant personnel (such as traffic police, surveyors, etc.) pass through, the reporting audio of the reporter and the corresponding relevant personnel are saved in a preset database, the reporting audio is obtained from the preset database, and the reporting audio is pre-processed. The pre-processing operation includes denoising, endpoint detection and pre-emphasis. The denoising process is used to remove noise in the reporting audio to reduce background noise. The endpoint detection process is used to check and determine the starting point and end point of the reporting audio, remove silent segments and non-voice segments, reduce the amount of calculation and improve the accuracy. The pre-emphasis process makes the signal spectrum of the reporting audio flatter, which is convenient for subsequent processing steps.

[0079] At least one voice sequence of the preprocessed report audio is extracted using a preset audio sequence algorithm, and the voice sequence belonging to the reporter is identified based on the voice sequence to determine the target audio.

[0080] In one embodiment, extracting at least one voice sequence from the case reporting audio data and determining target audio data according to the at least one voice sequence includes:

[0081] Extracting a speech sequence of at least one tone from the police report audio by a preset speech sequence recognition algorithm;

[0082] Calculating the timbre similarity between the speech sequence of the at least one timbre and the preset speech features in the speech feature library;

[0083] The target audio data is determined according to the timbre similarity.

[0084] In this embodiment, the report audio data must include the audio data of the reporter, and may include audio data of non-reporters (such as insurance surveyors, traffic police, etc.). Therefore, it is necessary to extract the target audio data belonging to the reporter from the report audio data to reduce the amount of calculation in subsequent steps.

[0085] In this embodiment, the preset voice features refer to the voice features of the reporter entered when the reporter took out insurance, and the voice features are stored in a voice feature library; since the report recording must contain the reporter's voice sequence, and there may also be voice sequences of interactions between non-reporters and reporters, after obtaining the report recording, a voice sequence of at least one timbre is extracted through a preset voice sequence recognition algorithm, and the voice sequence of each timbre is matched with each preset voice feature in the voice feature library for similarity, and the voice sequence with the highest timbre similarity is selected as the target audio data.

[0086] In this embodiment, the reporter's voice sequence is determined as the target sequence through similarity matching, and the target sequence is used as the basis for subsequent processing steps, which can improve the efficiency and accuracy of the drunk driving test.

[0087] In one embodiment, after determining the target audio data according to the at least one speech sequence, the method further comprises:

[0088] Determine whether the duration of the target audio data is greater than a preset duration threshold;

[0089] If the duration of the target audio data is greater than the preset duration threshold, a frame operation is performed on the target audio data according to a predetermined frame length and frame shift to obtain a target framed audio data set.

[0090] In this embodiment, since the characteristics of speech (such as syllables, etc.) remain relatively stable in a short period of time, it is determined whether the target audio data exceeds the preset time threshold according to a predetermined preset time threshold. When the target audio data exceeds the preset time threshold, the target audio data is framed according to a predetermined frame length and frame shift, and the serial numbers are marked to obtain multiple target framed audio data with serial numbers (framed speech segments). Multiple target framed audio data are collected to obtain a target framed audio data set. When the target audio data does not exceed the preset time threshold, the target audio data is not framed to ensure the continuity of the target audio data, so as to improve the subsequent continuity and thus improve the accuracy of subsequent drunk driving identification.

[0091] In this embodiment, according to the characteristics of the speech data, by determining whether the duration of the target audio data exceeds a preset duration threshold, the target audio data that exceeds the preset duration threshold is framed, which can improve the accuracy of feature extraction and thereby improve the accuracy of subsequent drunk driving identification.

[0092] In an implementable example A, Xiao Ming had an accident while driving an insured vehicle. Xiao Ming then called the traffic police and surveyor to report the accident through a handheld mobile terminal, saved the report audio of the report phone in a preset database, obtained the report audio from the preset database, extracted voice sequences of different tones in the report audio, obtained two voice sequences, matched the two voice sequences with each preset voice feature in the voice feature library respectively, determined Xiao Ming's voice sequence, used Xiao Ming's voice sequence as target audio data, determined whether the duration of the target audio data exceeds the duration threshold of 10 minutes, and when it exceeds the duration threshold of 10 minutes, the target audio data is framed to obtain multiple target framed audio data.

[0093] In this embodiment, by extracting at least one voice sequence from the report audio data and determining the target audio data based on the at least one voice sequence, the computational complexity of the drunk driving test can be reduced, the efficiency of the drunk driving test can be improved, and the accuracy of the drunk driving test can be improved.

[0094] S202. Extracting a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model.

[0095] In this embodiment, the target audio data is processed by a target drunk driving recognition model, and whether the driver is driving under the influence of alcohol is identified based on the target audio data; the target drunk driving recognition model is composed of a speech recognition module, a modal adapter and a large prediction model; the speech recognition module refers to a Whisper speech recognition module constructed and trained using a transformers architecture, and has the ability to extract speech feature vectors, transliterate sounds into words and recognize key texts; the large language model refers to an LLaMA model (Large Language Model Meta AI, alpaca large language model), which adopts a Transformer architecture and a neural network structure based on a self-attention mechanism. By stacking multiple layers of Transformers, LLaMA is capable of capturing complex relationships in the text and can analyze subsequent key text vectors in combination with preset weights.

[0096] In this embodiment, the speech feature vector refers to an emotional feature vector (such as nervousness, anxiety, anger, calmness, etc.), a speech rate feature vector (such as fast, smooth, slow, etc.) and a pitch feature vector (high, medium, low, etc.); the emotional feature vector, speech rate feature vector and pitch feature vector in the target audio data are extracted respectively through the Whisper speech recognition module.

[0097] Specifically, the short-time energy of the target audio data is identified by the speech recognition module of the target drunk driving recognition model, and the speaking rate feature vector is determined according to the short-time energy; the fundamental frequency value (F0 value) of each frame in the target audio data is extracted, and the fundamental frequency value is mapped to a two-dimensional coordinate system, the origin of the two-dimensional coordinate system is used as the starting point, and the fundamental frequency value of each frame is connected to obtain a intonation curve, and the intonation feature vector is extracted according to the intonation curve, and the emotional feature vector of the target audio data is determined based on the speaking rate feature vector and the intonation feature vector.

[0098] In one embodiment, before the speech recognition module of the target drunk driving recognition model is used to extract the speech feature vector of the target audio data, the method further includes:

[0099] Build an initial drunk driving recognition model using speech recognition modules, modality adapters, and large language models;

[0100] Acquire annotated speech training data, fix the parameters of the speech recognition module and the large language model of the initial drunk driving recognition model, and train the modality adapter of the initial drunk driving recognition model using the annotated speech training data to obtain an intermediate drunk driving recognition model;

[0101] The parameters of the speech recognition module of the intermediate drunk driving recognition model are fixed, and the modal adapter and the large language model of the intermediate drunk driving recognition model are trained using the annotated speech training data to obtain a target drunk driving recognition model.

[0102] In this embodiment, an initial drunk driving recognition model is constructed using a speech recognition module, a modal adapter and a large language model, and speech training data with emotion feature vectors, speech rate feature vectors, pitch feature vectors and keywords are obtained. The training process of the initial drunk driving recognition model is divided into two stages. In the first training stage, the parameters of the speech recognition module and the large language model are fixed, and the modal adapter in the initial drunk driving recognition model is trained separately through the annotated speech training data. After the first stage of training is completed, an intermediate drunk driving recognition model is generated, and the intermediate drunk driving recognition model is trained in the second stage. The parameters of the speech recognition module are fixed separately, and the parameters of the modal adapter and the large language model are updated through the annotated speech training data. The loss value of the model is calculated by a loss function calculation method. The loss function calculation method includes but is not limited to the Euclidean distance method, cosine similarity, etc. When the model loss value reaches a preset loss threshold, the second stage of training is completed to generate a target drunk driving recognition model.

[0103] In this embodiment, the training process of the drunk driving recognition model is divided into two stages. In the first stage of training, the parameters of the speech recognition module and the large language model are fixed, and the alignment of speech features and text embedding is focused on to improve the accuracy of the alignment of speech features and text embedding. In the second stage, the parameters of the speech recognition module are fixed, the modal adapter and the large language model are coordinated and optimized, and the capabilities of the two are better integrated, which can improve the accuracy of the drunk driving recognition model.

[0104] Continuing with the above-mentioned feasible example A, after acquiring the target audio data (including only Xiao Ming's speech sequence), the target audio data is input into the trained drunk driving recognition model, and the speech rate feature and pitch feature vector of the target audio data are extracted through the speech recognition module of the drunk driving recognition model, and the emotional feature vector of the target audio data is determined based on the speech rate feature vector and the pitch feature vector.

[0105] In this embodiment, the speech feature vector of the target audio data is extracted through a speech recognition module, which overcomes the low accuracy of drunk driving detection caused by the traditional drunk driving detection method that can only perform detection based on text, and improves the accuracy of drunk driving detection.

[0106] S203: Identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method.

[0107] In this embodiment, the target key text refers to key text that can reflect the central theme of the target audio data, including but not limited to keywords, key words and key phrases.

[0108] In this embodiment, the target audio data is converted into target text data through the speech recognition module of the target drunk driving recognition model, the first key text is determined from the target text data according to the speech feature vector, the second key text is determined from the target text data using a preset key text recognition method, the first key text and the second key text are collected to obtain the target key text of the target audio data.

[0109] Specifically, in one embodiment, identifying the target key text of the target audio data according to the speech feature vector and a preset key text recognition method includes:

[0110] Convert the target audio data into text to obtain target text data;

[0111] Extract key text from the target text data according to the change feature vector in the speech feature vector to obtain a first key text;

[0112] Extracting key text from the target text data using a preset key text recognition method to obtain a second key text;

[0113] The first key text and the second key text are collected to generate a target key text of the target audio data.

[0114] In this embodiment, the variable feature vector in the speech feature vector refers to part of the speech feature vector in the speech feature vector that exceeds the feature threshold corresponding to the feature, for example, part of the pitch feature vector that exceeds the preset pitch threshold, the emotional feature vector belonging to tension, uneasiness, etc., and part of the speech rate feature vector that exceeds the preset speech rate threshold. They are used as variable feature vectors, and the variable feature vector is identified from the target text data to determine the text corresponding to the variable feature vector to obtain the first key text; the preset key text recognition method includes but is not limited to extracting keywords (such as wine), key words (such as drinking, bar, etc.) or key phrases (coming out of the bar, etc.) sensitive text from the target text data based on the dictionary by a natural language processing (NLP) model, or a keyword extraction algorithm based on the frequency of occurrence (such as TF-IDF), etc.; using the preset key text recognition method to extract key text from the target text data to obtain a second key text, the first key text and the second key text are combined, and the overlapping key texts in the first key text and the second key text are deleted to generate the target key text.

[0115] In this embodiment, by extracting key text based on speech feature vectors and extracting key text based on a preset key text extraction method, it is possible to avoid keyword omissions, improve the accuracy of key text extraction, and thus improve the accuracy of drunk driving identification.

[0116] Continuing with the above-mentioned feasible example A, the target key text (including only Xiao Ming's speech sequence) is "I came out of XXX bar, followed the car in front for about 500 meters, and the other party suddenly braked and I hit it"; the first key text "the other party suddenly braked and I hit it" is determined according to the speech feature vector, and the second key text "XXX bar" is determined by the preset key text extraction algorithm, and "XXX bar" and "the other party suddenly braked and I hit it" are combined to generate the target key text.

[0117] In this embodiment, the target key text of the target audio data is identified by combining the speech feature vector with a preset key text recognition method, which can more accurately identify the key text from the target audio data, overcoming the problem that the traditional model can only process text, thereby improving the accuracy of subsequent drunk driving recognition.

[0118] S204: aligning the target key text with the speech feature vector using the modality adapter of the target drunk driving recognition model to obtain a multimodal feature vector.

[0119] In this embodiment, the target key text is converted into a target key text vector through a text vector conversion method (such as word2Vec technology), and the speech feature vector and the target key text vector are aligned through the modal adapter of the target drunk driving recognition model to obtain a multimodal feature vector including speech features and target key text vectors.

[0120] Specifically, in one embodiment, the modality adapter using the target drunk driving recognition model aligns the target key text with the speech feature vector to obtain a multimodal feature vector, including:

[0121] Convert the target key text into a vector to obtain a target key text vector;

[0122] Calculating the alignment similarity between the speech feature vector and the target key text vector;

[0123] Performing an alignment operation on the speech feature vector and the target key text vector according to the alignment similarity;

[0124] The aligned speech feature vector is fused with the target key text to obtain a multimodal feature vector.

[0125] In this embodiment, the target key text is converted into a target key text vector by a text vector conversion method (such as word2Vec technology), and the alignment similarity is obtained by directly calculating the similarity between the speech feature vector and the text feature vector, or respectively calculating the similarity between the emotion feature vector and the target key text vector, the similarity between the speech rate feature vector and the target key text vector, and the similarity between the pitch feature vector and the target key text vector. According to the alignment similarity, the speech feature vector and the target key text feature vector are fused or spliced, and the fusion or splicing operation includes but is not limited to a simple splicing method (Concatenation), a weighted sum method (Weighted Sum), a bilinear pooling method (Bilinear Pooling) etc., the simple concatenation method refers to performing head-to-tail concatenation of the speech feature vector and the target key text vector to obtain a long multimodal feature vector; the weighted summation method refers to performing weighted summation of the speech feature vector and the target key text vector according to a preset weight to obtain a multimodal feature vector; the bilinear pooling method refers to mapping the speech feature vector and the target key text vector into a high-dimensional space through an outer product operation, and then performing a dimensionality reduction operation on the high-dimensional vector to obtain a multimodal feature vector.

[0126] In this embodiment, by fusing or concatenating the target key text vector with the speech feature vector, the factors of the speech feature vector can be combined for analysis, thereby improving the accuracy of subsequent drunk driving detection.

[0127] Continuing with the above-mentioned feasible example A, the target key texts "XXX bar" and "The other party suddenly braked and I hit it" are converted into target key text vectors through Word2Vec technology, and the similarities between the target key text vector and the speech feature vector are calculated respectively. The highest similarity is selected as the alignment similarity, and the speech feature vector and the target key text vector are concatenated head to tail according to the alignment similarity to obtain a multimodal feature vector.

[0128] In this embodiment, the target key text is aligned with the speech feature vector through the modal adapter of the target drunk driving recognition model to obtain a multimodal feature vector, which can obtain more comprehensive data and improve the accuracy of subsequent models for drunk driving detection, thereby overcoming the problem of insufficient accuracy caused by traditional drunk driving methods that can only use text for analysis; on the other hand, it can also improve the robustness of the model.

[0129] S205. Analyze the multimodal feature vector using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

[0130] In this embodiment, the multimodal feature vector is analyzed using the LLaMA large language model, and the multimodal feature vector is analyzed through a predefined drunk driving key text library to identify the speech features of the multimodal feature vector, and the drunk driving score corresponding to the multimodal feature vector is calculated according to the predefined text vector weight and the speech feature weight, and a label of 0 or 1 is output according to the drunk driving score, where 0 represents no and 1 represents yes. When there are multiple multimodal feature vectors, the drunk driving score corresponding to each multimodal feature vector is calculated respectively, and the average of the multiple drunk driving scores is taken to obtain the drunk driving mean score, and a label of 0 or 1 is output using the drunk driving mean score.

[0131] In one embodiment, the large language model of the target drunk driving recognition model is used to analyze the multimodal feature vector to obtain a label of whether the reporting audio is drunk driving, including:

[0132] Analyze at least one of the multimodal feature vectors according to a predefined drunk driving key text library to obtain a drunk driving text score;

[0133] Performing speech feature analysis on at least one of the multimodal feature vectors to obtain a speech feature score;

[0134] Calculate the drunk driving text score and the voice feature score according to the predefined text vector weight and voice feature weight to obtain the drunk driving score corresponding to each multimodal feature vector;

[0135] All the drunk driving scores are taken to obtain the average drunk driving score, and a label of whether the reported audio data is drunk driving is output based on the average drunk driving score and a preset drunk driving score threshold.

[0136] In this embodiment, when there are multiple multimodal feature vectors, the drunk driving score of each multimodal feature vector is calculated, all the drunk driving scores are sorted, the highest drunk driving score is selected and compared with the preset drunk driving score threshold, and a label of whether the reported audio data is drunk driving is output.

[0137] In one embodiment, after obtaining the label of whether the reported audio data is drunk driving, the method further includes:

[0138] An analysis report is generated according to the analysis result of the large language model on the multimodal feature vector.

[0139] In this embodiment, after obtaining the label of whether the reported audio data is drunk driving, the reason corresponding to the output of the label is determined through the inverse process of the large language model of the target drunk driving recognition model, and an analysis report is generated according to the reason. For example, it is determined which key texts and corresponding voice features in the reported audio data are used to output the label corresponding to the reported audio data.

[0140] In this embodiment, by generating an analysis report, the transparency and credibility of the decision-making are improved, the accuracy of drunk driving detection is further improved, and it can also assist in decision-making and early warning.

[0141] Continuing with the above-mentioned feasible example A, after obtaining the multimodal feature vector obtained by aligning and splicing the key text vectors of "XXX bar" and "The other party suddenly braked and I hit it", and the corresponding speech feature vectors, the multimodal feature vector is input into the large language model, and "XXX bar" is identified and analyzed through the drunk driving key text library to calculate the drunk driving text score corresponding to "XXX bar"; the speech feature score of the speech feature vector corresponding to "XXX bar" is calculated; the multimodal feature vector corresponding to "XXX bar" is calculated by the weight corresponding to the drunk driving text score and the weight corresponding to the speech feature score. Similarly, calculate the drunk driving text score of the multimodal feature vector corresponding to "The other party suddenly braked and I hit him", take the mean or maximum value of the two drunk driving text scores, and compare it with the drunk driving score threshold. When the mean or maximum value of the two drunk driving text scores exceeds the drunk driving score threshold, output a 1 label, indicating that the report audio data detected drunk driving behavior. When the mean or maximum value of the two drunk driving text scores does not exceed the drunk driving score threshold, output a 0 label, indicating that the report audio data did not detect drunk driving behavior, and generate an analysis report based on the above detection process.

[0142] In this embodiment, the multimodal feature vector is analyzed by the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving. Multi-source information can be comprehensively analyzed to improve the accuracy of drunk driving detection, overcoming the problem of low accuracy of drunk driving detection caused by traditional models only targeting single feature detection.

[0143] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned reporting audio data and user information, the above-mentioned reporting audio data and user information can also be stored in a blockchain node.

[0144] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0145] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0146] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0147] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0148] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0149] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a drunk driving detection device based on audio data. Figure 2 Corresponding to the method embodiment shown, the apparatus can be specifically applied to various computer devices.

[0150] like Figure 3As shown, the drunk driving detection device 300 based on audio data described in this embodiment includes: an acquisition module 301, an extraction module 302, an alignment module 303, and a detection module 304.

[0151] in:

[0152] An acquisition module 301 is used to acquire the case reporting audio data, extract at least one voice sequence from the case reporting audio data, and determine the target audio data according to the at least one voice sequence;

[0153] The extraction module 302 is used to extract the speech feature vector of the target audio data by using the speech recognition module of the target drunk driving recognition model; and identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method;

[0154] An alignment module 303 is used to align the target key text with the speech feature vector using the modality adapter of the target drunk driving recognition model to obtain a multimodal feature vector;

[0155] The detection module 304 is used to analyze the multimodal feature vector using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

[0156] In this embodiment, by extracting at least one voice sequence from the reported audio data and determining the target audio data according to the at least one voice sequence, the calculation amount of the drunk driving test can be reduced, the efficiency of the drunk driving test can be improved, and the accuracy of the drunk driving test can be improved;

[0157] The speech feature vector of the target audio data is extracted by the speech recognition module of the target drunk driving recognition model, which overcomes the low accuracy of drunk driving detection caused by the traditional drunk driving detection method that can only be detected based on text, and improves the accuracy of drunk driving detection;

[0158] By combining the speech feature vector with a preset key text recognition method to identify the target key text of the target audio data, the key text can be more accurately recognized from the target audio data, overcoming the problem that the traditional model can only process the text, thereby improving the accuracy of subsequent drunk driving recognition;

[0159] By aligning the target key text with the speech feature vector through the modal adapter of the target drunk driving recognition model, a multimodal feature vector is obtained, which can obtain more comprehensive data and improve the accuracy of subsequent models for drunk driving detection, overcoming the problem of insufficient accuracy caused by traditional drunk driving methods that can only use text for analysis; on the other hand, the robustness of the model can also be improved;

[0160] The multimodal feature vector is analyzed through the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving. Multi-source information can be comprehensively analyzed to improve the accuracy of drunk driving detection and overcome the problem of low accuracy of drunk driving detection caused by traditional models only targeting single feature detection.

[0161] In order to solve the above technical problems, the present application embodiment also provides a device (computer device). Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0162] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 with a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0163] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0164] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of a drunk driving detection method based on audio data, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0165] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the method for reviewing the vehicle insurance document image.

[0166] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0167] During the implementation of the electronic device of the present application, by extracting at least one voice sequence of the reported audio data and determining the target audio data according to the at least one voice sequence, the calculation amount of the drunk driving test can be reduced, the efficiency of the drunk driving test can be improved, and the accuracy of the drunk driving test can be improved;

[0168] The speech feature vector of the target audio data is extracted by the speech recognition module of the target drunk driving recognition model, which overcomes the low accuracy of drunk driving detection caused by the traditional drunk driving detection method that can only be detected based on text, and improves the accuracy of drunk driving detection;

[0169] By combining the speech feature vector with a preset key text recognition method to identify the target key text of the target audio data, the key text can be more accurately recognized from the target audio data, overcoming the problem that the traditional model can only process the text, thereby improving the accuracy of subsequent drunk driving recognition;

[0170] By aligning the target key text with the speech feature vector through the modal adapter of the target drunk driving recognition model, a multimodal feature vector is obtained, which can obtain more comprehensive data and improve the accuracy of subsequent models for drunk driving detection, overcoming the problem of insufficient accuracy caused by traditional drunk driving methods that can only use text for analysis; on the other hand, the robustness of the model can also be improved;

[0171] The multimodal feature vector is analyzed through the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving. Multi-source information can be comprehensively analyzed to improve the accuracy of drunk driving detection and overcome the problem of low accuracy of drunk driving detection caused by traditional models only targeting single feature detection.

[0172] The present application also provides another embodiment, namely, providing a storage medium (computer-readable storage medium), wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned drunk driving detection method based on audio data.

[0173] During implementation, the computer-readable storage medium of the present application can reduce the amount of calculation of the drunk driving test, improve the efficiency of the drunk driving test, and improve the accuracy of the drunk driving test by extracting at least one voice sequence of the reported audio data and determining the target audio data according to the at least one voice sequence;

[0174] The speech feature vector of the target audio data is extracted by the speech recognition module of the target drunk driving recognition model, which overcomes the low accuracy of drunk driving detection caused by the traditional drunk driving detection method that can only be detected based on text, and improves the accuracy of drunk driving detection;

[0175] By combining the speech feature vector with a preset key text recognition method to identify the target key text of the target audio data, the key text can be more accurately recognized from the target audio data, overcoming the problem that the traditional model can only process the text, thereby improving the accuracy of subsequent drunk driving recognition;

[0176] By aligning the target key text with the speech feature vector through the modal adapter of the target drunk driving recognition model, a multimodal feature vector is obtained, which can obtain more comprehensive data and improve the accuracy of subsequent models for drunk driving detection, overcoming the problem of insufficient accuracy caused by traditional drunk driving methods that can only use text for analysis; on the other hand, the robustness of the model can also be improved;

[0177] The multimodal feature vector is analyzed through the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving. Multi-source information can be comprehensively analyzed to improve the accuracy of drunk driving detection and overcome the problem of low accuracy of drunk driving detection caused by traditional models only targeting single feature detection.

[0178] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0179] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0180] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A drunk driving detection method based on audio data, characterized in that: The steps include: Acquire audio data of a case report, extract at least one voice sequence of the audio data of the case report, and determine target audio data according to the at least one voice sequence; Extracting a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model; Identify the target key text of the target audio data according to the speech feature vector and a preset key text recognition method; Using the modality adapter of the target drunk driving recognition model, the target key text and the speech feature vector are aligned to obtain a multimodal feature vector; The multimodal feature vector is analyzed using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

2. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: The step of extracting at least one voice sequence from the case reporting audio data and determining target audio data according to the at least one voice sequence comprises: Extracting a speech sequence of at least one tone from the police report audio by a preset speech sequence recognition algorithm; Calculating the timbre similarity between the speech sequence of the at least one timbre and the preset speech features in the speech feature library; The target audio data is determined according to the timbre similarity.

3. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: After determining the target audio data according to the at least one speech sequence, the method further comprises: Determine whether the duration of the target audio data is greater than a preset duration threshold; If the duration of the target audio data is greater than the preset duration threshold, a frame operation is performed on the target audio data according to a predetermined frame length and frame shift to obtain a target framed audio data set.

4. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: Before extracting the speech feature vector of the target audio data using the speech recognition module of the target drunk driving recognition model, the method further includes: Build an initial drunk driving recognition model using speech recognition modules, modality adapters, and large language models; Acquire annotated speech training data, fix the parameters of the speech recognition module and the large language model of the initial drunk driving recognition model, and train the modality adapter of the initial drunk driving recognition model using the annotated speech training data to obtain an intermediate drunk driving recognition model; The parameters of the speech recognition module of the intermediate drunk driving recognition model are fixed, and the modal adapter and the large language model of the intermediate drunk driving recognition model are trained using the annotated speech training data to obtain a target drunk driving recognition model.

5. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: The step of identifying the target key text of the target audio data according to the speech feature vector and a preset key text recognition method includes: Convert the target audio data into text to obtain target text data; Extract key text from the target text data according to the change feature vector in the speech feature vector to obtain a first key text; Extracting key text from the target text data using a preset key text recognition method to obtain a second key text; The first key text and the second key text are collected to generate a target key text of the target audio data.

6. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: The modal adapter using the target drunk driving recognition model aligns the target key text with the speech feature vector to obtain a multimodal feature vector, including: Convert the target key text into a vector to obtain a target key text vector; Calculating the alignment similarity between the speech feature vector and the target key text vector; Performing an alignment operation on the speech feature vector and the target key text vector according to the alignment similarity; The aligned speech feature vector is fused with the target key text to obtain a multimodal feature vector.

7. The drunk driving detection method based on audio data as claimed in claim 1, characterized in that: The method of analyzing the multimodal feature vector using the large language model of the target drunk driving recognition model to obtain a label of whether the reporting audio is drunk driving includes: Analyze at least one of the multimodal feature vectors according to a predefined drunk driving key text library to obtain a drunk driving text score; Performing speech feature analysis on at least one of the multimodal feature vectors to obtain a speech feature score; Calculate the drunk driving text score and the voice feature score according to the predefined text vector weight and voice feature weight to obtain the drunk driving score corresponding to each multimodal feature vector; All the drunk driving scores are taken to obtain the average drunk driving score, and a label of whether the reported audio data is drunk driving is output based on the average drunk driving score and a preset drunk driving score threshold.

8. A drunk driving detection device based on audio data, characterized in that: The device comprises: An acquisition module, used to acquire the case reporting audio data, extract at least one voice sequence from the case reporting audio data, and determine the target audio data according to the at least one voice sequence; An extraction module, used to extract a speech feature vector of the target audio data using a speech recognition module of a target drunk driving recognition model; and identify a target key text of the target audio data according to the speech feature vector and a preset key text recognition method; An alignment module, used to align the target key text with the speech feature vector using the modality adapter of the target drunk driving recognition model to obtain a multimodal feature vector; The detection module is used to analyze the multimodal feature vector using the large language model of the target drunk driving recognition model to obtain a label of whether the reported audio data is drunk driving.

9. A computer device, characterized in that: The computer device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the drunk driving detection method based on audio data as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the drunk driving detection method based on audio data as described in any one of claims 1 to 7 is implemented.

Images