An event processing method and system based on a large language model

Abstract

This application relates to the field of natural language processing technology, and in particular to an event processing method and system based on a large language model, which addresses the problems of limited processing efficiency and handling effectiveness in related event processing systems. The method involves: real-time recognition and offline correction of received audio dialogue between the caller and the operator, followed by fusion to obtain the dialogue text; inputting the dialogue text into a large language model for multimodal information extraction to obtain key information about the event described in the audio dialogue; identifying the risk level of the event based on this key information, multiple recognition rules, and historical event knowledge, with each recognition rule corresponding to a risk level; determining a handling plan based on the risk level and sending the plan to the operator's terminal device and a processing terminal associated with the geographical location of the event; effectively improving recognition accuracy, processing efficiency, and handling effectiveness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to an event processing method and system based on a large language model. BACKGROUND

[0002] With the increase of the use of automobiles, the traffic flow in the road is gradually increasing, and the number of incoming calls about traffic accidents, traffic consultation, traffic complaints and the like also tends to increase, which increases the processing pressure of the event processing system.

[0003] In the related art, the traditional event processing system usually uses a single speech recognition module to perform speech recognition on the dialogue between the incoming caller and the operator, obtains keywords, and determines the risk level based on keyword matching rules to help the operator identify risk events. However, due to the sudden emergency of incoming calls, noisy environment, chaotic expression of the incoming caller, and use of dialect by the incoming caller, the traditional event processing system cannot accurately identify the key information about the risk level in the dialogue speech, cannot quickly and accurately identify the risk event, delays the key response time, and reduces the processing efficiency and disposal effect. There is an urgent need for a better event processing method for quickly and accurately identifying the risk level. SUMMARY

[0004] The embodiments of the present application provide an event processing method and system based on a large language model to improve the speech recognition accuracy, mine deep semantics, improve the sensitive event recognition accuracy, and achieve fast response.

[0005] The event processing method based on a large language model provided by the embodiments of the present application includes: receiving dialogue audio between an incoming caller and an operator; performing real-time recognition and offline correction on the dialogue audio, respectively, and fusing real-time text recognized in real time and offline text after offline correction to obtain dialogue text, the offline correction including text correction and dialect fine-tuning; inputting the dialogue text into a large language model to extract multi-modal information to obtain key information of an event described by the dialogue audio; identifying a risk level of the event based on the key information, a plurality of recognition rules and historical event knowledge, to obtain the risk level of the event, one recognition rule corresponding to one risk level; determining a disposal plan based on the risk level, and sending the disposal plan to a terminal device of the operator and a processing terminal associated with a geographical location of the event.

[0006] The above method fuses real-time recognition and offline correction of text, meets the rigid demand of millisecond-level response, and can improve the final recognition accuracy of the system in complex environments such as noise and accent through offline correction. At the same time, through the large language model driven information extraction, the context semantics of the dialogue audio can be deeply understood, and the key information in the unstructured dialogue can be accurately extracted. In the treatment decision, based on the key information, multiple recognition rules and historical event knowledge, the risk level of the event is identified, which can not only quickly match the known sensitive event characteristics, but also intelligently identify potential and implicit sensitive events through semantic analysis and context perception, realize the recognition accuracy, and greatly shorten the key response time. BRIEF DESCRIPTION OF DRAWINGS

[0007] Figure 1 An application scenario diagram of an event processing method based on a large language model in an embodiment of the present application;

[0008] Figure 2 A software architecture diagram of an event processing system based on a large language model in an embodiment of the present application;

[0009] Figure 3 A flowchart of an event processing method based on a large language model in an embodiment of the present application;

[0010] Figure 4 A processing layer architecture diagram in an embodiment of the present application;

[0011] Figure 5 A flowchart of fusing real-time recognition and offline correction of text in an embodiment of the present application;

[0012] Figure 6 A text correction flowchart in an embodiment of the present application;

[0013] Figure 7 A flowchart of determining a target word element and a score of each alternative word in an embodiment of the present application;

[0014] Figure 8 An application layer architecture diagram in an embodiment of the present application;

[0015] Figure 9 A flowchart of determining key information in an embodiment of the present application;

[0016] Figure 10 A flowchart of determining a risk level in an embodiment of the present application;

[0017] Figure 11 A flowchart of determining a rule score in an embodiment of the present application;

[0018] Figure 12 A schematic diagram of a risk level determination process in an embodiment of the present application;

[0019] Figure 13 A specific architecture schematic diagram of an event processing system based on a large language model in an embodiment of the present application;

[0020] Figure 14 A schematic diagram of an interaction process in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application. It should be noted that the terms “first”, “second”, “third” and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0022] In the related art, the traditional event processing system faces a series of technical bottlenecks in the level of intelligence, which seriously restricts the processing efficiency and disposal effect. Specifically, the related art has the following technical problems to be solved urgently: the traditional voice recognition system adopts a single recognition model, which makes it less adaptable and less accurate in complex scenarios; it lacks intelligent decision support; sensitive event recognition lags behind; the overall system architecture is rigid and cannot meet the dual needs of real-time processing and high-precision analysis.

[0023] Based on this, the embodiments of the present application provide an event processing method based on a large language model, which comprises the following steps: performing real-time recognition and offline correction on the dialogue audio between the caller and the operator received, and fusing the real-time text recognized in real time and the offline text corrected offline to obtain dialogue text, wherein the offline correction comprises text correction and dialect fine-tuning; inputting the dialogue text into a large language model to extract multi-modal information to obtain key information of the event described by the dialogue audio; identifying the risk level of the event based on the key information, a plurality of recognition rules and historical event knowledge, to obtain the risk level of the event, wherein one recognition rule corresponds to one risk level; determining a disposal plan based on the risk level, and sending the disposal plan to the terminal device of the operator and the processing terminal associated with the geographical location of the event.

[0024] In this application, by fusing real-time recognition and offline correction text, both the rigid demand of millisecond-level response and the final recognition accuracy in complex environments such as noisy and accent are improved through offline correction, effectively solving the performance bottleneck problem of a single model. At the same time, through the large language model driven extraction of key information, it realizes the qualitative change from "keyword matching" to "semantic understanding", can deeply understand the context semantics of the dialogue audio, accurately extract the key information in the unstructured dialogue, and provide solid and comprehensive data support for subsequent disposal decision. In disposal decision, based on key information, multiple identification rules and historical event knowledge, the risk level of the event is identified, realizing the leap from "passive response" to "active early warning", not only can quickly match known sensitive event features, but also can intelligently identify potential and implicit major events through semantic analysis and context perception, realize the identification accuracy, greatly shorten the key response time, and improve the processing efficiency and disposal effect.

[0025] The above method provided by the embodiments of the present application can be applied to a server in which a large language model-based event processing system is deployed. Referring to Figure 1 , the application scenario includes a mobile terminal 10 of a caller, a terminal device 20 of an operator, and a server 30. When the caller encounters a traffic accident, a traffic conflict, etc., the caller can dial a hotline to obtain help, etc. The operator side has a fixed phone, and a conversation between the two is established by connecting the caller's incoming call. The terminal device 20 sends the conversation audio between the caller and the operator to the server 30. After receiving the conversation audio between the caller and the operator, the server 30 triggers the execution of the above method in the embodiments of the present application, performs speech processing and multi-modal information extraction on the conversation audio, obtains the key information of the event described by the conversation audio, determines the risk level of the event based on the key information, determines a disposal plan, and sends the disposal plan to the terminal device 20 and a processing terminal associated with the geographical location of the event.

[0026] For example, the above server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system, providing cloud servers of basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0027] Of course, the method provided by the embodiments of the present application is not limited to Figure 1 The application scenario shown in the above embodiment can also be used in other possible application scenarios, which are not limited in the embodiments of the present application.

[0028] Referring to Figure 2The event processing system based on the large language model can include a voice access layer, a processing layer, an application layer, and a disposal layer. The voice access layer is configured to receive conversation audio between a caller and an operator. The processing layer is configured to perform real-time recognition and offline correction on the conversation audio respectively, and fuse real-time text recognized in real time and offline text after offline correction to obtain conversation text. The offline correction includes text correction and dialect fine-tuning. The application layer is configured to input the conversation text into a large language model to extract multi-modal information to obtain key information of an event described by the conversation audio. Based on the key information and a plurality of recognition rules, a risk level of the event is recognized to obtain the risk level of the event. One recognition rule corresponds to one risk level. A disposal plan is determined based on the risk level. The disposal layer is configured to send the disposal plan to a terminal device of the operator for display and to a processing terminal associated with a geographical location of the event.

[0029] After introducing the application scenarios of the embodiments of the present application, the preferred embodiments of the present application will be further described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application, and the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0030] Referring to Figure 3 The embodiments of the present application provide an event processing method based on a large language model. The specific process of the method is as follows:

[0031] Step 300: receiving conversation audio between a caller and an operator.

[0032] Still referring to Figure 1 In the embodiments of the present application, when step 300 is performed, the aforementioned event processing system based on a large language model (hereinafter referred to as system) can receive conversation audio between a caller and an operator through a voice access layer to trigger the execution of subsequent steps 310-340.

[0033] Step 310: performing real-time recognition and offline correction on the conversation audio respectively, and fusing real-time text recognized in real time and offline text after offline correction to obtain conversation text, wherein the offline correction includes text correction and dialect fine-tuning.

[0034] Referring to Figure 4 The application layer in the embodiments of the present application adopts a dual-model collaborative speech recognition architecture, that is, through the collaborative design of "real-time model + offline deep model", the response of milliseconds and the high demand for recognition accuracy are met. Figure 4In the implementation, the dialogue audio input is preprocessed by a dialogue audio input preprocessing module, the preprocessing module inputs the dialogue audio into a real-time recognition model and an offline processing queue respectively, the real-time recognition model performs real-time recognition on the input dialogue audio to obtain real-time text, and the offline processing queue inputs the dialogue audio into an offline deep model, and the offline deep model performs offline correction on the input dialogue audio to obtain offline text. Then, the real-time text and the offline text are fused by a text fusion algorithm to obtain the dialogue text of the dialogue audio, wherein the offline deep model has functions of text correction, speaker separation, and dialect fine-tuning.

[0035] In the implementation of step 310, referring to Figure 5 , the foregoing system can specifically perform the following process:

[0036] Step 3101: Real-time speech recognition is performed on the dialogue audio to obtain real-time text.

[0037] In the embodiment of the application, still referring to Figure 4 , when step 3101 is performed, the foregoing system can perform real-time speech recognition on the dialogue audio by using a real-time recognition model to obtain the real-time text. The real-time recognition model can be constructed by using a Transformer-based lightweight architecture to meet the requirement of millisecond-level response.

[0038] In the implementation, the foregoing system can send the real-time text recognized by the real-time recognition model to the terminal device of the operator to realize display of the real-time text.

[0039] Step 3102: Offline speech recognition is performed on the dialogue audio to obtain initial text.

[0040] In the embodiment of the application, still referring to Figure 4 , when step 3102 is performed, the foregoing system can perform offline speech recognition on the dialogue audio by using the speech recognition function of the offline deep model to obtain the initial text.

[0041] In the implementation, the offline deep model determines the voiceprint feature of the dialogue audio in parallel during the offline speech recognition process, compares the voiceprint feature with the voiceprint feature corresponding to the pre-set dialect model, fine-tunes the voiceprint feature based on the dialect model matched with the voiceprint feature after determining that the voiceprint feature of the dialogue audio matches the voiceprint feature corresponding to any dialect model, and then performs offline speech recognition on the fine-tuned voiceprint feature to obtain the initial text.

[0042] In some possible implementations, the foregoing system can be pre-configured with common dialect models. In the implementation, the following formula can be used to model to determine the dialect model matched with the voiceprint feature of the dialogue audio:

[0043]

[0044] wherein, represents audio (voiceprint) / text feature estimation, x represents audio (voiceprint) feature, d represents reference text signal; D represents reference text signal corresponding to a plurality of pre-configured dialect models; represents a function of taking the maximum value; represents the identification of the dialect model matched with the voiceprint feature of the dialogue audio.

[0045] After determining the dialect model matched with the voiceprint feature of the dialogue audio, the dialect feature fine-tuning is realized by the following formula:

[0046]

[0047] wherein, Merge() represents parameter injection / merging (such as Adapter injection or LoRA combination), represents the base model parameter on which the merging is performed represents the light adaptation parameter of the dialect model . represents the parameter of the merged model.

[0048] The above modeling manner can be obtained by pre-training, and the loss function during training can be represented by the following formula:

[0049]

[0050] wherein, represents the loss function of model training; represents an automatic speech recognition (ASR) constraint term, such as connectionist temporal classification (CTC) (negative log likelihood) or seq2seq (cross entropy); represents a dictionary / hotword constraint term, which makes the model strengthen the dictionary / hotword hit, which can be designed as a negative log probability penalty on the dictionary path, or a margin constraint after the score of the hotword is improved; represents the weight.

[0051] In the embodiments of the present application, an adapter is preloaded for a common dialect to reduce switching delay; meanwhile, when is insufficient or conflicts, the base model parameter is used as a fallback.

[0052] In this embodiment of the application, the aforementioned system supports dialect pre-selection. For example, through the settings interface, users can freely select the type of dialect to be used during reasoning. For example, for traffic in XX province, the dialect of a certain place in XX province can be selected. In this way, during reasoning, the aforementioned system can load the corresponding adaptation parameters and dictionary / hot word list under the selected dialect, and fine-tune the decoding search space to improve the adaptability of dialect vocabulary and pronunciation.

[0053] Step 3103: Perform audio analysis on the dialogue audio and mark the different speaker identifiers and timestamps in the dialogue audio.

[0054] In this embodiment of the application, when performing step 3103, the aforementioned system can use the speaker separation function of the offline deep model to mark the different speaker identifiers and timestamps in the dialogue audio.

[0055] Optionally, when performing step 3103, the aforementioned system may perform speech activity detection on the dialogue audio and segment the dialogue audio into multiple segments based on the detection results; then, based on the acoustic features corresponding to the multiple segments respectively, the multiple segments are clustered, and different speaker identifiers and timestamps are marked in the dialogue audio based on the clustering results, wherein the segment length of the clustered segments is not less than the segment length threshold and the classification confidence is not less than the classification confidence threshold.

[0056] In this embodiment of the application, the above clustering method can be modeled and implemented using the following formula:

[0057]

[0058]

[0059]

[0060]

[0061]

[0062] in, A segment or frame representing a time step t ( The speaker embedding vector extracted; Indicated and The cosine similarity matrix is ​​used for clustering or connectivity measurement, where i and j are the indices; In this embodiment, the optimal number of clusters is represented by... The quality of different K values ​​is evaluated using either the silhouette coefficient or BIC (Bayesian Information Criterion) / AIC (Akaike Information Criterion), and the optimal number of clusters is selected; the overall objective is to minimize the DER (Speaker Log Error Rate). The cluster centroid is represented by , used for subsequent alignment and redistribution; k represents the cluster index. This represents the set of sample indices for the k-th cluster, i.e., the set of all time points / segments assigned to cluster k in the current cluster. Indicates the number of its elements; This represents the value among all permutations π that makes the current centroid... With the center of history The optimal permutation with the minimum total distance is used to maintain the stability of speaker identification (ID); π represents a permutation mapping of a cluster label, which maps the current segment's cluster index set {1,…,K} to the historical global cluster index, and is used for cross-segment ID alignment; This represents the classification confidence score of the classification model. This represents the logits (raw predicted scores for each class) of the classification head U of the classification model. This indicates the target cluster index selected for this segment when a reassignment is triggered by low confidence or a short segment; This represents the average embedding vector within the segment; Represents the weights that control neighborhood consistency; This indicates the vote count for the cluster from neighboring (front and back windows) labels; This indicates a non-voice false positive. This indicates a missed voice message. This indicates that the speaker is confused. This indicates the total duration of the reference speech, such as the total duration of the dialogue audio.

[0063] against (Profile coefficient), for each sample point i', its profile coefficient s(i) is defined as follows:

[0064]

[0065] in, Indicates sample Average distance to other samples in the same cluster (cohesion); This represents the average distance (separation) from sample i' to all samples in the nearest other clusters. The silhouette coefficient S ranges from -1 to 1, with values ​​closer to 1 indicating better clustering performance.

[0066] Regarding BIC (Bayesian Information Criterion) / AIC (Akaike Information Criterion), in this embodiment, BIC / AIC, along with an assumed model such as a Gaussian Mixture Model (GMM), are used to evaluate the model's fit according to the following definitions of BIC and AIC, in order to select the optimal model:

[0067] BIC=-2lnL+plnn,AIC=-2lnL+2p

[0068] Where L represents the maximum likelihood value; p represents the number of model parameters; n represents the number of samples. Generally, the smaller the BIC / AIC value, the better the model.

[0069] In the implementation, the centroid of the initial label is calculated , and the Hungarian algorithm is used to solve the optimal matching to align the speaker identification (ID) on adjacent batches / segments and maintain ID stability. When the segment length is lower than the segment length threshold τ or the classification confidence is lower than the classification confidence threshold , the label sequence is re-allocated according to the formula , and a median filter with a window W is applied to the sub-label sequence to reduce jitter; the DER calculation includes a collar (boundary buffer) configuration to ignore time errors within δ seconds near the reference boundary, avoiding small jitter in boundary alignment from being counted as errors; a commonly used δ can be 0.25s or 0.5s.

[0070] For example, assuming there are 6 speech segments (segments), after unit norm embedding, the following is obtained:

[0071] e1=[0.80, 0.10,0.10],e2=[0.72,0.20,0.10],e3=[0.78,0.12,0.08]

[0072] e4=[-0.10,0.78,0.12],e5=[-0.18,0.70,0.20],e6=[-0.12,0.74,0.16]

[0073] Similarity examples are as follows: cos(e1,e2)≈0.97, cos(e1,e4)≈-0.01, showing that the two clusters are clearly separated.

[0074] Select the number of clusters: take spectral clustering or K-means with K∈{1,2,3} as an example, when K=2, the clusters obtained are: S1={e1,e2,e3}, S2={e4,e5,e6}. Centroid: μ1=(e1+e2+e3) / 3≈[0.7667,0.14,0.0933], μ2≈[-0.1333,0.74,0.16].

[0075] Simple silhouette: for e1, a(e1) is its distance to the cluster mean, b(e1) is its distance to the other cluster mean, if a ~ 0.03, b ~ 1.90 (approximated by angular distance), s(e1) ~ (b-a) / max ~ (1.87) / 1.90 ~ 0.984, overall mean is close to 0.9+, significantly better than K=1. BIC vs. AIC: if using GMM, assume log-likelihood lnL K increases with K, but p increases, BIC K = -2lnL K + p K lnn is minimized at K=2.

[0076] Cross-segment ID alignment (e.g., Hungarian algorithm): historical centroids: μ hist1 = [0.77, 0.13, 0.10], μ hist2 = [-0.14, 0.73, 0.17].

[0077] Cost matrix C[k,j] = ||μ k - μ hist_j ||2:

[0078] C[1,1] ~ ||[0.7667, 0.14, 0.0933] - [0.77, 0.13, 0.10]|| ~ 0.012

[0079] C[1,2] ~ ||[0.7667, 0.14, 0.0933] - [-0.14, 0.73, 0.17]|| ~ 1.06

[0080] C[2,1] ~ ||[-0.1333, 0.74, 0.16] - [0.77, 0.13, 0.10]|| ~ 1.09

[0081] C[2,2] ~ ||[-0.1333, 0.74, 0.16] - [-0.14, 0.73, 0.17]|| ~ 0.017

[0082] Optimal matching is μ1→hist1, μ2→hist2, with minimum total cost (~ 0.012 + 0.017 = 0.029).

[0083] Low-confidence reassignment and smoothing: average embedding = e2 for a short segment; classification head U of classification model produces logits z = [2.2, 0.3], softmax(z) = [0.90, 0.10], q t = 0.90 ≥ ρ(0.60), keep original cluster; another short segment logits z = [0.2, 0.0], softmax = [0.55, 0.45], qt = 0.55 < p, trigger redistribution: cos( , m1) ~ 0.99, cos( , m2) ~ 0.12;

[0084] Neighborhood voting: votes1 = 4, votes2 = 1, take l' = 0.1; scores: score1 = 0.99 + 0.1 4 = 1.39, score2 = 0.12 + 0.1 1 = 0.22, then, select k = 1.

[0085] Apply median filter with window W = 5 to the final label sequence to remove isolated short-time jumps.

[0086] DER (with collar): assume total reference speech duration T REF = 300s, false alarm T FA = 9s, miss T MISS = 12s, confusion T ERR = 15s, then, DER = (9 + 12 + 15) / 300 = 36 / 300 = 0.12 = 12%. With a collar of delta = 0.25s, the errors near the boundary are exempted, if 3s are exempted, the total error becomes 33s, DER = 33 / 300 = 11%. In the embodiments of the present application, the collar and the overlapping speaker scoring rule (whether to allow multiple speaker labels) are explicitly considered in DER evaluation.

[0087] In step 3104, text correction is performed on the initial text based on language prior knowledge to obtain an offline text, wherein the language prior knowledge includes at least one of synonym prior knowledge, easily confused word pair prior knowledge, or domain terminology prior knowledge.

[0088] In the embodiments of the present application, the aforementioned system is pre-equipped with language prior knowledge, such as a synonym replacement set (which can be constructed based on a synonym dictionary, a term library), an easily confused pair set (such as an error / confusion set, including homophones / near-homophones (common in ASR), similar characters, spelling / keyboard common errors, model confusion pairs, etc.), domain terminology (hot words / dictionary, including domain hot words and professional terms, such as "traffic", "congestion", "accident" in the traffic domain, containing high-value entities and collocations), etc.

[0089] In specific implementations, the aforementioned system is implemented through offline batch processing, and the text correction includes candidate set construction, weighted scoring, and constraint optimization; optionally, when step 3104 is performed, refer to Figure 6 , the offline deep model can specifically perform the following steps:

[0090] Step 31041: determining a candidate word set of the initial text based on language prior knowledge, wherein any candidate word in the candidate word set is a synonym, an ambiguous word or a domain hot word of a word unit in the initial text.

[0091] In the embodiments of the present application, when step 31041 is performed, based on language prior knowledge, the words related to the word units in the initial text in the language prior knowledge are taken as candidate words to construct a candidate word set, denoted as C:

[0092]

[0093] wherein t represents a time step; Syn(t) represents a candidate word from synonym substitution, which can be obtained based on synonym dictionaries, semantic equivalents or approximate words of word vectors, or term libraries in the embodiments of the present application; Err(t) represents a candidate word from common errors / confusions; and HF represents a candidate word from domain hot words / dictionaries.

[0094] Step 31042: performing text correction on at least one word unit in the initial text based on the candidate word set to obtain the offline text, wherein in the text correction for a target word unit, the target word unit is any one of the at least one word unit.

[0095] Step 1: determining a candidate word subset corresponding to a target word unit based on the candidate word set, wherein the candidate word subset includes at least one of synonyms, ambiguous words and domain hot words of the target word unit.

[0096] Step 2: scoring the target word unit and each candidate word in the candidate word subset that meets a preset constraint condition with the target word unit, wherein the constraint condition includes part or all of the following: an edit distance less than an edit distance threshold, a consistent part-of-speech and semantics, and a context that does not conflict.

[0097] In the embodiments of the present application, the constraint condition includes part or all of the following conditions:

[0098] Condition 1, edit distance: dist(t, w)≤ε (edit distance threshold), limiting the change range to avoid deviating too far from the original word, for example, the Chinese can use the word-level Levenshtein distance to judge;

[0099] Condition 2, consistent part-of-speech / semantics: POS(w)≈POS(t), sem(w)≈sem(t), ensuring that the replacement does not destroy the syntax and semantics, for example, POS tagging, named entity recognition (NER), and embedding similarity can be used to judge;

[0100] Condition 3, context does not conflict: no entity contradiction or collocation exception after replacement, which can be judged based on co-occurrence statistics, collocation dictionary, knowledge base constraints, etc.

[0101] In the implementation, when step 2 is performed, the candidate words in the candidate word subset are filtered based on the constraint condition, and each candidate word that satisfies the constraint condition with the target word is filtered to determine the score of the target word and each candidate word.

[0102] Optionally, referring to Figure 7 When step 2 is performed, the foregoing system can specifically perform the following steps:

[0103] Step 21: Determine the log probability of the target word and each candidate word in the initial text.

[0104] In the embodiment of the application, when step 21 is performed, the offline deep model determines the log probability of the target word and each candidate word in the initial text of the dialogue audio, that is, the context, to reflect the acceptability and fluency of the sentence after replacement. Taking the logarithm facilitates linear weighting and numerical stability.

[0105] In the embodiment of the application, the language model in the offline deep model can be an n-gram language model (Language Model, LM), or a neural LM such as a recurrent neural network (Recurrent Neural Network, RNN) / Transformer, etc.

[0106] Step 22: Determine the term frequency-inverse document frequency of the target word and each candidate word in the domain preset corpus set.

[0107] In the embodiment of the application, when step 22 is performed, the offline deep model determines the term importance and relevance of the target word and each candidate word in the corpus set, that is, the term frequency-inverse document frequency, based on the domain preset corpus set, to emphasize that professional terms are preferentially retained or corrected as professional words.

[0108] Step 23: Determine the hit intensity of the target word and each candidate word in the domain term set.

[0109] In the embodiment of the application, when step 23 is performed, the offline deep model determines whether the target word and each candidate word hit the traffic domain term based on the domain term set such as the traffic domain term set, to obtain the respective hit intensity, wherein the traffic domain term set can be pre-constructed by a traffic domain dictionary / hotword set and can include a public term library, an organization's name list, etc.

[0110] In a specific implementation, the hit strength can be set as a binary value, i.e., 1 if a hit occurs and 0 if a hit does not occur; of course, the specific value of the hit strength can also be determined according to hierarchical weighting of the dictionary source. For example, the hit strength is M when a public term library is hit, and the hit strength is N when an in-house whitelist is hit, where M is less than N.

[0111] Step 24: determining the weighted sum value of the log probability, the term frequency-inverse text frequency, and the hit strength of the target word item as the score of the target word item, and determining the weighted sum value of the log probability, the term frequency-inverse text frequency, and the hit strength of each candidate word as the respective score.

[0112] In the embodiment of the present application, when step 24 is performed, the score of the target word item / candidate word can be determined by the following formula:

[0113]

[0114] wherein, represents the score of w (such as the aforementioned target word item / candidate word) under the initial text, i.e., the context ctx; represents the log probability of w generated by the language model under the context ctx; represents the term importance and relevance of w on the domain corpus, i.e., the term frequency-inverse text frequency; represents the hit strength of w on the domain term set; and α, β, and γ respectively correspond to the language model probability, the domain weight, and the dictionary / hotword hit weight.

[0115] In the embodiment of the present application, α is used to dominate the readability of the context, β is used to dominate the domain relevance, and γ is used to dominate the priority of the key term, and the balance of the three can avoid “grammatically correct but not domain accurate” or “term correct but context not smooth”.

[0116] Step 3: determining that the difference between the score of the target candidate word and the score of the target word item is not less than a preset threshold, and replacing the target word item with the target candidate word as a new word item, wherein the target candidate word is the candidate word in the candidate word subset that meets the constraint condition with the target word item and has the maximum score.

[0117] In the embodiment of the present application, when step 3 is performed, first, the target candidate word is determined, which is the candidate word in the candidate word subset that meets the constraint condition with the target word item and has the maximum score. The target candidate word (denoted as ) can be represented by the following formula:

[0118]

[0119] Then, a difference value between the score of the target candidate word and the score of the target word unit, i.e., S(w )-S(t), is determined, and the difference value is compared with a preset threshold (δ). If the difference value is less than the preset threshold, i.e., S(w )-S(t)<δ, it indicates that the promotion is insufficient, and the original word unit is not replaced to prevent over-repair and introduction of errors. If the difference value is not less than the preset threshold, it indicates that the promotion effect is better, and a replacement operation is performed, i.e., the target candidate word is used to replace the target word unit.

[0120] In the embodiments of the present application, the original word / alternative word, score, and constraint check log are recorded, and rollback and quality evaluation are supported. In order to facilitate linear weighting, tfidf and hit_lex can be standardized or scaled to similar orders of magnitude. The above α, β, γ, and ε, δ can be enumerated on a validation set, and based on the enumeration, a linear / logistic regression is trained with annotated data to , tfidf, hit_lex, etc. as features, and the optimal combination of α, β, γ, and ε, δ is learned to maximize the correction accuracy / reservation rate. Different weights and dictionary source priorities can be set for different sub-domains (such as accidents, roads, and control), to improve the domain adaptability.

[0121] Step 3105: based on the different speaker identities and timestamps marked in the dialogue audio, text fusion is performed on the real-time text and the offline text to obtain the dialogue text.

[0122] In the embodiments of the present application, when step 3105 is performed, the aforementioned system performs text fusion and timestamp alignment on the real-time text and the offline text according to the results of speaker separation of the dialogue audio, i.e., the different speaker identities and timestamps marked in the dialogue audio in step 3103, and the timestamps of the real-time text and the offline text, to obtain the dialogue text corresponding to the dialogue audio.

[0123] Step 320: inputting the dialogue text into a large language model to extract key information of the event described by the dialogue audio.

[0124] In the embodiments of the present application, the powerful semantic understanding and reasoning ability of the pre-trained large language model is utilized to extract multi-modal information such as voice content, tone, and speech speed in the dialogue audio, and the multi-modal information is deeply analyzed to obtain the key information of the event.

[0125] Referring to Figure 8In the embodiment of the application, the aforementioned application layer is deployed with a multi-modal information extraction submodule and a sensitive event identification and handling recommendation submodule. Step 320 involves the function of the multi-modal information extraction submodule, and the following will be described in detail with respect to this part of the function. In the embodiment of the application, a large language model information extraction engine is constructed, which includes a text preprocessing module and a large language model module. The text preprocessing module performs text preprocessing, such as merging and deduplication, on the received dialogue text, and then inputs the large language model module. The large language model module extracts multi-modal information, deeply analyzes each modality information, and performs standardization processing to obtain the key information of the event.

[0126] In the implementation, when step 320 is performed, referring to Figure 9 The aforementioned system can specifically perform the following flow:

[0127] Step 3201: input the dialogue text into the aforementioned large language model, and extract multi-modal information from the dialogue text based on the large language model.

[0128] The multi-modal information includes part or all of entity modality information, relationship modality information, event detection modality information, and emotion state modality information representing the basic information of the event.

[0129] Step 3202: determine the scores corresponding to a plurality of candidate word units in the target modality information, wherein the target modality information is any modality information in the aforementioned multi-modal information, and the plurality of candidate word units are part or all of synonyms, ambiguous words, or confused words used to describe the same target.

[0130] In the embodiment of the application, when step 3202 is performed, the source confidence of each candidate word unit is adaptively weighted based on the context features, and the knowledge graph retrieval result is used for speech enhancement and term priority, and the key information in the unstructured dialogue (dialogue audio) is deeply analyzed and extracted, wherein the source is the acquisition approach of the candidate word unit, such as the aforementioned real-time recognition, offline recognition, and text correction recognition mode.

[0131] Based on this, in the embodiment of the application, taking any one of the aforementioned plurality of candidate word units as an example, denoted as a target candidate word unit, the score corresponding to the target candidate word unit is the sum of the recognition score of the target candidate word unit and the graph enhancement score.

[0132] Optionally, the above recognition score is a weighted sum of the pattern confidence and fusion weight corresponding to multiple recognition modes respectively. The multiple recognition modes include some or all of the real-time recognition mode, offline recognition mode, or text correction mode. The fusion weight corresponding to any recognition mode is determined based on the measurement information of the preset dimension of the recognition mode. The measurement information of the preset dimension may include some or all of the recognition confidence measurement information, context consistency measurement information, regularity / dictionary hit measurement information, and stability measurement information (such as recent error rate, candidate dispersion, etc.).

[0133] In practice, the source confidence of the target candidate word element is obtained. Assume the obtained source confidence of the target candidate word element is... Let represent the following: RT indicates real-time recognition, OFF indicates offline recognition (offline re-scoring / language model verification), and CORR indicates text correction (error correction / post-processing).

[0134] Then, the fusion weight corresponding to each recognition mode is obtained. In this embodiment of the application, for any recognition mode, denoted as S, its corresponding fusion weight can be based on the metric information of the recognition mode (S) at the current time t in a preset dimension ( To determine, denoted as It is understood that this fusion weight is an adaptive fusion weight, which is calculated in real time according to the current features of the recognition mode and is dynamically changing. In this embodiment, the fusion weight corresponding to the above recognition mode can be expressed by the following formula:

[0135]

[0136] Where u represents the pre-trained / calibrated parameters.

[0137] Obtain the pattern confidence score corresponding to each recognition pattern of the target candidate word, denoted as . ,in, This represents the target candidate word. Optionally, the pattern confidence of any recognition mode for this target candidate word is the confidence of the recognition mode for this target candidate word in the context (dialogue text) at the current time t. The scoring is based on dimensions such as acoustic confidence, language model probability, and contextual co-occurrence.

[0138] Therefore, in this embodiment of the application, the target candidate word element The recognition score can be represented by the following formula:

[0139]

[0140] Optionally, the graph enhancement score is the product of the gating value of the target candidate word in the dialogue text and the enhancement score. The gating value is determined based on the target candidate word and the dialogue text, and the graph enhancement score is determined based on the target candidate word, the dialogue text, and the domain corpus.

[0141] In practice, the graph augmentation score of the target candidate word is obtained and denoted as... This score is based on the results of Knowledge Graph Retrieval (KG), which is constructed from a domain corpus. The score can be obtained by weighting the following three parts:

[0142]

[0143] in, The entity link similarity is represented by the entity embedding e(w) of the target candidate noun w and the set of context entities. Similarity; This represents the path probability / reachability from a context entity to w in a knowledge graph; This indicates the hit score for terms / dictionaries, such as place name databases, license plate rules, and business terms in traffic scenarios. , , These are the weights.

[0144] Then, based on the context of the current time t With the target candidate word Triggered features Determine the gate value of the target candidate word, denoted as . The feature includes Out-of-Vocabulary (OOV) / term detection, number / place name patterns, contextual uncertainty (entropy), source divergence, and time urgency. For example, this feature can reflect whether the target candidate noun resembles an entity / number, whether a place name appears in the context, and whether the KG search hits. In this embodiment, the gating value of the target candidate noun can be represented by the following formula. :

[0145]

[0146] in, The parameter is σ, which can be pre-trained / calibrated; σ is a sigmoid function. In this embodiment, the value of the gate is in the range of (0, 1); an upper limit is set for the gate value to prevent excessive intervention caused by KG mislinking; in case of failure, it falls back to source fusion.

[0147] Then, in the embodiments of the present application, the graph enhancement score of the target candidate token can be expressed by the following formula:

[0148]

[0149] Then, in the embodiments of the present application, the score corresponding to the target candidate token w can be expressed by the following formula:

[0150]

[0151] In this way, the scores corresponding to multiple candidate tokens in the target modal information can be obtained.

[0152] Step 3203: Determine that the candidate token with the maximum score among the scores corresponding to the multiple candidate tokens is a keyword in the above key information.

[0153] In the embodiments of the present application, when performing step 3203, the candidate token with the maximum score among multiple candidate tokens in the target modal information can be obtained through the following formula, that is, a keyword in the key information, denoted as :

[0154]

[0155]

[0156] Among them, C t represents the set of candidate tokens at the current time t; is the edit distance.

[0157] Through the above词性 / 语义一致 + edit distance constraint, acoustic approximations but semantically incorrect alternatives can be effectively suppressed, such as, "洲 / 州", "到 / 道", "桥 / 道", etc.

[0158] It can be understood that the above set of candidate tokens can be each candidate token in any micro-batch input to the large language model module, and each candidate token comes from RT, OFF, CORR, and the set of candidate tokens is obtained by merging and deduplicating through the text preprocessing module. Optionally, in order to reduce the data processing volume, the above set of candidate tokens can include taking the top-N candidate tokens after merging and deduplicating during text preprocessing.

[0159] The following is an example for illustration. For example, in the traffic scenario, assuming that for the sentence fragment "A rear-end collision occurred at the intersection of Jianghan Road and Starlight Avenue in Binjiang District, Hangzhou", it is recognized that there is a word with ambiguity near "A rear-end collision occurred at the intersection of Jianghan Road and Starlight Avenue in Binjiang District, Hangzhou", and the acoustic output may be "Starlight Avenue / Starlight Dadao / Starlight Bridge", and the source confidence is as follows: S RT("Starlight Avenue") = 0.55, S OFF ("Starlight Avenue") = 0.60, S CORR ("Starlight Avenue") = 1.0 (hit lexicon); S RT ("Starlight Avenue") = 0.3, S OFF ("Starlight Avenue") = 0, S CORR ("Starlight Avenue") = 0; S RT ("Starlight Bridge") = 0.15, S OFF ("Starlight Bridge") = 0.40, S CORR ("Starlight Bridge") = 0, S

[0160] Suppose the obtained fusion weights are: ω RT = 0.4, ω OFF = 0.3, ω CORR = 0.3, where the RT quality is acceptable; the CORR has strong lexicon signals; and the parallel obtained KG bypass includes E ctx contains the identified "Hangzhou", "Binjiang District", "Jianghan Road", where the entity linking: e("Starlight Avenue") has a similarity of sim = 0.85 with the "Binjiang District" place name set; e("Starlight Bridge") has a weak match with "Binjiang District", sim = 0.40; the path probability: there is a high probability of intersection from "Jianghan Road" to "Starlight Avenue" in the county road network, and the probability (P G ) that the graph path exists or is reachable = 0.90; the path to "Starlight Bridge" is weak, P G = 0.35; the lexicon hit: hit_lex("Starlight Avenue") = 1, hit_lex("Starlight Avenue") = 0 (spelling error), hit_lex("Starlight Bridge") = 0 / weak hit. And suppose λ link = 0.5, λ path = 0.4, λ lex = 0.1, then

[0161] S KG ("Starlight Avenue") = 0.5 0.85 + 0.4 0.90 + 0.1 1.0 = 0.875;

[0162] S KG ("Starlight Bridge") = 0.5 0.40 + 0.4 0.35 + 0.1 0 = 0.335;

[0163] S KG ("Starlight Avenue") is very low (almost 0).

[0164] Gate value: context contains county and road network, and source confidence has divergence, f t Trigger place / road mode and uncertainty, get g t = 0.7.

[0165] Then, based on the foregoing source confidence, the identification score (source fusion score) of "Starlight Avenue" is: 0.4 0.55 + 0.3 0.60 + 0.3 1.0 = 0.22 + 0.18 + 0.30 = 0.70; the graph enhancement score (bypass enhancement) is: 0.7 0.875 ≈ 0.6125; then, the score of "Starlight Avenue" is: 0.70 + 0.6125 = 1.3125; the identification score of "Starlight Bridge" is: 0.4 0.15 + 0.3 0.40 + 0.3 0 ≈ 0.06 + 0.12 + 0 = 0.18; the graph enhancement score is: 0.7 0.335 ≈ 0.2345; then, the score of "Starlight Bridge" is: 0.18 + 0.2345 = 0.4145; "Starlight Avenue" is directly constrained and filtered due to the mismatch of word nature / semantic and the edit distance exceeding the threshold.

[0166] Among "Starlight Avenue" and "Starlight Bridge", the POS / SEM is consistent (place / road), the edit distance is acceptable, and finally = "Starlight Avenue"; and the context entity is updated to improve the stability of subsequent road segment / crossing recognition.

[0167] In the embodiments of the application, through adaptive fusion weight, the quality of different sources is changed with the scene, the softmax weight is self-adaptive in time sequence, and the instability of fixed weighting is avoided; the knowledge graph enhanced retrieval is taken as a parallel bypass, and the blocking caused by time-consuming KG retrieval is avoided; at the same time, through the gate t , the influence is dynamically scheduled, and is weakened in low risk and strengthened in high risk such as term, place name, and number. The constraint is set to effectively inhibit the alternative items with acoustic approximation but semantic error.

[0168] Through the above processing, the key information of the event can be obtained, including a plurality of key word units extracted from the candidate word units included in each of the aforementioned multi-modal information. Taking the multi-modal information including entity modal information, relationship modal information, event detection modal information and emotional state modal information as an example, the plurality of key word units included in the key information include personnel information, location information, time information and vehicle information in the entity modal information, causal relationship information and time sequence relationship information in the relationship modal information, accident type information and emergency level information in the event detection modal information, and emotional state information and tension level information in the emotional state modal information.

[0169] Step 330: identifying the risk level of the event based on the key information, a plurality of identification rules and historical event knowledge, to obtain the risk level of the event, wherein one identification rule corresponds to one risk level.

[0170] In the embodiments of the present application, step 330 involves Figure 8 The functions of the sensitive event identification and disposal recommendation sub-module are shown, and the following will be described in detail for this part of the function. Referring to Figure 8 In the embodiments of the present application, the sensitive event identification and disposal recommendation sub-module includes a rule engine, a classification model module, a graph retrieval module and a risk level judgment module, wherein the engine is used to identify the risk level of the event corresponding to the key information based on a plurality of identification rules and the key information, one identification rule corresponds to one risk level, and one risk level corresponds to one score threshold; the classification model module is used to identify the risk level of the event corresponding to the key information through machine learning, to obtain an event category probability that the risk level is consistent with the risk level corresponding to the identification rule identified by the rule engine; the graph retrieval module is used to identify the risk level of the event corresponding to the key information in combination with historical event knowledge, to obtain the similarity between the event and similar events; and the risk level judgment module is used to determine a level score based on the rule score, the event category probability and the similarity, and the respective dynamic weights, and determine the risk level of the event based on the level score and a plurality of score thresholds.

[0171] For example, taking a traffic scene as an example, in the embodiments of the present application, the plurality of identification rules can be defined based on the related knowledge of traffic event risk level identification, such as a first risk level, the trigger conditions of which include “death ≥ 3, serious injury ≥ 10, dangerous chemical leakage or explosion, high-speed continuous collision, and traffic paralysis caused by bad weather” and the like. Therefore, the core identification signal can include the number of casualties, dangerous chemical vehicles, explosion / leakage trigger words, road type and weather context; the risk score strategy can include rule strong matching, high event probability weight, context bonus and cross-speaker contradiction penalty; and the disposal suggestion can include emergency plan, rescue dispatch, warning zone setting, medical cooperation and the like.

[0172] In the embodiments of the present application, the above risk level, trigger condition, core identification signal and disposal suggestion are structured, stored by using structured DSL (Domain Specific Language), as shown in Table 1, which supports versioning, retrieval, auditing and online adjustment, and can be audited, released in gray scale, A / B tested, and supports online weight update and rollback. Optionally, the main table stores rule meta information and hierarchy; the sub-tables store trigger conditions, signal weights and disposal actions respectively; and the field types can be selected from general types to adapt to various types of databases such as MySQL, PostgreSQL and SQLite, and mapping processing is performed when necessary.

[0173] Table 1 Structured DSL of identification rule

[0174]

[0175] In the implementation, when step 330 is performed, referring to Figure 10 , the foregoing system can specifically perform the following process:

[0176] Step 3301: performing keyword matching based on the key information and the plurality of identification rules to obtain an identification rule matched with the key information, and scoring a matching degree between the event and the identification rule based on the key information and the identification rule to obtain a rule score.

[0177] In the embodiments of the present application, when step 3301 is performed, a rule engine is called to match the identification rule matched with the key information from the plurality of identification rules by using keyword matching, for example, the identification rule with the maximum matching degree or greater than a matching degree threshold is coarsely screened based on keyword matching, and then the matching degree between the key information and the identification rule is further analyzed and evaluated based on the key information and the identification rule, so as to obtain the rule score.

[0178] In the embodiments of the present application, the key information carries a speaker identifier and includes a plurality of keyword elements and identification confidence degrees corresponding to the plurality of keyword elements; and when the matching degree between the event and the identification rule is scored based on the key information and the identification rule in step 3301 to obtain the rule score, referring to Figure 11 , the foregoing system can specifically perform the following process:

[0179] Step 33011: scoring a rule matching degree between the key information and the identification rule based on the plurality of conditions and context constraint information included in the identification rule, and the plurality of keyword elements and identification confidence degrees to obtain a first score.

[0180] In the implementation, when step 33011 is performed, referring toFigure 12 , the signal input corresponding to the key information triggers the rule to cause the analysis of the identification rule (DSL), and the multiple conditions included in the identification rule are organized as all (AND) or any (OR) by block, wherein all (AND) represents "all meet", and any (OR) represents "any meet"; then, each block is aggregated by using a weighted extreme value approximation logic, and the aggregation scores of all the condition blocks are summed and normalized to obtain the first score, denoted as RuleHit.

[0181] In the embodiment of the application, all (AND) is denoted as , representing weighted minimum aggregation, and any (OR) is denoted as , representing weighted maximum aggregation, and can be respectively represented by the following formulas:

[0182]

[0183]

[0184] , wherein, represents the hit condition of the key information and the condition j in the condition block b, ∈ [0, 1], which can be obtained by using binary or fuzzy hit; is the weight of the condition j, that is, the weight of the DSL; , which embodies that all meet is more strict and the score is not higher than any meet; represents the total number of conditions in the condition block b.

[0185] The first score can be represented by the following formula:

[0186]

[0187] , wherein B represents the total number of condition blocks.

[0188] For example, the identification rule example, secondary (risk level): (multi-vehicle pileup + high speed + severe weather), "id": 002; "version": 1.1.0, "level": 2; "speaker_scope": caller; "time_window": 600; "context": high speed, [heavy rain, heavy snow, heavy fog]; "conditions": all (AND): "event": multi-vehicle pileup, 1.0 (weight); "entity": number of casualties, >= 10 (value), 0.8 (weight); "context_key": high speed, 0.5 (weight); any (OR): "context_key": ∈ ["heavy rain", "heavy snow", "heavy fog"], 0.4 (weight); "context_key": congestion_minutes, >= 30, 0.3 (weight).

[0189] Key information extracted from ASR+LLM (time window 600 seconds, main speaker caller) example: high speed; heavy rain; multi-vehicle pileup: 0.92 (confidence), 120 (ts); number of casualties: 12 (value), 0.85 (confidence); congestion duration: 45; speaker: caller; model_logits: major accident (1.5), general accident (0.3), minor accident (-0.2).

[0190] In the embodiments of the present application, if the event type condition is hit, = confidence, as in the above example "multi-vehicle pileup" hit, the recognition confidence of this keyword element is 0.92; Take 1.0 in the rule "event": multi-vehicle pileup, 1.0 (weight)"; numerical / context condition hit, binary or fuzzy hit can be used, if satisfied, =1.0, otherwise =0.0, as in the above example, "number of casualties", "high speed", "congestion duration", "severe weather contains".

[0191] Block aggregation calculation: (1) all satisfy block all (corresponding formula ): multi-vehicle pileup, 1.0 x 0.92 = 0.92; number of casualties >= 10, 0.8 x 1.0 = 0.8; high speed, 0.5 x 1.0 = 0.5; then = min(0.92, 0.8, 0.5) = 0.5; (2) any satisfy block any (corresponding formula ): bad weather contains, 0.4x1.0=0.4; congestion duration >=30, 0.3x1.0=0.3; then =max(0.4,0.3)=0.4; execute rule hit normalization (corresponding formula RuleHit): 0.5+0.4=0.9 when not normalized; all block upper bound is min(1.0,0.8,0.5)=0.5; any block upper bound is max(0.4,0.3)=0.4; upper bound sum is 0.5+0.4=0.9; after normalization, RuleHit=0.9 / 0.9=1.0 (falling in [0,1]).

[0192] Step 33012: Based on the key information and the above context constraint information, the confidence of the key information matching the recognition rule is scored to obtain a second score.

[0193] In the embodiments of the application, when step 33012 is executed, the key information is subjected to pattern recognition based on the context (context field) of the recognition rule, that is, the confidence of the key information matching the recognition rule is scored to obtain a second score, denoted as Context. In the above recognition rule example, "high speed + bad weather hit", Context is obtained based on experience value ≈0.8.

[0194] Step 33013: Based on the key information and the restriction condition included in the recognition rule, the confidence of the occurrence of the event is scored to obtain a third score, wherein the restriction condition includes a speaker restriction condition and a time restriction condition.

[0195] In the embodiments of the application, when step 33013 is executed, the key information is filtered according to the speaker_scope field and the time_window field in the aforementioned recognition rule to construct consistency check data to obtain a third score, denoted as Penalty. In the above recognition rule example, the key information is subjected to pattern recognition, and assuming that there is no conflict across speakers, Penalty=0.05, and the score is small.

[0196] In the embodiments of the application, the setting of the consistency constraint and the penalty limits the speaker range of the condition evaluation; when there is a contradiction across speakers, enter Penalty, Penalty=α'xcontradict+β'x incomplete, wherein contradict represents contradiction, which can be calculated based on the above constructed consistency check data, and incomplete represents incompleteness, which can also be calculated based on the above constructed consistency check data.

[0197] Step 33014: Based on the first score, the second score and the third score, the rule score is determined.

[0198] In the embodiment of the present application, when step 33014 is performed, the dynamic weight of the first score and the second score is determined based on the current risk level prior of the system and the speaker consistency check data, and is respectively denoted as 、 Then, the sum of the weighted sum of the first score and the dynamic weight corresponding thereto, the second score and the dynamic weight corresponding thereto, and the negative value of the third score is determined as the rule score, and the formula is expressed as follows:

[0199]

[0200] Step 3302: The key information is input into a preset classification model, and the risk level of the event is classified and recognized based on the preset classification model to obtain an event category probability that the risk level of the event is the risk level corresponding to the recognition rule.

[0201] In the embodiment of the present application, when step 3302 is performed, referring to Figure 12 , the risk level of the event is classified and recognized based on the key information by means of machine learning, so as to obtain the event category probability, denoted as .

[0202] In the specific implementation, the logits of the preset classification model are subjected to softmax with temperature T to obtain the event category probability , which can be expressed by the following formula:

[0203]

[0204] Wherein, the temperature T is a positive scalar, which is used for smoothing / sharpening calibration of the logits of the classification model; and z is the output value of the logits.

[0205] Taking a traffic scene as an example, it is known that the model logits ={“major accident”: 1.5, “general accident”: 0.3, “minor accident”: -0.2}, and T=1.0: exp(1.5)≈4.48, exp(0.3)≈1.35, exp(-0.2)≈0.82, and the total sum≈6.65; the event category probability is in turn: p(major)≈4.48 / 6.65≈0.674, p(general)≈0.203, and p(minor)≈0.123; if the target event is “major accident”, then ≈0.67.

[0206] In the embodiment of the present application, considering that the model also has an error possibility, the credibility of the current classification model is determined based on the current model historical false alarm situation of the system, and is denoted as , to correct the event category probability.

[0207] Step 3303: Based on the key information, retrieve a historical event similar to the event from the historical event knowledge, and determine the similarity between the event and the historical event.

[0208] In the embodiments of the present application, when step 3303 is performed, referring to Figure 12 , based on the key information, knowledge retrieval such as full-text retrieval (Elasticsearch, ES) / retrieval-augmented generation (RAG) is used to obtain the hit in the historical event knowledge, thereby obtaining the similarity, denoted as Retrieval. Taking the above identification rule example as an example, if the ES hit is “highway section + heavy rain accident history”, then Retrieval ≈ 0.6. Correspondingly, in the embodiments of the present application, the accuracy of retrieval is determined based on the historical false alarm rate of retrieval, denoted as , to correct the similarity.

[0209] Step 3304: Based on the rule score, the event category probability, and the similarity, score the risk level of the event to obtain a level score.

[0210] In the embodiments of the present application, after obtaining the rule score, the event category probability, and the similarity through the foregoing steps 3301-3303, referring to Figure 12 , the joint scoring and normalization can be performed through the following formula to obtain the level score, denoted as .

[0211]

[0212]

[0213] , wherein represents the mean, represents the standard deviation; other parameters have been introduced in the foregoing content and will not be described here. In the embodiments of the present application, the is de-meaned and scaled after sigmoid, to obtain a level score stabilized at (0, 1), which is convenient for cross-module and threshold strategy docking.

[0214] Taking the above identification rule example, RuleHit=1.0, Context≈0.8, Retrieval≈0.6, Penalty=0.05, and assuming that the preset classification model determines that the target event category corresponding to the key information is a major accident, then temperature T=1.0, and the output event category probability: ≈0.67; the weight setting assumption is: wrule =0.5, wmodel=0.3, wctx=0.15, wkg=0.1, then S raw =0.5×1.0+0.3×0.65+0.15×0.8+0.1×0.6-0.05=0.825. Assume normalization parameters: μ'=0.5, σ std =0.2, which can be obtained experimentally, then S'=sigmoid((0.825-0.5) / 0.2)= sigmoid(1.625)≈0.835.

[0215] In this embodiment of the application, rule hit, event trigger probability, context, and retrieval signal are unified into a learnable scoring function, namely the aforementioned level score, through rule-model joint scoring. Dynamic weights adapt to different scenarios; contextual elements serve as scoring items. Enter On the other hand, it can be used as a conditional gating mechanism (e.g., limiting road type, weather, or time window); by constraining speaker consistency, it can integrate speaker separation results to avoid misjudgments caused by cross-speaker splicing and improve the consistency between entities and events; by enhancing search engine optimization (RAG), it can perform ES search and hot word matching on locations, institutions, vehicle brands / models, etc., and use them as knowledge features for scoring and triggering correction; by setting scene temporal constraints, it can model time features such as "congestion duration" and "continuous call feedback" to avoid instantaneous anomalies being misjudged as sensitive events.

[0216] Step 3305: Determine the risk level of the event by the score threshold that has the smallest difference between the score and the score of the current risk level among the multiple risk levels.

[0217] See also Figure 12 By performing step 3305 and comparing the obtained rating with multiple rating thresholds, the risk level of the event can be determined, denoted as [risk level]. The formula is expressed as follows:

[0218]

[0219] Among them, Trigger ( ) indicates level The function, This indicates level two.

[0220] In this embodiment of the application, the level is used. The scoring threshold θ ( ) to perform binary triggering; a cost-sensitive threshold setting can be used, and θ (primary) ≥ θ (secondary) ≥ θ (tertiary) ≥ θ (quaternary) is ensured to guarantee the monotonicity of the classification. Assuming that the score thresholds are θ (1) = 0.9, θ (2) = 0.8, θ (3) = 0.7, and θ (4) = 0.6 based on the above-mentioned identification rule example, since 0.835 ≥ 0.8, it is determined that the risk level of the event is secondary "major". The definition of the risk level level can be: 1 = primary (particularly major), 2 = secondary (major), 3 = tertiary (relatively large), and 4 = quaternary (general).

[0221] Step 340: determining a handling plan based on the risk level, sending the handling plan to the terminal device of the operator, and sending the handling plan to the processing terminal associated with the geographical location of the event.

[0222] In the embodiments of the present application, still referring to Figure 8 The sensitive event identification and handling recommendation submodule further includes a handling recommendation module, which is configured to determine a handling plan based on the risk level. In specific implementation, when step 340 is performed, the handling plan can be formulated based on the handling suggestion corresponding to the determined risk level, such as that the primary (particularly major) corresponds to an emergency handling plan, the secondary (major) corresponds to a key attention handling plan, the tertiary (relatively large) corresponds to a regular enhanced processing plan, and the quaternary (general) corresponds to a standard handling plan (process), and the handling plan is sent to the terminal device of the operator through the handling layer for display, and is sent to the processing terminal associated with the geographical location of the event, so that the operator is aware of the handling suggestion and gives the caller a handling suggestion, and the relevant personnel of the processing terminal are linked to respond quickly.

[0223] The specific architecture of the foregoing event processing system based on a large language model will be described below with reference to a specific embodiment.

[0224] Referring to Figure 13The system includes a voice access layer that may include a voice access gateway and a load balancer, with the voice access gateway routing dialogue audio and the load balancer balancing the load; a processing layer that includes a real-time recognition service cluster and an offline processing service cluster; the real-time recognition service cluster is used for real-time recognition of dialogue audio; the offline processing service cluster is used for offline recognition and text correction of dialogue audio; an application layer that includes a large-scale model inference service, an information extraction service, a sensitive event recognition service, and a response recommendation service, where the large-scale model inference service is used for inference; the information extraction service is used to extract multimodal information; the sensitive event recognition service is used to identify risk levels; and the response recommendation service is used to determine response plans based on the identified risk levels; a response layer that includes an operator workbench, a management monitoring console, and a mobile application; and a data layer that provides data support for the system, such as the aforementioned language prior knowledge, corpus sets, and terminology sets; specifically, the data layer may include a real-time database, a historical data warehouse, and a knowledge graph, with the real-time database supporting the real-time recognition service cluster, the historical data warehouse supporting the offline processing service cluster, and the knowledge graph supporting subsequent graph-enhanced retrieval and knowledge retrieval.

[0225] See Figure 14 For callers, an event processing system based on a large language model (i.e.) Figure 14 The interaction process between the system, operators, and related processing terminals is explained. Optionally, the caller provides a voice description (step 1401); the system performs real-time voice recognition (step 1402); real-time text is sent to the operator's terminal device to display the real-time text to the operator (step 1403); the voice is processed through an offline model (step 1404); voice and text correction and speaker separation are performed (step 1405); large model information extraction is performed (step 1406); key information identification is performed (step 1407); event risk assessment is performed (step 1408); if it is a sensitive event (such as a level 1, 2, or 3 event), a red warning is sent to the operator's terminal device, and relevant processing terminals are automatically notified (step 1409); if it is a general event (such as a level 4 event), a routine handling prompt is sent to the operator's terminal device to enable the operator to assist the caller (step 1410); if it is not one of the above risks, a recommended handling action is sent to the operator's terminal device to enable the operator to assist the caller (step 1411); the operator sends a confirmation of the handling plan to the system (step 1412); the system records the handling process (step 1413).

[0226] Based on the same inventive concept, embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects.

Claims

1. A large language model-based event processing method, characterized in that, The method comprises the following steps: receiving conversation audio between a caller and an operator; real-time recognition and offline correction are performed on the conversation audio respectively, and real-time text recognized in real time and offline text after offline correction are fused to obtain conversation text, the offline correction comprises text correction and dialect fine-tuning; the conversation text is input into a large language model to extract multi-modal information, and based on acquisition approach confidence of a plurality of candidate word elements in target modal information and a knowledge graph constructed from domain corpus, scores corresponding to the plurality of candidate word elements are determined, the target modal information is any modal information in the multi-modal information extracted by the large language model, and the plurality of candidate word elements are all of synonyms, ambiguous words or confused words used to describe the same target; and a candidate word element with the maximum score in the scores corresponding to the plurality of candidate word elements is determined as one key word element in key information of an event described by the conversation audio, and the key information is obtained; based on the key information, a plurality of recognition rules and historical event knowledge, a risk level of the event is recognized to obtain the risk level of the event, and one recognition rule corresponds to one risk level; based on the risk level, a disposal plan is determined, the disposal plan is sent to a terminal device of the operator and a processing terminal associated with a geographical location of the event; wherein the score corresponding to the target candidate word element is the sum of an identification score of the target candidate word element and a graph enhancement score; the target candidate word element is any one of the plurality of candidate word elements; the identification score is a weighted sum of acquisition approach confidences corresponding to a plurality of acquisition approaches and fusion weights, the plurality of acquisition approaches include real-time recognition acquisition approach, offline recognition acquisition approach and text correction acquisition approach, and the fusion weight corresponding to any acquisition approach is determined based on measurement information of a preset dimension of the any acquisition approach, the measurement information of the preset dimension includes identification confidence measurement information, context consistency measurement information and regular / dictionary hit measurement information; the graph enhancement score is the product of a gating value of the target candidate word element in the conversation text and an enhancement score, and the gating value is calculated by the following formula: Wherein, the target candidate word unit is determined according to the following formula: The gating value of the target candidate word unit, the value range is (0, 1); the target candidate word unit is ; the is a parameter, which is pre-trained / calibrated; the sigma represents sigmoid; the characterizes the context based on the current time t The feature obtained by the target candidate word unit, including OOV / term detection, digital / place name pattern, time urgency feature outside the vocabulary ​ the enhancement score is calculated by the following formula: wherein the represents the enhanced score of the w; the represents the entity linking similarity, representing the similarity between the entity embedding e(w) of the w and the set of context entities ; the represents the path probability from the context entities to the w in the knowledge graph; the represents the matching degree score of the w with the term / dictionary; the , the , the are weights, respectively.

2. The method of claim 1, wherein, the conversation audio is recognized in real time and corrected offline respectively, and the first text recognized in real time and the second text corrected offline are fused to obtain the conversation text, comprising: real-time speech recognition is performed on the conversation audio to obtain the real-time text; offline speech recognition is performed on the conversation audio to obtain the initial text; audio analysis is performed on the conversation audio to mark different speaker identities and time stamps in the conversation audio; based on language prior knowledge, text correction is performed on the initial text to obtain the offline text, and the language prior knowledge includes at least one of synonym prior knowledge, easily confused word pair prior knowledge or domain term prior knowledge; based on the different speaker identities and time stamps marked in the conversation audio, text fusion is performed on the real-time text and the offline text to obtain the conversation text.

3. The method of claim 2, wherein, The text correction based on language prior knowledge is performed on the initial text to obtain the offline text, including: Based on language prior knowledge, a candidate word set of the initial text is determined, any candidate word in the candidate word set being a synonym, an ambiguous word or a domain hot word of a word element in the initial text; Based on the candidate word set, at least one word element in the initial text is subjected to text correction to obtain the offline text, wherein in the text correction for a target word element, the target word element is any one of the at least one word element: Based on the candidate word set, a candidate word subset corresponding to the target word element is determined, the candidate word subset including at least one of a synonym, an ambiguous word and a domain hot word of the target word element; From the candidate word subset, each candidate word satisfying a preset constraint condition with the target word element is screened out, the target word element and each screened candidate word are scored based on the initial text, a domain preset corpus and a domain term set, the constraint condition including an edit distance less than an edit distance threshold, a part of speech and a semantic consistency, a context coherence; A difference between a score of a target candidate word and a score of the target word element is determined to be not less than a preset threshold, the target candidate word being a candidate word in the candidate word subset satisfying the constraint condition with the target word element and having a maximum score, the target candidate word is used to replace the target word element, and the preset threshold is a minimum score difference for performing the replacement.

4. The method of claim 3, wherein, The scoring of the target word element and each screened candidate word based on the initial text, the domain preset corpus and the domain term set includes: Log probabilities of the target word element and each candidate word in the initial text are determined respectively; Term frequency-inverse document frequencies of the target word element and each candidate word in the domain preset corpus are determined respectively; Hit intensities of the target word element and each candidate word in the domain term set are determined respectively, the hit intensity representing whether any candidate word or the target word element is included in the domain term set, if included, the hit intensity is 1, and if not included, the hit intensity is 0; A weighted sum value of the log probability, the term frequency-inverse document frequency and the hit intensity of the target word element is determined as the score of the target word element, and a weighted sum value of the log probability, the term frequency-inverse document frequency and the hit intensity of each candidate word is determined as the score of each candidate word respectively.

5. The method of claim 2, wherein, Offline speech recognition is performed on the dialogue audio to obtain an initial text, including: A voiceprint feature of the dialogue audio is determined; Based on a dialect model matched with the voiceprint feature, the voiceprint feature is subjected to dialect feature fine-tuning, and offline speech recognition is performed on the fine-tuned voiceprint feature to obtain the initial text.

6. The method of claim 2, wherein, Audio analysis is performed on the dialogue audio, and different speaker identities and time stamps in the dialogue audio are marked, including: Voice activity detection is performed on the dialogue audio, and the dialogue audio is segmented into multiple segments based on a detection result; Based on the acoustic features corresponding to the plurality of segments respectively, the plurality of segments are clustered, and different speaker identities and timestamps are marked in the dialogue audio based on the clustering results, wherein the segment length of the clustered segments is not less than a segment length threshold and the classification confidence is not less than a classification confidence threshold.

7. The method according to any one of claims 1 to 6, wherein The risk level of the event is identified based on the key information, a plurality of identification rules and historical event knowledge, and the risk level of the event is obtained. Based on the key information and the plurality of identification rules, keyword matching is performed to obtain an identification rule matched with the key information, and based on the key information and the identification rule, the matching degree of the event and the identification rule is scored to obtain a rule score. The key information is input into a preset classification model, and the risk level of the event is classified and identified based on the preset classification model to obtain an event category probability that the risk level of the event is the risk level corresponding to the identification rule. Based on the key information, a historical event similar to the event is retrieved from the historical event knowledge, and the similarity between the event and the historical event is determined. Based on the rule score, the event category probability and the similarity, the risk level of the event is scored to obtain a level score. The risk level corresponding to the score threshold with the minimum difference value in the plurality of risk levels respectively corresponding to the score thresholds is determined as the risk level of the event.

8. A large language model-based intelligent event processing system, characterized by, It comprises: a voice access layer, a processing layer, an application layer and a handling layer, wherein The voice access layer is configured to receive dialogue audio between a caller and an operator. The processing layer is configured to perform real-time identification and offline correction on the dialogue audio respectively, and fuse real-time text identified in real time and offline text after offline correction to obtain dialogue text, wherein the offline correction includes text correction and dialect fine-tuning. The application layer is configured to input the dialogue text into a large language model to extract multi-modal information, and determine scores respectively corresponding to a plurality of candidate word elements based on the acquisition approach confidence of the plurality of candidate word elements in the target modal information and the knowledge graph constructed from the domain corpus, wherein the target modal information is any modal information in the multi-modal information extracted by the large language model, and the plurality of candidate word elements are all of the synonymous words, ambiguous words or confused words used to describe the same target; and determine a candidate word element with the maximum score in the scores respectively corresponding to the plurality of candidate word elements as one key word element in the key information of the event described by the dialogue audio, and obtain the key information; identify the risk level of the event based on the key information, a plurality of identification rules and historical event knowledge, and obtain the risk level of the event, wherein one identification rule corresponds to one risk level; and determine a handling plan based on the risk level. The handling layer is configured to send the handling plan to a terminal device of the operator for display, and to a processing terminal associated with the geographical location of the event. The score corresponding to the target candidate word unit is a sum of an identification score of the target candidate word unit and a graph enhancement score. The identification score is a weighted sum of acquisition approach confidences corresponding to multiple acquisition approaches and fusion weights, the multiple acquisition approaches include a real-time identification acquisition approach, an offline identification acquisition approach, and a text correction acquisition approach, a fusion weight corresponding to any acquisition approach is determined based on metric information of a preset dimension of the any acquisition approach, the metric information of the preset dimension includes identification confidence metric information, context consistency metric information, and regular / dictionary hit metric information; The graph enhancement score is a product of a gating value of the target candidate word unit in the dialogue text and an enhancement score, the gating value is calculated by the following formula: wherein the represents a gating value of the target candidate word piece, and the value range is (0, 1); the target candidate word piece is ; the is a parameter, which is pre-trained / calibrated; the sigma represents sigmoid; the characterizes a context based on a current time t and the feature obtained, including out-of-vocabulary (OOV) term detection, number / geo name pattern, and time urgency features. The enhancement score is calculated by the following formula: wherein the represents the enhanced score of the w; the represents the entity linking similarity, representing the similarity between the entity embedding e(w) of the w and the set of context entities ; the represents the path probability from the context entities to the w in the knowledge graph; the represents the matching degree score of the w with the term / dictionary; the , the , the are weights, respectively.

Images