A context-aware risk intervention method and system for streaming audio

Abstract

This invention belongs to the technical field of ride-hailing services, specifically relating to a context-aware risk intervention method and system for streaming audio. The method includes: receiving and processing in-vehicle streaming audio in real time, extracting multimodal features such as translated text and acoustic emotion; constructing a multidimensional context vector based on these features and real-time order metadata, performing initial risk screening using a lightweight classifier, and triggering deep analysis by a generative artificial intelligence model only when the risk probability exceeds a threshold; replacing the input data with a privacy information mask before deep analysis; the generative artificial intelligence model combining the context to output a structured risk assessment result; and finally triggering tiered automated intervention actions based on the result. This invention solves the computational performance bottleneck of deep semantic analysis in high-concurrency scenarios through a layered asynchronous processing architecture, achieving accurate, real-time identification and automated handling of metaphorical security risks.

Description

Technical Field

[0001] This invention belongs to the technical field of ride-hailing services, specifically relating to a context-aware risk intervention method and system for streaming audio. Background Technology

[0002] To improve the semantic understanding depth of safety monitoring of streaming audio (such as ride-hailing trip recordings), an advanced technical solution is to try to directly apply generative large language models (LLM) to risk analysis. This solution converts the entire audio stream during the trip into text, and without any pre-filtering, directly and completely inputs it into the generative large language model. Relying on the model's natural language understanding capabilities, it directly identifies risks such as conflicts and inducements to transactions from the dialogue text. It utilizes the powerful contextual understanding capabilities of generative artificial intelligence to overcome the shortcomings of traditional keyword matching and traditional classification models in recognizing complex and metaphorical content.

[0003] However, the aforementioned solution presents an irreconcilable contradiction between its enormous computational overhead and the high concurrency, low latency, and real-time processing requirements of ride-hailing services. Generative large language models require significant computational resources for inference and have high response latency. Performing full-volume, real-time model analysis on massive concurrent trip audio streams would immediately lead to system resource exhaustion and a surge in processing latency, failing to meet the high availability and millisecond-level response service level agreement (SLA) requirements of risk control systems. This means that although the technical solution theoretically has stronger semantic understanding capabilities, it cannot be deployed and applied in actual high-concurrency business scenarios due to performance bottlenecks. The core problem lies in the lack of a feasible architecture that can achieve a balance between resource consumption and audit depth. Summary of the Invention

[0004] The technical problem to be solved by this invention is the fundamental contradiction between the huge computational overhead brought about by the full use of a large model for deep understanding of dialogue semantics in high-concurrency real-time scenarios such as ride-hailing and the system performance requirements.

[0005] To address the aforementioned technical problems, a first aspect of this invention discloses a context-aware risk intervention method for streaming audio, the method comprising:

[0006] Receive real-time streaming audio data from the vehicle and process the streaming audio data concurrently to extract multimodal features, including translated text features and acoustic emotion features.

[0007] Based on the multimodal features and real-time order metadata, a multidimensional context vector is constructed, and a lightweight classifier is used to calculate the risk probability of the multidimensional context vector. Only when the risk probability exceeds a preset threshold is the generative artificial intelligence model triggered to perform deep analysis.

[0008] In response to deep analysis, the system inputs structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model and receives the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract.

[0009] Before inputting the structured prompts into the generative artificial intelligence model, the personal privacy information in the structured prompts is masked and replaced.

[0010] Based on the structured risk assessment results, trigger automated intervention actions corresponding to the risk assessment level.

[0011] As an optional implementation, in the first aspect of the present invention, receiving real-time streaming audio data from the vehicle and concurrently processing the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features, includes:

[0012] The streaming audio data is translated into time-stamped text segments in real time using a streaming speech recognition service.

[0013] In parallel, an acoustic sentiment analysis model is used to extract fundamental frequency, Mel-frequency cepstral coefficients, and energy features from the streaming audio data, and a normalized sentiment confidence score is output.

[0014] As an optional implementation, in the first aspect of the present invention, the step of constructing a multidimensional context vector based on the multimodal features and real-time order metadata includes:

[0015] Using the session memory module, a sliding window context containing the text of the most recent N rounds of dialogue is maintained for each driving order;

[0016] The text embedding vector of the sliding window context, the emotion feature vector extended by acoustic emotion features, and the order metadata vector encoded by real-time order status information are concatenated to generate a unified multidimensional context vector. ,in For text embedding vectors, For emotion feature vectors, This is the order metadata vector.

[0017] As an optional implementation, in the first aspect of the present invention, the real-time order status information includes a deviation status determined based on the vehicle's real-time geographical location, a status during late-night hours determined based on a timestamp, and the driver's historical complaint rate.

[0018] As an optional implementation, in the first aspect of the present invention, the step of inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model, and receiving a structured risk assessment result output by the model that conforms to a predetermined JSON Schema contract, includes:

[0019] Retrieve recent conversation context from session memory;

[0020] The recent dialogue context, the currently extracted acoustic emotion features, and risk metadata including yaw status are assembled into structured prompt words that conform to a preset template.

[0021] The structured cue words are input into a generative artificial intelligence model, and the output parser forces the model to return structured JSON data containing risk category enumeration values, risk score integers, and key evidence strings.

[0022] As an optional implementation, in the first aspect of the present invention, the masking and replacement of personal privacy information in the structured prompts includes:

[0023] By using a named entity recognition model and a regular expression pipeline, consecutive phone numbers and specific place names in text are identified and replaced with predefined mask labels, respectively. and .

[0024] As an optional implementation, in the first aspect of the present invention, triggering an automated intervention action corresponding to the risk assessment level based on the structured risk assessment result includes:

[0025] If the risk assessment level is low or medium risk, a text warning message will be sent to the driver's vehicle terminal via message queue telemetry transmission protocol and then broadcast to the driver's voice.

[0026] If the risk assessment level is high, the monitoring level of the corresponding order will be automatically upgraded, forced frame extraction of the in-vehicle video will be enabled, and the warning information containing the highlighted text will be pushed to the safety dispatch center in real time.

[0027] As an optional implementation, in the first aspect of the invention, in the step of inputting structured prompts into a preset generative artificial intelligence model, the inference temperature parameter of the generative artificial intelligence model is set to an extremely low value close to 0.

[0028] A second aspect of this invention discloses a context-aware risk intervention system for streaming audio, used to implement the context-aware risk intervention method for streaming audio described in any of the above embodiments, the system comprising:

[0029] The feature extraction module is used to receive real-time streaming audio data from the vehicle and perform concurrent processing on the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features.

[0030] The analysis trigger module is used to construct a multi-dimensional context vector based on the multimodal features and real-time order metadata, and to use a lightweight classifier to calculate the risk probability of the multi-dimensional context vector. Only when the risk probability exceeds a preset threshold is the generative artificial intelligence model triggered to perform deep analysis.

[0031] The risk reasoning module is used to respond to deep analysis triggers by inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model, and receiving the structured risk assessment results output by the model that conform to a predetermined JSONSchema contract.

[0032] The processing module is used to perform masking and replacement processing on the personal privacy information in the structured prompts before the generative artificial intelligence risk reasoning module performs reasoning.

[0033] The execution module is used to trigger automated intervention actions corresponding to the risk assessment level based on the structured risk assessment results.

[0034] A third aspect of the present invention discloses another context-aware risk intervention system for streaming audio, the system comprising:

[0035] Memory containing executable program code;

[0036] A processor coupled to the memory;

[0037] The processor calls the executable program code stored in the memory to execute a context-aware risk intervention method for streaming audio disclosed in the first aspect of the present invention.

[0038] The fourth aspect of the present invention discloses a computer-readable storage medium storing computer instructions, which, when invoked by a processor, are used to execute a context-aware risk intervention method for streaming audio disclosed in the first aspect of the present invention.

[0039] Compared with the prior art, the beneficial effects of the present invention are:

[0040] The technical solution provided by this invention effectively solves the contradiction between the huge computational overhead caused by directly and fully calling generative large language models and the requirements for high concurrency and real-time performance. By using a lightweight classifier to perform real-time initial screening of feature vectors that integrate multi-dimensional context, this design can intercept most low-risk dialogues and trigger generative AI deep analysis only for a small number of high-risk scenarios. This significantly reduces computational resource consumption at the source, making it possible to deploy deep semantic analysis in high-concurrency streaming audio scenarios. Based on this, for the triggered scenarios, generative AI is used for deep reasoning, combined with contextual constraints such as order status, to achieve accurate identification of complex and metaphorical risks, improving audit depth. Simultaneously, the method performs mandatory personal information mask replacement processing before data is input into the AI ​​model, ensuring privacy compliance throughout the process. Finally, based on the structured risk assessment results output by AI, automated and tiered intervention actions can be triggered, forming a complete and efficient closed loop from risk perception, judgment to handling. Attached Figure Description

[0041] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings, wherein:

[0042] Figure 1 This is a flowchart illustrating a context-aware risk intervention method for streaming audio disclosed in an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of the structure of a context-aware risk intervention system for streaming audio disclosed in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of another context-aware risk intervention system for streaming audio disclosed in an embodiment of the present invention. Detailed Implementation

[0045] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0047] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0048] This invention discloses a context-aware risk intervention method and system for streaming audio. It utilizes a lightweight classifier to perform real-time initial screening of feature vectors incorporating multi-dimensional context. This design can intercept most low-risk dialogues, triggering generative AI deep analysis only for a small number of high-risk scenarios. This significantly reduces computational resource consumption at the source, making it possible to deploy deep semantic analysis in high-concurrency streaming audio scenarios. Based on this, for triggered scenarios, generative AI is used for deep reasoning, combined with contextual constraints such as order status, to achieve accurate identification of complex and metaphorical risks, improving audit depth. Simultaneously, the method performs mandatory personal information mask replacement before data is input into the AI ​​model, ensuring privacy compliance throughout the entire process.

[0049] Example 1

[0050] Please see Figure 1 , Figure 1 This is a flowchart illustrating a context-aware risk intervention method for streaming audio disclosed in an embodiment of the present invention. Figure 1 The described context-aware risk intervention method for streaming audio is applied to a data processing chip, processing terminal, or processing server, and the processing server can be a local server or a cloud server; this embodiment of the invention does not limit the application. Figure 1 As shown, this context-aware risk intervention method for streaming audio may include the following operations:

[0051] 101. Receive real-time streaming audio data from the vehicle and perform concurrent processing on the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features.

[0052] Specifically, through a concurrent processing flow, features of two different modalities—transcribed text and acoustic emotion—are extracted simultaneously from the same audio stream, achieving efficient and parallel perception of semantic information of audio content and speaker's emotional state.

[0053] It is evident that this method provides comprehensive and real-time foundational data for subsequent fusion analysis, avoiding the limitations of single-modal analysis.

[0054] 102. Based on the multimodal features and real-time order metadata, construct a multidimensional context vector, and use a lightweight classifier to calculate the risk probability of the multidimensional context vector. Only when the risk probability exceeds a preset threshold, trigger the generative artificial intelligence model to perform deep analysis.

[0055] Specifically, by fusing text features, acoustic emotion features, and real-time order metadata into a unified multi-dimensional context vector, and using a lightweight classifier for rapid risk assessment, a pre-filtering mechanism is implemented. This mechanism can perform real-time, low-latency preliminary risk assessment on continuously flowing audio data.

[0056] As can be seen, this design allows subsequent computationally intensive generative AI models to avoid processing all the data, and are only triggered when the scenario is initially identified as high-risk. This achieves on-demand allocation of computing resources at the system architecture level, meeting the requirements of high-concurrency real-time processing while providing a feasible basis for deep semantic analysis.

[0057] 103. In response to deep analysis triggers, the system inputs structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model and receives the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract.

[0058] Specifically, when the triggering conditions are met, structured instructions that integrate dialogue history and real-time status information are input into the generative artificial intelligence model, and the model is constrained to output structured data with a fixed format, so that deep semantic analysis can be carried out in a rich context and produce machine-readable and clear results.

[0059] As can be seen, this step ensures that the analysis results of the artificial intelligence model have a high degree of certainty and consistency, and its standardized data structure can be directly parsed and called by the downstream risk decision-making system automatically, thus opening up the key link from intelligent analysis to automatic handling.

[0060] 104. Before inputting the structured prompts into the generative artificial intelligence model, the personal privacy information in the structured prompts is masked and replaced.

[0061] Specifically, before data is fed into an external generative artificial intelligence model, the personal identity information in the input is forcibly desensitized, creating a barrier in the technical process to prevent the original sensitive data from being directly exposed to subsequent external computing stages.

[0062] It is evident that this approach isolates the risk of privacy breaches at the source, enabling the entire analysis process based on external artificial intelligence models to meet the basic compliance requirements of data security laws and regulations regarding the processing of personal information.

[0063] 105. Based on the structured risk assessment results, trigger automated intervention actions corresponding to the risk assessment level.

[0064] Specifically, by using standardized and structured risk assessment results as input, the system automatically maps and executes preset intervention strategies, achieving seamless integration and automatic driving from risk analysis to response actions.

[0065] It is evident that this method completes a closed loop from risk perception and intelligent assessment to final handling, enabling security protection to move away from purely manual reliance and achieve real-time, automated response based on event levels, thereby improving the efficiency of security operations.

[0066] As an optional embodiment, the step described above, receiving real-time streaming audio data from the vehicle and concurrently processing the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features, includes:

[0067] The streaming audio data is translated into time-stamped text segments in real time using a streaming speech recognition service.

[0068] In parallel, an acoustic sentiment analysis model is used to extract fundamental frequency, Mel-frequency cepstral coefficients, and energy features from the streaming audio data, and a normalized sentiment confidence score is output.

[0069] In this embodiment of the invention, through a concurrent processing mechanism, two types of features, namely textual semantics and acoustic emotion, are extracted simultaneously from the same audio stream, realizing multi-dimensional parallel perception of audio content. This provides comprehensive and real-time basic data input for subsequent comprehensive risk assessment, overcoming the limitations of single-modal analysis.

[0070] Specifically, this step involves two parallel processing branches. One branch converts the continuous audio stream into timestamped text segments in real time by calling a streaming speech recognition service based on a modern speech recognition architecture, ensuring that the temporal sequence of the dialogue content is preserved and providing raw materials for semantic analysis. The other branch uses a pre-trained acoustic sentiment analysis model to extract key physical features from the original audio signal. These features mainly include the fundamental frequency reflecting the pitch of the sound, the Mel-frequency cepstral coefficient representing the spectral characteristics of the sound, and the energy features representing the intensity of the sound. The model performs calculations and analyses based on these features and finally outputs a value between 0 and 1 to quantify the confidence level of the specific emotion carried by the current audio segment, such as an index of anger or fear.

[0071] It is evident that this parallel processing architecture can simultaneously produce textual records of dialogues and quantitative indicators of the speaker's emotional state within millisecond-level latency. Textual features serve to understand literal and metaphorical semantics, while acoustic emotional features reveal the speaker's emotional state from the perspective of paralinguistic cues. The combination of the two constructs a feature foundation for subsequent steps that is closer to real-world interaction scenarios and integrates "implied meaning" and "implied emotion," enabling the system to perform more refined and accurate risk context understanding, rather than relying solely on isolated text keywords.

[0072] As an optional embodiment, the step described above, namely constructing a multidimensional context vector based on the multimodal features and real-time order metadata, includes:

[0073] Using the session memory module, a sliding window context containing the text of the most recent N rounds of dialogue is maintained for each driving order;

[0074] The text embedding vector of the sliding window context, the emotion feature vector extended by acoustic emotion features, and the order metadata vector encoded by real-time order status information are concatenated to generate a unified multidimensional context vector. ,in For text embedding vectors, For emotion feature vectors, This is the order metadata vector.

[0075] In this embodiment of the invention, by constructing a unified multidimensional context vector, features from different sources and modalities are fused to provide comprehensive contextual information for risk assessment that goes beyond a single dialogue fragment. This elevates the analysis of isolated current statements to a coherent risk assessment that combines recent dialogue history, real-time emotional state, and dynamic travel scenarios, thereby enhancing the system's ability to perceive and understand complex and hidden risk patterns.

[0076] Specifically, the process first utilizes a session memory module to maintain a dynamically updated dialogue history for each independent order session. This history is stored in the form of a sliding window, saving the dialogue text between passengers and drivers in the most recent rounds. This allows the system to move away from isolated analysis of single sentences and instead understand them within a coherent dialogue sequence, thereby capturing semantics, referential relationships, and potential conflict escalation processes that vary depending on the context.

[0077] Based on this, the system generates a unified multidimensional feature representation, which is composed of three parts:

[0078] The first part is the text embedding vector. It uses a pre-trained language model to encode the context of the dialogue text within the sliding window into a high-dimensional dense vector, capturing its deep semantic information.

[0079] The second part is the emotion feature vector. It is derived from the acoustic emotion confidence score extracted in the steps and quantitatively represents the speaker's emotional state in the corresponding time period.

[0080] The third part is the order metadata vector. It encodes real-time order status information, such as whether a deviation has occurred or whether it is a late-night period, into machine-readable feature vectors.

[0081] Final fusion vector From the formula Generate, where symbols This represents the vector concatenation operation, which is mathematically a chaining process that connects three vectors end-to-end in a dimension to form a longer new vector that contains all the information sources.

[0082] It is evident that this construction method creates a three-dimensional feature representation that includes semantics, emotion, and real-time context, enabling the subsequent lightweight classifier to make decisions based on a complete context. For example, the system can identify the potential risk that is far greater than any single dimension when the text content seems ordinary but the emotion vector shows high tension and the metadata vector indicates that the vehicle is deviating. This solves the problem of misjudgment and missed judgment caused by the lack of multi-dimensional context fusion in traditional methods.

[0083] As an optional embodiment, the real-time order status information in the above steps includes the deviation status determined based on the vehicle's real-time geographical location, the status during late-night hours determined based on timestamps, and the driver's historical complaint rate.

[0084] In this embodiment of the invention, the defined real-time order status information, by integrating spatiotemporal scenarios and historical behavior data, injects a dynamic and personalized contextual dimension into risk assessment, enabling the system to go beyond isolated analysis of the current audio segment itself, and to comprehensively assess interactive behavior within a specific physical environment, temporal context, and historical pattern, thereby improving the accuracy of risk screening and contextual awareness.

[0085] Specifically, these status information profiles depict the real-time risk profile of the trip from different perspectives. The deviation status is calculated by comparing the GPS location information reported by the vehicle in real time with the navigation planned path or the conventionally defined standard path. When the vehicle trajectory deviates from the preset path for a certain period of time, this status is triggered. This is usually highly related to potential safety hazards or service disputes. The late-night status is determined by comparing the current timestamp with the preset late-night time interval (e.g., 11 PM to 5 AM the next day). This period is generally considered to be a period with relatively higher safety risks due to factors such as low visibility and low traffic. The driver's historical complaint rate is statistical data extracted from the driver's file. It is usually calculated as the proportion of valid complaint orders received in a certain period of time to the total number of completed orders. It quantifies the probability that the driver will cause service conflicts or safety problems in advance.

[0086] As can be seen, by encoding three types of information—vehicle deviation status, late-night time status, and driver historical complaint rate—into order metadata vectors, the system can simultaneously consider environmental anomalies, time risk factors, and the inherent risk tendency of the target during initial risk screening. For example, even if the current conversation sounds normal, if the three conditions of vehicle deviation, late-night time, and high driver historical complaint rate are met simultaneously, the lightweight classifier is more likely to calculate a high-risk probability, thus triggering further in-depth analysis. This mechanism ensures that the system can identify complex scenarios that appear normal under pure audio analysis but are actually high-risk in the overall context, allowing resources to be allocated to situations that truly require in-depth analysis, thus optimizing the overall efficiency of computing resource allocation and the comprehensiveness of risk coverage.

[0087] As an optional embodiment, the step described above, of inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model and receiving the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract, includes:

[0088] Retrieve recent conversation context from session memory;

[0089] The recent dialogue context, the currently extracted acoustic emotion features, and risk metadata including yaw status are assembled into structured prompt words that conform to a preset template.

[0090] The structured cue words are input into a generative artificial intelligence model, and the output parser forces the model to return structured JSON data containing risk category enumeration values, risk score integers, and key evidence strings.

[0091] In this embodiment of the invention, a highly structured and controllable data exchange mechanism drives a generative artificial intelligence model to complete complex risk assessments, and transforms the model's open-domain natural language capabilities into deterministic signals that can be directly utilized by downstream systems. This solves key problems such as uncontrollable output results, inconsistent formats, and difficulty in integrating with automated systems in actual industrial deployments of generative models, reliably transforming advanced semantic understanding capabilities into structured and operable risk assessments.

[0092] Specifically, the process begins by retrieving the text history of recent conversations from the session memory, providing the model with a coherent context rather than isolated sentences. Subsequently, the system assembles this text history, the acoustic sentiment quantification score corresponding to the current time slice, and key risk metadata, including yaw status, into a structured prompt word according to a preset, carefully designed instruction template. This prompt word not only contains the data to be analyzed but also clearly defines the task, background, and requirements of the analysis through the template. For example, it instructs the model to "act as a security auditor and conduct an assessment based on the following conversation, user sentiment, and trip status."

[0093] The most crucial operation lies in the mandatory constraints on the model's output. The system uses an output parser to require the generative AI model to organize its responses strictly according to a predefined JSON Schema contract. This contract explicitly stipulates that the output must be a JSON object and must contain several fixed fields: an enumerated value representing the risk category, a risk score integer representing the degree of risk, and a string of key evidence extracted from the input context to support the judgment.

[0094] It is evident that this end-to-end design, from structured input to structured output, achieves a dual objective. On the one hand, structured prompts inject precise domain knowledge and multi-dimensional context into the model, guiding it to conduct professional and focused analysis, thereby improving the relevance and accuracy of judgments. On the other hand, the mandatory structured output completely eliminates the randomness and openness of the model's responses, ensuring that each return is data with a unified format, clearly defined fields, and direct parsing and consumption by the program. This enables the downstream risk management module to seamlessly and stably trigger corresponding automated actions based on the risk categories and scores output by the model, completing the key transformation from intelligent analysis to automated execution, and truly realizing the industrial application of large language models in rigorous risk control processes.

[0095] As an optional embodiment, the step of masking and replacing the personal privacy information in the structured prompt words in the above steps includes:

[0096] By using a named entity recognition model and a regular expression pipeline, consecutive phone numbers and specific place names in text are identified and replaced with predefined mask labels, respectively. and .

[0097] In this embodiment of the invention, before the streaming audio data is sent to an external generative artificial intelligence model for in-depth analysis after real-time feature extraction, context construction, and initial risk screening, a deterministic privacy filtering layer is inserted. Through automated technical means, while giving full play to the semantic analysis capabilities of the external large model, the risk of leakage of users' sensitive personal information is fundamentally eliminated, achieving a balance between data processing utility and privacy security compliance.

[0098] Specifically, this step scans and cleans the text content that will form the prompt word, employing a hybrid recognition strategy: on the one hand, it uses a pre-trained named entity recognition model to identify specific place names with semantic meaning in the text, such as city names, business district names, or landmark building names; on the other hand, it uses regular expressions with specific patterns to accurately match consecutive digit phone numbers appearing in the text. When these predefined privacy entities are detected, the system does not simply delete them, but instead uses predefined, meaningless mask labels for in-situ replacement. For example, a dialogue containing "I'll be at the North Gate of People's Square in a bit, my phone number is 1380013xxxx" is replaced with "I'll be at..." The phone number is ".

[0099] It is evident that this processing method achieves multiple objectives: it severs the path from the source of the original personally identifiable information to the external computing environment, meeting the mandatory requirements of data security regulations for the "de-identification" of personal information; secondly, the mask replacement method preserves the syntactic structure and contextual coherence of the text, which is beneficial for generative artificial intelligence models. and These placeholders indicate the presence of an entity and its category, allowing the model to understand the semantic logic of the conversation (e.g., discussing meeting at a certain location), but without obtaining the actual location and number. This ensures that the model's understanding ability is not compromised while protecting privacy. The process is fully automated and embedded in the data processing pipeline, requiring no human intervention. This ensures efficient and standardized privacy processing and provides a compliance foundation for the entire external AI-based security audit solution.

[0100] As an optional embodiment, the step described above, triggering an automated intervention action corresponding to the risk assessment level based on the structured risk assessment result, includes:

[0101] If the risk assessment level is low or medium risk, a text warning message will be sent to the driver's vehicle terminal via message queue telemetry transmission protocol and then broadcast to the driver's voice.

[0102] If the risk assessment level is high, the monitoring level of the corresponding order will be automatically upgraded, forced frame extraction of the in-vehicle video will be enabled, and the warning information containing the highlighted text will be pushed to the safety dispatch center in real time.

[0103] In this embodiment of the invention, this step completes the execution loop from intelligent risk analysis to automated security intervention. Based on standardized and structured risk assessment results, it automatically executes preset and differentiated response strategies, transforming intelligent analysis capabilities into security protection actions that can take effect immediately. This significantly shortens the response time from risk identification to platform intervention, and realizes the automation and intelligent upgrade of security operation and maintenance.

[0104] Specifically, this step executes a tiered handling process based on the risk level field in the structured assessment results output by the generative artificial intelligence model. For low-risk or medium-risk assessments, the system sends an encrypted instruction to the driver's in-vehicle infotainment system via a message queue telemetry transmission protocol. Upon receiving the instruction, the in-vehicle infotainment system will broadcast the text content in real time using speech synthesis technology, such as playing warnings like "Please pay attention to driving regulations and communication tone," aiming to remind and correct the driver's behavior. This method of intervention is direct, fast, and has minimal disruption to the trip.

[0105] For high-risk assessments, the system triggers a series of upgraded composite responses. First, the monitoring level of the order is automatically upgraded from normal to high in the background. This means that the system will allocate more computing and storage resources to intensively monitor the subsequent data of the order. Next, the system sends instructions to the audio and video equipment in the vehicle to enable the high-frequency frame-sampling upload function of the video stream, preserving key visual evidence for possible subsequent investigations. At the same time, a detailed warning message integrating risk category, risk score, time, location, and highlighted text of key evidence will be pushed to the work interface of the human safety dispatch center in real time, enabling the safety officer to grasp the complete context of the high-risk order as soon as possible and quickly decide whether to take further measures such as human voice intervention or contacting the police.

[0106] It is evident that this automated intervention mechanism based on clear rules establishes a scalable response system that matches the risk level. It not only enables immediate and automated handling of low- to medium-risk situations, reducing the pressure on manual monitoring, but more importantly, it constructs an efficient handling process for truly high-risk security incidents, characterized by "automatic system discovery, immediate evidence preservation, and real-time information synchronization." This transforms passive post-incident tracing into proactive in-incident intervention and protection, improving the timeliness and effectiveness of overall security protection.

[0107] As an optional embodiment, in the step above, in the step of inputting structured prompts into a preset generative artificial intelligence model, the inference temperature parameter of the generative artificial intelligence model is set to an extremely low value close to 0.

[0108] In this embodiment of the invention, by specifically configuring a key internal parameter of the generative artificial intelligence model, its output behavior in the serious scenario of security auditing is optimized. Specifically, the inference temperature parameter of the model is set to an extremely low value that is infinitely close to zero. This operation, in terms of the mathematical mechanism inside the model, will make it almost always select the one with the highest predicted probability from all possible choices when generating each subsequent word, while almost completely ignoring other options with lower probabilities.

[0109] This configuration greatly suppresses the inherent randomness of generative models. With fixed structured prompts as input, the output generated by the model each time it runs will have high consistency, determinism and repeatability. For risk assessment tasks, this means that after the same risk scenario is analyzed by the model, it will produce the same or extremely similar structured judgment results each time, including completely consistent risk category enumeration values ​​and highly similar risk scores.

[0110] It is evident that this parameter setting fundamentally ensures the stability and reliability of the risk assessment results. In automated safety systems, subsequent intervention decisions and actions heavily rely on the output of this risk assessment. If the model output is random due to a high temperature parameter, it may lead to the same dialogue being judged as high-risk at times and low-risk at others, making the entire automated handling chain unpredictable and unreliable. By setting the temperature close to zero, it ensures that the generative artificial intelligence model behaves like a deterministic function in this process, with stable and predictable output. This meets the stringent requirements of industrial applications for the stability of core judgment modules, enabling the deep analysis based on generative models to be seamlessly integrated into the automated risk management workflow in a reliable and trustworthy manner. This is a crucial technical guarantee for the entire system to be put into practical operation.

[0111] Furthermore, the following is an application scenario example demonstrating the complete process from audio stream input to automatic system intervention:

[0112] Scene background:

[0113] Time: 1:30 AM.

[0114] Location: A city.

[0115] Trip: Passenger A placed an order through a ride-hailing platform, and driver B accepted the order. After the trip started, the vehicle gradually deviated from the navigation route and drove towards a more remote area.

[0116] Example of the entire system processing flow:

[0117] Step 1: Real-time Feature Extraction

[0118] The in-car audio system continuously recorded the conversation. Passenger A said, "Driver, did we take the wrong road? The navigation doesn't seem to take this route." Driver B replied in a slightly nervous tone, "Hey, that route on the navigation is congested at night. I know this route well; it's faster and saves fuel."

[0119] The system processes this audio segment concurrently:

[0120] 1. The streaming speech recognition service translates it into the above text in real time and adds a precise timestamp to each sentence.

[0121] 2. Meanwhile, the acoustic sentiment analysis model extracts acoustic features from driver B's speech segments and calculates a high tension index (e.g., a tension index of 0.8).

[0122] This step produces time-series dialogue text and quantified sentiment data in parallel.

[0123] Step 2: Context building and initial risk screening triggering

[0124] The system synchronously obtains the real-time metadata of the order and confirms the current status as follows: the vehicle has deviated 2 kilometers, it is late at night, and driver B has a moderate historical complaint rate.

[0125] Next, the system uses session memory to obtain recent dialogue context and concatenates the current text embedding vector, driver sentiment vector, and order metadata vector encoding information such as deviation, late night, and complaint rate into a unified multi-dimensional feature vector. This vector is then input into a lightweight classifier (such as XGBoost) for real-time computation.

[0126] The calculation results show that the risk probability of the current segment is 0.76, which exceeds the preset trigger threshold (e.g., 0.6). Therefore, the system determines that this scenario is suspected to be high-risk and immediately triggers the generative artificial intelligence model to perform in-depth analysis.

[0127] Step 3: Deep Reasoning and Privacy Desensitization

[0128] Before calling the large model, the system scanned and desensitized the text used to assemble the prompt words (no specific phone number or address appeared in the dialogue in this example, so the original text was approved).

[0129] The system retrieves dialogue history from memory, the current sentiment score, and key metadata about the "vehicle has deviated" status. Following a pre-defined template, it assembles this into a structured prompt, explicitly instructing the model to analyze the deviation. This prompt is then fed into a generative artificial intelligence model.

[0130] Under the constraint of low inference temperature parameters, the model outputs structured results that strictly conform to the predetermined JSON format:

[0131] {

[0132] "risk_category": "Inducing offline transactions",

[0133] "risk_score": 82,

[0134] “reasoning_evidence”: “When the driver was deviating from the route late at night, he explained that he was taking a detour by saying that it would save fuel. Combined with his nervous tone, this is consistent with a common sales pitch used to induce passengers to accept transactions outside the platform.”

[0135] }

[0136] Step 4: Automated Tiered Intervention

[0137] Upon receiving the risk assessment results in the aforementioned JSON format, the risk decision engine automatically executes a pre-defined high-risk intervention strategy based on the high-risk level (risk_score: 82) and the specific category (inducing offline transactions).

[0138] 1. Automatically upgrade monitoring level: Mark the monitoring status of the order as "high risk" in the background and allocate more resources for tracking.

[0139] 2. Enable video evidence preservation: Send a command to the vehicle to turn on the in-vehicle camera and perform high-frequency video frame extraction and uploading.

[0140] 3. Real-time push of manual warnings: A warning message containing key evidence text highlighted (such as "fuel-saving", "I know it well"), risk score, specific time, real-time vehicle location and deviation trajectory is pushed to the monitoring screen of the platform's safety dispatch center in real time.

[0141] Step 5: Security specialist intervenes

[0142] The on-duty specialist at the safety dispatch center immediately received a pop-up alert on the screen. The specialist could quickly browse the complete analysis context (dialogue records, sentiment analysis, vehicle trajectory) and then use the system to call the driver in real time for questioning and warnings. If necessary, the specialist could also contact the police.

[0143] This invention efficiently extracts multimodal features through concurrent processing, and then innovatively integrates voice, emotion, and real-time order context for rapid initial screening. This ensures that most low-risk dialogues do not consume a lot of computing power, solving the performance bottleneck under high concurrency. For the few high-risk scenarios that are triggered, a large model is used to perform deep semantic analysis in rich context, accurately identifying metaphorical inducements such as "saving fuel". Finally, the system automatically executes graded intervention based on the structured judgment results, forming a second-level closed loop from risk perception, intelligent analysis to automatic handling, improving the efficiency of safe operation.

[0144] Example 2

[0145] Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a context-aware risk intervention system for streaming audio disclosed in an embodiment of the present invention. Figure 2 The described context-aware risk intervention system for streaming audio can be applied to data processing chips, processing terminals, or processing servers, and the processing server can be a local server or a cloud server; this embodiment of the invention does not limit the application. Figure 2 As shown, this context-aware risk intervention system for streaming audio may include the following operations:

[0146] The feature extraction module 201 is used to receive real-time streaming audio data from the vehicle and perform concurrent processing on the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features.

[0147] Specifically, through a concurrent processing flow, features of two different modalities—transcribed text and acoustic emotion—are extracted simultaneously from the same audio stream, achieving efficient and parallel perception of semantic information of audio content and speaker's emotional state.

[0148] It is evident that this method provides comprehensive and real-time foundational data for subsequent fusion analysis, avoiding the limitations of single-modal analysis.

[0149] The analysis trigger module 202 is used to construct a multi-dimensional context vector based on the multimodal features and real-time order metadata, and use a lightweight classifier to calculate the risk probability of the multi-dimensional context vector. Only when the risk probability exceeds a preset threshold, the generative artificial intelligence model is triggered to perform deep analysis.

[0150] Specifically, by fusing text features, acoustic emotion features, and real-time order metadata into a unified multi-dimensional context vector, and using a lightweight classifier for rapid risk assessment, a pre-filtering mechanism is implemented. This mechanism can perform real-time, low-latency preliminary risk assessment on continuously flowing audio data.

[0151] As can be seen, this design allows subsequent computationally intensive generative AI models to avoid processing all the data, and are only triggered when the scenario is initially identified as high-risk. This achieves on-demand allocation of computing resources at the system architecture level, meeting the requirements of high-concurrency real-time processing while providing a feasible basis for deep semantic analysis.

[0152] The risk reasoning module 203 is used to respond to deep analysis triggers by inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model, and receiving the structured risk assessment results output by the model that conform to a predetermined JSONSchema contract.

[0153] Specifically, when the triggering conditions are met, structured instructions that integrate dialogue history and real-time status information are input into the generative artificial intelligence model, and the model is constrained to output structured data with a fixed format, so that deep semantic analysis can be carried out in a rich context and produce machine-readable and clear results.

[0154] As can be seen, this step ensures that the analysis results of the artificial intelligence model have a high degree of certainty and consistency, and its standardized data structure can be directly parsed and called by the downstream risk decision-making system automatically, thus opening up the key link from intelligent analysis to automatic handling.

[0155] The processing module 204 is used to perform masking and replacement processing on the personal privacy information in the structured prompts before the generative artificial intelligence risk reasoning module performs reasoning.

[0156] Specifically, before data is fed into an external generative artificial intelligence model, the personal identity information in the input is forcibly desensitized, creating a barrier in the technical process to prevent the original sensitive data from being directly exposed to subsequent external computing stages.

[0157] It is evident that this approach isolates the risk of privacy breaches at the source, enabling the entire analysis process based on external artificial intelligence models to meet the basic compliance requirements of data security laws and regulations regarding the processing of personal information.

[0158] The execution module 205 is used to trigger automated intervention actions corresponding to the risk assessment level based on the structured risk assessment results.

[0159] Specifically, by using standardized and structured risk assessment results as input, the system automatically maps and executes preset intervention strategies, achieving seamless integration and automatic driving from risk analysis to response actions.

[0160] It is evident that this method completes a closed loop from risk perception and intelligent assessment to final handling, enabling security protection to move away from purely manual reliance and achieve real-time, automated response based on event levels, thereby improving the efficiency of security operations.

[0161] Example 3

[0162] Please see Figure 3 , Figure 3 This is a schematic diagram of another context-aware risk intervention system for streaming audio disclosed in an embodiment of the present invention. Figure 3 As shown, the device may include:

[0163] Memory 301 storing executable program code;

[0164] Processor 302 coupled to memory 301;

[0165] The processor 302 calls the executable program code stored in the memory 301 to execute some or all of the steps in the context-aware risk intervention method for streaming audio disclosed in Embodiment 1 of the present invention.

[0166] Example 4

[0167] This invention discloses a computer storage medium storing computer instructions. When these computer instructions are invoked, they are used to execute some or all of the steps in a context-aware risk intervention method for streaming audio disclosed in Embodiment 1 of this invention.

[0168] Example 5

[0169] This invention discloses a computer program product comprising a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform the steps of a context-aware risk intervention method for streaming audio as described in Embodiment 1.

[0170] The system embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0171] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0172] Finally, it should be noted that the context-aware risk intervention method and system for streaming audio disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A context-aware risk intervention method for streaming audio, characterized in that, The method includes: Receive real-time streaming audio data from the vehicle and process the streaming audio data concurrently to extract multimodal features, including translated text features and acoustic emotion features. Based on the multimodal features and real-time order metadata, a multidimensional context vector is constructed, and a lightweight classifier is used to calculate the risk probability of the multidimensional context vector. Only when the risk probability exceeds a preset threshold is the generative artificial intelligence model triggered to perform deep analysis. In response to deep analysis, the system inputs structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model and receives the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract. Before inputting the structured prompts into the generative artificial intelligence model, the personal privacy information in the structured prompts is masked and replaced. Based on the structured risk assessment results, trigger automated intervention actions corresponding to the risk assessment level.

2. The context-aware risk intervention method for streaming audio according to claim 1, characterized in that, The process involves receiving real-time streaming audio data from the vehicle and concurrently processing the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features, including: The streaming audio data is translated into time-stamped text segments in real time using a streaming speech recognition service. In parallel, an acoustic sentiment analysis model is used to extract fundamental frequency, Mel-frequency cepstral coefficients, and energy features from the streaming audio data, and a normalized sentiment confidence score is output.

3. The context-aware risk intervention method for streaming audio according to claim 1, characterized in that, The construction of a multidimensional context vector based on the multimodal features and real-time order metadata includes: Using the session memory module, a sliding window context containing the text of the most recent N rounds of dialogue is maintained for each driving order; The text embedding vector of the sliding window context, the emotion feature vector extended by acoustic emotion features, and the order metadata vector encoded by real-time order status information are concatenated to generate a unified multidimensional context vector. ,in For text embedding vectors, For emotion feature vectors, This is the order metadata vector.

4. The context-aware risk intervention method for streaming audio according to claim 3, characterized in that, The real-time order status information includes deviation status determined based on the vehicle's real-time geographical location, status during late-night hours determined based on timestamps, and the driver's historical complaint rate.

5. A context-aware risk intervention method for streaming audio according to claim 1, characterized in that, The process of inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model and receiving the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract includes: Retrieve recent conversation context from session memory; The recent dialogue context, the currently extracted acoustic emotion features, and risk metadata including yaw status are assembled into structured prompt words that conform to a preset template. The structured cue words are input into a generative artificial intelligence model, and the output parser forces the model to return structured JSON data containing risk category enumeration values, risk score integers, and key evidence strings.

6. The context-aware risk intervention method for streaming audio according to claim 1, characterized in that, The process of masking and replacing the personal privacy information in the structured prompts includes: By using a named entity recognition model and a regular expression pipeline, consecutive phone numbers and specific place names in text are identified and replaced with predefined mask labels, respectively. and .

7. A context-aware risk intervention method for streaming audio according to claim 1, characterized in that, The step of triggering automated intervention actions corresponding to the risk assessment level based on the structured risk assessment results includes: If the risk assessment level is low or medium risk, a text warning message will be sent to the driver's vehicle terminal via message queue telemetry transmission protocol and then broadcast to the driver's voice. If the risk assessment level is high, the monitoring level of the corresponding order will be automatically upgraded, forced frame extraction of the in-vehicle video will be enabled, and the warning information containing the highlighted text will be pushed to the safety dispatch center in real time.

8. A context-aware risk intervention method for streaming audio according to claim 1, characterized in that, In the step of inputting structured prompts into a preset generative artificial intelligence model, the inference temperature parameter of the generative artificial intelligence model is set to an extremely low value close to 0.

9. A context-aware risk intervention system for streaming audio, used to implement the context-aware risk intervention method for streaming audio as described in any one of claims 1-8, characterized in that, The system includes: The feature extraction module is used to receive real-time streaming audio data from the vehicle and perform concurrent processing on the streaming audio data to extract multimodal features, including translated text features and acoustic emotion features. The analysis trigger module is used to construct a multi-dimensional context vector based on the multimodal features and real-time order metadata, and to use a lightweight classifier to calculate the risk probability of the multi-dimensional context vector. Only when the risk probability exceeds a preset threshold is the generative artificial intelligence model triggered to perform deep analysis. The risk reasoning module is used to respond to deep analysis triggers by inputting structured prompts containing dialogue context and risk metadata into a preset generative artificial intelligence model, and receiving the structured risk assessment results output by the model that conform to a predetermined JSON Schema contract. The processing module is used to perform masking and replacement processing on the personal privacy information in the structured prompts before the generative artificial intelligence risk reasoning module performs reasoning. The execution module is used to trigger automated intervention actions corresponding to the risk assessment level based on the structured risk assessment results.

10. A context-aware risk intervention system for streaming audio, characterized in that, The system includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute a context-aware risk intervention method for streaming audio as described in any one of claims 1-8.

Images