Intelligent shunting method and device based on multi-modal information processing, equipment and medium

Abstract

The invention relates to the technical field of intelligent decision making, can be applied to business scenes of financial science and technology, medical treatment and health and the like, and discloses an intelligent shunting method, device, equipment and medium based on multi-modal information processing. And completing living body identity verification based on a three-dimensional structure and dynamic actions, identifying acoustic emotions and semantic emotions, fusing the acoustic emotions and the semantic emotions with interactive behaviors to generate a comprehensive emotional state, generating an access decision in combination with task complexity, executing an automatic service or a seat service, and providing a corresponding service response. According to the method, multi-modal information is formed by fusing vision, voice and interaction behaviors, user emotion and task complexity are obtained on the premise that living body safety verification is guaranteed, accurate access decision making is achieved, and therefore the intelligent level of service distribution and user experience are improved while identity verification safety and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of intelligent decision-making technology, and in particular to an intelligent traffic diversion method, apparatus, device, and medium based on multimodal information processing. Background Technology

[0002] In the fintech business sector, existing intelligent customer service systems generally rely on speech recognition and natural language processing technologies to complete business inquiries, identity verification, and process guidance. However, in actual service, they still face problems such as cumbersome identity verification methods, insufficient liveness detection, and weak emotion understanding capabilities. Identity verification still mainly relies on inputting static data such as ID numbers, account information, or policy information. Users on mobile devices need to frequently switch between different interfaces, which easily leads to input errors or interruptions, resulting in low interaction efficiency. At the same time, the two-dimensional image liveness detection methods relied upon by mainstream systems have limited protection against attacks such as photo re-capture, video replay, or screen simulation, and cannot meet the high requirements for identity security in financial service scenarios. In addition, existing customer service systems mainly remain at the semantic understanding level in terms of emotion recognition and user state analysis. They cannot effectively recognize emotional fluctuations in speech, nor can they judge the user's true emotional state based on the user's interaction behavior. This can easily lead to responses that are inconsistent with the user's emotions in high-pressure or high-risk scenarios, thus reducing the service experience.

[0003] In the healthcare sector, intelligent consultation systems also need to identify users, understand their needs, and provide treatment guidance or health advice. However, existing technologies suffer from similar shortcomings in terms of interaction security and emotion understanding. Identity verification typically relies on patients providing personal information, a complex process whose accuracy is heavily influenced by human factors. In telemedicine or online consultations, two-dimensional image-based liveness detection methods are ill-equipped to defend against deepfakes and face replacements, potentially leading to misidentification. Furthermore, users may experience significant emotions such as anxiety, tension, or worry during medical consultations, but existing systems lack multimodal emotion recognition mechanisms. This prevents the system from adjusting its interaction style based on emotional states, impacting user trust and acceptance. Simultaneously, the lack of a comprehensive judgment mechanism based on user emotional states and semantic complexity often hinders the system from guiding users into human assistance at appropriate times, affecting consultation efficiency and service continuity. Summary of the Invention

[0004] The main objective of this invention is to provide an intelligent traffic splitting method, apparatus, device, and storage medium based on multimodal information processing. This invention aims to solve the technical problems of existing technologies being unable to simultaneously achieve secure and reliable liveness verification, accurate emotional state recognition, and dynamic access decision-making based on multimodal information, resulting in low identity verification efficiency, insufficient security, and unintelligent service splitting.

[0005] To achieve the above objectives, the present invention provides an intelligent traffic splitting method based on multimodal information processing, comprising: Collect users' multimodal interaction information, which includes visual information, voice information, and interaction behavior information; Extract facial 3D structure information and dynamic facial movement information from the visual information; The user's liveness status is determined based on the dynamic facial movement information, and if the user is determined to be live, the facial 3D structure information is compared with a preset feature template to complete the identity verification. Acoustic emotional state is identified based on the speech information, and semantic emotional state is identified based on the semantic text information converted from the speech information. The acoustic emotional state, the semantic emotional state, and the interaction behavior information are fused to generate a comprehensive emotional state. The task complexity is analyzed based on the semantic text information, and the access decision is generated by combining the task complexity with the comprehensive emotional state. Based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process.

[0006] Furthermore, to achieve the above objectives, the present invention provides an intelligent traffic diversion device based on multimodal information processing, comprising: A multimodal acquisition module is used to collect multimodal interaction information of users, including visual information, voice information, and interaction behavior information; The face structure and motion analysis module is used to extract three-dimensional facial structure information and dynamic facial motion information from the visual information. The liveness and identity verification module is used to determine the user's liveness status based on the dynamic facial movement information, and, if the user is determined to be alive, to compare the facial three-dimensional structure information with a preset feature template to complete the identity verification. The emotion state fusion module is used to identify acoustic emotion state based on the speech information, and to identify semantic emotion state based on semantic text information converted from the speech information, and to fuse the acoustic emotion state, the semantic emotion state and the interaction behavior information to generate a comprehensive emotion state. The access decision generation module is used to parse the task complexity based on the semantic text information and integrate the task complexity with the comprehensive emotional state to generate an access decision. The service execution and switching module is used to execute automated services or switch to agent services based on the access decision, and to provide corresponding service responses during or after the switching process.

[0007] Furthermore, to achieve the above objectives, the present invention also provides a computer device, the computer device including a memory, a processor, and an intelligent traffic splitting program based on multimodal information processing stored in the memory and executable on the processor, wherein when the intelligent traffic splitting program based on multimodal information processing is executed by the processor, it implements the steps of the intelligent traffic splitting method based on multimodal information processing as described above.

[0008] Furthermore, to achieve the above objectives, the present invention also provides a non-volatile computer-readable storage medium storing an intelligent traffic splitting program based on multimodal information processing, wherein the intelligent traffic splitting program based on multimodal information processing, when executed by a processor, implements the steps of the intelligent traffic splitting method based on multimodal information processing as described above.

[0009] Beneficial Effects: This invention relates to the field of intelligent decision-making technology and can be applied to business scenarios such as fintech and healthcare. It discloses an intelligent traffic routing method, device, equipment, and medium based on multimodal information processing, including: collecting multimodal information composed of visual, voice, and interactive behavior; completing liveness authentication based on three-dimensional structure and dynamic actions; identifying acoustic and semantic emotions and fusing interactive behavior to generate a comprehensive emotional state; generating an access decision based on task complexity; executing automated or agent services and providing corresponding service responses. This invention, by fusing multimodal information formed by visual, voice, and interactive behavior, obtains user emotions and task complexity while ensuring secure liveness authentication, achieving accurate access decisions. This improves the security and efficiency of identity verification while enhancing the intelligence level of service routing and user experience. Attached Figure Description

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a schematic diagram of an application environment for an intelligent traffic splitting method based on multimodal information processing according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an embodiment of the intelligent traffic splitting method based on multimodal information processing according to the present invention. Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the intelligent diversion device based on multimodal information processing of the present invention; Figure 4 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; Figure 5 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0011] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0012] The intelligent traffic splitting method based on multimodal information processing provided in this invention can be applied to applications such as... Figure 1 In this application environment, the client communicates with the server via a network. The server can collect multimodal information composed of visual, voice, and interactive behaviors from the client to complete liveness authentication based on three-dimensional structure and dynamic actions, identify acoustic and semantic emotions, and integrate interactive behaviors to generate a comprehensive emotional state. It then combines this with task complexity to generate an access decision, execute automated or agent-based services, and provide corresponding service responses. This invention, by integrating multimodal information formed by visual, voice, and interactive behaviors, obtains user emotions and task complexity while ensuring secure liveness authentication, achieving accurate access decisions. This improves the security and efficiency of identity verification while enhancing the intelligence level of service routing and user experience. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented using a standalone server or a server cluster consisting of multiple servers. The following detailed description of specific embodiments further illustrates this invention.

[0013] Please see Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the intelligent traffic splitting method based on multimodal information processing provided by the present invention. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0014] like Figure 2 As shown, the intelligent traffic splitting method based on multimodal information processing proposed in this invention includes the following steps: S10, Collect the user's multimodal interaction information, which includes visual information, voice information, and interaction behavior information; In this embodiment, the visual information comes from optical imaging of the user's facial region and upper body posture, constructed through two-dimensional images or three-dimensional data containing depth structure. The process of acquiring visual information relies on camera imaging equipment, which can be a camera with infrared depth measurement capabilities or an acquisition module with structured light or binocular parallax calculation capabilities. The visual information may include facial texture, key point motion trajectories, contour deformation, head posture changes, and background brightness changes, etc., and is constructed as a temporally correlated image sequence using consecutive frames. During visual information acquisition, exposure adjustment, illumination adaptation, and image distortion correction need to be performed to ensure the data meets the alignment requirements for subsequent feature extraction.

[0015] Speech information originates from the user's verbal expressions during interaction and is received by a microphone acquisition device. The speech information includes the original sound wave signal, sound pressure changes, spectral energy distribution, and multi-channel spatial sound field differences, which are converted into digital waveforms using an analog-to-digital converter. Speech information acquisition requires sampling rate control, noise suppression preprocessing, and echo cancellation to ensure the signal possesses distinguishable features suitable for acoustic state recognition, speech content analysis, and vocalization mode detection. In subsequent processing, the speech information can be used as input for acoustic feature extraction to construct frequency distribution maps or Mel-scale feature matrices.

[0016] Interactive behavior information originates from user actions within the interface environment, forming recordable interaction trajectories through touchscreens, buttons, mice, or other input devices. This information includes touch position changes, click frequency, pause duration, input speed, window switching trajectories, and page scrolling speed. Collecting interactive behavior information requires obtaining the raw behavioral event stream from the operating system input layer and appending structured content such as timestamps, event type numbers, and interface element identifiers. This imbues the data with behavioral characteristics suitable for subsequent emotional feature inference or task state understanding.

[0017] To avoid time shifts between different modalities during multimodal interaction information acquisition, time alignment is required. This includes attaching a unified time identifier to the visual frame sequence, audio frame sequence, and behavioral event stream based on the system clock, and using interpolation, frame locking, or event completion mechanisms to ensure they form comparable information groups within the same time window. After alignment, each segment of visual, audio, and interactive behavioral information can be grouped into a context-sensitive data fragment for subsequent state analysis or authentication processing by window segmentation.

[0018] This embodiment, by simultaneously collecting visual, voice, and interactive behavior information, can construct a multi-source data foundation for user status from different dimensions, enabling higher accuracy and stability in identity verification, emotion recognition, and task status understanding. The multimodal data is organized into structured fragments within a unified time window, which is beneficial for subsequent liveness detection, emotion state inference, and access strategy judgment, significantly improving the overall security performance and response consistency of the interactive system.

[0019] S20, extract facial three-dimensional structure information and dynamic facial movement information from the visual information; In this embodiment, achieving face recognition from visual information requires performing a face detection operation to locate facial regions in an image frame and generate bounding boxes containing coordinate information. Face detection can be based on a feature pyramid structure, a deep convolutional network, or a lightweight keypoint network, but the explanation of the structure is not limited to a specific network; rather, it emphasizes the method of calculating facial positions in the image region. After detection, facial keypoints are located based on the detected regions, including the two-dimensional coordinates of positions such as the corners of the eyes, brow peaks, nose tip, and corners of the mouth. The results of keypoint localization constitute the basic reference for the facial structure, used for subsequent geometric shape reconstruction.

[0020] Facial 3D structural information originates from the geometric inference process of keypoint coordinates and depth data. If the visual information includes depth maps or structured light projection results, a dense point cloud can be formed by combining depth pixels with keypoints. If the visual information only contains two-dimensional image frames, a 3D point set can be obtained using binocular parallax structures or inferential reconstruction methods. After the point cloud is formed, spatial optimization is required, including noise point removal, point cloud densification, and surface reconstruction, to form facial 3D structural information suitable for subsequent identity verification and motion analysis. This 3D structural information includes three-dimensional coordinates, facial contours, and information on local facial protrusions or depressions, serving as an important reference for user characteristics.

[0021] Dynamic facial motion information is derived from the coordinate changes of keypoints between consecutive frames. Motion trajectories are obtained by tracking the positional changes of keypoints across different frames. The movement of each keypoint over time constitutes the basic expression of facial movements, manifested as dynamic changes such as raised eyebrows, lowered corners of the mouth, closed eyes, and head rotation. To achieve more refined motion descriptions, motion smoothing can be applied to the trajectory to reduce the impact of image noise and momentary jitter. Furthermore, a motion coding method based on a facial muscle physiological model is introduced to convert the motion trajectory into motion parameters representing the direction and amplitude of muscle pulling. This conversion method aids in subsequent behavior judgment and exhibits higher representational stability in emotion recognition and liveness detection.

[0022] The process of extracting facial 3D structure information and dynamic facial motion information from visual information is continuous. It is necessary to ensure that the input of each calculation step comes from the processed results. For example, key point coordinates come from face detection, motion trajectory comes from key point tracking, and 3D structure comes from the fusion of key point and depth data.

[0023] This embodiment extracts facial 3D structure information and dynamic facial movement information from visual information, enabling the simultaneous acquisition of facial spatial morphology and temporal change trajectory. This provides subsequent identity verification and status analysis with stable structural representation capabilities and high-resolution movement description capabilities. This processing method, which interweaves spatial and temporal features, improves the accuracy of liveness detection and enhances the ability to analyze subtle facial movement changes, thereby providing more reliable support for subsequent decision-making.

[0024] S30, determine the user's liveness status based on the dynamic facial movement information, and if the user is determined to be live, compare the facial three-dimensional structure information with a preset feature template to complete the identity verification. In this embodiment, the determination of a liveness state relies on the consistency between dynamic facial motion information and the expected motion pattern. The system can issue motion commands to the user during interaction, such as blinking, opening the mouth, nodding, or combinations of multiple actions, guiding the user to perform the corresponding action within a limited time window. Dynamic facial motion information records the trajectory of key points when the user performs the action. The system analyzes the periodic changes in eyelid opening and closing, the angular change path of head posture, and the changes in the opening and closing amplitude of the mouth contour to calculate whether the action execution sequence and timing conform to the command sequence, whether the action amplitude is within the allowable range, and whether the start and end times of the action match the command issuance time. Based on indicators such as the continuity of the motion trajectory, the rationality of the action amplitude, and the degree of timing matching, a liveness detection score can be constructed. When the score is higher than a threshold, it indicates that the user has completed a complex, real-time action according to the command, reflecting real facial muscle activity, thus determining a liveness state.

[0025] In the process of using facial 3D structural information for identity verification, it is necessary to characterize the 3D geometric shape. Facial 3D structural information includes the 3D coordinates of the facial surface, local curvature distribution, facial contour lines, and the spatial relative positions of areas such as the eye sockets, bridge of the nose, and cheekbones. For identity verification, facial 3D structural information can be encoded into high-dimensional feature vectors. For example, a set of stable geometric feature points can be obtained by locally sampling a 3D mesh. Then, by calculating the relative distances, angles, and curvatures between these feature points, a geometric description with identity discrimination can be formed. This geometric description is independent of texture information, unaffected by lighting, skin color, and makeup, and is more suitable for long-term identity recognition.

[0026] Preset feature templates can be understood as a set of facial feature vectors generated and persistently stored for users during the registration process. Each user corresponds to at least one facial 3D structure feature vector. Where permissible, multiple sets of feature vectors extracted under different poses and expressions can be saved to form a feature template set, in order to cope with viewpoint shifts and expression changes in actual use. Preset feature templates are usually normalized and can be accompanied by quality labels, such as image sharpness and pose angle range, to select the most suitable template for calculation during comparison.

[0027] During the comparison of facial 3D structural information with preset feature templates, it is necessary to calculate the similarity between the currently acquired facial 3D structural feature vector and the template vector. Similarity can be calculated using Euclidean distance, cosine similarity, or a learned metric function, mapping the result to a score. To improve robustness, a set of templates closest to the current pose can be selected for comparison. The similarity scores from multiple comparisons are weighted and summed to obtain the final identity score. This identity score is compared with a preset identity threshold; if it exceeds the threshold, the current 3D structure is considered highly consistent with the registered template.

[0028] Authentication success depends on the simultaneous fulfillment of two results: liveness detection and facial structure comparison. Liveness detection excludes non-authentic user scenarios such as photos or replayed videos, while facial structure comparison with a preset feature template confirms whether the user is a registered user. Authentication is only successful when the liveness status obtained from dynamic facial motion information is true and the similarity between the facial structure information and the preset feature template meets the requirements.

[0029] This embodiment uses dynamic facial motion information for liveness detection and compares the three-dimensional facial structure information with a preset feature template only when the liveness status is established. This reduces the success rate of attacks such as image forgery and video replay without increasing the complexity of user operations. At the same time, it uses three-dimensional geometric features to improve the stability and anti-interference ability of the identity verification results, so that a single interaction can satisfy both security and verification speed and user experience.

[0030] S40, based on the voice information, identify the acoustic emotional state, and based on the semantic text information converted from the voice information, identify the semantic emotional state, and fuse the acoustic emotional state, the semantic emotional state, and the interaction behavior information to generate a comprehensive emotional state; In this embodiment, speech information typically consists of continuous audio sampling points and can include various acoustic attributes such as pitch, speech rate, energy, formants, and spectral envelope. To form an analytical basis suitable for emotion recognition from the speech information, the audio stream needs to be segmented according to overlapping time windows of fixed length. Each segment can be transformed into a time-frequency matrix through short-time Fourier transform and combined with multi-dimensional acoustic representations such as Mel-frequency cepstral coefficients, fundamental frequency trajectory, energy variation graph, and zero-crossing rate distribution to form a cross-feature vector. The acoustic emotion state is derived from the classification calculation of these feature vectors, typically including emotion category labels and intensity values, such as "angry, moderate intensity" and "anxious, high intensity," to reflect the emotional tendency conveyed by the voice itself.

[0031] Semantic text information derived from speech information provides emotional cues at the linguistic content level. To obtain accurate textual representation, acoustic decoding of the audio stream is required. This involves converting the corresponding speech segments into text sequences using an end-to-end recognition model or a decoder based on both acoustic and language models. Semantic text information includes word sequences, semantic structures, syntactic relationships, and logical word order. To identify semantic emotional states, semantic emotional features need to be extracted from the text. These features can be constructed based on syntactic tree structures, word vector sequences, emotional lexicon matching distributions, and emotional tendency markers, forming emotional expression vectors corresponding to the linguistic semantics. Semantic emotional states are generated based on the intent of the textual expression, the density of emotional words, and the emotional tendency of the sentence structure. They are typically composed of emotional categories and intensity values, complementing acoustic emotional states.

[0032] Interactive behavior information comes from user actions on the interface, including input latency, swiping speed, and the number of erroneous operations. This behavioral information is highly coupled with emotions; for example, in an anxious state, input latency is shortened and touch frequency increases, while in a confused or uncertain state, the probability of returning to the previous action or clicking repeatedly is higher. To form a behavioral representation usable for fusion computing, behavioral records need to be transformed into behavioral feature vectors in chronological order, including a temporal structure of rhythm changes, input interval changes, and operation frequency. The behavioral features are aligned with acoustic and semantic features in the temporal dimension, making the three emotional cues comparable within the same time window.

[0033] Acoustic emotional state, semantic emotional state, and interactive behavioral information need to be integrated through a unified fusion mechanism to generate a comprehensive emotional state. The fusion process can be based on weighted computation, attention mechanisms, or hierarchical emotion understanding structures, transforming emotional information from the three dimensions of sound, language, and behavior into a set of emotion vectors, and then generating the final comprehensive emotional state through emotion intensity mapping. The fusion mechanism needs to consider the independent contributions of the three dimensions and the relationships between them simultaneously. For example, when the voice is loud but the text is calm, behavioral information is needed to determine whether it indicates tension, excitement, or agitation during an argument. The comprehensive emotional state is a unified description that includes emotion category and emotion intensity, used to express the user's overall emotional tendency in the current interaction scenario and providing input for subsequent interaction decisions.

[0034] This embodiment analyzes and generates a comprehensive emotional state by unifying emotional cues from three dimensions: sound, language, and behavior. This reduces the bias caused by a single emotion recognition dimension, making emotion judgment more comprehensive and reliable. Simultaneously, the multimodal fusion mechanism can capture users' genuine emotional changes in different types of interactions without relying on additional human input, achieving a sensitive ability to capture emotions such as tension, confusion, anger, and anxiety.

[0035] S50, based on the semantic text information, analyze the task complexity, and integrate the task complexity with the comprehensive emotional state to generate an access decision; In this embodiment, semantic text information is used to express the user's true needs in communication, and its content includes word structure, semantic relationships, and expression logic. To obtain a quantifiable analytical basis from the text, intent analysis and semantic structure recognition are required to map the text information into business semantics relevant to the task category. Task complexity measures the difficulty of understanding the text expression and can be derived from features such as text length, semantic level, word density, number of key entities, or syntactic structure proportion. The difficulty level can also be determined by considering the business context to which the text belongs, ensuring that the complexity reflects both linguistic complexity and business processing requirements.

[0036] The overall emotional state is derived from the fusion of acoustic, semantic, and behavioral cues, encompassing both emotion category and intensity, and can characterize a user's psychological tendencies in communication. Emotion intensity can influence the urgency expressed by the user; therefore, after task complexity assessment, it needs to be considered in conjunction with emotional intensity to influence subsequent decision-making outcomes. Historical response success rate is used to provide a stability reference, judging the reliability of automated processing by combining historical interaction performance.

[0037] By simultaneously using task complexity, emotional intensity, and historical response success rate as computational inputs, an access decision that can adapt to different outputs can be formed, enabling the system to formulate appropriate service allocation strategies under different expression difficulties and emotional conditions.

[0038] This embodiment uses quantifiable task complexity derived from semantic text information and integrates it with the overall emotional state to participate in decision-making calculations. This allows the system to simultaneously consider the difficulty of user expression and the level of emotional tension, resulting in more stable and reasonable access judgments. Introducing historical response success rates improves decision reliability, ensures consistency of results across different environments, and enhances both the user experience and system efficiency.

[0039] S60, based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process.

[0040] In this embodiment, executing automated services means establishing an automated execution chain after decision-making. This chain includes a response generation module, a knowledge retrieval module, or a business operation module, constructing the final output content through rule matching, semantic matching, or task sequence execution. Switching to a human agent service indicates a shift in the service processing path to a manual processing chain. This can be achieved by loading available agent information through a scheduling system, selecting a suitable human agent terminal through routing strategies, and establishing a real-time interactive channel. Providing corresponding service responses during the switchover process means that continuous feedback can be obtained at the user's end regardless of whether the channel switchover is complete. Feedback may include queuing status, processing status, initial guidance content, or system prompts. The service response output after the switchover is completed is generated by the automated execution end or the human agent end, including the final business content, explanation content, or operation execution result.

[0041] Implementing this link requires maintaining stable state management capabilities. The state management component records channel switching phases, completed interactions, and the number of pending tasks. Changes in state trigger the output module to update the user interface or provide voice feedback. The actual service response can be generated through a text processing module, a speech synthesis module, or an interface rendering module, and transmitted to the user terminal via the communication channel.

[0042] This embodiment maintains continuity of interaction and reduces uncertainty caused by user waiting by directly connecting access decisions to either an automated or manual link and providing continuous feedback during the switching phase. Automated execution modules improve the processing efficiency of simple tasks, while manual links enhance the processing quality of complex tasks. Dynamic switching between the two is achieved through this process, thereby improving overall service capabilities and user experience.

[0043] In one embodiment, step S10 above includes: S101, acquire raw visual information containing depth information through the camera module, the raw visual information including face image sequence and environmental background data; S102, acquire multi-channel raw speech information through the microphone module, the raw speech information including raw audio waveform and acoustic feature parameters; S103, collect raw interactive behavior information through the interactive interface module, the raw interactive behavior information including click coordinate sequence, input response time and operation frequency characteristics; S104, perform time synchronization alignment processing on the original visual information, original voice information and original interactive behavior information to generate synchronized visual information, synchronized voice information and synchronized interactive behavior information. S105, extract visual information segments, voice information segments, and interactive behavior information segments from the synchronized visual information, synchronized voice information, and synchronized interactive behavior information according to a preset time window. S106, add timestamp identifiers and session context labels to the visual information fragments, voice information fragments and interactive behavior information fragments to generate labeled visual information, labeled voice information and labeled interactive behavior information.

[0044] In this embodiment, the visual information originates from the image frame sequence output by the camera module. The camera module can be a front-facing camera, an external high-definition camera, or an integrated terminal with infrared depth acquisition capabilities. During the acquisition process, it not only acquires two-dimensional color images but also simultaneously outputs depth information related to the distance to objects. In the original visual information, the face image sequence corresponds to a partial image cropped from the face detection area in consecutive frames, used for subsequent processing such as identity recognition, liveness detection, and expression recognition. The environmental background data corresponds to the content in the entire frame excluding the face area, including indoor and outdoor scenes, lighting distribution, and the position of occluding objects, used to assist in determining environmental stability, noise interference, or the existence of multi-user scenarios. The original visual information is stored in a buffer in chronological order, retaining the original resolution and depth channels to prevent information loss during subsequent reconstruction of the stereo structure or extraction of local area fragments.

[0045] Speech information is acquired by the microphone module. Multi-channel raw speech information refers to audio signals acquired simultaneously by multiple physical or virtual channels, such as multiple channels output from the terminal's built-in microphone array, or spatially distributed signals generated by front and rear microphones. The raw audio waveform represents a continuous sequence of sampled values ​​in the time domain, used for subsequent time-domain or frequency-domain analysis. Acoustic feature parameters are sets of feature vectors extracted from the waveform, such as spectral envelope parameters, fundamental frequency trajectory, energy distribution parameters, formant descriptors, Mel-frequency cepstral coefficients, etc., obtained through processing such as sliding window framing, window function weighting, and fast transform, and used for subsequent emotion recognition, speech activity detection, or speaker state judgment. The multi-channel structure allows for subsequent suppression of background noise based on beamforming or spatial filtering, improving the stability of emotion recognition and speech transcription.

[0046] Interactive behavior information is collected by the interactive interface module, which can be a mobile application interface, a web page interface, or a self-service terminal touch interface. The click coordinate sequence in the raw interactive behavior information comes from the click event records of interface controls. Each click generates a record containing horizontal and vertical coordinates, control identifiers, and event trigger times, used to reconstruct the user's operation trajectory on the interface. Input response time is obtained by recording the time interval between the display of the prompt information on the interface and the user completing the input. It can be calculated based on the difference in event timestamps and is used to reflect the user's reaction speed when faced with different questions or different business prompts. Operation frequency characteristics are obtained by statistically analyzing behavioral events such as the number of clicks, swipes, and page switches per unit time, and can be further transformed into rhythmic characteristics, such as short bursts of clicks and long periods of stagnation, for subsequent joint judgment of emotional state and task complexity. During the collection process, the interactive interface module outputs a uniformly formatted event record structure to the behavior collection unit to ensure that the visual, audio, and behavioral data can be aligned in subsequent stages.

[0047] Time synchronization and alignment is the fundamental process for aligning visual, audio, and interactive behavior information to the same instant of interaction on the timeline. In practice, timestamp queues are maintained for each of the three types of data. These timestamps can originate from the terminal system clock, server time synchronization system, or network synchronization source. By unifying the time zone and time precision, each visual frame, each audio segment, and each behavioral event is mapped to a common timeline. Time synchronization and alignment can employ a window matching approach, defining a time period as a synchronization window and using the earliest and latest timestamps within the window as upper and lower bounds. Visual frames, audio frames, and behavioral events are collected within this interval to form synchronized visual, audio, and interactive behavior information. Interpolation or alignment correction mechanisms can also be introduced. When data from a particular path is missing within a certain time period, it is filled in using nearest neighbors or marked as missing, ensuring that subsequent processing links do not fail entirely due to a temporary interruption of a single path.

[0048] The preset time window is a parameter configuration used to segment the timeline, controlling the duration covered by each synchronization unit. For example, it can be set to several windows in the hundreds of milliseconds range to capture short-term emotional fluctuations, or to windows in the seconds range to cover a complete question-and-answer round. When extracting visual, audio, and interactive information fragments from synchronized visual, audio, and interactive behavior information according to the preset time window, each window can be mapped to a group of fragments. Each fragment has a fixed duration and contains all valid data within that time period. Visual information fragments can contain multiple frames of images and their depth channels; audio information fragments can contain the audio waveform and related acoustic features for the corresponding time period; and interactive behavior information fragments can contain one or more click events, input events, or swipe events.

[0049] A timestamp is a time information appended to each segment, used to indicate the segment's position in the global interaction process. It can be represented by a combination of fields such as start time, end time, window sequence number, and session offset time. Session context tags are information related to the business context appended to the timestamp, such as the current business category tag, current round number, previous round response summary identifier, current authentication status, and current risk level tag. These tags can be obtained from the semantic parsing module, business process engine, or history management module. After appending timestamps and session context tags to visual information segments, voice information segments, and interactive behavior information segments, labeled visual information, labeled voice information, and labeled interactive behavior information are formed. These labeled data units have clear time and context positioning capabilities and can be retrieved, filtered, aggregated, or replayed in subsequent links.

[0050] This embodiment reduces the recognition bias caused by time misalignment between different channels through the above steps, enabling liveness detection, emotion recognition, and access decisions to make judgments based on consistent data fragments. This improves the stability and accuracy of identity verification, security assessment, and service routing, while also providing fine-grained data support for historical session backtracking, risk tracking, and service quality analysis.

[0051] In one embodiment, step S20 above includes: S201, Perform face detection and key point localization on the visual information to generate a face detection result containing face bounding box and facial key point coordinates; S202, Based on the coordinates of the facial key points, preliminary three-dimensional facial structure information is generated through the structured light three-dimensional reconstruction module; S203, perform point cloud optimization and mesh reconstruction processing on the preliminary facial 3D structure information to generate facial 3D structure information; S204, extract the motion trajectory of the facial key point coordinates from the continuous frame sequence of the visual information to generate original dynamic facial motion information; S205, Perform motion smoothing and noise filtering on the original dynamic facial motion information to generate refined dynamic facial motion information; S206, Based on the facial muscle movement model, the refined dynamic facial movement information is encoded into action units to generate dynamic facial movement information.

[0052] In this embodiment, the processing steps for extracting facial 3D structure information and dynamic facial motion information from visual information use continuous image frames as input, explicitly modeling the changes in the face in both spatial and temporal dimensions. After face detection and keypoint localization, the visual information generates a face detection result containing face bounding boxes and facial keypoint coordinates. The face bounding boxes can be provided by a face detection network based on a convolutional neural network, which delineates the face region by outputting a rectangular or rotated rectangular region on the original image. The facial keypoint coordinates correspond to discrete anatomical positions such as the corners of the eyes, brow peaks, nose tip, corners of the mouth, and jawline. The keypoint branch of the detection network outputs a set of two-dimensional or two-dimensional plus depth coordinates, which serves as geometric anchors in subsequent 3D reconstruction and motion analysis. The face detection results correspond one-to-one with the visual information frames on the timeline, ensuring that the face region and keypoint positions in each subsequent frame have an accurate source.

[0053] When generating preliminary facial 3D structural information based on facial key point coordinates using a structured light 3D reconstruction module, the module can combine depth channel information, projected light stripe patterns, or binocular parallax information to project each key point and its neighboring pixels into a 3D coordinate space. Specifically, the facial key point coordinates can be aligned with the depth map, and the pixel coordinates can be converted into a 3D point cloud in the camera coordinate system using the camera intrinsic matrix and distortion parameters. Alternatively, the key points can be matched with the structured light coded pattern, the stripe phase can be analyzed to obtain the depth, and then the 3D position can be reconstructed. This preliminary facial 3D structural information contains a discrete set of 3D points, which contains spatial features such as facial contours, nasal bridge height, and cheekbone undulations, providing a foundation for subsequent higher-precision surface modeling.

[0054] The initial facial 3D structural information undergoes point cloud optimization and mesh reconstruction. Point cloud optimization during the generation of facial 3D structural information can include operations such as noise point removal, outlier removal, point density resampling, and smoothing filtering. Point cloud quality can be improved through neighborhood statistics, voxel meshing, or normal-vector-based filtering algorithms. Mesh reconstruction transforms the optimized point cloud into a topologically continuous triangular or quadrilateral mesh. Poisson reconstruction, Delaunay triangulation, or level set-based reconstruction algorithms can be used to generate surface models. The generated facial 3D structural information is stored in mesh form, containing vertex coordinates, facet connectivity, and optional texture coordinates. This 3D representation preserves facial depth and shape while facilitating affine transformations, rigid body registration, or local deformation estimation during liveness detection, identity verification, and expression analysis.

[0055] When extracting motion trajectories of facial keypoint coordinates from a continuous sequence of visual information to generate raw dynamic facial motion information, the system correlates the positions of the same keypoint in different frames along the time axis. This can be achieved by assigning a fixed index to each keypoint, using the coordinates of the keypoint from the previous moment as the predicted position during inter-frame tracking, and then precisely correcting it through local search or optical flow tracking, thus obtaining a time-varying coordinate sequence. The raw dynamic facial motion information consists of multiple trajectories, each corresponding to the motion curve of a keypoint along the time axis, reflecting actions such as eyelid closure and opening, mouth drooping and lifting, and changes in jaw opening and closing amplitude. To ensure trajectory continuity, interpolation or trajectory stitching strategies can be used in cases of frame rate fluctuations or short-term occlusion.

[0056] When generating refined dynamic facial motion information by performing motion smoothing and noise filtering on the original dynamic facial motion information, motion smoothing can utilize low-pass filtering, Kalman filtering, or spline curve fitting methods to suppress high-frequency oscillations introduced by acquisition jitter and detection errors. Noise filtering can correct for sudden jumps, abnormal peaks, or short-term missing points. Through threshold judgment, neighborhood statistics, or trajectory repair mechanisms based on velocity and acceleration constraints, the motion trajectory is restored to a smoother curve that more closely matches the physiological characteristics of the movement.

[0057] Based on a facial muscle motion model, refined dynamic facial motion information is encoded into action units. When generating dynamic facial motion information, the facial muscle motion model can refer to the action unit classification system to map the displacement patterns of different key point combinations into a set of discrete action unit labels. In specific implementation, the motion trajectory of key points is converted into displacement vectors, velocity vectors, or deformation parameters within a time window, and input into a rule-based or learning-based classification module. This is then matched with pre-labeled action unit templates to obtain the activation intensity of action units such as eyebrow raising, brow tightening, corners of the mouth turning up, lips closing, and chin lowering. The action unit encoding process preserves the temporal and amplitude information of the action occurrence, and the encoding results can be output in time series form to form dynamic facial motion information. This information, together with the facial 3D structure information, constitutes the dynamic 3D facial description required for subsequent liveness detection and identity verification.

[0058] For example, in structured light 3D reconstruction: the system emits infrared structured light through a front-end camera to acquire depth images of the user's face, and then uses point cloud reconstruction algorithms to generate a high-precision 3D face mesh model. The core algorithm employs:

[0059] in, Represents the coordinates of the image plane. The depth error evaluation value at a given location is used to measure whether the current depth estimate is reasonable. The entire reconstruction process minimizes this error. The accumulated value is used to optimize the three-dimensional structure information of the face; Indicates in The reference light intensity at the location is usually the light intensity distribution of a known structured light coding pattern under ideal projection conditions, which can be obtained during the calibration stage; Indicates in The observed light intensity at a location is the pixel value in the actual structured light reflection image captured by the camera; Indicates in The face depth value at a given location represents the distance of the face surface relative to the camera coordinate system, and is the depth map that needs to be solved. Indicates depth map in The gradient at a given location reflects the magnitude of depth variation within the neighborhood and is used to constrain the smoothness of a 3D surface. The norm representing the depth gradient is generally the L1 or L2 norm. The larger the value, the more drastic the surface changes. This represents the smoothing constraint coefficient, used to adjust the weights of the data consistency term and the smoothing constraint term. The larger the value, the more the reconstruction result resembles a smooth surface; the preceding coefficient 1 / 2 is a scaling factor, which facilitates subsequent differentiation and optimization, and does not affect the position of the optimal solution.

[0060] This embodiment, by continuously executing face detection and key point localization, structured light 3D reconstruction, point cloud optimization and mesh reconstruction, key point motion trajectory extraction, motion smoothing and noise filtering, and action unit encoding based on facial muscle motion models in the visual information processing stage, can simultaneously obtain high-precision three-dimensional facial structure information and dynamic facial motion information under the same visual channel. This not only improves the ability of liveness detection and identity comparison to resist attacks such as photo and video replay, but also provides a more granular action expression basis for subsequent emotion recognition and service triage, enabling the intelligent customer service system to have higher reliability and distinguishability in the three aspects of identity security, attack protection, and emotion perception.

[0061] In one embodiment, step S30 above includes: S301, present the user with a preset sequence of liveness detection action instructions, the sequence of liveness detection action instructions including at least two actions among blinking, opening the mouth and nodding; S302, Based on the dynamic facial motion information, analyze the continuity of the motion trajectory and the consistency of facial micro-movements of the user executing the liveness detection action command sequence, and generate a liveness detection score; S303, compare the liveness detection score with a preset liveness threshold, and when the liveness detection score exceeds the preset liveness threshold, determine that the user is alive; S304, After determining that the user is a living person, extract the facial feature vector from the facial 3D structure information; S305, determine the similarity score between the face feature vector and the registered face feature vector in the preset feature template; S306, compare the similarity score with a preset authentication threshold, and when the similarity score exceeds the preset authentication threshold, determine that the authentication is successful.

[0062] In this embodiment, when determining a user's liveness status based on dynamic facial motion information, a preset sequence of liveness detection action instructions is first presented to the user. This sequence consists of a set of pre-configured facial or head movements, including at least two of blinking, opening the mouth, and nodding. It is displayed to the user through voice announcements, text prompts, or interface animations. Each instruction has a clearly defined action type, start time, and duration requirement. Random order or combinations can be configured to reduce the success rate of script attacks and pre-recorded video attacks. The instruction sequence is recorded internally by the system as an action tag sequence on a timeline for subsequent alignment with the dynamic facial motion information.

[0063] Dynamic facial motion information originates from the keypoint time trajectories and motion unit codes obtained in the previous stage, encompassing continuous changes in areas such as eyelids, lips, eyebrows, jaw, and head posture over time. When analyzing the user's execution of liveness detection action command sequences, each action in the sequence needs to be matched with the dynamic facial motion information within the corresponding time period. Motion trajectory continuity is evaluated by assessing the displacement curve of the same action throughout the entire execution process. For example, in a blinking motion, the displacement of the upper eyelid keypoint during the closing and opening phases should exhibit a smooth contraction and rebound trajectory; in a nodding motion, the angular changes in jaw and head posture should exhibit acceleration and deceleration processes consistent with physical inertia. The velocity, acceleration, and trajectory smoothness of keypoints between consecutive frames can be calculated to check for sudden jumps, repetitive directions, or abnormal segments that do not conform to physiological movements.

[0064] Facial micro-motion consistency focuses on fine-grained natural tremors and subtle changes, including slight eyelid tremors, minor facial muscle displacements, slight head movements, and minute undulations in the depth direction. High-temporal-resolution trajectories contained in dynamic facial motion information can be decomposed into high-frequency micro-motion components through high-pass filtering, spectral analysis, or short-time energy analysis, and then checked for consistency with macroscopic motion trajectories. If macroscopic motion is present but micro-motion components are consistently absent, or if the temporal relationship between micro-motion components and macroscopic motion deviates significantly from true physiological characteristics (e.g., eyelid position changes and micro-motion frequencies are completely decoupled during blinking), it can be determined that there is a risk of screen replay or video re-enactment. By jointly evaluating the continuity of motion trajectories and facial micro-motion consistency, a liveness detection score can be constructed. Parameters such as the continuity index, micro-motion consistency index, and motion response delay of multiple motion segments are weighted and summed or mapped to a numerical score through a trained classification model. The score positively corresponds to the probability of liveness and is normalized to a fixed interval for easy comparison with thresholds.

[0065] The preset liveness threshold is provided during the system configuration phase. It can be selected through statistical analysis based on the distribution of real users and attack samples in historical data, balancing the false rejection rate and false acceptance rate. When comparing the liveness detection score with the preset liveness threshold, if the score exceeds the threshold, the current dynamic facial movement information is considered to correspond to a real user's movement with high confidence, and the user is determined to be alive. If the score is below the threshold, a retry process can be triggered or subsequent authentication requests can be directly rejected. This creates a hard filtering condition in the decision-making chain, ensuring that only users who pass the liveness detection proceed to the next stage of identity verification.

[0066] After determining that a user is alive, facial feature vectors need to be extracted from the 3D facial structure information. This 3D facial structure information is a previously reconstructed 3D mesh or point cloud representation, containing geometric information such as facial contours, nasal bridge height, jawline shape, and cheekbone contours. During feature vector extraction, the 3D face can first undergo pose and scale normalization to align it to a unified 3D reference coordinate system. Then, the normalized mesh or point cloud is input into a 3D convolutional network, graph convolutional network, or point cloud-based embedding network, outputting a facial feature vector in fixed-dimensional vector form. This vector encodes the facial structure shape in geometric space. Through training, vectors obtained from multiple acquisitions of the same user can be made close to each other, while vectors from different users can be far apart, making it suitable as input for identity verification.

[0067] The preset feature template refers to the set of facial structural features generated and stored during user registration or initial authentication. During the registration process, multiple sets of facial 3D structural information are collected from the user in a live state, and facial feature vectors are extracted from each. These vectors are then averaged, clustered, or selected based on their quality to form the user's registered facial feature vector set in the feature database. This set is associated with the user account and business identifier and serves as the comparison target during the authentication phase. To determine the similarity score between the current facial feature vector and the registered facial feature vectors in the preset feature template, measures such as cosine similarity, Euclidean distance inverse mapping, or Mahalanobis distance can be used. When multiple registered vectors exist in the template, the similarity can be calculated for each vector and the maximum value taken, or multiple results can be weighted and fused to obtain the final similarity score. This ensures that the comparison process takes into account deformation and noise from multiple data collections.

[0068] The similarity score is compared with a preset identity verification threshold to determine the identity verification result. The preset identity verification threshold is also determined through historical data analysis. Different thresholds can be selected on the training or validation set based on security level requirements to control the false rejection rate and false acceptance rate. When the similarity score exceeds the threshold, it indicates that the geometric difference between the current 3D facial structure and the registration template is within an acceptable range, and identity verification can be considered successful. If the similarity score does not reach the threshold, it is considered that the current user and the registered user do not match, and highly sensitive business applications can be rejected or the user can be guided to other supplementary verification processes. The entire processing chain logically forms a two-layer filtering structure of "liveness detection first, then comparison." Dynamic facial action information is used to exclude sources of non-genuine actions, while facial 3D structure information and preset feature templates are used to establish geometric identity consistency judgment. The two types of information are linked through pre- and post-conditional constraints, preventing any path that bypasses liveness detection to directly perform template comparison, or any situation where authorization is based solely on action features without identity comparison.

[0069] This embodiment tightens authentication conditions in both time and space dimensions by setting two numerical gating thresholds: a liveness threshold and an identity verification threshold. This improves authentication security while keeping the user operation process simple, reduces the probability of misjudgment caused by relying solely on static images or single-dimensional features, and provides a reliable identity prerequisite for subsequent emotion recognition and intelligent triage processes.

[0070] In one embodiment, step S40 above includes: S401, perform frame segmentation and acoustic feature extraction on the speech information to generate an acoustic feature vector; S402, Based on the acoustic feature vector, the acoustic emotion state, including emotion category and intensity value, is identified by the acoustic emotion recognition model; S403, Perform speech recognition processing on the speech information to generate semantic text information; S404, perform text parsing and feature extraction on the semantic text information to generate a text feature vector; S405, Based on the text feature vector, identify the semantic emotion state including emotion category and intensity value through a semantic emotion recognition model; S406, extract temporal features from the interactive behavior information to generate a behavior feature vector; S407, the acoustic emotional state, the semantic emotional state, and the behavioral feature vector are fused through a weighted fusion mechanism to generate a comprehensive emotional state.

[0071] In this embodiment, when recognizing acoustic emotional states based on speech information, frame segmentation is first performed on the continuous speech signal. The entire speech information is divided into a series of short segments with fixed durations and partial overlap, with each frame corresponding to tens of milliseconds of speech content. The purpose of frame segmentation is to utilize the approximately stationary nature of speech over a short period of time, allowing subsequent feature extraction steps to calculate indicators such as energy distribution and frequency composition within each frame. When performing acoustic feature extraction on each frame of speech information, parameters such as Mel frequency cepstral coefficients, log-Mel filter bank energy, fundamental frequency profile, formant position, short-term energy changes, and spectral centroid can be calculated. Then, the features from multiple frames within the same time window are concatenated into a vector sequence in chronological order, and compressed into a fixed-dimensional acoustic feature vector through dimensionality reduction or a coding network. The acoustic feature vector carries information on speech pitch, timbre changes, speech rate rhythm, and energy fluctuations in the numerical space, providing a basis for analyzing patterns such as tension, anger, calmness, and anxiety in emotional states.

[0072] Acoustic emotion recognition models receive acoustic feature vectors as input and aggregate the feature sequences over time using multi-layer convolutional networks, recurrent networks, or structures with attention mechanisms to output acoustic emotion states. An acoustic emotion state comprises two parts: emotion category and intensity value. The emotion category can use predefined labels, such as anger, sadness, anxiety, calmness, joy, and confusion, with the probability distribution for each category provided by the final classifier layer. The intensity value can be obtained by setting a continuous emotion dimension in the output layer or by mapping a function to the category probabilities, and is represented by a scalar or vector to indicate the intensity of the emotion. Thus, the acoustic emotion state not only reflects the speaker's current emotional inclination but also the strength of the emotional expression.

[0073] At the semantic level, speech information needs to be converted into semantic text information. The speech recognition processing stage performs acoustic and language modeling on the speech information, mapping continuous speech waveforms into character sequences, word sequences, or sub-word sequences, generating semantic text information containing complete sentence structures. This text information preserves the user's original vocabulary choices, grammatical structure, and instruction intent, providing a foundation for identifying emotions from a content perspective. In the text parsing and feature extraction process, word segmentation, punctuation correction, and spelling correction can be performed first. Dependency parsing and semantic role labeling are then performed on the sentences to identify subject-verb-object relationships, modification relationships, and business-related entities and events. Subsequently, through pre-trained language models, word vector models, or sentence vector encoding networks, the processed text is mapped to a high-dimensional semantic space to obtain text feature vectors. Text feature vectors reflect the semantic similarity of sentences in geometric space; for example, complaints and emergency requests for help will cluster in different regions of the vector space.

[0074] Semantic emotion recognition models take text feature vectors as input and use feedforward networks, attention networks, or classifiers based on transformer structures to comprehensively analyze emotional words, negative structures, exaggerated expressions, and contextual cues in the text to identify semantic emotional states. Semantic emotional states are also expressed using emotion categories and intensity values. Categories can cover labels such as anger, resentment, anxiety, disappointment, and rational consultation, while intensity values ​​reflect the degree of intensity of the semantic emotional tendency. For example, when the text feature vector contains a large number of strongly negative words and frequently contains expressions demanding complaints or escalation, the probability of the negative emotion category in the semantic emotional state increases, and the intensity value approaches its upper limit.

[0075] Interactive behavior information records user actions within the interface or interaction channels, including click coordinate sequences, input response times, operation frequency characteristics, interface scrolling behavior, and time spent on the same screen. The system generates event timestamps and event parameters. The temporal feature extraction stage integrates the interactive behavior information chronologically, calculating the number of triggers per unit time, the distribution of consecutive operation intervals, the trend of dwell time, and patterns such as sudden interruptions for each event type. These statistics are encoded into behavioral feature vectors. These behavioral feature vectors reflect whether user operations are smooth, whether there are frequent accidental touches, whether the user repeatedly returns to the same page, and whether the user spends a long time in key input areas. These behaviors are often correlated with emotions such as tension, hesitation, and anger.

[0076] The generation of the comprehensive emotional state relies on a weighted fusion mechanism. During fusion, the emotion category and intensity value in the acoustic emotion state are encoded into vector form, for example, by concatenating the category probability distribution with the intensity value or obtaining a unified dimensional representation through linear mapping; the semantic emotion state is encoded into a vector in the same way; and the behavioral feature vector is mapped to a scale range compatible with the first two types of vectors through a linear layer or normalization. Then, the features from the three sources are aligned on the time axis to ensure that speech segments, text segments, and behavioral segments within the same time period participate in the same round of fusion operation. The weighted fusion mechanism can use fixed weights, summing the three vectors proportionally, or it can use trainable weights, dynamically adjusting the contributions of acoustic emotion, semantic emotion, and behavioral features to the total output based on the current scenario through a fully connected network or attention network. The output of the fusion operation serves as a vector representation of the comprehensive emotional state, which is then passed through a classification layer to obtain a unified emotion label and overall emotion intensity, or directly outputs multi-dimensional emotion coordinates for subsequent use in decision-making models.

[0077] This embodiment extracts acoustic feature vectors, text feature vectors, and behavioral feature vectors in parallel during the same interaction process, and constructs a weighted fusion relationship between acoustic emotional state, semantic emotional state, and interactive behavior information. This unifies three types of information—voice tone changes, content expression tendencies, and actual operational behaviors—into a comprehensive emotional state. Since the acoustic channel is more sensitive to changes in pitch, speech rate, and energy, the semantic channel can capture language patterns such as complaining, seeking help, and negation, and the behavioral channel reflects changes in click frequency and dwell time, the weighted fusion mechanism integrates these sources in the vector space. The comprehensive emotional state is more stable and granular than the emotional judgment of any single channel, thus providing more reliable emotional input for subsequent task complexity analysis and access decision-making. This enables automated service triggering, human agent transfer, and response strategy adjustment to fully consider the user's true emotional state, reduce the probability of mis-triage, and improve the interactive experience in high-pressure scenarios.

[0078] In one embodiment, step S50 above includes: S501, Perform intent recognition and business domain analysis on the semantic text information to generate task type classification results; S502, Based on the task type classification result and the semantic complexity features of the semantic text information, determine the task complexity; S503, Extract the emotional intensity from the comprehensive emotional state; S504, Based on the task complexity, the emotion intensity, and the historical response success rate, an access decision is generated through a probabilistic decision model.

[0079] In this embodiment, semantic text information originates from the speech recognition or text input results of the previous processing stage. This stage considers it as the content carrier of the user's business needs and requires transformation into a computable structured representation through intent recognition and business domain analysis. Intent recognition maps entire or multiple sentences of semantic text to intent labels in a finite set, such as policy changes, claims consultation, premium payment, product interpretation, health management consultation, and remote follow-up assistance. Specifically, a pre-trained language model can be used to extract sentence vectors, and then a fully connected network can be used to output the intent probability distribution. The intent category with the highest probability or the top few high-probability categories can be selected for subsequent calculations. Business domain analysis further distinguishes the relevant business segments based on intent recognition. For example, in the fintech business domain, this includes insurance underwriting, claims, and pre-insurance consultation; in the healthcare business domain, it includes health assessment, medication guidance, and chronic disease follow-up. Based on keywords, entity recognition results, and contextual conversation trajectories, semantic text is mapped to business domain labels. Intent labels and business domain labels together constitute the task type classification result, providing prior information on the semantic category dimension for complexity assessment.

[0080] Determining task complexity relies on the task type classification results and the semantic complexity features of the semantic text information. Semantic complexity features characterize the amount of information required to complete the current request, the number of business rules, and the depth of cross-system calls, and can be constructed from multiple perspectives. Structural features such as text length, number of sentences, and number of nested clauses can be statistically analyzed to reflect the simplicity of the expression structure. Keywords related to business rules can be extracted, such as "multiple insured persons," "policy amendment," and "cross-institutional inspection report," and the number of rule entries involved can be determined through a thesaurus or knowledge graph. Furthermore, by combining the operation sequences of similar text in historical sessions, the number of interface calls triggered in the workflow, the number of form fields, and the number of business validations can be statistically analyzed to obtain a complexity reference value associated with historical work order processing time and failure rate. In implementation, features such as intent label, business domain label, text length, keyword count, rule entry count, and historical processing depth can be concatenated into a task feature vector, which is then input into a regression network or a segmented mapping function to output a task complexity score. The score can fall within a preset range, such as a real number between zero and one after normalization. The larger the value, the longer the operation chain and the more business constraints are required to complete the request.

[0081] The overall emotional state is derived from the emotion fusion processing in the previous stage, including fields such as emotion category and emotion intensity. In this stage, emotion intensity needs to be extracted as a crucial factor in the input decision. Emotion intensity can be directly derived from the amplitude in the overall emotion vector, or it can be obtained by linearly combining the intensities of acoustic and semantic emotions. For example, aggregating the probability and intensity of negative categories yields a single scalar representing the degree of anxiety, anger, or panic. To ensure numerical scale compatibility with task complexity scoring, emotion intensity can be normalized or range stretched to unify emotion indicators from different sources into the same numerical range, preventing any single source from dominating the weights in subsequent models.

[0082] Historical response success rate is used to reflect the processing effectiveness of similar tasks under different service paths in the past, providing empirical data support for probabilistic decision-making models. Historical response success rate can be statistically analyzed across multiple dimensions. For example, using task type classification results and business domain tags as query conditions, the completion rate of automated services, the resolution rate by human agents, and the proportion of events such as secondary callbacks, complaints, and timeout closures can be statistically analyzed from historical interaction records. The proportion of issues resolved in one go under the automated service path can be calculated and compared with the corresponding proportion under the human agent service path, forming a success rate vector related to task characteristics. For use in models, this vector can be compressed into a bias value reflecting whether the current task is more suitable for automated or human processing, or two dimensions can be retained separately to represent the success rate of the automated path and the success rate of the human path, allowing subsequent models to learn the weights.

[0083] The probabilistic decision model generates access decisions based on three input factors: task complexity, emotional intensity, and historical response success rate. The output typically includes category labels such as automated service, agent service, or hybrid strategy, along with corresponding probabilities. In implementation, the task complexity score, processed emotional intensity, and historical response success rate features are concatenated into a decision input vector. Embedded vectors of task type classification results and business domain labels can be incorporated to form a more comprehensive representation of decision features. The probabilistic decision model can employ logistic regression networks to fit the effects of manual allocation decisions and actual processing through training on historical labeled data; it can also use Bayesian networks to explicitly model the conditional probability relationships between events such as excessively high task complexity, excessively high emotional intensity, and low success rates of automated paths; or it can use gradient boosting trees or deep neural networks to fit complex nonlinear relationships. During model training, historical interaction samples are used, including task features, emotional indices, historical success rates, actual access paths used, and user satisfaction. Under a supervised learning framework, the model aims to minimize the difference between predicted access decisions and actual decisions, or maximize the probability within the overall success rate and satisfaction thresholds. During the online inference phase, for each newly arrived request, the task complexity score, emotional intensity, and historical success rate features are obtained through a unified interface. These features are then input into a probabilistic decision model to obtain the probability of automated service and the probability of agent service. Based on a preset strategy, the service path corresponding to the highest probability is selected, or a threshold strategy is adopted within a certain range to directly assign high-complexity, high-emotion-intensity requests to agent service, while low-complexity, low-emotion-intensity requests with a high success rate on the automated path are left on the automated service path.

[0084] For example, decision-making logic includes three layers: Task complexity determination layer: Calculates task complexity score based on semantic parsing results. Standardized transactions (such as checking balances) receive low scores, while non-standard transactions (such as claims disputes) receive high scores.

[0085] Emotion Intensity Assessment Layer: Based on the intensity output by the emotion recognition module. To determine the customer's emotional state. When Furthermore, when the emotion is negative, the system prioritizes transferring the case to a human agent.

[0086] Multi-factor fusion decision-making layer: comprehensive , Compared with historical response success rate The probability of human intervention is calculated using a Bayesian network:

[0087] Here, H represents an event or hypothesis that requires human intervention. This represents the task complexity metric obtained at the current time t, derived from intent recognition and business domain analysis. This represents the emotional intensity index obtained at the current time t, corresponding to the "emotional intensity" extracted from the comprehensive emotional state mentioned earlier. This represents the historical response success rate or historical abnormal response rate obtained at the current time t, and is used to reflect the success rate of similar requests under the automated service path. Indicates the complexity of a given task Emotional intensity and historical response success rate / anomaly rate metrics Under these conditions, the posterior probability that requires human intervention is the value ultimately used to make the access decision. This indicates that, under the premise that "human intervention is already required," the observed probability of the current task complexity, emotional intensity, and historical performance falling into this set of values ​​is used to characterize the typical combination of these factors in a human intervention scenario. This indicates the prior probability that human intervention is required, and can be set based on long-term statistics of the proportion of human intervention, business strategies, or service level strategies. This represents the overall probability of the current set of task complexity, emotional intensity, and historical performance occurring. It is a normalization factor used to ensure that the sum of the posterior probabilities of all possible events is 1. In practice, it can be estimated through full probability expansion or numerical approximation. For example, when the probability exceeds a threshold (e.g., 0.6), a human agent is automatically triggered.

[0088] This embodiment identifies the intent and analyzes the business domain of semantic text information to construct task type classification results. It calculates task complexity by combining semantic complexity features, extracts emotional intensity from the overall emotional state, and inputs it into the probabilistic decision model along with the historical response success rate. Under a unified numerical framework, it can simultaneously consider the processing difficulty of the task itself, the user's current emotional pressure, and the real effects of different service paths in similar past scenarios. The access decision no longer relies solely on a single keyword or call duration threshold, but provides the optimal path selection in a multi-dimensional feature space. This allows for priority allocation of human agents in complex and high-pressure scenarios, avoiding conflicts caused by repeated failures of automated services. In simple and stable scenarios, it fully utilizes automated services to improve processing efficiency, achieving a comprehensive effect of improved traffic triage accuracy, increased utilization of human resources, and improved user satisfaction.

[0089] In one embodiment, step S60 above includes: S601, parse the access decision to obtain the service type indication; S602, when the service type is indicated as an automated service, an automated service process including retrieving matching response content from the knowledge base and generating an automated service response is executed; S603, when the service type is indicated as agent service, execute the agent service switching process including generating transition scripts, initializing agent service connection, and transmitting user context information to the agent service terminal; S604, During the agent service switching process, the transition script is presented to the user; S605 provides corresponding service responses in real time based on the service type during service execution and monitors the service status. S606 records service execution process data.

[0090] In this embodiment, access decisions can exist in the form of tags, score vectors, or structured records. They contain at least a service type field and a confidence field related to that type, and may also include extended information such as priority, service level, and estimated processing time. The process of parsing the access decision involves mapping this structured result into executable control instructions within the system. The service type field and confidence field are read through a rule engine or decision parsing module, and a service type indication is generated by combining this with the current business strategy table or routing configuration table. The service type indication is typically represented by an enumerated value from a finite set, such as "automated service," "agent service," or "hybrid service." In this processing flow, "automated service" and "agent service" are the two main categories used to control subsequent branch logic. The parsing process can also perform a threshold judgment on the confidence of the access decision. When the confidence is insufficient, the service type indication is adjusted to a more conservative manual access type to reduce the risk of incorrect traffic routing.

[0091] Automated services refer to service processing paths completed without human intervention, relying on knowledge bases and task execution components. The automated service process is triggered by service type indications. First, a search request is constructed based on the semantic text information of the current session, user identity tags, and business context, and then input into the knowledge base retrieval module. The knowledge base can include question-and-answer pairs, process guidance documents, business rule descriptions, interface call configuration items, etc., and can be stored in a vector index, inverted index, or graph structure. During the retrieval phase, several candidate responses are selected through similarity calculations, field matching, or multi-field fusion scoring. Based on this, the results can be reordered, and candidates irrelevant to the current business can be filtered according to the current session context. The automated service response is the final output generated based on the selected response content. It can directly use existing text from the knowledge base, or the language generation module can rewrite the search results, supplement business parameters, and insert dynamic data to form a response content adapted to the current user's question and business status. The generated automated service response is bound to the current session identifier and sent to the user via a text channel or a speech synthesis channel.

[0092] Agent service is a service path where interaction is completed via a human customer service terminal. The agent service switching process is also driven by service type indication. First, a transitional script is generated on the dialogue side to inform the user that they are about to be transferred to human service, the estimated waiting time, or that a suitable service personnel is being matched for them. The transitional script can be templated and parameter-filled based on the overall emotional state and task complexity; for example, reassuring wording can be added in high-emotional-intensity scenarios, and explanatory content can be added in high-task-complexity scenarios. Then, the agent service connection is initialized. At the communication layer, the target agent server is selected through the session routing module. Appropriate agent terminals can be assigned based on business domain, agent skill tags, current load, and service level policies. The initialization process includes assigning a new routing identifier to the session, creating a session context on the agent's end, and establishing a two-way communication channel. User context information is a set of structured data constructed to ensure the continuity of human access. It typically includes historical dialogue text, semantic text information summaries, overall emotional state, task type classification results, task complexity, user basic attribute tags, and the most recent access decision. The process of transmitting user context information to the agent server requires encapsulating this data according to a predefined structure, desensitizing sensitive fields, and sending it to the agent system through a secure channel so that human service personnel can directly understand the user's problem development trajectory and current mood upon access.

[0093] The system handles transitional messages during agent service switching, emphasizing continuous interaction with the user until network routing, resource allocation, and agent response are complete. After detecting that the service type indicator is agent service and initiating the switching process, the system outputs transitional text or voice prompts in the user interface or voice channel. These prompts may include information such as the reason for the transfer, estimated waiting time, and queue order. In scenarios with high emotional stress, reassuring and reassuring expressions can also be added. Presentation methods can include text pop-ups, voice announcements, and combined text and image prompts, aligned with the current conversation timeline to avoid prolonged periods of silence without feedback.

[0094] During service execution, corresponding service responses are provided in real time based on the service type. This is manifested in the automated service response module continuously processing new user input under the automated service path, and the agent terminal inputting response content in real time and sending it to the user through the system under the agent service path. The system differentiates its processing for different paths based on service type indicators. Under the automated service path, it can repeatedly trigger knowledge base retrieval or business operation calls for continuous user queries, while under the agent service path, it synchronizes new user input to the agent terminal interface. Service status monitoring is carried out throughout the entire service execution period, forming a status indicator set by collecting operational parameters such as response time, success flags, error codes, disconnection events, and retries. The monitoring module can determine whether the connection is normal, whether the response has timed out, and whether high-frequency errors have occurred based on the status indicator set, and trigger alarm mechanisms or switching strategies when necessary. For example, it may attempt to switch to agent service when the automated service fails continuously, or trigger backup agent reallocation when the agent service is unresponsive for a long time.

[0095] Recording service execution process data involves writing information such as access decisions, service type indications, automated service response summaries, agent service dialogue summaries, service status change sequences, user emotional state changes, and final business results to logs or persistent storage at the end of the service processing chain or at the end of a phase. Service execution process data can be indexed multiple times according to session, user, and business dimensions for subsequent service quality assessment, policy model updates, security tracing, and anomaly analysis. By associating access decisions with actual service paths and results, high-value supervisory data can be provided for subsequent training or adjustment of probabilistic decision models, forming a closed loop from multimodal perception and decision generation to service execution and feedback.

[0096] This embodiment integrates the execution paths of decision-driven automated services and agent services, and truly implements the task complexity and comprehensive emotional state obtained from the preceding multimodal analysis into the service routing and response output stages, thereby achieving collaborative matching between service models and user states.

[0097] In one embodiment, a smart traffic splitting device based on multimodal information processing is provided, which corresponds one-to-one with the smart traffic splitting method based on multimodal information processing described in the above embodiments. (Refer to...) Figure 3 , Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the intelligent traffic diversion device based on multimodal information processing of the present invention. The modules include: multimodal acquisition module 10, face structure and action analysis module 20, liveness and identity verification module 30, emotion state fusion module 40, access decision generation module 50, and service execution and switching module 60. Detailed descriptions of each functional module are as follows: The multimodal acquisition module 10 is used to acquire the user's multimodal interaction information, which includes visual information, voice information and interaction behavior information; The face structure and motion analysis module 20 is used to extract three-dimensional face structure information and dynamic facial motion information from the visual information; The liveness and identity verification module 30 is used to determine the user's liveness status based on the dynamic facial movement information, and under the condition that the user is determined to be alive, compare the facial three-dimensional structure information with the preset feature template to complete the identity verification. The emotion state fusion module 40 is used to identify acoustic emotion state based on the speech information, and to identify semantic emotion state based on semantic text information converted from the speech information, and to fuse the acoustic emotion state, the semantic emotion state and the interaction behavior information to generate a comprehensive emotion state. Access decision generation module 50 is used to parse the task complexity based on the semantic text information and integrate the task complexity with the comprehensive emotional state to generate an access decision; The service execution and switching module 60 is used to execute automated services or switch to agent services according to the access decision, and provide corresponding service responses during or after the switching process.

[0098] In one embodiment, the multimodal acquisition module 10 is specifically used for: The camera module collects raw visual information containing depth information, including facial image sequences and environmental background data. Multi-channel raw speech information is acquired through a microphone module, and the raw speech information includes raw audio waveforms and acoustic feature parameters. The original interactive behavior information is collected through the interactive interface module, including click coordinate sequence, input response time and operation frequency characteristics. The original visual information, original voice information, and original interactive behavior information are time-synchronized and aligned to generate synchronized visual information, synchronized voice information, and synchronized interactive behavior information. Visual information segments, audio information segments, and interactive behavior information segments are extracted from the synchronized visual information, synchronized audio information, and synchronized interactive behavior information according to a preset time window. Add timestamps and session context labels to the visual information fragments, voice information fragments, and interactive behavior information fragments to generate labeled visual information, labeled voice information, and labeled interactive behavior information.

[0099] In one embodiment, the face structure and motion analysis module 20 is specifically used for: Perform face detection and key point localization on the visual information to generate a face detection result containing face bounding boxes and facial key point coordinates; Based on the coordinates of the facial key points, preliminary three-dimensional facial structure information is generated through the structured light 3D reconstruction module. The preliminary facial 3D structure information is subjected to point cloud optimization and mesh reconstruction processing to generate facial 3D structure information; The motion trajectory of the facial key point coordinates is extracted from the continuous frame sequence of the visual information to generate the original dynamic facial motion information; The original dynamic facial motion information is processed by motion smoothing and noise filtering to generate refined dynamic facial motion information. Based on the facial muscle motion model, the refined dynamic facial motion information is encoded into motion units to generate dynamic facial motion information.

[0100] In one embodiment, the liveness and identity verification module 30 is specifically used for: Present the user with a preset sequence of liveness detection action instructions, the sequence of liveness detection action instructions including at least two actions among blinking, opening the mouth and nodding; Based on the dynamic facial motion information, the continuity of the motion trajectory and the consistency of facial micro-movements of the user executing the liveness detection action command sequence are analyzed to generate a liveness detection score. The liveness detection score is compared with a preset liveness threshold. When the liveness detection score exceeds the preset liveness threshold, the user is determined to be alive. After determining that the user is alive, facial feature vectors are extracted from the three-dimensional facial structure information; Determine the similarity score between the facial feature vector and the registered facial feature vector in the preset feature template; The similarity score is compared with a preset authentication threshold. When the similarity score exceeds the preset authentication threshold, authentication is deemed successful.

[0101] In one embodiment, the emotion state fusion module 40 is specifically used for: The speech information is processed by frame segmentation and acoustic feature extraction to generate an acoustic feature vector; Based on the acoustic feature vector, an acoustic emotion recognition model is used to identify acoustic emotion states, including emotion categories and intensity values. The speech information is processed by speech recognition to generate semantic text information; The semantic text information is parsed and features are extracted to generate a text feature vector; Based on the text feature vector, a semantic emotion recognition model is used to identify semantic emotion states, including emotion categories and intensity values. Temporal features are extracted from the interactive behavior information to generate a behavior feature vector; A comprehensive emotional state is generated by fusing the acoustic emotional state, the semantic emotional state, and the behavioral feature vector through a weighted fusion mechanism.

[0102] In one embodiment, the access decision generation module 50 is specifically used for: The semantic text information is subjected to intent recognition and business domain analysis to generate task type classification results; Based on the task type classification results and the semantic complexity features of the semantic text information, the task complexity is determined. Extract the emotional intensity from the overall emotional state; Based on the task complexity, the emotion intensity, and the historical response success rate, an access decision is generated through a probabilistic decision model.

[0103] In one embodiment, the service execution and switching module 60 is specifically used for: The access decision is analyzed to obtain the service type indication; When the service type is indicated as an automated service, an automated service process is executed, which includes retrieving matching response content from the knowledge base and generating an automated service response. When the service type is indicated as agent service, an agent service switching process is executed, including generating transition scripts, initializing agent service connection, and transmitting user context information to the agent service terminal. During the switchover process, the aforementioned transition script is presented to the user; During service execution, provide corresponding service responses in real time based on the service type and monitor the service status; Record data during the service execution process.

[0104] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a server-side intelligent traffic offloading method based on multimodal information processing.

[0105] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides determination and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the client-side functions or steps of an intelligent traffic splitting method based on multimodal information processing.

[0106] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Collect users' multimodal interaction information, which includes visual information, voice information, and interaction behavior information; Extract facial 3D structure information and dynamic facial movement information from the visual information; The user's liveness status is determined based on the dynamic facial movement information, and if the user is determined to be live, the facial 3D structure information is compared with a preset feature template to complete the identity verification. Acoustic emotional state is identified based on the speech information, and semantic emotional state is identified based on the semantic text information converted from the speech information. The acoustic emotional state, the semantic emotional state, and the interaction behavior information are fused to generate a comprehensive emotional state. The task complexity is analyzed based on the semantic text information, and the access decision is generated by combining the task complexity with the comprehensive emotional state. Based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process.

[0107] In one embodiment, a non-volatile computer-readable storage medium is provided, which may be non-volatile or volatile, and stores a computer program thereon. When the computer program is executed by a processor, it performs the following steps: Collect users' multimodal interaction information, which includes visual information, voice information, and interaction behavior information; Extract facial 3D structure information and dynamic facial movement information from the visual information; The user's liveness status is determined based on the dynamic facial movement information, and if the user is determined to be live, the facial 3D structure information is compared with a preset feature template to complete the identity verification. Acoustic emotional state is identified based on the speech information, and semantic emotional state is identified based on the semantic text information converted from the speech information. The acoustic emotional state, the semantic emotional state, and the interaction behavior information are fused to generate a comprehensive emotional state. The task complexity is analyzed based on the semantic text information, and the access decision is generated by combining the task complexity with the comprehensive emotional state. Based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process.

[0108] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0109] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0110] It should be noted that if any software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. The embodiments described above are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0111] The user personal information involved in this application embodiment is all authorized (knowing and consenting) by the relevant parties or fully authorized by all parties, and the executing entity can obtain it through various open, legal and compliant means. The collection, storage, use, processing, transmission, provision and disclosure of the information, data and signals involved all comply with the relevant laws and regulations of the relevant countries and regions, and do not violate public order and good morals.

Claims

1. A smart traffic splitting method based on multimodal information processing, characterized in that, Includes the following steps: Collect users' multimodal interaction information, which includes visual information, voice information, and interaction behavior information; Extract facial 3D structure information and dynamic facial movement information from the visual information; The user's liveness status is determined based on the dynamic facial movement information, and if the user is determined to be live, the facial 3D structure information is compared with a preset feature template to complete the identity verification. Acoustic emotional state is identified based on the speech information, and semantic emotional state is identified based on the semantic text information converted from the speech information. The acoustic emotional state, the semantic emotional state, and the interaction behavior information are fused to generate a comprehensive emotional state. The task complexity is analyzed based on the semantic text information, and the access decision is generated by combining the task complexity with the comprehensive emotional state. Based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process.

2. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Collect user's multimodal interaction information, which includes visual information, voice information, and interaction behavior information, including: The camera module collects raw visual information containing depth information, including facial image sequences and environmental background data. Multi-channel raw speech information is acquired through a microphone module, and the raw speech information includes raw audio waveforms and acoustic feature parameters. The original interactive behavior information is collected through the interactive interface module, including click coordinate sequence, input response time and operation frequency characteristics. The original visual information, original voice information, and original interactive behavior information are time-synchronized and aligned to generate synchronized visual information, synchronized voice information, and synchronized interactive behavior information. Visual information segments, audio information segments, and interactive behavior information segments are extracted from the synchronized visual information, synchronized audio information, and synchronized interactive behavior information according to a preset time window. Add timestamps and session context labels to the visual information fragments, voice information fragments, and interactive behavior information fragments to generate labeled visual information, labeled voice information, and labeled interactive behavior information.

3. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Extracting facial 3D structure information and dynamic facial movement information from the visual information includes: Perform face detection and key point localization on the visual information to generate a face detection result containing face bounding boxes and facial key point coordinates; Based on the coordinates of the facial key points, preliminary three-dimensional facial structure information is generated through the structured light 3D reconstruction module. The preliminary facial 3D structure information is subjected to point cloud optimization and mesh reconstruction processing to generate facial 3D structure information; The motion trajectory of the facial key point coordinates is extracted from the continuous frame sequence of the visual information to generate the original dynamic facial motion information; The original dynamic facial motion information is processed by motion smoothing and noise filtering to generate refined dynamic facial motion information. Based on the facial muscle motion model, the refined dynamic facial motion information is encoded into motion units to generate dynamic facial motion information.

4. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Based on the dynamic facial motion information, the user's liveness status is determined. If the user is determined to be live, the facial 3D structure information is compared with a preset feature template to complete identity verification, including: Present the user with a preset sequence of liveness detection action instructions, the sequence of liveness detection action instructions including at least two actions among blinking, opening the mouth and nodding; Based on the dynamic facial motion information, the continuity of the motion trajectory and the consistency of facial micro-movements of the user executing the liveness detection action command sequence are analyzed to generate a liveness detection score. The liveness detection score is compared with a preset liveness threshold. When the liveness detection score exceeds the preset liveness threshold, the user is determined to be alive. After determining that the user is alive, facial feature vectors are extracted from the three-dimensional facial structure information; Determine the similarity score between the facial feature vector and the registered facial feature vector in the preset feature template; The similarity score is compared with a preset authentication threshold. When the similarity score exceeds the preset authentication threshold, authentication is deemed successful.

5. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Based on the speech information, an acoustic emotional state is identified, and based on the semantic text information converted from the speech information, a semantic emotional state is identified. The acoustic emotional state, the semantic emotional state, and the interaction behavior information are then fused to generate a comprehensive emotional state, including: The speech information is processed by frame segmentation and acoustic feature extraction to generate an acoustic feature vector; Based on the acoustic feature vector, an acoustic emotion recognition model is used to identify acoustic emotion states, including emotion categories and intensity values. The speech information is processed by speech recognition to generate semantic text information; The semantic text information is parsed and features are extracted to generate a text feature vector; Based on the text feature vector, a semantic emotion recognition model is used to identify semantic emotion states, including emotion categories and intensity values. Temporal features are extracted from the interactive behavior information to generate a behavior feature vector; A comprehensive emotional state is generated by fusing the acoustic emotional state, the semantic emotional state, and the behavioral feature vector through a weighted fusion mechanism.

6. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Based on the semantic text information parsing task complexity, and by integrating the task complexity with the comprehensive emotional state to generate an access decision, the following is included: The semantic text information is subjected to intent recognition and business domain analysis to generate task type classification results; Based on the task type classification results and the semantic complexity features of the semantic text information, the task complexity is determined. Extract the emotional intensity from the overall emotional state; Based on the task complexity, the emotion intensity, and the historical response success rate, an access decision is generated through a probabilistic decision model.

7. The intelligent traffic splitting method based on multimodal information processing as described in claim 1, characterized in that, Based on the access decision, execute automated services or switch to agent services, and provide corresponding service responses during or after the switching process, including: The access decision is analyzed to obtain the service type indication; When the service type is indicated as an automated service, an automated service process is executed, which includes retrieving matching response content from the knowledge base and generating an automated service response. When the service type is indicated as agent service, an agent service switching process is executed, including generating transition scripts, initializing agent service connection, and transmitting user context information to the agent service terminal. During the switchover process, the aforementioned transition script is presented to the user; During service execution, provide corresponding service responses in real time based on the service type and monitor the service status; Record data during the service execution process.

8. A smart traffic diversion device based on multimodal information processing, characterized in that, The intelligent traffic diversion device based on multimodal information processing includes: A multimodal acquisition module is used to collect multimodal interaction information of users, including visual information, voice information, and interaction behavior information; The face structure and motion analysis module is used to extract three-dimensional facial structure information and dynamic facial motion information from the visual information. The liveness and identity verification module is used to determine the user's liveness status based on the dynamic facial movement information, and, if the user is determined to be alive, to compare the facial three-dimensional structure information with a preset feature template to complete the identity verification. The emotion state fusion module is used to identify acoustic emotion state based on the speech information, and to identify semantic emotion state based on semantic text information converted from the speech information, and to fuse the acoustic emotion state, the semantic emotion state and the interaction behavior information to generate a comprehensive emotion state. The access decision generation module is used to parse the task complexity based on the semantic text information and integrate the task complexity with the comprehensive emotional state to generate an access decision. The service execution and switching module is used to execute automated services or switch to agent services based on the access decision, and to provide corresponding service responses during or after the switching process.

9. A computer device, characterized in that, The computer device includes a memory, a processor, and a multimodal information processing-based intelligent traffic splitting program stored in the memory and executable on the processor. When executed by the processor, the multimodal information processing-based intelligent traffic splitting program implements the steps of the multimodal information processing-based intelligent traffic splitting method as described in any one of claims 1-7.

10. A non-volatile computer-readable storage medium, characterized in that, The storage medium stores an intelligent traffic splitting program based on multimodal information processing. When the processor executes the intelligent traffic splitting program based on multimodal information processing, it implements the steps of the intelligent traffic splitting method based on multimodal information processing as described in any one of claims 1-7.

Images