A sim identity multi-modal perception business hall multi-terminal collaborative initiative service method

Abstract

The application discloses a SIM identity multi-modal perception business hall multi-terminal collaborative active service method, relates to the technical field of digital identity authentication, multi-modal perception and business hall intelligent service, and comprises the following steps: collecting SIM encryption identity and equipment identification by means of a business hall entrance near field communication module; checking and uploading a temporary session identification to an intelligent service agent to complete identity authentication and call a customer business portrait; synchronously collecting visual, voice and location multi-modal behavior data; distributing each modal attention weight based on the business portrait, dynamically fusing data, predicting customer service demand; combining terminal load and location to screen a main service terminal; generating a service handover path, preconfiguring each terminal interface, issuing a guide instruction to complete customer active reception, and realizing cross-terminal non-susceptible handover according to a session package. The application realizes business hall whole-process intelligent service through near field non-susceptible identity verification, attention multi-modal prediction, multi-terminal collaborative scheduling and robot active guidance.

Description

Technical Field

[0001] This invention relates to the fields of digital identity authentication, multimodal perception, and intelligent service in business halls, and particularly to a method and apparatus for multi-terminal collaborative proactive service in business halls based on SIM identity multimodal perception. Background Technology

[0002] As telecommunications operators deepen their digital transformation in their service halls, various intelligent service terminals have been gradually deployed, including bipedal service robots, desktop service terminals, AI self-service card writing machines, and service personnel wearing electronic badges. This has formed a comprehensive service capability system for customers, enabling self-service inquiries, business processing, and SIM card writing. These terminals constitute the core service nodes of the operators' AI service hall product system, providing customers with diversified business processing channels.

[0003] Currently, existing technical solutions related to the above-mentioned technical background mainly include the following categories: The first type is a smart device voice interaction solution based on multi-task dialect recognition. This solution realizes the voice interaction of smart devices through a multi-task dialect recognition model, covering two parallel tasks: dialect recognition and voice intent understanding. However, this type of solution requires the customer to actively initiate a voice command to trigger the interaction process, which is a passive service mode and cannot complete the identity perception and demand prediction when the customer enters the business hall.

[0004] The second category is interactive solutions that integrate audio and video, using an attention mechanism to fuse the multimodal features of audio and video data. This type of solution is primarily geared towards multimedia content processing scenarios and does not yet cover integrated applications such as multi-terminal collaborative service scheduling in service halls and customer identity awareness.

[0005] The third type is Bluetooth pairing solutions based on voice commands and voiceprint authentication. These solutions utilize voice commands and voiceprint authentication to authenticate and connect Bluetooth devices, solving the device authentication problem in a single scenario. However, this solution only covers the device pairing process and lacks the ability for cross-terminal service collaboration and session handover.

[0006] The fourth category is smart card-based electronic card application solutions. These solutions build application modules within the smart card security chip based on the JavaCard specification and national cryptographic algorithms, achieving contactless verification and data recording via NFC. While this type of solution shares some technical similarities with the scenarios mentioned above in terms of smart card identity carriers and NFC near-field communication, its application scenarios are concentrated on single-service verification and do not involve multi-terminal collaborative scheduling or multi-modal perception fusion.

[0007] The fifth category is traditional customer identification solutions in service halls, which mainly rely on customers actively presenting identification, scanning codes, or undergoing manual questioning to complete identity verification. This method has low identification efficiency and requires active customer cooperation, making it difficult to achieve seamless entry identification. The sixth category is online application solutions for SIM digital identity. Currently, the SIM digital identity authentication capabilities of operator authentication platforms are only applied to online scenarios and have not yet extended to the areas of customer identity perception and service scheduling in offline service halls. There is a significant technological gap between offline near-field perception and multi-terminal collaborative services for SIM digital identity.

[0008] In summary, the existing intelligent service solutions for business halls have the following main technical shortcomings: First, the service process is mainly reactive, and the intelligent terminals cannot identify the customer's identity and service needs before the customer arrives. Customers need to find the terminal themselves or wait in line, failing to achieve a seamless "identification upon entry" experience. Second, there is a lack of effective collaboration mechanisms between the various intelligent terminals. Bipedal robots, desktop service terminals, and AI self-service card writing machines operate independently and cannot perform cross-terminal task scheduling and process arrangement according to the customer's multi-stage service needs. When handling multiple services, customers need to switch repeatedly between different terminals, and each switch requires re-authentication and information entry, resulting in insufficient service continuity. Third, the SIM digital identity authentication capability is limited to online applications. The operator's unique SIM card security element identity carrier has not yet been realized in the offline business hall scenario for perception and scheduling.

[0009] Therefore, there is an urgent need to provide a multi-terminal collaborative proactive service method based on the fusion of SIM digital identity offline near-field perception and multimodal behavior perception data, so as to realize the whole process of proactive service, including customer entry identification, accurate demand prediction, intelligent terminal scheduling, and seamless handover across terminals. Summary of the Invention

[0010] The main objective of this invention is to provide a SIM identity multimodal perception service method for multi-terminal collaborative proactive service in a business hall.

[0011] Another objective of this invention is to propose a SIM identity multimodal perception service hall multi-terminal collaborative proactive service device.

[0012] The third objective of this invention is to provide an electronic device.

[0013] A fourth objective of this invention is to provide a non-transitory computer-readable storage medium.

[0014] To achieve the above objectives, a first aspect of the present invention proposes a SIM identity multimodal perception service hall multi-terminal collaborative proactive service method, comprising: The near-field communication module deployed at the entrance of the business hall detects the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal, and binds and verifies it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated. The temporary session identifier and the encrypted identity identifier are sent to the business service agent. The business service agent calls the digital identity authentication interface of the authentication platform to complete the identity verification and obtains the customer's business profile data after the identity verification is successful. Collect multimodal behavior perception data of the customer in the business hall, including visual trajectory data, voice intent data, and dwell location data; Based on the business profile data, the attention weights of each modality are determined. The attention weights are then used to dynamically weight and fuse the multimodal behavior perception data and the business profile data to generate customer service demand prediction results. Based on the prediction results and the load and location information of each smart terminal, the main service terminal is selected. Based on the customer service demand prediction results, a service handover path is generated. The interface pre-configuration is performed on each smart terminal along the service handover path in sequence, and an active guidance instruction is sent to the main service terminal to enable the main service terminal to approach the customer and output guidance messages to complete the response to customer service demand and achieve seamless service handover across terminals.

[0015] Optionally, a near-field communication module deployed at the entrance of the business hall detects the encrypted identity identifier stored in the SIM card security element of the customer's mobile terminal, and binds and verifies it with the synchronously acquired device identifier. After successful verification, a temporary session identifier is generated, including: By deploying an NFC sensing array in the entrance area of ​​the business hall, the encrypted identity identifier stored in the ISD-P field of the SIM card security element in the customer's mobile terminal is read. The encrypted identity identifier conforms to the GlobalPlatform specification. By deploying Bluetooth beacon arrays in the same area, the BLE MAC address of the customer's mobile terminal is synchronously obtained as the device identifier; The encrypted identity identifier and the device identifier are sent to the local verification module to verify whether the correspondence between the two exists in the pre-registration whitelist; After successful verification, a temporary session identifier is generated. The temporary session identifier will automatically expire after a preset time after the customer leaves the NFC sensing range.

[0016] Optionally, the temporary session identifier and the encrypted identity identifier are sent to the service agent. The service agent calls the digital identity authentication interface of the authentication platform to complete the identity verification, and after successful identity verification, obtains the customer's business profile data, including: The temporary session identifier and the encrypted identity identifier are sent to the service agent via an HTTPS secure channel with TLS / SSL encryption. The encrypted identity identifier is submitted to the authentication platform by calling the SIM digital identity authentication interface of the authentication platform through the RESTful API. Identity verification is completed based on the two-way authentication between the SIM card security element and the operator's core network, and the authentication result and corresponding customer mobile phone number returned by the authentication platform are obtained. After identity verification, the customer's business profile data is obtained from the business profile database. The business profile data includes historical business processing records, current pending business, contract status, and preference information.

[0017] Optionally, the multimodal behavior perception data of the customer in the business hall is collected, including: High-definition camera arrays deployed in the business hall are used to collect customer images. Customer skeleton information is extracted based on human posture recognition algorithm. Combined with trajectory tracking algorithm, the customer's walking trajectory and facial expression are obtained as visual trajectory data. Voice segments of the customer are collected using a directional microphone array. Based on an intent classification model, keywords are extracted and intents are classified from the voice segments, and the voice segments are converted into intent tags as voice intent data. The real-time coordinates of customers within the business hall are obtained through indoor positioning sensors as data on their location.

[0018] Optionally, attention weights for each modality are determined based on the business profile data, and the attention weights are used to dynamically weight and fuse the multimodal behavior perception data and the business profile data to generate customer service demand prediction results, including: Visual trajectory data, voice intent data, dwell location data, and business profile data are each converted into feature vector form; Based on the feature vectors of the business profile data, the attention weights of each modality are calculated using a normalized exponential function. When the business profile indicates that the customer has a clear business goal, the attention weights of voice intent and business profile are increased. When the customer is a first visitor or the business goal is unclear, the attention weights of visual trajectory and dwell position are increased. The feature vectors of each modality are weighted and summed according to their corresponding attention weights to generate a fused feature vector; The fused feature vectors are input into the classification network, which outputs the customer service demand prediction results and their confidence scores.

[0019] Optionally, a primary service terminal may be selected based on the prediction results combined with the load and location information of each smart terminal, including: Based on the business types included in the customer service demand prediction results, a set of candidate terminals with corresponding service capability tags is matched; Obtain the current load rate and geographic location coordinates of each smart terminal; The load balancing score is calculated based on the load rate of each terminal, and the distance score is calculated based on the distance between each terminal and the customer. Based on the preset load balancing weight and distance weight, each terminal in the candidate terminal set is weighted and comprehensively scored, and the terminal with the best score is selected as the main service terminal.

[0020] Optionally, a service handover path is generated based on the customer service demand prediction results. The interface is pre-configured for each smart terminal along the path, and an active guidance command is sent to the main service terminal to bring it closer to the customer and output guiding messages, thus responding to the customer service demand. This includes: When the customer service demand forecast results involve multiple business processes, a service handover path is generated according to the order of each business process. According to the service handover path, the interactive interface of each smart terminal on the path is pre-configured in sequence before the customer arrives. The content of the pre-configuration is determined according to the device type of the terminal and the business process it undertakes. When the main service terminal is a mobile robot, an active guidance instruction containing the customer's real-time location and guidance target is sent to the main service terminal; The system plans walking paths based on indoor navigation algorithms, receives updated customer location coordinates in real time and dynamically adjusts the path, and controls the main service terminal to actively approach the customer. Upon reaching the vicinity of the customer, a personalized guiding script is output through the voice interaction module to guide the customer to the target terminal location on the service handover path; After the customer arrives at the target terminal, a secondary identity verification is completed through the customer's mobile terminal to establish a service interaction session; During service handover, the session state, processed business data, and pending business list are encapsulated into a session handover packet and forwarded to the next service terminal through a unified message bus. This enables the next terminal to restore the service interaction interface based on the session handover packet, achieving seamless service handover.

[0021] To achieve the above objectives, a second aspect of the present invention provides a SIM identity multimodal perception service hall multi-terminal collaborative proactive service device, comprising: The near-field perception and detection unit is used to detect the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal through the near-field communication module deployed at the entrance of the business hall, and bind and verify it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated. The identity authentication profiling unit is used to send the temporary session identifier and the encrypted identity identifier to the business service intelligent agent. The business service intelligent agent calls the digital identity authentication interface of the authentication platform to complete the identity authentication and obtains the customer's business profile data after the identity authentication is successful. A multimodal perception acquisition unit is used to collect multimodal behavior perception data of the customer in the business hall. The multimodal behavior perception data includes visual trajectory data, voice intent data, and dwell position data. The fusion scheduling and allocation unit is used to determine the attention weight of each modality based on the business profile data, use the attention weight to dynamically weight and fuse the multimodal behavior perception data and business profile data to generate customer service demand prediction results, and select the main service terminal based on the prediction results combined with the load information and location information of each smart terminal. The collaborative service execution unit is used to generate service handover paths based on the predicted customer service needs, perform interface pre-configuration on each smart terminal along the service handover path, and send an active guidance instruction to the main service terminal to enable the main service terminal to approach the customer and output guidance messages, thereby responding to customer service needs and achieving seamless service handover across terminals.

[0022] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0023] To achieve the above objectives, a third aspect of this application provides an electronic device, including a processor and a memory; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, for implementing a SIM identity multimodal perception business hall multi-terminal collaborative proactive service method as described in the first aspect embodiment.

[0024] To achieve the above objectives, the fourth aspect of this application proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements a SIM identity multimodal perception business hall multi-terminal collaborative proactive service method as described in the first aspect embodiment.

[0025] The embodiments of the present invention have the following beneficial effects: 1. Extend SIM digital identity authentication capabilities from online to offline service hall scenarios, enabling seamless identity recognition for customers upon entry through near-field communication.

[0026] 2. Construct a multimodal fusion prediction method based on attention mechanism, using business profile features as the basis for weight allocation, and adaptively weightedly fuse four types of data: visual trajectory, voice intent, dwell position and business profile, to improve the accuracy of demand prediction.

[0027] 3. Establish a multi-terminal collaborative service scheduling mechanism, select the main service terminal based on the terminal status matrix and generate a service handover path; through session handover packet encapsulation and recovery, and with the secondary confirmation of the SIM dynamic authentication token, achieve seamless service transmission across terminals and eliminate the sense of process disconnect between customers' repeated authentication and information entry.

[0028] 4. Simultaneously, a mobile robot proactive guidance mechanism is configured, transforming the traditional passive service mode into a proactive intelligent guidance mode. This invention achieves a fully proactive service process of "identification upon entry, prediction of demand, scheduling of terminals, and handover of services." Attached Figure Description

[0029] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a SIM identity multimodal perception service hall multi-terminal collaborative proactive service method provided by an embodiment of the present invention; Figure 2 This is a schematic diagram of a hierarchical architecture for near-field detection of SIM digital identity provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the multimodal perception attention fusion network structure provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the multi-terminal collaborative service scheduling logic for a business hall provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the module architecture of the SIM identity multimodal perception business hall collaborative service system provided in an embodiment of the present invention; Figure 6 This is a structural diagram of a SIM identity multimodal perception service hall multi-terminal collaborative proactive service device provided in an embodiment of the present invention. Detailed Implementation

[0030] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] The following description, with reference to the accompanying drawings, describes a method and apparatus for multi-terminal collaborative proactive service in a SIM identity multimodal perception service hall, according to an embodiment of the present invention.

[0033] Example 1 This invention provides a method for SIM identity multimodal perception-based multi-terminal collaborative proactive service in a business hall. Figure 1 This is a flowchart illustrating a SIM identity multimodal perception service hall multi-terminal collaborative proactive service method provided by an embodiment of the present invention. Figure 1 As shown, the method includes the following steps: Step S1: The near-field communication module deployed at the entrance of the business hall detects the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal, and binds and verifies it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated.

[0034] It should be understood that this step corresponds to Figure 2 The architecture consists of a perception layer and a processing layer front-end module. The perception layer is the hardware equipment in the entrance area of ​​the business hall. It is composed of an NFC sensing array and a Bluetooth beacon array to form a near-field communication module, which is responsible for collecting the original identity data of customers entering the hall. The collected identity data is uniformly transmitted to the binding verification module and SIM authentication module in the processing layer to complete the pre-verification work.

[0035] It should be noted that the detailed execution process in this step includes three parts: multi-channel parallel signal acquisition, local binding whitelist verification, and temporary session identifier generation. All original identity data follows the requirements noted at the bottom of the attached diagram and is uniformly encrypted using the national cryptographic standard SM4. Before the authorization operation is completed, the system will not persistently store any original identity data to ensure privacy and security during the entry stage.

[0036] For example, when a customer enters the entrance area of ​​the business hall with a mobile terminal equipped with a SIM card, two arrays in the sensing layer simultaneously start collecting data. One is an NFC sensing array, which follows the ISO / IEC 14443 communication protocol and reads the encrypted identity identifier stored in the ISD-P field of the Security Element (SE) of the customer's mobile terminal SIM card. This encrypted identity identifier is a unique identification code after MSIN hashing, and the generation standard strictly follows the GlobalPlatform industry specifications. The other is a Bluetooth beacon array deployed in the same area. The array adopts the BLE 5.0 communication standard and simultaneously captures the BLE MAC address of the customer's mobile terminal as a unique device identifier.

[0037] It should be noted that after the two arrays have completed the data collection, the encrypted identity identifier and device identifier will be transmitted to the binding verification module inside the processing layer. The core working logic of the binding verification module is to match the pre-registered whitelist and verify whether there is a compliant binding relationship between the encrypted identity identifier and the corresponding MAC device identifier. The binding verification process is considered to have passed only if the matching record exists in the whitelist.

[0038] To further explain, after the binding verification module completes the verification, it synchronously pushes the verification pass signal to the temporary session identifier generation module in the processing layer. This module generates a unique session identifier (TID) with the naming format TID_YYYYMMDD_CUSTOMER. This TID is the unique session identifier for the entire subsequent process. At the same time, the authorization determination module in the attached figure will synchronously receive the binding verification result, providing a basis for subsequent authorization determination.

[0039] Optionally, this application sets up an expiration control logic for the temporary session identifier. When a customer leaves the NFC sensing array coverage area at the entrance of the business hall, the system starts timing. After a preset duration of 30 minutes, the temporary session identifier generated this time will automatically expire. After the expiration, all service sessions based on this TID will be terminated, thus eliminating the data security risks caused by the long-term retention of identity credentials.

[0040] It is worth mentioning that while the processing layer completes the TID generation, the authorization determination module inside the processing layer pushes a trigger authorization delivery signal to the output layer. After receiving the signal, the identity awareness authorization prompt module inside the output layer pushes a pop-up authorization prompt to the customer's mobile terminal. The pop-up displays the message "China Unicom AI Business Hall provides you with intelligent proactive services. Do you authorize the identification of your SIM digital identity?" Only after the customer manually confirms the authorization will the subsequent S2 to S5 full-process service link be officially started. If the customer chooses to refuse authorization, the system directly switches to the regular self-service mode without identity recognition and no longer executes the subsequent identity authentication, demand prediction, and intelligent scheduling related processes.

[0041] It should be noted that the near-field communication module used for entry identity perception in this application can be replaced by other equivalent hardware devices, specifically a WiFi probe array or a UWB positioning array. The WiFi probe array achieves device identification by detecting the MAC address of the mobile phone's WiFi signal, while the UWB positioning array completes entry perception by detecting the UWB tag signal on the terminal. Both replacement schemes can basically achieve the function of customer entry identity perception, but the overall recognition accuracy and security level are lower than the NFC+Bluetooth combination scheme of this application. The WiFi probe can only obtain the publicly available MAC address and cannot read the encrypted unique identifier in the SIM card's security element, which has the security flaw of single identity recognition dimension and easy for forgery.

[0042] Step S2: The temporary session identifier and the encrypted identity identifier are sent to the service agent. The service agent calls the digital identity authentication interface of the authentication platform to complete the identity verification and obtains the customer's business profile data after the identity verification is successful.

[0043] It should be understood that this step is connected. Figure 2 The processing layer output link and the top-level operational service intelligent agent module of the output layer are connected. The session identifier TID and encrypted identity identifier generated by the processing layer are transmitted to the operational service intelligent agent of the output layer through the authorization judgment module triggering the authorization delivery link. The operational service intelligent agent is the core scheduling and computing unit of the entire process of this application, and undertakes all core computing tasks such as identity authentication, profile retrieval, multimodal fusion and terminal scheduling.

[0044] It should be noted that the data transmission links for the two types of sensitive identity data, temporary session identifiers and encrypted identity identifiers, are protected by dual encryption. After data encapsulation, the data is transmitted through an HTTPS secure channel with an additional TLS / SSL encryption suite to avoid the risk of data eavesdropping and tampering during transmission.

[0045] To further explain, after receiving the encrypted identity identifier and TID, the service agent calls the built-in SIM digital identity authentication interface of the operator's authentication platform through a standardized RESTful API interface. It submits the encrypted identity identifier to the authentication platform to complete the authentication process. The authentication process strictly follows the 3GPP TS 33.401 communication specification and relies on the two-way encrypted authentication mechanism between the SIM card security element and the operator's core network to complete the verification of the real identity. After the verification is completed, the authentication platform sends back two core data to the service agent, namely the identity authentication result and the customer's corresponding mobile phone number.

[0046] It should be further explained that the service intelligence agent can only access the backend business profile database and retrieve the complete business profile data of the customer when the identity authentication result is passed. The profile data contains multi-dimensional business information, specifically divided into four categories: historical business processing records, currently unfinished pending business, various contract operation statuses (including the expiration time of the smart life card contract), and customer service preference information. The preference information is further subdivided into sub-tags such as self-service device service preferences and voice interaction service preferences, providing basic prior features for subsequent S3 and S4 multimodal fusion and demand prediction.

[0047] It is worth mentioning that the SIM digital identity near-field authentication method of this application can be equivalently replaced by other civilian identity recognition schemes, specifically QR code scanning authentication or facial recognition authentication. Both alternative schemes can achieve the basic function of customer entry identity recognition, but both have obvious shortcomings. QR code scanning authentication requires the customer to manually cooperate in scanning the code, which cannot achieve seamless recognition; facial recognition authentication has the risk of facial privacy data leakage, and the privacy protection level is lower. Overall, its convenience and security are inferior to the encryption authentication scheme based on SIM security elements of this application.

[0048] Step S3: Collect the customer's multimodal behavior perception data in the business hall. The multimodal behavior perception data includes visual trajectory data, voice intent data, and dwell location data.

[0049] It should be understood that this step is the full-domain data collection stage after the customer enters the business hall. Multimodal perception data is the input basis of the S4 attention fusion network. The three types of perception data are collected by different hardware devices. Each hardware device follows the corresponding industry communication and algorithm specifications. The dimensions of the collected data do not overlap and can completely restore all the behavioral characteristics of the customer in the hall.

[0050] For example, the visual trajectory data is collected by a 1080P high-definition camera array deployed throughout the business hall. The camera array has an indoor space coverage of over 95%. The real-time customer images collected are input based on the human posture recognition algorithm of OpenPose skeleton detection, which extracts the key point information of the customer's human skeleton. Combined with the real-time trajectory tracking algorithm, two types of visual features are output simultaneously: the customer's entire walking trajectory and real-time facial expression change data, which together constitute the complete visual trajectory data.

[0051] Optionally, the voice intent data is collected using a 6-channel directional microphone array. The directional sound pickup range of the microphone array is 3 meters, capturing only customer voice segments within the range. After collection, an intent classification algorithm based on a BERT pre-trained model is used to complete two stages of processing. The first step is to extract keywords from the voice text, and the second step is to perform standardized intent classification based on the extracted keywords. Finally, the original voice segments are converted into standardized structured intent labels. For example, if the customer's voice content is "I want to change a card", the algorithm extracts the keyword "change card" and outputs the standardized intent label "SIM card replacement".

[0052] It is worth mentioning that the location data is collected in real time by UWB indoor positioning sensors. The sensor positioning accuracy can reach ±10cm. The hardware communication standard follows the IEEE 802.15.4z specification and continuously outputs the customer's two-dimensional real-time coordinates in the business hall. The coordinate data is synchronously pushed to the business service intelligent agent for location feature calculation.

[0053] Step S4: Determine the attention weight of each modality based on the business profile data, use the attention weight to dynamically weight and fuse the multimodal behavior perception data and business profile data to generate customer service demand prediction results, and select the main service terminal based on the prediction results combined with the load information and location information of each smart terminal.

[0054] It should be understood that this step consists of two core computations: the first part is the multimodal attention fusion network computation, corresponding to... Figure 3 The schematic diagram of the multimodal perception data fusion analysis network structure shows the complete logic. The back end is the comprehensive scoring and screening logic of the main service terminal, which provides the screening basis for S5 terminal scheduling. Both segments of the operation are completed independently by the service agent.

[0055] It should be noted that the input of the multimodal fusion network includes four types of standardized feature vectors: a 128-dimensional visual trajectory feature vector V, a 64-dimensional speech intent feature vector A, a 32-dimensional dwell position feature vector L, and a 256-dimensional business profile feature vector P. These four types of vectors are uniformly input into the attention weight calculation layer. The weight calculation layer uses the business profile feature vector P as the core benchmark and dynamically calculates the attention weights specific to the four modalities through a softmax normalized exponential function. The weight allocation rules are divided into two types of scene adaptive adjustment.

[0056] To further explain, the first scenario involves business profile data indicating that the customer has a clear business objective, such as a customer's pending business tag being SIM card replacement. In this case, the system automatically increases the weight of voice intent Wa to 0.45 and the weight of business profile Wp to 0.35, prioritizing the customer's voice requests and historical business data to determine the needs. The second scenario involves a customer visiting the business hall for the first time and whose profile does not indicate a clear business objective. In this case, the system increases the weight of visual trajectory Wv to 0.40 and the weight of dwelling location Wl to 0.30, predicting potential service needs based on the customer's walking trajectory and dwelling location behavior.

[0057] It should be noted that after the weight calculation is completed, the system enters the weighted fusion layer, where the vectors are weighted and summed according to a fixed fusion calculation formula, outputting a unique fused feature vector F. The formula is as follows:

[0058] in, The fused feature vector is a unified feature representation that integrates information from four modalities and will be directly fed into the downstream classification network to complete demand prediction. The visual trajectory attention weight is a floating-point value in the range of 0 to 1, and takes the value of 0.40 when the customer has no clear business objective. A 128-dimensional visual trajectory feature vector is used to store the customer skeleton, trajectory, and quantitative features of facial expressions. This represents the attention weight for voice intent, a floating-point value in the range of 0 to 1, with a value of 0.45 when the customer has a specific pending task. It is a 64-dimensional voice intent feature vector, storing voice keywords and quantitative indicators of business intent; The attention weight for the location of the visitor is a floating-point value in the range of 0 to 1, which increases the weight when the customer's visit goal is unclear. A 32-dimensional feature vector representing the dwell location is used to store quantitative data on the customer's indoor coordinates and dwell time. The attention weight for business profiles is a floating-point value in the range of 0 to 1, with a value of 0.35 when the customer has a clear business need. A 256-dimensional business profile feature vector is generated, storing anonymized and quantified features of customer historical business transactions, contracts, and interaction preferences. It should be noted that... The sum of the four weights after softmax normalization is 1. All operations are element-wise weighted operations on vectors, so there will be no deviation in dimension calculation.

[0059] The fused feature vector F is fed into a three-layer fully connected MLP classification network. The network uses the ReLU activation function to perform nonlinear operations and finally outputs the customer service demand prediction results, along with the corresponding demand confidence values. For example, the primary demand is "SIM card replacement" with a confidence of 0.92, and the secondary demand is "contract renewal consultation" with a confidence of 0.65.

[0060] It is worth mentioning that after completing the demand forecast, the main service terminal screening process begins. The business service intelligent agent pre-builds a global terminal status matrix for the business hall. The matrix stores four core parameters for each intelligent terminal: a unique identifier for the type, real-time load rate, indoor geographic coordinates, and service capability tags. During the screening, all terminals with corresponding service capability tags are first selected based on the business type in the forecast results, forming a candidate terminal set.

[0061] Optionally, the terminal comprehensive score is calculated by weighting two dimensions. The load balancing dimension has a fixed weight of 0.4, and the load balancing single score is calculated as (1 - terminal real-time load rate) × 0.4. The distance dimension has a fixed weight of 0.6, and the distance single score is calculated based on the spatial distance between the terminal and the customer's real-time coordinates, and then multiplied by a weight coefficient of 0.6. The two scores are added together to obtain the terminal comprehensive score. All terminal scores in the candidate set are sorted, and the terminal with the highest comprehensive score is selected as the main service terminal for this service.

[0062] It should be noted that the attention mechanism fusion network used in this application can be replaced by other equivalent fusion structures. Specifically, it can be replaced by a feature concatenation + fully connected network structure or a gating mechanism fusion structure. The feature concatenation scheme directly concatenates the feature vectors of the four modalities and inputs them into a three-layer MLP network to complete the classification. The gating mechanism fusion controls the transmission throughput of each modality information through the gating unit. Both alternative schemes can realize the basic functions of multimodal data fusion and demand prediction, but they lack the ability to dynamically and adaptively adjust weights. Their adaptability to new customers, regular customers, and different business scenarios is far weaker than the attention mechanism fusion scheme in this application, and their demand prediction accuracy and scenario generalization are worse.

[0063] Step S5: Generate a service handover path based on the customer service demand prediction results, perform interface pre-configuration on each smart terminal along the service handover path in sequence, and send an active guidance command to the main service terminal to enable the main service terminal to approach the customer and output guidance messages to complete the response to customer service demand and achieve seamless service handover across terminals.

[0064] It should be understood that this step corresponds to Figure 4 A schematic diagram illustrating the logic of multi-terminal collaborative service scheduling strategy generation, showing the complete execution chain and its interconnection. Figure 2 All modules in the output layer send dispatch instructions to the multi-terminal dispatch module and the operational service intelligent agent interaction unit. The multi-terminal dispatch module uniformly receives the session identifier TID and completes the distribution of instructions across all terminals. The whole process is divided into five sub-processes: path generation, terminal pre-configuration, robot active guidance, secondary identity verification, and seamless cross-terminal handover.

[0065] It should be noted that when the customer service demand forecast results include two or more consecutive business processes, the service intelligence agent generates a complete service handover path according to the standardized processing sequence of the business. Distance constraints are set during the path generation process, and the physical straight-line distance between two adjacent service terminals within the path shall not exceed a preset threshold of 5 meters, so as to ensure that customers can complete the terminal switching in a short distance and reduce the sense of disjointed handover experience.

[0066] To further explain, before a customer arrives at the first service terminal, the intelligent service agent pre-configures all terminals along the handover path according to the handover sequence. The pre-configuration content varies depending on the type of terminal: desktop service terminals pre-load a customer-specific interactive interface with built-in high-frequency service entry points, contract expiration reminders, and personalized service recommendations. Simultaneously, relevant customer service data is pre-loaded and cached locally on the terminal via an HTTPS channel. The AI ​​self-service card writing machine pre-configures the APDU standard instruction stream based on the customer's SIM card type and contract status, following the GlobalPlatform and JavaCard specifications. The instructions include card-writing-specific commands such as SELECT AID and WRITE BINARY, and simultaneously pre-loads optional service packages, customer numbers, and contract cache data. The wearable electronic badge (Youai Xiaoban) pushes customer identity summaries and standardized reception guidance messages through its display module. An example push message is: "Ms. Zhang, 186XXXXXXXX, applying for SIM card replacement. Contract is about to expire; we suggest renewal," assisting offline service personnel in quickly identifying customer needs.

[0067] It should be further explained that if the main service terminal selected by S4 is the bipedal robot T1, the service agent sends a complete active guidance command to the robot. The command data packet encapsulates two types of location information: the customer's real-time coordinates and the coordinates of the target terminal on the handover path. After receiving the command, the robot's built-in indoor navigation algorithm plans the optimal barrier-free walking path in real time. During the walking process, it continuously receives the customer's real-time coordinates updated synchronously by the service agent and dynamically corrects the walking route until it reaches the designated interaction range around the customer. After the robot reaches the target location, it outputs customized personalized greetings and service guidance messages through its own voice interaction module and actively guides the customer to the first business terminal on the handover path.

[0068] It is worth mentioning that after the customer arrives at the target service terminal within the path, the system initiates a secondary identity verification process. The target terminal pushes a secondary authentication request to the customer's mobile terminal. The SIM digital identity authentication module inside the customer's mobile phone calls the SIM security element to generate a dynamic authentication token. The token is signed using the national cryptographic SM2 algorithm and a timestamp is attached. The token's validity period is set to 60 seconds. After the target terminal verifies that the token signature and timestamp are both legal and valid, a dedicated service interaction session is established. The session is bound to the temporary session identifier TID generated by S1 throughout the process, ensuring that the identity information is continuously traceable throughout the entire process.

[0069] For example, during the service interaction process, the terminal continuously collects customer interaction behavior characteristics. The collected indicators include three categories: real-time progress of business processing, duration of single interface interaction, and facial expression recognition results. The system presets three types of service handover trigger judgment conditions. Cross-terminal handover is initiated when any one of them is met: First, all business links corresponding to the current terminal are completed, and there are subsequent pending business links; Second, the duration of a single interaction exceeds a preset threshold of 90 seconds, and the facial expression recognition determines that the user is in a confused state; Third, the customer actively issues an interaction command to switch service terminals via voice.

[0070] Optionally, after the handover logic is triggered, the current service terminal encapsulates the complete interaction data into a JSON format session handover packet. The handover packet always contains three core types of content: session_id (session identifier), completed_tasks (completed business fields), and pending_tasks (pending business fields). The handover packet is forwarded to the next service terminal in the handover path through the unified message bus built by the service agent. After receiving the handover packet, the next terminal directly reads all session status and business data, quickly restores the unfinished interaction interface, and does not require the customer to repeatedly complete identity authentication and manually enter personal information, thus achieving seamless handover across terminals.

[0071] It should be noted that the bipedal robot proactive guidance mode of this application can be equivalently replaced by other guidance methods. Specifically, it can be replaced by displaying guidance information on a fixed guidance screen at the entrance of the business hall, or by triggering service personnel to proactively greet customers through information push on electronic badges. Both alternative solutions can achieve basic customer guidance services, but they can only provide standardized and undifferentiated general guidance content. They cannot output personalized guidance services based on the customer's real-time location, behavior status, and business needs. The service accuracy, intelligence, and personalization are significantly lower than the bipedal robot proactive guidance solution.

[0072] It should be understood that this application is equipped with a complete and compliant privacy protection mechanism throughout the entire process, covering the entire lifecycle from S1 entry to S5 service termination. During the entry phase, the system can only initiate SIM identity detection after the customer manually confirms and authorizes the process. During the operation of the service intelligent agent, business profiles and multimodal behavior perception data are uniformly desensitized. The desensitized data is stored using the national cryptographic standard SM4, and only the feature information necessary for demand prediction calculation is retained. After the service session ends, the temporary session identifier automatically expires after 30 minutes, and the stored desensitized feature data is automatically deleted by the system after 7 days. The data processing rules throughout the entire process fully comply with the relevant compliance and control requirements of the Personal Information Protection Law.

[0073] To facilitate the explanation of the technical solution of the present invention, the following description is based on three specific application embodiments.

[0074] First implementation method: Existing customers clearly define the intelligent business processing scenarios.

[0075] This implementation method is applicable to existing customers in service halls with complete business profiles and clearly defined pending tasks, representing the most typical conventional application scenario for this application. When a customer enters the service hall entrance area with their mobile phone, the NFC sensing array and Bluetooth beacon array deployed at the entrance simultaneously collect data, reading the encrypted identity identifier from the customer's SIM card's secure element and collecting the phone's BLE MAC device identifier. Both sets of data are sent to a local verification module for whitelist binding verification. Upon successful verification, the system generates an automatically expired temporary session identifier. After the customer authorizes SIM digital identity recognition, the encrypted identity identifier and temporary session identifier are uploaded to the service agent via a TLS / SSL encrypted HTTPS channel.

[0076] The service agent calls the authentication platform interface to complete two-way authentication. After successful authentication, it retrieves customer profile data such as historical business records, pending business, contract status, and service preferences. Inside the business hall, the multimodal perception module simultaneously collects customer visual trajectory, voice intent, and dwell location data. Based on the customer's clearly defined pending business tag of SIM card replacement, the attention mechanism fusion network adaptively allocates weights, setting the voice intent weight Wa to 0.45 and the business profile weight Wp to 0.35. Combined with the visual trajectory weight and dwell location weight, multimodal feature weighted fusion is completed. The trained three-layer MLP fully connected classification network outputs the customer's main need as SIM card replacement, along with a high-confidence prediction result.

[0077] Furthermore, the service intelligence agent constructs a global terminal status matrix, combining load balancing weights and distance weights to complete a comprehensive terminal score, selecting the optimal AI self-service card writer as the primary service terminal, and pre-loading the corresponding SIM card writing APDU instructions, contract packages, and customer business data. Simultaneously, a bipedal robot is dispatched to provide proactive path guidance. Upon arrival at the terminal, the customer completes secondary identity verification using a national cryptographic SM2 dynamic token generated by the SIM security element, establishing a continuous service session. If there is a subsequent contract renewal consultation requirement, the system automatically generates a terminal handover path, completing a seamless cross-terminal handover through standardized session handover packages. The entire process requires no repeated customer authentication or information entry. After the service is completed, the temporary session identifier automatically expires, and de-identified data is periodically cleared, meeting privacy compliance requirements.

[0078] Second implementation method: Intelligent guidance scenario for first-time visitors with blank customer profiles.

[0079] This implementation method corresponds to the scenario of a first-time customer visit. Specifically, when Mr. Li enters the business hall for the first time, the system's business profile database contains no historical business data for him, only basic information such as his mobile phone number. The overall business profile data is almost blank. After the customer enters, the S1 to S3 processes of this application are executed normally, completing the SIM digital identity near-field verification, cloud identity authentication, and multimodal behavioral data collection.

[0080] It should be understood that, since the customer has no historical business records and no preset business goals, the attention mechanism of the service agent integrates the network to adaptively adjust the weight ratio of each modality, weakening the reference ratio of business profile data. The specific weights are set as follows: visual trajectory weight Wv=0.40, dwelling position weight Wl=0.30, voice intent weight Wa=0.20, and business profile weight Wp=0.10, relying on real-time customer behavior data to guide demand prediction. The high-definition camera array in the business hall, relying on the OpenPose skeleton detection algorithm and trajectory tracking algorithm, continuously captures Mr. Li's on-site behavior, detecting that the customer continuously lingers and wanders back and forth in the business area of ​​the business hall, with facial expressions showing a typical confused and bewildered state. UWB positioning data simultaneously confirms that the customer has no fixed service location and is in a state of aimless wandering.

[0081] Multimodal perception data and lightweight business profile feature vectors are substituted into an attention fusion formula to complete weighted calculations, generating a fused feature vector which is then input into a classification network. This ultimately accurately predicts the customer's primary need for new customer guidance and consultation. Based on this, the service agent retrieves the terminal state matrix to complete a comprehensive score, selecting a bipedal robot T1 as the primary service guidance terminal and issuing proactive guidance commands containing the customer's real-time UWB coordinates. The robot dynamically plans and corrects its walking path in real time using indoor navigation algorithms, proactively approaching the customer and outputting personalized guidance scripts. Simultaneously, throughout the entire human-computer interaction process, a 6-channel directional microphone array continuously collects customer voice segments in real time, dynamically updates voice intent features, and continuously corrects the customer service need prediction results, providing pre-scheduling support for subsequent potential business needs such as card application, consultation, and account opening.

[0082] Third implementation method: Seamless manual handover scenario for customers experiencing confusion.

[0083] This implementation method addresses seamless handover scenarios where customer service processing is hindered or confusion triggers manual intervention. Specifically, after Mr. Wang completes SIM digital identity authentication upon entering the premises, the system predicts his need for a plan change based on his business profile. It then dispatches desktop terminal T2 as the primary service terminal, pre-loading the plan change interface and customer business data. Upon Mr. Wang's arrival at the terminal, he completes secondary identity verification, and the plan change process proceeds normally.

[0084] During business interactions, the system continuously monitors the duration of customer interactions and their facial expressions. When a customer's single desktop terminal interaction time exceeds 120 seconds, and the high-definition camera array combined with the FACS facial motion coding system determines that the customer exhibits a typical confused expression with furrowed brows and shifty eyes, the system determines that the preset service interaction trigger conditions are met and automatically initiates the manual intervention dispatch process. The service intelligence agent searches the load status and location information of each electronic badge terminal in the business hall in real time, filtering out the on-site service personnel corresponding to the electronic badge W2 that is closest and has the lowest load.

[0085] Furthermore, the electronic work badge W2 instantly receives precise handover information pushed by the service intelligence agent. The push message specifically states: "Mr. Wang encountered difficulties in the package selection process; recommended script: Based on usage habits, we recommend package XX." This allows service personnel to quickly grasp the customer's current bottleneck, business scenario, and standardized reception plan. Upon arrival at the desktop terminal T2, the system automatically encapsulates the current session's session_id, the customer's selected package information, and the list of incomplete business transactions into a JSON-formatted session handover package, which is synchronized to the terminal in real time via a unified message bus. Service personnel can then directly continue providing business guidance from the existing interactive interface. This achieves seamless business takeover throughout the process. Mr. Wang does not need to repeatedly enter personal information or repeat his business needs, completely resolving customer processing bottlenecks and achieving a seamless transition between human-computer interaction and manual service.

[0086] Example 2 This invention also relates to a SIM identity multimodal perception service hall multi-terminal collaborative proactive service system, such as... Figure 5 As shown, the system is divided into six core functional modules that work together. All modules rely on the local area network inside the business hall and the unified message bus built on the MQTT protocol within the business service intelligent agent module to complete cross-module data interaction and command transmission, ensuring the real-time performance and encryption security of the entire data transmission chain. The six core modules are: near-field communication module 501, business service intelligent agent module 502, multimodal perception module 503, multi-terminal scheduling and execution module 504, service interaction and handover module 505, and privacy protection module 506. The composition, execution logic, and interaction relationship of each module will be explained in detail below.

[0087] It should be understood that the near-field communication module 501 is the system's entrance sensing hardware unit, deployed in the entrance area of ​​the business hall. The module integrates two parallel acquisition arrays: an NFC sensing array and a Bluetooth beacon array. The NFC sensing array follows the ISO / IEC 14443 communication protocol, with an effective sensing distance of no more than 10cm, and is used to read the encrypted identity identifier stored in the secure element of the customer's mobile terminal SIM card at close range. The Bluetooth beacon array adopts the BLE 5.0 communication standard, with an effective sensing distance controlled within 5m, and simultaneously acquires the BLE MAC device identifier of the customer's mobile phone. The encrypted identity data acquired by the two arrays is sent to the module's built-in verification unit to complete the whitelist verification of the binding between the identity and the device identifier. After the verification is successful, a globally unique temporary session identifier is generated. The module pushes the identity data packet containing the encrypted identity identifier, device identifier, and temporary session identifier to the identity authentication submodule of the business service intelligent agent module 502 through a unified message bus.

[0088] It should be noted that the Business Service Intelligent Agent Module 502 is the core decision-making and computation module of this system, internally divided into four independent functional sub-modules: identity authentication sub-module, profile acquisition sub-module, fusion analysis sub-module, and scheduling strategy sub-module. The identity authentication sub-module connects to the China Unicom authentication platform via a standardized RESTful API, receiving encrypted identity data uploaded by the near-field communication module 501 to complete two-way authentication of the SIM digital identity. After successful identity authentication, the profile acquisition sub-module retrieves full-dimensional customer business profile data stored in the business profile database via an HTTPS channel with TLS / SSL encryption. The fusion analysis sub-module receives visual, voice, and location-based perceptual feature vectors pushed by the multimodal perception module 503, and uses an attention mechanism fusion network to perform dynamic weighted fusion of multimodal data, outputting customer service demand prediction results. The scheduling strategy sub-module constructs a global business hall terminal status matrix, combines load balancing weights and customer distance weights to complete a comprehensive terminal score, generates a standardized terminal scheduling strategy, and finally distributes the scheduling strategy to the multi-terminal scheduling execution module 504 via a unified message bus.

[0089] To further clarify, the multimodal perception module 503 is a full-area behavioral data acquisition unit within the hall, comprising three types of acquisition sub-modules: a visual perception sub-module, a voice perception sub-module, and a location perception sub-module. The visual perception sub-module relies on a 1080P high-definition camera array deployed throughout the hall to acquire customer images, and uses OpenPose skeleton detection and trajectory tracking algorithms to extract customer walking trajectories and facial expression visual features. The voice perception sub-module uses a 6-channel directional microphone array to acquire customer voice segments within a 3-meter range, and uses the BERT intent classification model to complete keyword extraction and business intent label conversion. The location perception sub-module uses a UWB indoor positioning sensor conforming to the IEEE 802.15.4z standard to achieve high-precision real-time coordinate acquisition within ±10cm. The structured feature vectors acquired by the three sub-modules are uniformly encapsulated and continuously pushed to the fusion analysis sub-module of the service agent module 502 as the input data source for demand prediction.

[0090] It should be further explained that the multi-terminal scheduling and execution module 504 is the system instruction issuing and execution unit. It receives the scheduling strategy data packet issued by the business service intelligent agent module 502. The module connects to four types of business hall service terminal hardware, namely bipedal robots, desktop service terminals, AI self-service card writing machines, and electronic badges. According to the business type, terminal priority, and customer location information in the scheduling strategy, the module issues differentiated instructions to the corresponding terminals. For business terminals, it issues instructions for interface pre-configuration and business data caching. For bipedal robots, it issues active guidance instructions containing real-time customer coordinates and target locations. For electronic badges, it pushes customer identity summaries and standardized reception guidance scripts, completing the pre-scheduling and pre-preparation work of all terminals.

[0091] It's worth noting that the Service Interaction and Handover Module 505 is responsible for the full-process interaction control and seamless cross-terminal handover of customer business transactions. Internally, it is divided into a secondary authentication submodule, an interaction monitoring submodule, an interaction trigger determination submodule, and a session handover submodule. The secondary authentication submodule receives the SIM dynamic authentication token uploaded by the target terminal, verifies the validity of the national cryptographic SM2 signature and timestamp, and establishes a continuous service session bound to a temporary session identifier. The interaction monitoring submodule collects real-time data on customer business transaction progress, interface dwell time, and facial expression data. The interaction trigger determination submodule presets three types of interaction trigger conditions and determines in real-time whether to initiate the cross-terminal handover process. After the interaction triggers, the session handover submodule encapsulates the session_id, completed tasks, and pending tasks into a JSON-formatted session handover packet, which is forwarded to the next service terminal via a unified message bus, achieving seamless restoration of the interactive interface.

[0092] Optionally, the privacy protection module 506 serves as a full-process data compliance management unit, covering the entire lifecycle from customer access authorization, data processing and storage, to session termination and cleanup. The module includes an authorization management submodule, a de-identification processing submodule, and an automatic clearing submodule. The authorization management submodule controls the SIM digital identity recognition startup permission; the system can only initiate the identity collection process after the customer manually confirms authorization. The de-identification processing submodule uses the national cryptographic SM4 symmetric encryption algorithm to de-identify and encrypt the original data of business profiles and multimodal perception, retaining only the feature information necessary for demand prediction. The automatic clearing submodule has built-in time-limited control logic; the temporary session identifier automatically expires 30 minutes after the customer leaves the NFC sensing range, and the de-identified feature data stored after the session ends is automatically cleared by the system 7 days later, fully complying with the requirements of the Personal Information Protection Law.

[0093] It should be understood that, in combination with, for example Figure 5 The module interaction link shown in the diagram is completed through a unified message bus output by the service agent module 502, where data and command interactions between modules are performed. The near-field communication module 501 and the multimodal perception module 503 upload raw perception data to the service agent module 502, while the service agent module 502 outputs scheduling strategies to the multi-terminal scheduling execution module 504. The multi-terminal scheduling execution module 504 and the service interaction and handover module 505 simultaneously transmit terminal operating status and interaction session data back to the service agent module 502. At the same time, all sensitive customer data generated throughout the entire link is simultaneously accessed by the privacy protection module 506 to complete desensitization and timeliness control, forming a complete closed-loop SIM identity multimodal perception business hall multi-terminal collaborative proactive service system.

[0094] Example 3 This invention provides a SIM identity multimodal perception service hall multi-terminal collaborative proactive service device. Figure 6This is a schematic diagram of the structure of a SIM identity multimodal perception service hall multi-terminal collaborative proactive service device provided in an embodiment of the present invention. Figure 6 As shown, the device includes: The near-field perception and detection unit 100 is used to detect the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal through the near-field communication module deployed at the entrance of the business hall, and bind and verify it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated. The identity authentication and profiling unit 200 is used to send the temporary session identifier and the encrypted identity identifier to the business service intelligent agent. The business service intelligent agent calls the digital identity authentication interface of the authentication platform to complete the identity authentication and obtains the customer's business profile data after the identity authentication is successful. The multimodal perception acquisition unit 300 is used to collect the multimodal behavior perception data of the customer in the business hall. The multimodal behavior perception data includes visual trajectory data, voice intent data, and dwell position data. The fusion scheduling and allocation unit 400 is used to determine the attention weight of each modality based on the business profile data, use the attention weight to dynamically weight and fuse the multimodal behavior perception data and business profile data to generate customer service demand prediction results, and select the main service terminal based on the prediction results combined with the load information and location information of each smart terminal. The collaborative service execution unit 500 is used to generate a service handover path based on the customer service demand prediction results, perform interface pre-configuration on each smart terminal on the path in sequence according to the service handover path, and send an active guidance instruction to the main service terminal so that the main service terminal approaches the customer and outputs guidance words to complete the response to customer service demand and realize seamless service handover across terminals.

[0095] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0096] Example 4 To implement the methods of the above embodiments, the present invention also provides an electronic device, which includes a memory and a processor; wherein the processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the various steps of the methods described above.

[0097] Example 5 To implement the above embodiments, this application also proposes a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in the foregoing embodiments.

[0098] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0099] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0100] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A method for multi-terminal collaborative proactive service in a SIM-based identity multimodal perception service hall, characterized in that: include: The near-field communication module deployed at the entrance of the business hall detects the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal, and binds and verifies it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated. The temporary session identifier and the encrypted identity identifier are sent to the business service agent. The business service agent calls the digital identity authentication interface of the authentication platform to complete the identity verification and obtains the customer's business profile data after the identity verification is successful. Collect multimodal behavior perception data of the customer in the business hall, including visual trajectory data, voice intent data, and dwell location data; Based on the business profile data, the attention weights of each modality are determined. The attention weights are then used to dynamically weight and fuse the multimodal behavior perception data and the business profile data to generate customer service demand prediction results. Based on the prediction results and the load and location information of each smart terminal, the main service terminal is selected. Based on the customer service demand prediction results, a service handover path is generated. The interface pre-configuration is performed on each smart terminal along the service handover path in sequence, and an active guidance instruction is sent to the main service terminal to enable the main service terminal to approach the customer and output guidance messages to complete the response to customer service demand and achieve seamless service handover across terminals.

2. The method according to claim 1, characterized in that, The near-field communication module deployed at the entrance of the service hall detects the encrypted identity identifier stored in the SIM card security element of the customer's mobile terminal, and binds and verifies it with the synchronously acquired device identifier. Upon successful verification, a temporary session identifier is generated, including: By deploying an NFC sensing array in the entrance area of ​​the business hall, the encrypted identity identifier stored in the ISD-P field of the SIM card security element in the customer's mobile terminal is read. The encrypted identity identifier conforms to the GlobalPlatform specification. By deploying Bluetooth beacon arrays in the same area, the BLE MAC address of the customer's mobile terminal is synchronously obtained as the device identifier; The encrypted identity identifier and the device identifier are sent to the local verification module to verify whether the correspondence between the two exists in the pre-registration whitelist; After successful verification, a temporary session identifier is generated. The temporary session identifier will automatically expire after a preset time after the customer leaves the NFC sensing range.

3. The method according to claim 1, characterized in that, The temporary session identifier and the encrypted identity identifier are sent to the service agent. The service agent calls the digital identity authentication interface of the authentication platform to complete the identity verification, and after successful identity verification, obtains the customer's business profile data, including: The temporary session identifier and the encrypted identity identifier are sent to the service agent via an HTTPS secure channel with TLS / SSL encryption. The encrypted identity identifier is submitted to the authentication platform by calling the SIM digital identity authentication interface of the authentication platform through the RESTful API. Identity verification is completed based on the two-way authentication between the SIM card security element and the operator's core network, and the authentication result and corresponding customer mobile phone number returned by the authentication platform are obtained. After identity verification, the customer's business profile data is obtained from the business profile database. The business profile data includes historical business processing records, current pending business, contract status, and preference information.

4. The method according to claim 1, characterized in that, Collect multimodal behavior perception data of the customer in the business hall, including: High-definition camera arrays deployed in the business hall are used to collect customer images. Customer skeleton information is extracted based on human posture recognition algorithm. Combined with trajectory tracking algorithm, the customer's walking trajectory and facial expression are obtained as visual trajectory data. Voice segments of the customer are collected using a directional microphone array. Based on an intent classification model, keywords are extracted and intents are classified from the voice segments, and the voice segments are converted into intent tags as voice intent data. The real-time coordinates of customers within the business hall are obtained through indoor positioning sensors as data on their location.

5. The method according to claim 1, characterized in that, Based on the business profile data, attention weights for each modality are determined. These attention weights are then used to dynamically weight and fuse the multimodal behavior perception data and the business profile data to generate customer service demand prediction results, including: Visual trajectory data, voice intent data, dwell location data, and business profile data are each converted into feature vector form; Based on the feature vectors of the business profile data, the attention weights of each modality are calculated using a normalized exponential function. When the business profile indicates that the customer has a clear business goal, the attention weights of voice intent and business profile are increased. When the customer is a first visitor or the business goal is unclear, the attention weights of visual trajectory and dwell position are increased. The feature vectors of each modality are weighted and summed according to their corresponding attention weights to generate a fused feature vector; The fused feature vectors are input into the classification network, which outputs the customer service demand prediction results and their confidence scores.

6. The method according to claim 1, characterized in that, The primary service terminal is selected based on the prediction results, combined with the load and location information of each smart terminal, including: Based on the business types included in the customer service demand prediction results, a set of candidate terminals with corresponding service capability tags is matched; Obtain the current load rate and geographic location coordinates of each smart terminal; The load balancing score is calculated based on the load rate of each terminal, and the distance score is calculated based on the distance between each terminal and the customer. Based on the preset load balancing weight and distance weight, each terminal in the candidate terminal set is weighted and comprehensively scored, and the terminal with the best score is selected as the main service terminal.

7. The method according to claim 1, characterized in that, Based on the customer service demand forecast, a service handover path is generated. The interface of each smart terminal along the path is pre-configured sequentially, and a proactive guidance command is sent to the main service terminal. This prompts the main service terminal to approach the customer and output guiding messages, thus responding to the customer service demand. This includes: When the customer service demand forecast results involve multiple business processes, a service handover path is generated according to the order of each business process. According to the service handover path, the interactive interface of each smart terminal on the path is pre-configured in sequence before the customer arrives. The content of the pre-configuration is determined according to the device type of the terminal and the business process it undertakes. When the main service terminal is a mobile robot, an active guidance instruction containing the customer's real-time location and guidance target is sent to the main service terminal; The system plans walking paths based on indoor navigation algorithms, receives updated customer location coordinates in real time and dynamically adjusts the path, and controls the main service terminal to actively approach the customer. Upon reaching the vicinity of the customer, a personalized guiding script is output through the voice interaction module to guide the customer to the target terminal location on the service handover path; After the customer arrives at the target terminal, a secondary identity verification is completed through the customer's mobile terminal to establish a service interaction session; During service handover, the session state, processed business data, and pending business list are encapsulated into a session handover packet and forwarded to the next service terminal through a unified message bus. This enables the next terminal to restore the service interaction interface based on the session handover packet, achieving seamless service handover.

8. A SIM identity multimodal perception service hall multi-terminal collaborative proactive service device, characterized in that, include: The near-field perception and detection unit is used to detect the encrypted identity identifier stored in the SIM card security element in the customer's mobile terminal through the near-field communication module deployed at the entrance of the business hall, and bind and verify it with the synchronously acquired device identifier. After the verification is successful, a temporary session identifier is generated. The identity authentication profiling unit is used to send the temporary session identifier and the encrypted identity identifier to the business service intelligent agent. The business service intelligent agent calls the digital identity authentication interface of the authentication platform to complete the identity authentication and obtains the customer's business profile data after the identity authentication is successful. A multimodal perception acquisition unit is used to collect multimodal behavior perception data of the customer in the business hall. The multimodal behavior perception data includes visual trajectory data, voice intent data, and dwell position data. The fusion scheduling and allocation unit is used to determine the attention weight of each modality based on the business profile data, use the attention weight to dynamically weight and fuse the multimodal behavior perception data and business profile data to generate customer service demand prediction results, and select the main service terminal based on the prediction results combined with the load information and location information of each smart terminal. The collaborative service execution unit is used to generate service handover paths based on the predicted customer service needs, perform interface pre-configuration on each smart terminal along the service handover path, and send an active guidance instruction to the main service terminal to enable the main service terminal to approach the customer and output guidance messages, thereby responding to customer service needs and achieving seamless service handover across terminals.

9. An electronic device, characterized in that, Including processor and memory; The processor reads executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method as described in any one of claims 1-7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-7.

Images