A multimodal biometric enrollment method, apparatus, robot, and medium

Abstract

The application relates to the field of embodied intelligence, and particularly discloses a multi-modal biological feature input method, device, robot and medium, wherein the method comprises the following steps: firstly, an equipment message routing unit receives an input request sent by a mobile terminal through near field wireless communication; secondly, an input arrangement unit checks an account authorization state and a binding relationship, and if the input condition is met, the input arrangement unit proceeds; subsequently, the input arrangement unit maintains four types of input sessions according to a path, only one session is allowed at the same time, and a new request is rejected when the session is occupied; an entity input activates a local equipment to extract a feature vector, and a remote input forwards a terminal to collect data and then extract a feature; finally, after a completion instruction is received, a target template slot is determined, and a feature vector and a user identifier are written into a trusted execution environment security world in the form of an encrypted persistent object. Compared with the prior art, the application has the advantages of being capable of uniformly arranging and managing multi-modal input paths, high safety and the like.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence, and in particular to a method, apparatus, robot, and medium for recording multimodal biometric features. Background Technology

[0002] In recent years, the global robotics industry has maintained rapid growth. As embodied intelligent robots gradually enter home environments, the demand for shared use by multiple users, including master accounts and family members, is increasingly prominent. Facial and voiceprint biometric authentication, with its convenience and security advantages, is gradually becoming a core capability for robot interaction security. At the same time, AI-generated forgery technology is rapidly iterating, and the security protection capabilities of single-modal biometric authentication are continuously declining. Multimodal fusion authentication has become the industry-recognized technological development direction, placing higher demands on the supporting biometric data entry system.

[0003] Currently, some consumer-grade embody smart products have been equipped with facial recognition or voice interaction functions, and a few products have set up facial and voiceprint registration entry points on mobile devices. However, existing product solutions generally have shortcomings: incomplete biometric coverage, with most only supporting a single facial modality; limited registration paths, mostly supporting only device-based collection and failing to achieve collaboration between device-based collection and remote collection from mobile devices; weak multi-user management capabilities, lacking a hierarchical authorization system and authorization lifecycle linkage mechanism; and some solutions store biometric features in the cloud, which does not comply with biometric data security compliance requirements.

[0004] At the technical level, existing biometric authentication solutions based on Trusted Execution Environments (TEEs) mostly deploy the TrustZone trusted environment on cloud servers. User biometric data needs to be uploaded to the cloud for storage and verification, with the overall architecture centered on the cloud as the core of trust. For example, Chinese patent application CN112491844A proposes a voiceprint and facial recognition verification system and method based on a Trusted Execution Environment. This system applies TrustZone technology to smart home scenarios, isolating the cloud server environment into a normal world and a secure world. Users collect voice and facial images via mobile devices and upload them to the cloud. Within the Trusted Execution Environment, voiceprint features are extracted using Mel-frequency cepstral coefficients, and facial features are extracted using circular local binary patterns. Identity verification is then completed through two-factor matching, thereby improving the security of remote control of smart homes. However, this solution still has obvious limitations: its trusted execution environment and biometric database are both deployed in the cloud, and all biometric data must be uploaded and stored over the network, which poses a risk of transmission leakage and cannot meet the compliance requirements of biometric data retention on the device side and prohibition of uploading to the cloud; it only supports a single collection path for mobile terminals, does not cover the device itself collection scenario, and does not have the ability to uniformly orchestrate and manage session mutual exclusion for multi-path input.

[0005] Overall, existing technologies have the following significant drawbacks in practical applications: First, cloud-based trusted architectures are insufficient to meet on-device compliance requirements. Solutions that deploy biometric storage and verification in the cloud require the transmission of sensitive biometric data across networks, posing a risk of data leakage. They cannot meet the compliance constraints of not allowing facial and voiceprint data to be stored in the cloud, and also make it difficult to achieve near-field closed-loop security control.

[0006] Second, there is a lack of unified orchestration capabilities for multi-path data entry. Existing solutions do not perform unified session scheduling and path distribution for body data acquisition and mobile terminal near-field remote data acquisition, making it impossible to achieve mutual exclusion control of multiple data entry paths, which can easily lead to concurrent conflicts and make it difficult to ensure the ease of data entry in different scenarios.

[0007] Third, the account lifecycle is disconnected from the data entry process. The existing data entry mechanism is not deeply bound to the account authorization status. There is a lack of verification of account permissions, authorization scope and validity period before data entry. There is also no mechanism to automatically terminate the data entry session and synchronize and clean up templates when authorization expires or the account is deleted. This can easily lead to security risks such as unauthorized data entry and residual expired templates.

[0008] Fourth, the security protection layers for remote data entry are insufficient. Existing solutions rely solely on link-layer pairing mechanisms to ensure connection security, without designing source authentication, replay protection, and block-level integrity verification mechanisms for the biometric data itself. Consequently, the authenticity and integrity of remotely entered data lack effective protection.

[0009] Against this backdrop, existing biometric data entry solutions are no longer adequate to meet the development needs of embodied intelligent robots for edge-side compliance, multi-user sharing, and multimodal collaboration, and a new edge-side multimodal biometric data entry method is urgently needed. Summary of the Invention

[0010] The purpose of this invention is to provide a multimodal biometric data entry method, device, robot, and medium.

[0011] The objective of this invention can be achieved through the following technical solutions: A multimodal biometric data entry method is executed on an embodied intelligent robot computing platform. The computing platform includes a device message routing unit, an entry orchestration unit, a face feature processing unit, a voiceprint feature processing unit, a wireless communication module, at least one camera, at least one microphone array, and a processor supporting TrustZone hardware security extensions. The computing platform is divided into a rich execution environment and a trusted execution environment. The method includes the following steps: Step S1: The device message routing unit obtains the input request sent by the mobile terminal via the near-field wireless communication link and forwards the input request to the input orchestration unit. The input request carries path type parameters, target user identifier and mobile terminal current login account identifier. The path type parameters indicate one of the following: face input, voiceprint input, remote face input or remote voiceprint input. Step S2: The input and arrangement unit reads the account authorization status and verifies the account binding relationship and input authorization in conjunction with the input request. If the input conditions are met, proceed to step S3. The account authorization status includes device binding status, main account flag, authorized account flag, authorized function scope and authorization expiration time. Step S3: The input arrangement unit maintains input sessions for four paths based on path type parameters and performs path distribution according to the path type parameters. Only one input session is allowed to be established at a time to activate the corresponding path. When an input request for another path arrives and an input session is already occupied, the input request is rejected and a path occupancy indication is returned. If the path type parameter indicates body face input or body voiceprint input, a body face input session or body voiceprint input session for the corresponding path is established, and a start input command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The corresponding feature processing unit activates the camera or microphone array to collect raw biological data and extracts biometric embedding vectors. If the path type parameter indicates remote face input or remote voiceprint input, a corresponding remote face input session or remote voiceprint input session is established, and a remote start input command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The remote collection data message sent by the mobile terminal is forwarded to the corresponding feature processing unit, and the feature processing unit extracts biometric embedding vectors from the forwarded raw biological data. Step S4: After receiving the input completion indication reported by the feature processing unit, the input arrangement unit determines the target template slot according to the path type parameter, and transmits the biometric embedding vector and the target user identifier based on the target template slot and writes them into the trusted execution environment security world in the form of an encrypted persistent object to complete the biometric input.

[0012] In step S2, if the device is not bound to the main account, the target user has not obtained identity entry permission, the authorization is out of scope, or the authorization has expired, it is determined that the entry conditions are not met; when the path type parameter indicates remote face entry or remote voiceprint entry, the consistency between the mobile terminal's current login account identifier and the target user identifier is also verified. If they are inconsistent, it is determined that the entry conditions are not met.

[0013] In step S3, the remote input session also includes device mutual trust authentication before forwarding the remote collection data packet: the input orchestration unit generates a random challenge value and sends it to the mobile terminal, receives the signed data packet and sequentially performs public key verification, hash consistency verification and challenge validity period check. After all passes, the data source is determined to be trustworthy, and then the remote collection data packet is forwarded; the signed data packet is generated by the mobile terminal by concatenating the input destination identifier, the challenge value and the feature hash value and signing it with the device trust ring private key.

[0014] In step S3, the remote input session transmits data in blocks to the corresponding feature processing unit. After each data block is processed, the feature processing unit reports the block processing result. The input orchestration unit receives the block processing result and sends a process synchronization request to the mobile terminal. It receives the synchronization receipt returned by the mobile terminal and relays the synchronization receipt to the feature processing unit until all data blocks are processed. The process synchronization request carries a synchronization data identifier and the hash value of the current data block after processing; the mobile terminal compares the hash value locally and returns a synchronization receipt indicating to continue or to re-collect; when the instruction to re-collect is received, the input and arrangement unit notifies the corresponding feature processing unit to reprocess the current data block.

[0015] The input and arrangement unit performs owner verification during remote data packet forwarding and process synchronization receipt. It verifies whether the communication source identifier, the mobile terminal's current login account identifier, and the target user identifier are all consistent with the session record of the current activation path. If they are inconsistent, it refuses to execute and returns an owner mismatch indication.

[0016] When the cloud pushes an account list update to the input and arrangement unit, the cloud account metadata and the locally stored biometric references are merged according to the target user identifier, without overwriting the locally generated feature identifiers, and the version broadcast identifiers of the account, face, or voiceprint are incremented; when a biometric deletion request, account unbinding request, authorization deletion request, or factory reset request is received from the mobile terminal, the persistent object corresponding to the biometric stored in the trusted execution environment is deleted at the user granularity or device granularity, and the corresponding biometric reference in the account configuration is cleared.

[0017] Each user's official template slots include a face slot, an application-side synchronous voiceprint slot, and an end-side enhanced voiceprint slot. The on-the-spot face registration and remote face registration update the same face slot mutually exclusively, and the on-the-spot voiceprint registration and remote voiceprint registration update the same application-side synchronous voiceprint slot mutually exclusively. The method further includes a shadow mode voiceprint enhancement step: when the time since the last end-side enhanced voiceprint recording reaches a preset interval and candidate voiceprint features are generated during daily recognition, the candidate voiceprint features are written into a short-term cache; within the cache validity period, face matching results, lip movement detection results, and voiceprint matching results are obtained; when the face matching result indicates that the user in the current image is the user identified by the corresponding user, the lip movement detection result indicates that the user in the current image is speaking, and the voiceprint matching result indicates that the candidate voiceprint features match the application-side synchronous voiceprint template or the end-side enhanced voiceprint template, the end-side enhanced voiceprint copy of the same user is replaced with the candidate voiceprint features, while keeping the application-side synchronous voiceprint copy unchanged; when any determination fails or the cache times out, the candidate voiceprint features are discarded.

[0018] A multimodal biometric data entry device is applied to an embodied intelligent robot computing platform. The computing platform includes a device message routing unit, a data entry and arrangement unit, a facial feature processing unit, a voiceprint feature processing unit, a wireless communication module, at least one camera, at least one microphone array, and a processor supporting TrustZone hardware security extensions. The computing platform is divided into a rich execution environment and a trusted execution environment. The device includes: The device message routing unit is used to obtain the input request sent by the mobile terminal via the near-field wireless communication link and forward the input request to the input orchestration unit. The input request carries a path type parameter, a target user identifier and a mobile terminal current login account identifier. The path type parameter indicates one of the following: on-body face input, on-body voiceprint input, remote face input or remote voiceprint input. The input orchestration unit is used to maintain input sessions for four paths based on path type parameters, read account authorization status, and verify account binding relationships and input authorization in conjunction with input requests. If the input conditions are met, path distribution is performed according to the path type parameters. The account authorization status includes device binding status, master account flag, authorized account flag, authorized function scope, and authorization expiration time. Only one input session is allowed to be established at a time to activate the corresponding path. When an input request for another path arrives and an input session is already occupied, the input request is rejected and a path occupancy indication is returned. If the path type parameters indicate the main body... If face registration or voiceprint registration is performed, a corresponding face registration session or voiceprint registration session is established, and a start registration command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. If the path type parameter indicates remote face registration or remote voiceprint registration, a corresponding remote face registration session or remote voiceprint registration session is established, and a remote start registration command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. At the same time, the registration completion indication is received from the feature processing unit, the target template slot is determined according to the path type parameter, and the TEE secure storage module is called to store the template. The feature processing unit includes a face feature processing unit and a voiceprint feature processing unit, which are used to collect or receive raw biological data based on remote collection data packets and extract the corresponding biological feature embedding vectors, respectively. After the data is entered, the unit reports the data entry completion instruction to the data entry arrangement unit. The TEE secure storage module is used to transmit the biometric embedding vector and the target user identifier based on the target template slot and write them into the Trusted Execution Environment secure world in the form of an encrypted persistent object.

[0019] An embodied intelligent robot includes a processor, a camera, a microphone array, a wireless communication module, and a memory. The processor supports TrustZone hardware security extensions, and the processor implements the method when executing a computer program stored in the memory.

[0020] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor supporting TrustZone hardware security extensions, implements the method.

[0021] Compared with the prior art, the present invention has the following beneficial effects: (1) Existing technologies deploy trusted execution environments on cloud servers, which poses a risk of data transmission leakage and fails to meet the end-side compliance constraints of not allowing biological data to be stored in the cloud. This invention deploys a trusted execution environment that supports TrustZone extensions on the end-side computing platform of an embodied intelligent robot. All biometric embedding vectors are directly written into the local secure world in the form of encrypted persistent objects. The entire process does not pass through any cloud nodes, realizing a complete closed loop for biological data at the end-side. This eliminates the risk of cloud leakage from the architecture and fully complies with the compliance requirements of not allowing near-field data to leave the end-side.

[0022] (2) Existing technologies only support the collection of biometric features through a single path on mobile terminals, which cannot meet the concurrent management requirements of multiple input entry points in home scenarios. This invention manages four input paths—on-the-spot face, on-the-spot voiceprint, remote face, and remote voiceprint—through an input orchestration unit, distributes input tasks according to path type, and allows only one input session to be established at a time. When a path is occupied, new requests are directly rejected. This can fully cover both on-the-spot and remote input scenarios, avoid concurrent conflicts of multiple sessions, and realize integrated management of multi-path input on the terminal side.

[0023] (3) The existing technology lacks account verification function. This invention verifies the account binding status, main account / authorized account flag, authorized function scope and authorization expiration time before the input starts. At the same time, it associates the authorization expiration, account deletion and input session status. When the authorization expires, the input session is automatically terminated. It can strictly control input permissions from the entry layer, prevent unauthorized input behavior, and adapt to the family multi-user hierarchical sharing scenario.

[0024] (4) The security level of existing technologies is weak. This invention superimposes a challenge-response and synchronous confirmation mechanism in the remote data entry path to verify the source legality and data integrity of remote data acquisition messages sent by the mobile terminal. This can effectively resist replay attacks and data tampering, and improve the credibility and transmission security of remote data entry.

[0025] (5) In scenarios such as account unbinding, authorization deletion, and device factory reset, existing technologies may leave expired biometric templates in the trusted environment, posing security risks. This invention achieves linked management of account lifecycle and trusted templates. When an account is unbound, authorization expires, or the device is reset to factory, the corresponding template object in the TEE security world is cleared simultaneously, completely avoiding the residue of expired biometric features, ensuring that the template's entire lifecycle is consistent with the account permission status, and strengthening the security of trusted storage on the device side. Attached Figure Description

[0026] Figure 1 This is a system architecture diagram of the present invention; Figure 2 This is a layered diagram of the data entry link from the mobile terminal to the embodied intelligent robot body according to the present invention; Figure 3 This is a flowchart of the method of the present invention; Figure 4 This is a schematic diagram of the mutual exclusion state machine for the four paths of the recording session of the present invention; Figure 5 This is a flowchart illustrating the device mutual trust authentication process for remote input sessions according to the present invention. Figure 6 This is a timing diagram of the remote input main link of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0029] Example 1

[0030] Figure 1 This is an architecture diagram of the system upon which the multimodal biometric data entry method provided in an embodiment of the present invention is based. Figure 2 This document presents a hierarchical diagram illustrating the data entry link from the mobile terminal to the embodied intelligent robot body based on this system. (Example:) Figure 1 and Figure 2 As shown, the multimodal biometric data entry method in this embodiment of the invention is executed on an embodied intelligent robot computing platform. The computing platform includes a device message routing unit, a data entry and arrangement unit, a face feature processing unit, a voiceprint feature processing unit, a wireless communication module, at least one camera, at least one microphone array, and a processor that supports TrustZone hardware security extensions. The computing platform is divided into an enriched execution environment (REE) and a trusted execution environment (TEE).

[0031] This embodiment uses a quadruped robot dog as an example. The computing platform uses an NVIDIA Jetson AGX Orin as the main processor. This processor supports ARM TrustZone hardware security extensions, which can isolate a rich execution environment (REE, i.e., a normal operating system environment running Ubuntu) and a trusted execution environment (TEE, secure world) through hardware isolation. The storage medium uses eMMC memory with a built-in RPMB (Replay Protection Block) security partition, which can only be directly accessed by the TEE secure world for trusted persistent storage of biometric templates. The computing platform peripherals include: one RGB camera with a resolution of 1920×1080, a frame rate of 30fps, and a horizontal field of view of 85°; one four-channel microphone ring array with a sampling rate of 16kHz and a quantization bit width of 16bit; and a dual-mode wireless communication module that supports WiFi Direct, Bluetooth Low Energy, and NearLink near-field communication. All biometric data transmission is completed through the near-field link, without passing through a cloud server.

[0032] The system adopts a layered and decoupled design. Mobile terminals cannot directly access the device sensor drivers and trusted storage interfaces. The responsibilities of each layer are as follows: The first layer, mobile terminal side: deployed on mobile phones / tablet terminals, responsible for initiating input requests and displaying input progress; collecting raw biological data under remote paths, and participating in device mutual trust authentication and process synchronization receipt.

[0033] The second layer, the near-field communication link, enables bidirectional message transmission based on WiFi Direct / Bluetooth Low Energy / Starlink, ensuring that neither raw biological data nor control messages are relayed through the cloud. When adapted to the Starlink link, its physical layer anti-relay attack, link layer SM4 national cryptographic encryption, and high concurrency and low latency features can be reused to further improve transmission security and stability.

[0034] The third layer, the device message routing unit, is deployed on the REE side of the computing platform. It is responsible for maintaining the connection between the mobile terminal and the computing platform, uniformly receiving all input messages, extracting the communication source identifier, and forwarding them to the input orchestration unit.

[0035] The fourth layer, the input and orchestration unit, is deployed on the REE computing platform and serves as the core scheduling hub of the entire input system. It is responsible for four-path distribution, global session mutual exclusion control, owner consistency verification, remote session tunnel management, process synchronization relay, voiceprint enhancement judgment, trusted storage call encapsulation, database entry triggering, and status broadcasting.

[0036] The fifth layer, the feature processing unit, includes a face feature processing unit and a voiceprint feature processing unit, deployed on the REE computing platform. The face feature processing unit is responsible for driving the camera to capture face images or receiving forwarded remote face images, and extracting a 512-dimensional normalized face feature embedding vector; the voiceprint feature processing unit is responsible for driving the microphone array to capture speech or receiving forwarded remote speech data, and extracting a 512-dimensional normalized voiceprint feature embedding vector.

[0037] Layer 6, Device Mutual Trust Verification Unit: Deployed on the REE side of the computing platform, only remotely entered paths are effective, and it is responsible for performing challenge-response signature verification, hash integrity verification, and challenge validity period verification.

[0038] Layer 7, Account Configuration Persistence Unit: Deployed on the REE side of the computing platform, it stores account metadata such as user identifier, feature identifier, creation timestamp, and authorization information, but does not store any biometric plaintext.

[0039] The eighth layer, TEE secure storage layer: deployed within the secure world of the computing platform's TEE side, receives embedded vectors and metadata submitted by trusted storage clients within the orchestration unit, performs multi-level parameter verification and object management, and then writes them to the RPMB secure partition.

[0040] Figure 3 This is a flowchart illustrating a multimodal biometric data entry method according to an embodiment of the present invention. Based on this, the method may include the following steps: Step S1: The device message routing unit obtains the input request sent by the mobile terminal via the near-field wireless communication link and forwards the input request to the input orchestration unit. The input request carries path type parameters, target user identifier and mobile terminal current login account identifier. The path type parameters indicate one of the following: on-site face input, on-site voiceprint input, remote face input or remote voiceprint input.

[0041] After receiving the request, the device message routing unit completes the initial verification of the message format and extraction of the communication source identifier, and forwards the complete request to the input and arrangement unit.

[0042] Step S2: The input and arrangement unit reads the account authorization status and verifies the account binding relationship and input authorization in conjunction with the input request. If the input conditions are met, proceed to step S3. The account authorization status includes device binding status, main account flag, authorized account flag, authorized function scope and authorization expiration time.

[0043] The data entry and arrangement unit performs two levels of verification: 1. Basic general verification: When the device is not bound to the main account, the target user has not obtained identity entry permission, the authorization is out of scope or the authorization has expired, it is determined that the entry conditions are not met, the entry session is refused to be established and the reason for failure is returned; 2. Remote Path Exclusive Verification: When the path type parameter indicates remote face recognition or remote voiceprint recognition, the consistency between the mobile terminal's currently logged-in account identifier and the target user identifier is verified. If they do not match, it is determined that the recognition conditions are not met, and the establishment of a remote recognition session is refused. This ensures that remote recognition is initiated by the target user's own account and prevents unauthorized registration.

[0044] Step S3: The input and arrangement unit maintains input sessions for four types of paths based on the path type parameters and performs path distribution according to the path type parameters.

[0045] like Figure 4 As shown, the data entry and arrangement unit implements global mutual exclusion control through an internally maintained four-path structured session registry. This registry contains four optional session registration fields, corresponding to the subject's face, subject's voiceprint, remote face, and remote voiceprint, respectively; the remote path registration field simultaneously records the communication source device address, the mobile terminal's currently logged-in account identifier, and the target user identifier. The mutual exclusion judgment logic is as follows: (1) Traverse all registration positions. If any non-empty registration position exists, reject the new request and return the path occupancy indication; (2) If the target path registration bit is already occupied, if the source address and user identifier of the new request are completely consistent with the registration bit, it is considered an idempotent request and a success is returned directly; otherwise, an owner mismatch indication is returned. (3) After the verification is successful, the request information is written to the corresponding registration position, the preset session timeout timer is started, and the start command is sent to the downstream.

[0046] Based on this logic, only one entry session can be established at any given time to activate the corresponding path.

[0047] Based on the path type, the processing can be divided into the following two branches: I. Ontology Input Branch.

[0048] If the path type parameter indicates body face recording or body voiceprint recording, a body face recording session or body voiceprint recording session corresponding to the path is established, and a start recording command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The corresponding feature processing unit then activates the camera or microphone array to collect raw biological data and extracts the biometric feature embedding vector.

[0049] II. Remote data entry branch.

[0050] If the path type parameter indicates remote face registration or remote voiceprint registration, a corresponding remote face registration session or remote voiceprint registration session is established, and a remote start registration instruction is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The remote acquisition data message sent by the mobile terminal is forwarded to the corresponding feature processing unit, and the feature processing unit extracts the biometric feature embedding vector from the forwarded raw biometric data.

[0051] In a preferred embodiment, the remote input session further includes device trust authentication before forwarding the remote acquisition data packets: The input and arrangement unit generates a random challenge value and sends it to the mobile terminal. It receives the signed data packet and sequentially performs public key verification, hash consistency verification, and challenge validity period check. After all checks pass, the data source is determined to be trustworthy, and then the remote collection data packet is forwarded. The signed data packet is generated by the mobile terminal by concatenating the input destination identifier, challenge value, and feature hash value and signing it with the device trust ring private key.

[0052] Data transmission employs a block-level transmission and block-level synchronization confirmation mechanism, possessing complete transaction semantics: The mobile terminal sends the raw data in sequential numbered blocks. After the input and arrangement unit verifies that the communication source identifier, login account identifier, target user identifier and session record are consistent (owner verification), the data is forwarded to the corresponding feature processing unit through the session tunnel. After processing each data block, the feature processing unit reports the block identifier and the processed hash value; the input and arrangement unit encapsulates this into a process synchronization request and sends it to the mobile terminal. If the hash match is found locally on the mobile terminal, a continuation receipt is returned; otherwise, a resampling receipt is returned. The orchestration unit relays the receipt to the feature processing unit, which reprocesses the current data block under the resampling instruction. Each data block carries a session transaction identifier and an event type identifier: intermediate data blocks trigger a timeout timer refresh, and the end data block triggers a full verification pipeline; if a single block synchronization times out by 5000 milliseconds, the request is resent, and the session is actively terminated after a total of 3 timeouts.

[0053] In addition, the data entry and arrangement unit performs owner verification during the remote acquisition data packet forwarding and process synchronization receipt. It verifies whether the communication source identifier, the mobile terminal's current login account identifier, and the target user identifier are all consistent with the session record of the current activation path. If they are inconsistent, it refuses to execute and returns an owner mismatch indication.

[0054] Step S4: After receiving the input completion indication reported by the feature processing unit, the input and arrangement unit determines the target template slot according to the path type parameter, and transmits the biometric embedding vector and the target user identifier to the TEE secure world based on the target template slot. The biometric embedding vector and the target user identifier are written to the RPMB partition of the eMMC storage in the form of encrypted persistent objects, and the corresponding biometric reference and creation timestamp are written in the account configuration. An account status change notification is issued.

[0055] When the cloud pushes an updated account list to the input orchestration unit, it merges the cloud account metadata with the locally stored biometric references according to the target user identifier, without overwriting locally generated feature identifiers, and increments the version broadcast identifier of the account, face, or voiceprint. Each user's official template slots include a face slot, an application-side synchronous voiceprint slot, and an end-side enhanced voiceprint slot. On-the-spot face input and remote face input mutually exclude each other from updating the same face slot, and on-the-spot voiceprint input and remote voiceprint input mutually exclude each other from updating the same application-side synchronous voiceprint slot.

[0056] When a biometric deletion request, account unbinding request, authorization deletion request, or factory reset request is received from a mobile terminal, the persistent objects of the face, application-side synchronized voiceprint, and terminal-side enhanced voiceprint in the TEE are deleted at the user or device level, and the corresponding biometric references in the account configuration are cleared.

[0057] In this embodiment, a persistent object identifier in triplet format is constructed using the target user identifier, biometric type identifier (such as face, voiceprint) and template sequence number. First, the old object with the same name is deleted, and then a new encrypted persistent object is created. The input and arrangement unit first writes the account configuration carrying the new feature identifier into the account configuration persistence unit, and then submits the biometric feature embedding vector to the TEE persistent object through its internal trusted storage client. If the TEE write fails, the account configuration is rolled back to the state before the write.

[0058] Specifically, the data entry process is as follows: 1) The global unique feature identifier and creation timestamp generated by the input and arrangement unit are first written to the account configuration persistence unit; 2) Through the internal trusted storage client, the user identifier, biometric type, template number, and feature embedding vector are encapsulated as cross-world calling parameters and transmitted to the TEE secure world; 3) After the trusted application within the TEE performs magic number verification and dimension range verification, it constructs a triplet object identifier and writes it to the RPMB partition using an anti-rollback strategy of first deleting the old object and then creating a new encrypted persistent object. 4) If the TEE write fails or times out, immediately roll back the account configuration to the state before the write to avoid a dangling state where the account has a feature reference but the trusted storage has no corresponding template. 5) After successful writing, increment the version broadcast flag of the corresponding modality, publish the account status change notification, and release the session registration bit.

[0059] In a preferred embodiment, the method further includes a shadow pattern voiceprint enhancement step to improve the long-term environmental adaptability of voiceprint recognition without modifying the baseline template. This step specifically includes the following steps: When the time since the last enhanced voiceprint recording reaches a preset interval and candidate voiceprint features are generated during routine recognition, the candidate voiceprint features are written into a short-term cache. Retrieve face matching results, lip movement detection results, and voiceprint matching results within the cache validity period; When the face matching result indicates that the user in the current image is the user identified by the corresponding user, the lip movement detection result indicates that the user in the current image is speaking, and the voiceprint matching result indicates that the candidate voiceprint feature matches the application-side synchronous voiceprint template or the terminal-side enhanced voiceprint template, the terminal-side enhanced voiceprint copy of the same user is replaced with the candidate voiceprint feature, while the application-side synchronous voiceprint copy remains unchanged. If any decision fails or the cache times out, the candidate voiceprint feature is discarded.

[0060] In another preferred embodiment, a template lifecycle management mechanism is provided, including: (1) Local metadata retention under distributed account synchronization.

[0061] When the cloud pushes an update to the account list, the data is merged according to the target user identifier: only cloud metadata such as nickname and authorization scope is updated, while locally stored biometric references, feature identifiers and creation timestamps are retained, without overwriting local input results; after merging, the version broadcast identifiers of account, face and voiceprint are incremented respectively, and downstream modules can perceive changes by dimension without full reloading.

[0062] (2) Account lifecycle coordinated cleanup The entry and arrangement unit scans for account authorization expiration times every 3600 seconds; when an account authorization expires, an account is deleted, or authorization is revoked: Iterate through all face, application-side synchronized voiceprint, and terminal-side enhanced voiceprint template objects under the user at the user granularity, and permanently delete the corresponding persistent objects in the TEE (the terminal-side enhanced voiceprint is deleted along with the base template and must not remain). Clear the biometric references for the corresponding user in the account configuration; If the user has an ongoing data entry session, immediately terminate the session, notify the feature processing unit to stop processing, and return an account expiration indication.

[0063] When a request to delete biometrics, unbind a device, or restore factory settings is received, the above cleanup process is executed at the user or device level to ensure that no invalid templates remain.

[0064] (3) Activate the two-way alignment mechanism When the system starts, the input orchestration unit reads the valid biometric template references in the current account configuration, and at the same time enumerates all persistent objects in the TEE and performs two-way cleanup: deleting template objects that are not within the scope of the account configuration, and clearing dangling feature references in the account configuration that have no corresponding trusted objects, to ensure that the state is consistent after restart.

[0065] (4) Trusted storage session abnormal recovery The input orchestration unit periodically checks the TEE secure storage response status. After multiple consecutive failed checks, the secure storage session is automatically rebuilt. After the session is restored, the input, deletion, and query processes can be executed normally, reducing the impact of short-term anomalies in trusted storage.

[0066] This embodiment also provides an embodied intelligent robot, including a processor, a camera, a microphone array, a wireless communication module, and a memory. The processor supports TrustZone hardware security extensions, and the processor implements the method described when executing a computer program stored in the memory.

[0067] Example 2

[0068] This embodiment is based on embodiment 1 and provides a description of the specific implementation process of triggering face recognition on a mobile terminal application.

[0069] Link: Mobile terminal application → WiFi direct connection → Device message routing unit → Input and arrangement unit → Face feature processing unit → Trusted storage client within the input and arrangement unit → TEE trusted application.

[0070] S1 (Send Request): The main account sends a face recognition start request via the mobile terminal; the device message routing unit forwards it to the recognition and arrangement unit.

[0071] S2 (Entry Verification): The entry and arrangement unit verifies that the account binding relationship and entry authorization are correct, and then proceeds to step S3.

[0072] S3 (Voice Acquisition): The input and arrangement unit establishes a voice input session for face recognition and sends a start command to the face feature processing unit. The face feature processing unit activates the RGB camera and acquires image frames at 30fps. The face detector performs detection on each frame: when the detected face bounding box area occupies no less than 10% of the screen area, the face center is located within ±15% of the center of the screen, and the ambient light intensity is no less than 100 lux, a 128×128 pixel aligned image is extracted from the face region and input into the face feature extraction neural network to output a 512-dimensional normalized embedding vector. Within a 3-second acquisition window, if 10 consecutive frames contain valid faces and the mean cosine similarity of the feature vectors is no less than 0.85, the middle frame is selected as the final input data. If the conditions are not met within the acquisition window, guidance information is broadcast through speech synthesis and acquisition is restarted.

[0073] S4 (Process Synchronization and Convergence): The face feature processing unit reports the acquisition progress to the input orchestration unit; the input orchestration unit sends a process synchronization request to the mobile terminal via the routing unit, and the mobile terminal responds with a continue instruction. The face feature processing unit reports the completion of input and the embedded vector; the input orchestration unit writes the persistent face TEE object of the main account, updates the account configuration, publishes an account status change notification, and returns a success instruction to the mobile terminal via the routing unit.

[0074] Example 3

[0075] This embodiment is based on embodiment 1 and provides a description of the specific implementation process of triggering the voiceprint recording of the subject by a mobile terminal application.

[0076] Link: Mobile terminal application → WiFi direct connection → Device message routing unit → Input and arrangement unit → Voiceprint feature processing unit → Trusted storage client within the input and arrangement unit → TEE trusted application.

[0077] S1 (Send Request): The main account is within the effective acquisition range of the four-channel microphone ring array of the quadruped robot dog. It sends a voiceprint recording start request via the mobile terminal, with the target user identifier being the main account. The device message routing unit forwards the recording request to the recording orchestration unit.

[0078] S2 (Entry Verification): The entry and arrangement unit verifies that the account binding relationship and entry authorization are correct, and then proceeds to step S3.

[0079] S3 (Voice Acquisition): The recording and arrangement unit establishes a voiceprint conversation and sends a start command to the voiceprint feature processing unit. Simultaneously, the speech synthesis module broadcasts the guidance message "Please speak clearly to the robot dog." The voiceprint feature processing unit activates four microphones to acquire 3 seconds of speech at 16kHz and 16bit. When the signal-to-noise ratio is not less than 15dB and the effective speech duration is not less than 2.5 seconds, a 512-dimensional normalized voiceprint embedding vector is extracted from the audio stream; if the conditions are not met, a re-acquisition guidance is broadcast and re-acquisition is performed.

[0080] S4 (Process Synchronization and Convergence): The voiceprint feature processing unit reports the acquisition progress percentage to the input and arrangement unit; the input and arrangement unit sends a process synchronization request to the mobile terminal via the device message routing unit, and the mobile terminal responds with a continue instruction. The voiceprint feature processing unit reports the completion of input and the embedded vector; the input and arrangement unit writes the triplet identifier into the main account's voiceprint TEE persistent object, updates the account configuration, increments the voiceprint version broadcast identifier, publishes an account status change notification, and returns a success instruction to the mobile terminal via the device message routing unit.

[0081] Example 4

[0082] This embodiment is based on embodiment 1 and provides a description of the specific implementation process of remote voiceprint recording triggered by a mobile terminal application.

[0083] Link: The mobile terminal application sends the message to the device message routing unit via WiFi direct connection, and is uniformly scheduled by the recording and arrangement unit; the original audio is collected by the mobile terminal application, and the voiceprint feature processing unit completes feature extraction in the dog body; the trusted storage client in the recording and arrangement unit initiates a cross-boundary call → the TEE trusted application completes the database entry, and the account configuration persistence unit records the feature reference.

[0084] S1: A family member authorized by the main account logs into the mobile terminal application using their own account and initiates a remote recording start request. The path type parameter indicates remote voiceprint recording, and both the target user identifier and the currently logged-in account identifier on the mobile terminal are the family member's account. The device message routing unit forwards the request to the recording orchestration unit.

[0085] S2: After verifying the account binding relationship and entry authorization of the entry and arrangement unit, proceed to step S3.

[0086] S3: Establish a remote voiceprint session and send a remote start command to the voiceprint feature processing unit.

[0087] S31: The mobile terminal application sends raw PCM audio in chunks.

[0088] S32: Device Trust Authentication.

[0089] S321: The computing platform calls the hardware random number generator to generate a 256-bit random challenge value, records the generation timestamp, and sets a 30-second validity period.

[0090] S322: Send the challenge value and remote voiceprint recording destination identifier to the mobile terminal application via WiFi link.

[0091] S323: The mobile terminal application collects a 3-second voice sample and extracts a 512-dimensional voiceprint embedding vector. It calculates the SHA256 hash value of the embedding vector, concatenates the recorded target identifier, challenge value and hash value in sequence to form a signature payload, signs the signature payload using the device trust ring elliptic curve private key, and sends back the trust ring public key material.

[0092] S324: The computing platform device mutual trust verification unit sequentially performs: public key signature verification, hash consistency verification, and challenge value validity check. Once all three checks pass, the data source is deemed trustworthy.

[0093] S33: After the input and arrangement unit verifies that the communication source identifier, the mobile terminal's current login account identifier, and the target user identifier are all consistent with the activated session, it forwards the remotely collected data packet to the voiceprint feature processing unit; if the owner does not match or a session has not been established, it refuses to forward the packet.

[0094] S34: Process synchronization relay.

[0095] S341: After processing each block of raw audio, the voiceprint feature processing unit reports the block identifier and the processed hash to the recording and arrangement unit.

[0096] S342: The orchestration unit encapsulation process synchronization request is sent to the mobile terminal via the device message routing unit; after local comparison, the mobile terminal sends a confirmation to continue or re-sample.

[0097] S343: The orchestration unit relays the acknowledgment to the voiceprint feature processing unit; the current block is reprocessed during re-sampling. Single block synchronization confirmation timeout is 5000 milliseconds.

[0098] S344: After all blocks are confirmed to be consistent, the voiceprint feature processing unit reports the completion of the recording and the 512-dimensional embedding vector to the arrangement unit.

[0099] S4: The orchestration unit generates a feature identifier, writes it to the persistent TEE object of the family member's voiceprint through its internal trusted storage client, updates the account configuration, increments the voiceprint version broadcast identifier, and returns a successful recording indication to the mobile terminal via the routing unit.

[0100] Example 5

[0101] This embodiment, based on embodiment 1, provides a detailed description of the implementation process of remote face recognition with challenge-based mutual trust and block-level synchronization normal path. The specific steps are as follows: Figure 6As shown.

[0102] Link layering: Mobile terminal application connects directly via WiFi → Device message routing unit → Input and arrangement unit → Face feature processing unit → Trusted storage client within the input and arrangement unit → TEE trusted application. The original image is acquired by the mobile terminal application, while feature extraction and database storage are completed on the dog itself.

[0103] S1: A family member authorized by the main account initiates remote face registration by logging into the mobile terminal application using their own account. The path type parameter indicates remote face registration, and both the target user identifier and the currently logged-in account identifier on the mobile terminal are the family member's account. The device message routing unit forwards the message to the registration orchestration unit.

[0104] S2: Enter the arrangement unit. After verifying the account binding relationship and the entry authorization, proceed to step S3.

[0105] S3: After the input and arrangement unit confirms that no other paths are currently occupied, it establishes a remote face session and sends a remote start command to the face feature processing unit.

[0106] S31: Question-Response Mutual Trust. For example... Figure 5 As shown, it includes the following steps: S311: The orchestration unit calls the hardware random number generator to generate a 256-bit challenge value, records the generation timestamp T0, and has a validity period of 30 seconds.

[0107] S312: The routing unit sends the challenge value and the remote face registration destination identifier to the mobile terminal.

[0108] S313: The mobile terminal collects facial images, extracts 512-dimensional embedding vectors, and calculates SHA256 hashes; it then concatenates the destination identifier, challenge value, and hash value, signs the result using the device trust ring private key, and sends it back along with the public key material.

[0109] S314: The device mutual trust verification unit sequentially performs public key signature verification, hash consistency verification, and challenge validity period check (the difference between the current time and T0 does not exceed 30 seconds). After all three items pass, proceed to S32.

[0110] S32: The mobile terminal application sends the original JPEG image in 64KB blocks, numbered sequentially. The input and arrangement unit performs ownership verification on each block (the communication source identifier, the mobile terminal's currently logged-in account identifier, and the target user identifier must match the active session), and forwards it to the face feature processing unit after passing the verification.

[0111] S33: After processing each block, the face feature processing unit reports the block sequence number and processed hash to the orchestration; the orchestration encapsulation process synchronization request is sent to the mobile terminal application via the device message routing unit; if the mobile terminal application's local comparison matches, the receipt continues to indicate; if they do not match, the receipt is re-sampling indicated; the orchestration relays the receipt to the face feature processing unit. If the single block waits for a receipt for 5000 milliseconds, the synchronization request is resent; if there are 3 timeouts, the session is terminated.

[0112] S4: After all blocks are completed, the face feature processing unit reports the 512-dimensional embedding vector; it arranges and writes the family member face TEE persistent object, updates the account configuration, publishes the status change notification, and returns a success indication and feature identifier to the mobile terminal application via the routing unit.

[0113] Example 6

[0114] This embodiment is based on embodiment 1 and describes in detail the implementation process of multi-user hierarchical management and automatic expiration cleanup.

[0115] In a family setting, the main account is permanently authorized (with an expiration time of 0), family member accounts have an expiration time set to 30 days after the current time, and temporary visitor accounts have an expiration time of 24 hours later.

[0116] Before establishing an input session, the input orchestration unit reads the local account configuration and cloud-synchronized account metadata to determine whether the device is already bound to a main account, whether the target user belongs to the main account or a valid authorized account, whether the authorized function scope includes identity input capability, and whether the authorization expiration time is still valid. If any condition is not met, the orchestration unit refuses to establish an input session and returns a failure indication to the mobile terminal via the device message routing unit.

[0117] Once the authorization status verification is successful, the input orchestration unit verifies that the number of currently valid users on the device does not exceed 10, and confirms that the target user's current input object belongs to one of the three preset template slots: face, application-side synchronous voiceprint, or terminal-side enhanced voiceprint. If it is a re-entry or enhancement of the same slot, the corresponding old template is deleted before entering the new template input process; if a request is made to add a non-preset template slot, the new process is rejected.

[0118] The computing platform performs an authorization expiration check every 3600 seconds. (1) Get the current system time.

[0119] (2) Traverse registered user accounts: Skip permanent authorization for main accounts; for expired family member accounts, traverse all face, application-side synchronous voiceprint and terminal-side enhanced voiceprint template objects under their user identifier, open them one by one in TEE and permanently delete the corresponding persistent objects (the terminal-side enhanced voiceprint must be cleared along with the application-side synchronous voiceprint and must not remain); skip non-expired accounts.

[0120] (3) Clear the local configuration data of expired accounts.

[0121] (4) Issue a notification of account status change, indicating the invalid user and the characteristics of the removed user.

[0122] The biometric templates of expired users are irreversibly deleted from the RPMB partition without manual intervention from the main account. If the user has an ongoing data entry session, the orchestration unit synchronously and forcibly terminates the session and notifies the feature processing unit to stop.

[0123] Example 7

[0124] This embodiment is based on embodiment 1 and describes in detail the implementation process of four-path mutual exclusion and cloud synchronization to retain local templates.

[0125] Phase A (Mutual Exclusion) (A1) At time T0: The main account initiates the body face registration via the mobile terminal, the orchestration unit establishes a body face session, and the session status changes from idle to body face activation.

[0126] (A2) T0+5 seconds: A family member initiates voiceprint registration via another mobile terminal. The scheduling unit detects that an existing path session is occupied, rejects the voiceprint initiation request, and returns a path occupancy indication to the mobile terminal. The voiceprint session is not established.

[0127] (A3) Within T0+40 seconds: The main account's face registration is completed, the orchestration unit is written to the main account's face TEE persistent object, and the session returns to idle; thereafter, family members can successfully start the body voiceprint registration.

[0128] Phase B (Cloud Synchronization and Merging) (B1) Before the main account's face is entered, there is already a binding relationship between the main account's face feature identifier and the main account's user identifier locally.

[0129] (B2) The cloud pushes an update to the account list, carrying metadata such as user nicknames and permission levels, but not carrying local biometric templates.

[0130] (B3) Arrangement units are merged by user identifier: the nickname and permissions pushed by the cloud are used to cover the local same name field, the facial feature identifier and creation timestamp of the local main account are retained, and the generated feature identifier is not overwritten.

[0131] (B4) After the merge is completed, increment the face version broadcast identifier (e.g., change from 12 to 13). After the downstream display module subscribes to the identifier, refresh the family member list, and the interface displays the new nickname while still being associated with the original face template.

[0132] Phase C (Owner Verification Linkage) If a remote data collection message for the main account is received during cloud synchronization, the orchestration unit verifies whether the communication source identifier, the mobile terminal's currently logged-in account identifier, and the target user identifier are all consistent with the active session; if there is no active session or the owner is inconsistent, forwarding is refused and an owner mismatch indication is returned.

[0133] Example 8

[0134] This embodiment is based on embodiment 1 and explains in detail how the configuration writing and TEE writing are kept consistent after the data entry is completed.

[0135] (1) The voiceprint feature processing unit reports the completion indication of the family member account and the 512-dimensional embedding vector. The entry arrangement unit generates a new voiceprint feature identifier for the account and records the identifier and creation timestamp in the account configuration to be written.

[0136] (2) The entry orchestration unit submits the account configuration to be written to the account configuration persistence unit. After successful submission, the trusted storage client in the entry orchestration unit packages and transmits the target user identifier, biometric identifier, template sequence number and embedding vector to the TEE secure storage module.

[0137] (3) If the TEE secure storage module is successfully written, the recording and arrangement unit will update the local account status to the new configuration, increment the voiceprint version broadcast identifier and issue a status change notification.

[0138] (4) If the TEE secure storage module fails to write, the entry orchestration unit immediately rolls back the account configuration persistence unit to the old configuration before (1) and returns an entry failure indication to the mobile terminal. As a result, the voiceprint feature identifier that does not have a corresponding trusted storage object will not be retained in the account configuration.

[0139] This consistency commit process keeps configuration-side feature references synchronized with TEE objects, preventing template references from becoming dangling due to device power outages, short-term unavailability of trusted storage, or failures in cross-world calls.

[0140] Example 9

[0141] This embodiment is based on embodiment 1 and describes in detail the implementation process of startup alignment and trusted storage session recovery.

[0142] (1) After the quadruped robot dog computing platform is started, the input and arrangement unit first reads the account configuration to obtain the current set of valid user identifiers and the facial and voiceprint features of each user.

[0143] (2) Enter the template objects that already exist in the TEE secure storage module by enumerating the orchestration unit. If the user identifier to which the object belongs is not in the current account configuration, call the deletion process to clear the object; if the biometric reference that exists in the account configuration but is not enumerated by the TEE, clear the reference and update the account configuration.

[0144] (3) After the alignment is completed, the input orchestration unit periodically checks the response status of the TEE secure storage module. If the check fails multiple times in a row, the secure storage session is re-established through its internal trusted storage client; after the session is restored, subsequent input, deletion and status query continue to use the restored trusted storage link.

[0145] Through the above process, the system can maintain consistency between account configuration, template objects, and status broadcasts even after restarting, temporary loss of trusted storage connection, or account synchronization.

[0146] Example 10

[0147] This embodiment is based on embodiment 1 and provides a detailed description of the four-path session registry maintained within the input and arrangement unit and its linkage with the account lifecycle.

[0148] (A) Four-path structured mutual exclusion state machine The entry orchestration unit maintains a registry in memory containing four optional session registration bits, each corresponding to an entry path:

[0149] When a startup request arrives for any path, the orchestration unit performs the following check sequence: (A1) Iterate through all four session registration slots. If any non-empty registration slot exists, reject the new request and return the "Other paths are being entered" indication. (A2) If the target path registration bit is already occupied (such as repeated startup of the same path), check whether the communication source address and user identifier of the new request are completely consistent with the occupied registration bit - if they are consistent, it is considered an idempotent request and returns success directly; if they are inconsistent, return the "owner mismatch" indication. (A3) After all checks pass, the source address and user identifier of the request are written into the corresponding registration bit, the timeout timer for this path is started (default 40 seconds), and a start command is sent to the downstream feature processing unit.

[0150] When a path completes, fails, times out, or is terminated externally, the corresponding registration bit is cleared, and the timeout timer stops. This structured design allows mutual exclusion checks to be performed by traversing only a fixed number of session registration bits, and it naturally supports any combination of the four path exclusions—for example, when a face scan is in progress, neither the voiceprint scan, remote voiceprint scan, nor remote face scan can be initiated.

[0151] (B) Account deletion: voluntary termination of ongoing data entry. The entry orchestration unit triggers an "active slot account validity scan" every time it receives a cloud account synchronization update or performs an authorization expiration check: (B1) Construct a set of user identifiers from the current list of valid accounts; (B2) Check in turn whether the target user identifier of each non-empty registration bit in the four session registration bits belongs to the above set; (B3) If a user ID for a particular registration slot is not in the valid set (e.g., the user has been deauthorized by the main account or automatically removed upon authorization expiration), immediately execute the following termination sequence: (B31) Stop the timeout timer corresponding to this registration bit; (B32) Send a stop processing command to the corresponding feature processing unit; (B33) The device message routing unit returns a "Account has been deleted" failure indication and a reason explanation to the mobile terminal that initiated the entry; (B34) Clear this registration slot.

[0152] This self-termination mechanism ensures that changes to the account lifecycle are immediately reflected in the data entry layer, preventing inconsistencies such as "the user has been removed but their biometrics are still being entered." This mechanism is driven by a background timer (triggered by default every 5 seconds for account expiration checks, or immediately upon each cloud account synchronization push), requiring no manual intervention.

[0153] Example 11

[0154] This embodiment further refines the configuration-trusted storage consistency commit process mentioned in Embodiment 8 from the perspective of verification pipeline and transaction semantics, focusing on demonstrating the multi-step verification link that the remote entry path goes through after the data arrives at the orchestration unit, as well as the transaction integrity guarantee of block-level relay.

[0155] (1) End-to-end verification pipeline Taking remote face recognition as an example, when the feature processing unit reports the completion message, the orchestration unit verifies each item in the following order:

[0156] If any step C1–C4 fails, the data entry and arrangement unit will not execute C5–C8, but will instead return a failure indication and the reason for the failure to the mobile terminal. This "verification-database entry-notification" pipeline ensures that only fully verified data can be written to the account configuration and trusted storage, preventing invalid data from contaminating the account status due to internal anomalies in the feature processing unit.

[0157] (2) Block-level relay transaction integrity For chunked data transmission along remote paths, each data block carries a session transaction identifier and an event type identifier. The entry orchestration unit, acting as a relay node, maintains the current transaction mapping for each active remote session. (21) Intermediate data block: After the data entry and arrangement unit forwards the data to the feature processing unit, it resets the timeout timer of the path (to prevent false timeouts caused by long-term transmission); at the same time, it sends a synchronization confirmation request back to the mobile terminal and waits for the mobile terminal's continuation / re-acquisition receipt. (22) End data block: The input and arrangement unit clears the corresponding registration position, stops the timer, and executes the above verification pipeline C1-C8; if the verification is successful, a success confirmation is returned to the mobile terminal; if the verification fails, a failure confirmation and the reason for failure are returned. (23) The mobile terminal fails to acknowledge synchronization within 5000 milliseconds: The orchestration unit resends the synchronization request. After accumulating 3 timeouts, it actively terminates the session, clears the slots, and returns a timeout failure.

[0158] The aforementioned transaction semantics ensure reliable delivery of remote data entry: the transmission results of each piece are clearly confirmed, and the success or failure status of the final result must reach the mobile terminal side, eliminating information asymmetry where the data entry is completed on the terminal side but unknown on the mobile terminal side.

[0159] Example 12

[0160] This embodiment is based on embodiment 1 and verifies a security attack scenario.

[0161] Attack Scenario A: Remote link man-in-the-middle tampering attack.

[0162] An attacker intercepts a signature data packet on the WiFi link between the mobile terminal and the computing platform, attempting to replace the feature embedding vector with the attacker's own features. When the computing platform performs a hash consistency check, it finds that the hash value of the replaced feature does not match the hash value bound to the signature and refuses to input it. If the attacker also tamperes with the signature, the public key verification will also fail because the attacker does not possess the device's trust ring private key.

[0163] Attack Scenario B: Replay Attack.

[0164] The attacker records network packets from a legitimate remote input session and replays them after a 30-second validity period. The computing platform checks that the difference between the challenge value generation time and the current time exceeds 30 seconds, causing the validity period check to fail; furthermore, the random challenge value generated in the new session is inconsistent with the old challenge value in the replayed packet, so the input is rejected.

[0165] Attack Scenario C: Unauthorized devices impersonate data entry.

[0166] An attacker implements the protocol on an unregistered mobile terminal and sends data packets with fake signatures. The device does not possess a pre-set trust ring private key and cannot generate a legitimate digital signature; even if a self-made public key is attached, it is not on the device's trust ring whitelist, and public key verification fails. This scenario contrasts with the normal mutual trust path in Embodiment 4, supporting the technical effect of claim 3.

[0167] Example 13

[0168] This embodiment is based on embodiments 1-12. A full-process test of four-path orchestration was conducted on a Jetson AGX Orin test bench. The mobile terminal and computing platform were directly connected via WiFi. The sample size was 30 times per path. The results are as follows:

[0169] Experiments show that the four paths can stably merge and be stored in the database under unified orchestration. The mutual exclusion, owner binding, cloud synchronization retention and authorization linkage, structured mutual exclusion of the four paths, end-to-end verification pipeline and block-level reliable relay are consistent with the descriptions in Examples 1-12 above.

[0170] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0171] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0172] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0173] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0174] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0175] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0176] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0177] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0178] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0179] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0180] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0181] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0182] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A multimodal biometric data entry method, executed on an embodied intelligent robot computing platform, characterized in that, The computing platform includes a device message routing unit, an input and arrangement unit, a face feature processing unit, a voiceprint feature processing unit, a wireless communication module, at least one camera, at least one microphone array, and a processor supporting TrustZone hardware security extensions. The computing platform is divided into rich execution environments and trusted execution environments. The method includes the following steps: Step S1: The device message routing unit obtains the input request sent by the mobile terminal via the near-field wireless communication link and forwards the input request to the input orchestration unit. The input request carries path type parameters, target user identifier and mobile terminal current login account identifier. The path type parameters indicate one of the following: face input, voiceprint input, remote face input or remote voiceprint input. Step S2: The input and arrangement unit reads the account authorization status and verifies the account binding relationship and input authorization in conjunction with the input request. If the input conditions are met, proceed to step S3. The account authorization status includes device binding status, main account flag, authorized account flag, authorized function scope and authorization expiration time. Step S3: The input arrangement unit maintains input sessions for four paths based on path type parameters and performs path distribution according to the path type parameters. Only one input session is allowed to be established at a time to activate the corresponding path. When an input request for another path arrives and an input session is already occupied, the input request is rejected and a path occupancy indication is returned. If the path type parameter indicates body face input or body voiceprint input, a body face input session or body voiceprint input session for the corresponding path is established, and a start input command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The corresponding feature processing unit activates the camera or microphone array to collect raw biological data and extracts biometric embedding vectors. If the path type parameter indicates remote face input or remote voiceprint input, a corresponding remote face input session or remote voiceprint input session is established, and a remote start input command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. The remote collection data message sent by the mobile terminal is forwarded to the corresponding feature processing unit, and the feature processing unit extracts biometric embedding vectors from the forwarded raw biological data. Step S4: After receiving the input completion indication reported by the feature processing unit, the input arrangement unit determines the target template slot according to the path type parameter, and transmits the biometric embedding vector and the target user identifier based on the target template slot and writes them into the trusted execution environment security world in the form of an encrypted persistent object to complete the biometric input.

2. The multimodal biometric data entry method according to claim 1, characterized in that, In step S2, if the device is not bound to the main account, the target user has not obtained identity entry permission, the authorization is out of scope, or the authorization has expired, it is determined that the entry conditions are not met; when the path type parameter indicates remote face entry or remote voiceprint entry, the consistency between the mobile terminal's current login account identifier and the target user identifier is also verified. If they are inconsistent, it is determined that the entry conditions are not met.

3. The multimodal biometric data entry method according to claim 1, characterized in that, In step S3, the remote input session also includes device mutual trust authentication before forwarding the remote collection data packet: the input orchestration unit generates a random challenge value and sends it to the mobile terminal, receives the signed data packet and sequentially performs public key verification, hash consistency verification and challenge validity period check. After all passes, the data source is determined to be trustworthy, and then the remote collection data packet is forwarded; the signed data packet is generated by the mobile terminal by concatenating the input destination identifier, the challenge value and the feature hash value and signing it with the device trust ring private key.

4. The multimodal biometric data entry method according to claim 1, characterized in that, In step S3, the remote input session transmits data in blocks to the corresponding feature processing unit. After each data block is processed, the feature processing unit reports the block processing result. The input orchestration unit receives the block processing result and sends a process synchronization request to the mobile terminal. It receives the synchronization receipt returned by the mobile terminal and relays the synchronization receipt to the feature processing unit until all data blocks are processed. The process synchronization request carries a synchronization data identifier and the hash value of the current data block after processing; the mobile terminal compares the hash value locally and returns a synchronization receipt indicating to continue or to re-collect; when the instruction to re-collect is received, the input and arrangement unit notifies the corresponding feature processing unit to reprocess the current data block.

5. The multimodal biometric data entry method according to claim 4, characterized in that, The input and arrangement unit performs owner verification during remote data packet forwarding and process synchronization receipt. It verifies whether the communication source identifier, the mobile terminal's current login account identifier, and the target user identifier are all consistent with the session record of the current activation path. If they are inconsistent, it refuses to execute and returns an owner mismatch indication.

6. The multimodal biometric data entry method according to claim 1, characterized in that, When the cloud pushes an account list update to the input and arrangement unit, the cloud account metadata and the locally stored biometric references are merged according to the target user identifier, without overwriting the locally generated feature identifiers, and the version broadcast identifiers of the account, face, or voiceprint are incremented; when a biometric deletion request, account unbinding request, authorization deletion request, or factory reset request is received from the mobile terminal, the persistent object corresponding to the biometric stored in the trusted execution environment is deleted at the user granularity or device granularity, and the corresponding biometric reference in the account configuration is cleared.

7. The multimodal biometric data entry method according to claim 1, characterized in that, Each user's official template slots include a face slot, an application-side synchronous voiceprint slot, and an end-side enhanced voiceprint slot. The on-the-spot face registration and remote face registration update the same face slot mutually exclusively, and the on-the-spot voiceprint registration and remote voiceprint registration update the same application-side synchronous voiceprint slot mutually exclusively. The method further includes a shadow mode voiceprint enhancement step: when the time since the last end-side enhanced voiceprint recording reaches a preset interval and candidate voiceprint features are generated during daily recognition, the candidate voiceprint features are written into a short-term cache. Retrieve face matching results, lip movement detection results, and voiceprint matching results within the cache validity period; When the face matching result indicates that the user in the current image is the user identified by the corresponding user identifier, the lip movement detection result indicates that the user in the current image is speaking, and the voiceprint matching result indicates that the candidate voiceprint feature matches the application-side synchronous voiceprint template or the terminal-side enhanced voiceprint template, the terminal-side enhanced voiceprint copy of the same user is replaced with the candidate voiceprint feature, while the application-side synchronous voiceprint copy remains unchanged; when any determination fails or the cache times out, the candidate voiceprint feature is discarded.

8. A multimodal biometric data entry device, applied to an embodied intelligent robot computing platform, characterized in that, The computing platform includes a device message routing unit, an input and arrangement unit, a face feature processing unit, a voiceprint feature processing unit, a wireless communication module, at least one camera, at least one microphone array, and a processor supporting TrustZone hardware security extensions. The computing platform is divided into a rich execution environment and a trusted execution environment. The device includes: The device message routing unit is used to obtain the input request sent by the mobile terminal via the near-field wireless communication link and forward the input request to the input orchestration unit. The input request carries a path type parameter, a target user identifier and a mobile terminal current login account identifier. The path type parameter indicates one of the following: on-body face input, on-body voiceprint input, remote face input or remote voiceprint input. The input orchestration unit is used to maintain input sessions for four paths based on path type parameters, read account authorization status, and verify account binding relationships and input authorization in conjunction with input requests. If the input conditions are met, path distribution is performed according to the path type parameters. The account authorization status includes device binding status, master account flag, authorized account flag, authorized function scope, and authorization expiration time. Only one input session is allowed to be established at a time to activate the corresponding path. When an input request for another path arrives and an input session is already occupied, the input request is rejected and a path occupancy indication is returned. If the path type parameters indicate the main body... If face registration or voiceprint registration is performed, a corresponding face registration session or voiceprint registration session is established, and a start registration command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. If the path type parameter indicates remote face registration or remote voiceprint registration, a corresponding remote face registration session or remote voiceprint registration session is established, and a remote start registration command is sent to the corresponding face feature processing unit or voiceprint feature processing unit. At the same time, the registration completion indication is received from the feature processing unit, the target template slot is determined according to the path type parameter, and the TEE secure storage module is called to store the template. The feature processing unit includes a face feature processing unit and a voiceprint feature processing unit, which are used to collect or receive raw biological data based on remote collection data packets and extract the corresponding biological feature embedding vectors, respectively. After the data is entered, the unit reports the data entry completion instruction to the data entry arrangement unit. The TEE secure storage module is used to transmit the biometric embedding vector and the target user identifier based on the target template slot and write them into the Trusted Execution Environment secure world in the form of an encrypted persistent object.

9. A embodied intelligent robot, comprising a processor, a camera, a microphone array, a wireless communication module, and a memory, characterized in that, The processor supports the TrustZone hardware security extension, and when the processor executes a computer program stored in the memory, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor that supports TrustZone hardware security extensions, it implements the method as described in any one of claims 1-7.

Images