Self-service check-in terminal interactioner identity binding method and system based on deep learning

Abstract

The application discloses a medical self-service reporting terminal interactor identity binding method and system based on deep learning, and aims at solving the problem of determining the cause-effect association of touch events and interactors caused by multi-user concurrency, shielding and asynchronous acquisition. The application realizes the high-reliability binding of touch events and unique interactors, automatic maintenance of conversation state and robust anti-interference technical effects by collecting video and touch data with hardware time stamps, unifying the time base and camera-to-screen projection mapping, estimating the human body and hand perception and contact state, constructing event candidate features, and outputting dynamic causal masks from the mask generation network to constrain the attention reasoning and conversation management of the causal Transformer.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of identity recognition, and in particular to a medical self-service check-in terminal interactor identity binding method and system based on deep learning. BACKGROUND

[0002] With the advancement of hospital informatization construction, medical self-service check-in terminals are widely used in scenarios such as registration, check-in, and payment. To ensure process accuracy and privacy security, the terminal needs to reliably bind the touch events on the touch screen with the actual interactor identity under complex crowd and multi-user concurrent conditions. Existing technologies mainly evolve along three paths: one is the geometric and rule method, which is based on human body detection and hand positioning, combines the spatial distance and time window from the fingertip to the touch point for nearest neighbor matching, and relies on the fixed projection calibration of the camera to the screen; the second is the deep learning perception fusion method, which uses target detection, target tracking, hand key points, and contact state estimation to comprehensively score the events and candidate interactors; the third is the multi-modal representation learning method, which uses attention mechanism and Transformer structure to jointly model video information and touch data.

[0003] However, the above-mentioned solutions still have deficiencies in determining the causal relationship between the interaction action and the identity:

[0004] 1. The correlation mechanism focuses more on correlation than causality: there is a lack of explicit causal constraints on people, hands, and touch events, and in the case of simultaneous touch or close-range interference, mispairing and unstable confidence are prone to occur;

[0005] 2. Insufficient cross-modal time and space alignment: there is a hardware clock offset and jitter between the camera and the touch driver, there is a lack of unified time base, the projection mapping from the camera to the screen is mostly offline static calibration, and there is a lack of online micro-correction in the running stage, which leads to projection error accumulation and affects distance and accessibility evaluation;

[0006] 3. Insufficient modeling of occlusion and accessibility: usually, semantic segmentation and relative depth are not combined for occlusion probability estimation, human kinematics constraints are not fully applied, and attention fusion stage lacks prior gating, which is prone to cross-person misbinding and affects the reliability of conversation management.

[0007] Therefore, a medical self-service check-in terminal interactor identity binding method and system that can solve the above-mentioned deficiencies of existing technologies is a problem that needs to be solved by those skilled in the art. SUMMARY

[0008] An object of the present application is to provide a medical self-service check-in terminal interactor identity binding method and system based on deep learning, aiming at the problem that the prior art is difficult to accurately determine the cause-effect correlation of touch events and interactors caused by multi-user concurrency, occlusion, and asynchronous camera and touch control driving, a technical solution is proposed, which collects and maps hardware timestamps to a unified time base, estimates the projection mapping of the camera to the screen based on the screen visual anchor point, performs human body detection and tracking, hand key point and contact state estimation, projects the fingertip to the screen coordinates and calculates the projection distance and time difference, and forms event candidate features by combining kinematics accessibility and occlusion probability, generates dynamic causal masks by a mask generation network, and performs causal attention reasoning in the causal Transformer and automatically unbinds combined with the session management, The present application has the technical effects of realizing high-reliability binding of touch events and unique interactors in complex crowd scenes, reducing cross-person misconfiguration, improving interaction continuity and terminal service reliability.

[0009] According to an embodiment of the present application, a medical self-service check-in terminal interactor identity binding method based on deep learning is provided, characterized by comprising the following steps:

[0010] S1, the camera and touch control driver of the medical self-service check-in terminal collect video frame sequences and touch event sequences with hardware timestamps respectively, and combine them in time sequence to generate original time sequences;

[0011] S2, taking the original time sequence as input, time calibration and mapping to a unified time base are performed on the video frame sequence and the touch event sequence, and the projection mapping of the camera to the screen is estimated, the video frame sequence of the unified time base, the touch event sequence of the unified time base and the projection mapping of the camera to the screen are combined to form aligned data;

[0012] S3, taking the aligned data as input, human body detection and tracking are performed on the video frame sequence in it, and hand key point and fingertip positioning, contact state probability estimation are performed, the spatiotemporal trajectory, appearance feature, hand key point, fingertip coordinate and contact state probability of each candidate interactor are organized into a candidate interactor description;

[0013] S4, taking the candidate interactor description as input, for each touch event, projecting the fingertip coordinates to the screen coordinate system, calculating the projection distance and time difference of the touch event and each candidate interactor, and obtaining the accessibility score and the occlusion probability, combining the contact state probability, and organizing the event candidate features according to the pairing index of the touch event and the candidate interactor;

[0014] S5, taking the event candidate feature as input, generating a dynamic causal mask matrix based on the event candidate feature by the mask generation network, encoding the event candidate feature into an event representation vector, a candidate interactor representation vector and a hand representation vector respectively, and assembling the event representation vector, the candidate interactor representation vector and the hand representation vector into a representation vector sequence in time sequence and with a pairing index, and combining the dynamic causal mask matrix and the representation vector sequence into a causal mask and a representation vector;

[0015] S6, taking the causal mask and the representation vector as input, performing causal attention reasoning on the representation vector under the constraint of the dynamic causal mask, outputting the unique interactor identity and the corresponding confidence of each touch event, and generating an identity binding result;

[0016] S7, taking the identity binding result as input, updating the current session state of the medical self-check-in terminal, and automatically releasing the identity binding according to the unbinding rule when the interactor leaves or there is no touch event within the preset time window, and generating a session state result.

[0017] Optionally, step S1 is specifically:

[0018] The video frame sequence with a hardware timestamp is collected through the camera of the medical self-check-in terminal;

[0019] The touch event sequence with a hardware timestamp and containing screen coordinates is collected through the touch driving of the touch screen, wherein the screen coordinates are two-dimensional coordinates of the touch point on the touch screen;

[0020] The video frame sequence and the touch event sequence are combined according to the time sequence of their hardware timestamps to generate an original time sequence.

[0021] Optionally, step S2 is specifically:

[0022] Taking the original time sequence as input, reading the hardware timestamps of the video frame sequence and the touch event sequence, estimating and calibrating the time offset between the camera time and the touch driving time, mapping the video frame sequence and the touch event sequence to a unified time base, obtaining a unified time base video frame sequence and a unified time base touch event sequence;

[0023] Stable visual anchors such as screen boundaries are extracted from the unified time base video frame sequence, and the projection mapping from the camera to the screen is estimated based on the stable visual anchors and is corrected online during running;

[0024] The unified time base video frame sequence, the unified time base touch event sequence and the projection mapping from the camera to the screen are combined into alignment data.

[0025] Optionally, step S3 is specifically:

[0026] Taking the alignment data as input, human upper body detection and multi-target tracking are performed on the video frame sequence in the unified time base to obtain the spatiotemporal trajectory and appearance feature of each candidate interactor;

[0027] Hand key point detection and fingertip positioning are performed on the video frame sequence in the unified time base to output the contact state probability of each candidate interactor;

[0028] The spatiotemporal trajectory, appearance feature, hand key point, fingertip coordinate and contact state probability of each candidate interactor are merged with the alignment data to generate a candidate interactor description.

[0029] Optionally, step S4 is specifically:

[0030] Taking the candidate interactor description as input, a touch event sequence in the unified time base, hand key points, fingertip coordinates, contact state probabilities of the candidate interactor and a camera-to-screen projection mapping are read;

[0031] The fingertip coordinates of each candidate interactor are projected to a screen coordinate system based on the camera-to-screen projection mapping;

[0032] For each touch event in the unified time base, the projection distance and time difference between the touch event and each candidate interactor are calculated, kinematic features are extracted, reachability scores are estimated according to joint angles and upper limb lengths, and occlusion probabilities are estimated according to segmentation and relative depth, and the contact state probability is combined to obtain event candidate features according to the pairing index of the touch event and the candidate interactor.

[0033] Optionally, step S5 is specifically:

[0034] Taking the event candidate features as input, a mask value is calculated for each touch event and each candidate interactor by a mask generation network according to the projection distance, time difference, reachability score, occlusion probability and contact state probability, and a temperature soft gate is used to form a dynamic causal mask matrix;

[0035] Meanwhile, the event candidate features are encoded into event representation vectors, candidate interactor representation vectors and hand representation vectors, and are assembled into a representation vector sequence in time sequence and pairing index;

[0036] The dynamic causal mask matrix and the representation vector sequence jointly constitute a causal mask and a representation vector.

[0037] Optionally, step S6 is specifically:

[0038] Taking the causal mask and the representation vector as input, wherein the causal mask and the representation vector include the dynamic causal mask matrix and the representation vector sequence;

[0039] inputting the event representation vector in the representation vector sequence as a query, and inputting the candidate interactor representation vector and the hand representation vector in the representation vector sequence as a key and a value into an attention calculation module of the causal Transformer;

[0040] obtaining a candidate interactor probability distribution for each touch event by taking the dynamic causal mask matrix as a prior gate and performing temperature scaling in attention calculation;

[0041] selecting a candidate interactor with the maximum probability as the unique interactor identity based on the candidate interactor probability distribution, outputting a corresponding confidence, and generating an identity binding result.

[0042] Optionally, step S7 specifically includes:

[0043] inputting the identity binding result as an input, reading the unique interactor identity and the corresponding confidence of each touch event, registering the unique interactor identity as a current session state of the medical self-check-in terminal, and recording a time of the latest touch event;

[0044] when a new touch event is not generated within a preset time window, automatically releasing the current identity binding according to a release rule;

[0045] when the identity binding result does not contain the unique interactor identity of the current session state within the preset time window, determining that an interactor leaves a visible area and automatically releasing the current identity binding according to the release rule;

[0046] when a new touch event occurs and the identity binding result points to a unique interactor identity different from the current session state, releasing the existing identity binding and registering a new unique interactor identity as the current session state;

[0047] organizing a state after releasing or maintaining the identity binding into a session state result.

[0048] A medical self-check-in terminal interactor identity binding system based on deep learning, characterized in that it includes:

[0049] a collection module configured to collect a time-stamped video frame sequence and a touch event sequence and combine them into an original time sequence;

[0050] an alignment module configured to perform time calibration on the sequence and map it to a unified time base, estimate a camera-to-screen mapping, and output alignment data;

[0051] a perception module configured to perform human body detection and tracking, hand key point and fingertip positioning, and contact state estimation on the video frame sequence, and output a candidate interactor description;

[0052] The association module is configured to project the fingertip coordinates to a screen coordinate system, calculate association features of the candidate interactors for the touch event, and output event candidate features in combination with a contact state;

[0053] The mask and representation module is configured to generate a dynamic causal mask and construct a representation vector sequence based on the event candidate features, and output the causal mask and the representation vector;

[0054] The identity binding module is configured to perform causal attention reasoning under the constraint of the dynamic causal mask, output the unique interactor identity and confidence of each touch event, and generate an identity binding result;

[0055] The session management module updates the current session state according to the identity binding result, and automatically releases or maintains the identity binding according to the unbinding rule, and outputs a session state result.

[0056] The beneficial effects of the present application are:

[0057] 1. Based on the ternary space-time joint modeling of people, hands and touch events, the dynamic causal mask is introduced, and the priori gating and temperature scaling are implemented in the causal Transformer, in combination with the contact state, the accessibility score and the occlusion probability, so that the cross-person misbinding and concurrent interference are significantly reduced, and the high-confidence binding of the touch event and the unique interactor is ensured;

[0058] 2. Through the unified time base calibration driven by the hardware timestamp and the online micro-correction of the camera-to-screen projection mapping, the projection distance, the time difference and the kinematics feature calculation are more accurate, and higher robustness and stability are achieved under the conditions of clock drift, view angle change and environmental disturbance;

[0059] 3. The identity binding result and the session management are linked, the identity is automatically unbound or switched according to the timeout and leaving rules, the current session state is continuously maintained, the misoperation and unauthorized risk are reduced, and the interaction continuity and terminal business reliability are improved. BRIEF DESCRIPTION OF DRAWINGS

[0060] The accompanying drawings are included to provide a further understanding of the present application, and constitute a part of the specification, and are used to explain the present application together with embodiments of the present application, and do not constitute a limitation of the present application. In the drawings:

[0061] Figure 1 A flowchart of a medical self-service check-in terminal interactor identity binding method and system based on deep learning is provided. DETAILED DESCRIPTION

[0062] The present application will now be described in further detail with reference to the drawings. These drawings are simplified schematic diagrams, and only illustrate the basic structure of the present application in a schematic manner, and therefore only show the components related to the present application.

[0063] Reference Figure 1 A medical self-check-in terminal interactioner identity binding method based on deep learning, characterized in that it comprises the following steps:

[0064] S1, the camera and touch driver of the medical self-check-in terminal respectively collect video frame sequences and touch event sequences with hardware time stamps, and combine them in time sequence to generate original time sequences;

[0065] S2, taking the original time sequence as input, time calibrating and mapping the video frame sequence and the touch event sequence to a unified time base, while estimating the camera-to-screen projection mapping, the unified time base video frame sequence, the unified time base touch event sequence and the camera-to-screen projection mapping form the alignment data;

[0066] S3, taking the alignment data as input, human body detection and tracking are performed on the video frame sequence in it, and hand key point and fingertip positioning, contact state probability estimation are performed, the spatiotemporal trajectory, appearance feature, hand key point, fingertip coordinate and contact state probability of each candidate interactioner are organized into candidate interactioner description;

[0067] S4, taking the candidate interactioner description as input, for each touch event, projecting the fingertip coordinates to the screen coordinate system, calculating the projection distance and time difference of the touch event and each candidate interactioner, and obtaining the reachability score and the occlusion probability, combining the contact state probability, and organizing the event candidate features according to the pairing index of the touch event and the candidate interactioner;

[0068] S5, taking the event candidate feature as input, generating a dynamic causal mask matrix based on the event candidate feature through a mask generation network, while encoding the event candidate feature into an event representation vector, a candidate interactioner representation vector and a hand representation vector, and assembling them into a representation vector sequence according to the time sequence and the pairing index, the dynamic causal mask matrix and the representation vector sequence together form the causal mask and the representation vector;

[0069] S6, taking the causal mask and the representation vector as input, performing causal attention reasoning on the representation vector under the constraint of the dynamic causal mask, outputting the unique interactioner identity and the corresponding confidence of each touch event, and generating the identity binding result;

[0070] S7, taking the identity binding result as input, updating the current session state of the medical self-check-in terminal, and automatically releasing the identity binding according to the unbinding rule when the interactioner leaves or there is no touch event within the preset time window, and generating the session state result.

[0071] In this specific embodiment, S1 is specifically:

[0072] In this embodiment, independent acquisition threads and circular buffers are configured for the camera and touch driver in the medical self-service check-in terminal, respectively, to ensure that the data is attached with the timestamp of the source hardware and written to the memory queue at the same time it is generated at the source, thereby avoiding the interference of application layer time drift on the order judgment.

[0073] Specifically, the camera outputs image frames at a fixed frame rate and the driver layer records the hardware counter value as a timestamp at the point of interruption. After successful capacitive sampling, the touch driver packages the screen coordinates of the touch point and the event identifier into an event record and adds the timestamp generated by the hardware timing in the driver layer.

[0074] To ensure seamless integration with the time calibration in subsequent step S2, this step only performs "acquisition + time sequence combination" without any cross-modal clock domain correction or interpolation, and adopts a single-consumer merging strategy under concurrent conditions to maintain the stability of the original sequence.

[0075] Let the video frame sequence be:

[0076] ;

[0077] The touch event sequence is as follows:

[0078] ;

[0079] in Indicates the first Frame image data, This indicates that the frame is generated by the hardware timestamp of the camera's clock field. For frame index, The number of frames captured. Indicates the first The two-dimensional screen coordinates of a touch event on the touchscreen. This indicates that the event was generated by a hardware timestamp from the touch-driven clock field. For touch event indexing, The number of touch events collected;

[0080] The terminal's data merging thread reads the latest records incrementally from two buffer queues and performs a stable, non-descending sort based on the timestamp field within each record to obtain the original time series, denoted as:

[0081] and and satisfy ;

[0082] in Indicates the sorted order of the first... 1 record To merge sequence indexes, This represents the total number of records after merging. represents a union operation that merges the two types of records into a single collection, represents a stable sort operator in ascending order of the timestamp field within the record (when the timestamps are equal, the relative order of the original source queue is preserved);

[0083] In the present embodiment, S2 is specifically:

[0084] The terminal takes the original time sequence as input, first reads the two-way hardware timestamps, and estimates the constant offset of the camera time relative to the touch driving time in the running period in the sliding window cross-correlation and peak positioning mode, so as to unify the two-way data to the same time base for subsequent cross-modal reasoning;

[0085] An additive calibration model is adopted, which unifies the time base timestamp:

[0086] ;

[0087] Wherein is the hardware timestamp of the first frame video in the camera clock domain, is the video frame index, is the additive offset amount for the camera clock domain to be translated to the touch driving clock domain, is the hardware timestamp of the first touch event in the touch driving clock domain, is the touch event index, and is the timestamp under the unified time base;

[0088] In the unified time base video frame, the system estimates the projection mapping of the camera to the screen based on the stable visual anchor points such as the screen frame and the corner point detection and matching, adopts a homography model:

[0089] Complete the coordinate transformation from the camera image plane to the screen coordinate system;

[0090] Wherein is the homography matrix representing the projection mapping of the camera to the screen, is the homogeneous coordinate on the image plane, is the homogeneous coordinate under the screen coordinate system, is the scale factor introduced by the homogeneous normalization; In order to suppress the cumulative error caused by the slight change of the installation posture and the viewing angle, an online micro-correction strategy is adopted to update the mapping smoothly, which is specifically:

[0091]

[0092] ; ​

[0093] wherein is the current projection mapping at time is the current projection mapping at time is the time index, is the projection mapping at the previous time, is the projection mapping fitted from the current frame stable anchor observations, is the updated smoothing coefficient.

[0094] Finally, the video frame sequence with uniform time base, the touch event sequence with uniform time base and the camera-to-screen projection mapping maintained online are jointly organized as the alignment data:

[0095] .

[0096] wherein denotes the video frame sequence mapped to uniform time base, denotes the touch event sequence mapped to uniform time base, denotes the projection mapping after online micro-correction.

[0097] In the specific embodiment, the S3 is specifically:

[0098] With the alignment data as input, the upper body human detection and multi-target tracking are firstly performed on the video frame sequence with uniform time base to obtain stable cross-frame identities, and the deep convolution network is used in the detection stage to output the candidate box and the appearance vector, and the spatio-temporal and appearance joint correlation is used in the tracking stage to maintain the trajectory continuity of the same person.

[0099] For the convenience of formal description, the spatio-temporal trajectory and appearance feature of the first candidate interactive person are recorded as:

[0100] .

[0101] wherein denotes the spatio-temporal trajectory set of the candidate interactive person , denotes the candidate interactive person index, denotes the time index under the uniform time base, denotes the visible time index set of the candidate interactive person , denotes the upper body detection box (including the center coordinates and the width and height) at time , denotes the appearance feature vector extracted at time ;

[0102] Subsequently, the spatio-temporal trajectory and appearance feature of the second hand keypoint and fingertip localization in the region of the hand to obtain a hand keypoint coordinate set and a fingertip two-dimensional coordinate, wherein the fingertip localization is expressed in a mapping form taking the keypoint as input ;

[0103] wherein represents the fingertip two-dimensional coordinate at time , represents the hand keypoint coordinate set (containing the two-dimensional coordinates of several joint points) at time , represents a function mapping realized by the fingertip localization network;

[0104] On this basis, the state probability of whether contact with the screen is evaluated, which is expressed by a logistic regression type estimation model as follows:

[0105] ;

[0106] wherein represents the contact state probability at time , represents a Sigmoid activation function, represents a weight vector of the contact state classifier, represents a feature vector obtained by splicing the hand keypoint, fingertip motion and local appearance, represents a bias term of the classifier;

[0107] Finally, the spatiotemporal trajectory and appearance feature , hand keypoint , fingertip coordinate and contact state probability of each candidate interactor at a unified time base are aligned by time and associated with the frame index of the aligned data to organize a candidate interactor description for subsequent correlation feature calculation and causal reasoning.

[0108] In the specific embodiment, the S4 is specifically:

[0109] Taking the candidate interactor description as input and reading the touch event sequence of the unified time base and the camera-to-screen projection mapping, first, the fingertip of each candidate interactor at the unified time base is projected from the camera image coordinate to the screen coordinate to realize the same domain comparison with the touch event, which is expressed by a homography transformation as follows:

[0110] ;

[0111] wherein represents a camera-to-screen homography matrix, represents the candidate interactor at time homogeneous coordinates of the fingertip in the camera image plane, denotes the corresponding projection to the homogeneous coordinates in the screen coordinate system, denotes the homogeneous normalization scale factor, is the index of the candidate interactor, is the time index under the unified time base;

[0112] Subsequently, a spatio-temporal pairing is constructed around each touch event with the candidate interactor, and the geometric distance and time difference are calculated to quantify the degree of "close and synchronous", defined as:

[0113] and ;

[0114] where denotes the projection Euclidean distance between the touch event and the candidate interactor , denotes the two-dimensional Euclidean norm, denotes the two-dimensional coordinates of the touch event on the touch screen, denotes the absolute time difference between the unified time base timestamp of the touch event and the time , denotes the timestamp of the touch event mapped to the unified time base, is the index of the touch event;

[0115] At the kinematics and visibility level, the joint angle vector estimated based on the hand key points, the equivalent length of the upper limbs, and the short-time motion of the fingertips constitute the kinematic description and are output by the learning scorer to obtain the reachability score , while the semantic segmentation and relative depth estimation are used to obtain the occlusion probability ;

[0116] where denotes the possibility score of reaching the touch event, denotes the probability of being occluded by other people or their own limbs;

[0117] Combining the contact state prior, the above quantities are organized into an event candidate feature vector:

[0118] ;

[0119] where denotes the feature vector for pairing index , denotes the candidate interactor at time The contact state probabilities, all of the above quantities are organized by paired indexes after being aligned with a unified time base for subsequent mask generation and representation encoding.

[0120] In this specific embodiment, S5 specifically includes:

[0121] Using paired feature vectors as input, a masking network first applies temperature-based soft gating to each touch event and each candidate interactor to form dynamic causal constraints. Based on this, it constructs a sequence of representation vectors for events, candidates, and hands for subsequent causal attention inference. The core computation of the mask is... All mask values ​​are then organized into a dynamic causal mask matrix based on the number of events and candidates. ;

[0122] in Indicates that the paired index is The gate mask value, Indicates the touch event index under a unified time base. Represents the candidate interactor index under a unified time base. Represents the Sigmoid function, This represents a scalar scoring function implemented by a mask generation network. This represents the event candidate feature vector constructed in step S4. It indicates temperature parameters and controls the sharpness of the gating. Represents a dynamic causal mask matrix, Indicates the total number of touch events, Indicates the total number of candidate interactors;

[0123] To obtain a compact representation, this embodiment averages the paired features across candidates on the event side. The event representation vector is obtained through the encoding function. The mean value of the paired features across events is obtained for the candidate side. Candidate representation vectors are obtained through the encoding function. To characterize the influence of local hand states on causal relationships, the paired features are directly encoded using hand representation. ;

[0124] in Indicates touch event Average features across all candidates Indicates candidate interactors Average characteristics across all events Event representation encoding function, Candidate representation encoding function, Hand representation encoding function Indicates touch event Event representation vector, Indicates candidate interactors Candidate representation vectors, Indicates paired index The hand representation vector;

[0125] Finally, the dynamic causal mask matrix and the representation vector sequence organized by time order and paired index are output together, denoted as causal mask and representation vector respectively, for the next step of causal attention inference.

[0126] In this specific embodiment, S6 specifically refers to:

[0127] Taking a causal mask and representation vector as input, causal constraints are applied in the time dimension, allowing event queries to focus only on their history and current moment. The event representation vector serves as the query, while candidate interactor representation vectors and hand representation vectors serve as keys and supplementary causal evidence. First, a linear mapping is performed on events and candidates to obtain the query and keys required for attention, and hand representations provide additional scoring. The query and key are defined as follows: Candidate distributions are obtained using temperatureization and prior gating. Then, the unique interactor identity and confidence level are obtained using the maximum a posteriori criterion. ;

[0128] in Indicates the first Event representation vector of each touch event, Indicates the touch event index, Indicates the first Candidate representation vectors of candidate interactors Indicates the candidate interactor index, Indicates paired index Hand representation vector, Indicates an event query vectors This represents a linear transformation matrix that maps an event representation to a query. Indicates candidate key vector, This represents the linear transformation matrix that maps candidate representations to keys. Representing the dimension of the attention latent space and used for scaling the inner product, This indicates the weighting factor of the hand gesture in the scoring. This represents mapping the hand representation to a weight vector of scalar scores. Represents temperature parameters and is used to control the smoothness of the distribution. This indicates that the dynamic causal mask generated in step S5 is used in pairing. The gate value at the location, Indicates to avoid Numerical stability constant, Indicating in candidate index Softmax normalization operator for dimension, Represents the natural logarithm, Indicates touch event Unique Interactor Index This indicates the corresponding confidence level and is used to output the identity binding results in subsequent session management.

[0129] In this specific embodiment, S7 specifically refers to:

[0130] Using the identity binding result as input, the system maintains the dual-element status of "current session identity" and "last touch time" according to a unified time base, and automatically unbinds and switches based on a preset time window.

[0131] First, let's denote any current moment as a unified time base time. The set of bound event indexes within the nearest time window is defined as follows:

[0132] ;

[0133] in Indicates time The set of event indexes within the window Indicates the touch event index, Indicates touch event Unified time base timestamp, Indicates the duration of the preset time window;

[0134] Based on in-window events, the current session is assessed in three ways: automatic unbinding upon timeout of inactivity, automatic unbinding upon leaving the visible area, and binding switching triggered by different identities. Specifically, the session identity is updated in segments. Represented as:

[0135] ;

[0136] in Indicates time Current session identity, Indicates the session identity at the previous moment. Indicates an unbound state. Indicates the timestamp of the most recent touch event, The operator representing the summation of set indices, This indicates an indicator function (1 for true, 0 for false). Indicates touch event The unique interactor identity obtained through binding Indicates at time the index of the touch event just arrived or newly processed;

[0137] When binding or switching occurs, the latest touch time is updated as and the state consistency within the window is maintained, and finally the session state result is recorded as ;

[0138] wherein represents the session state result pair at the moment , and and have the same meaning as before.

[0139] The above description is merely preferred specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art, according to the technical range disclosed by the present application and the inventive concept, can make equivalent replacements or changes within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for binding an identity of an interactor of a medical self-service check-in terminal based on deep learning, characterized in that, The method comprises the following steps: S1, collecting video frame sequences and touch event sequences with hardware time stamps by the camera and touch driver of the medical self-check-in terminal respectively, and combining them in time sequence to generate original time sequences; S2, taking the original time sequences as input, time calibrating and mapping the video frame sequences and touch event sequences to a unified time base, while estimating the projection mapping from the camera to the screen, and combining the unified time base video frame sequences, the unified time base touch event sequences and the projection mapping from the camera to the screen to form alignment data; S3, taking the alignment data as input, performing human body detection and tracking on the video frame sequences in the alignment data, and performing hand key point and fingertip positioning, contact state probability estimation, organizing the spatiotemporal trajectory, appearance feature, hand key point, fingertip coordinate and contact state probability of each candidate interactor into candidate interactor description; S4, taking the candidate interactor description as input, projecting the fingertip coordinates to the screen coordinate system for each touch event, calculating the projection distance and time difference between the touch event and each candidate interactor, and obtaining reachability score and occlusion probability, combining the contact state probability, and organizing the event candidate features according to the pairing index of the touch event and the candidate interactor; S5, taking the event candidate features as input, generating a dynamic causal mask matrix based on the event candidate features by a mask generation network, while encoding the event candidate features into event representation vectors, candidate interactor representation vectors and hand representation vectors, and assembling them into representation vector sequences according to time sequence and pairing index, and combining the dynamic causal mask matrix and the representation vector sequence into causal mask and representation vector; S6, taking the causal mask and representation vector as input, performing causal attention reasoning on the representation vector under the constraint of dynamic causal mask, outputting the unique interactor identity and corresponding confidence of each touch event, and generating identity binding result; S7, taking the identity binding result as input, updating the current session state of the medical self-check-in terminal, and automatically releasing the identity binding according to the unbinding rule when the interactor leaves or there is no touch event within the preset time window, and generating the session state result. 2.The method of claim 1, wherein, S1 specifically comprises: collecting video frame sequences with hardware time stamps through the camera of the medical self-check-in terminal; collecting touch event sequences with hardware time stamps and containing screen coordinates through the touch driver of the touch screen, wherein the screen coordinates are two-dimensional coordinates of the touch point on the touch screen; combining the video frame sequences and the touch event sequences according to the time sequence of their hardware time stamps to generate original time sequences. 3.The method of claim 1, wherein, S2 specifically comprises: taking the original time sequences as input, reading the hardware time stamps of the video frame sequences and the touch event sequences, estimating the time offset between the camera time and the touch driver time and performing time calibration, mapping the video frame sequences and the touch event sequences to a unified time base to obtain unified time base video frame sequences and unified time base touch event sequences; extracting screen boundary stable visual anchor points in the unified time base video frame sequences, estimating the projection mapping from the camera to the screen based on the stable visual anchor points and performing online micro-correction during running; The sequence of video frames with unified time base, the sequence of touch events with unified time base, and the camera-to-screen projection mapping jointly constitute alignment data. 4.The method of claim 1, wherein, S3 specifically is: Taking the alignment data as input, human upper body detection and multi-target tracking are performed on the sequence of video frames with unified time base to obtain the spatiotemporal trajectory and appearance feature of each candidate interactor; Hand key point detection and fingertip positioning are performed on the sequence of video frames with unified time base to output the contact state probability of each candidate interactor; The spatiotemporal trajectory, appearance feature, hand key point, fingertip coordinates, and contact state probability of each candidate interactor are merged with the alignment data to generate a candidate interactor description.

5. The medical self-service check-in terminal interactor identity binding method based on deep learning according to claim 1, characterized in that, S4 specifically is: Taking the candidate interactor description as input, the sequence of touch events with unified time base, the hand key point, fingertip coordinates, and contact state probability of the candidate interactor, and the camera-to-screen projection mapping are read; The fingertip coordinates of each candidate interactor are projected to the screen coordinate system based on the camera-to-screen projection mapping; For each touch event with unified time base, the projection distance and time difference between the touch event and each candidate interactor are calculated, kinematic features are extracted, reachability scores are estimated based on joint angles and upper limb lengths, and occlusion probabilities are estimated based on segmentation and relative depth, and the contact state probability is combined to obtain event candidate features according to the pairing index of the touch event and the candidate interactor. 6.The method of claim 1, wherein, S5 specifically is: Taking the event candidate features as input, mask values are calculated for each touch event and each candidate interactor by a mask generation network based on the projection distance, time difference, reachability score, occlusion probability, and contact state probability, and a dynamic causal mask matrix is formed by temperature soft gating; Meanwhile, the event candidate features are encoded into event representation vectors, candidate interactor representation vectors, and hand representation vectors, and are assembled into a representation vector sequence in time order and pairing index; The dynamic causal mask matrix and the representation vector sequence jointly constitute a causal mask and a representation vector.

7. The method of claim 1, wherein the method further comprises: S6 specifically is: Taking the causal mask and the representation vector as input, wherein the causal mask and the representation vector include the dynamic causal mask matrix and the representation vector sequence; The event representation vectors in the representation vector sequence are taken as queries, and the candidate interactor representation vectors and hand representation vectors in the representation vector sequence are taken as keys and values, which are input into the attention calculation module of the causal Transformer; In the attention calculation, the dynamic causal mask matrix is taken as a priori gating and is temperature scaled to obtain a candidate interactor probability distribution for each touch event; Based on the candidate interactor probability distribution, the candidate interactor with the maximum probability is selected as the unique interactor identity, and the corresponding confidence is output to generate an identity binding result. 8.The method of claim 1, wherein, S7 specifically is: Taking the identity binding result as input, the unique interactor identity and the corresponding confidence of each touch event are read, the unique interactor identity is registered as the current session state of the medical self-service check-in terminal, and the time of the last touch event is recorded; When no new touch event is generated within a preset time window, the current identity binding is automatically released according to the unbinding rule. When the identity binding result does not contain the unique interactor identity of the current session state within a preset time window, it is determined that the interactor leaves the visual area and the current identity binding is automatically released according to the unbinding rule; When a new touch event occurs and the identity binding result points to a unique interactor identity different from the current session state, the existing identity binding is released and the new unique interactor identity is registered as the current session state; The state after releasing or maintaining the identity binding is organized as a session state result.

9. A deep learning-based medical self-service check-in terminal interactor identity binding system, which executes the deep learning-based medical self-service check-in terminal interactor identity binding method according to any one of claims 1 to 8, characterized in that, Comprise: A collection module for collecting a time-stamped video frame sequence and a touch event sequence and combining them into an original time sequence; An alignment module for time calibration of the sequence and mapping to a unified time base, estimating camera-to-screen mapping, and outputting alignment data; A perception module for human body detection and tracking, hand key point and fingertip positioning, and contact state estimation on the video frame sequence, and outputting candidate interactor descriptions; An association module for projecting fingertip coordinates to a screen coordinate system, calculating association features of the candidate interactor for the touch event, and combining the contact state, and outputting event candidate features; A mask and representation module for generating a dynamic causal mask based on the event candidate features and constructing a representation vector sequence, and outputting the causal mask and the representation vector; An identity binding module for causal attention reasoning under the constraint of the dynamic causal mask, outputting the unique interactor identity and the confidence of each touch event, and generating an identity binding result; A session management module for updating the current session state according to the identity binding result, and automatically releasing or maintaining the identity binding according to the unbinding rule, and outputting a session state result.

Images