Multi-modal identity authentication method fusing human face and gait features

Abstract

This invention discloses a multimodal identity authentication method that integrates facial and gait features. The method includes: acquiring a video frame sequence and generating human trajectory units; extracting facial image sequences and gait contour sequences based on the human trajectory units to generate a first identity candidate, a first confidence trajectory, a second identity candidate, and a second confidence trajectory; constructing a first binding evidence chain and a second binding evidence chain, and inputting them into a binding decision model to generate a binding state vector and a cross-modal binding identifier; establishing an authentication object record based on the cross-modal binding identifier, generating an identity conflict record, and calling an authentication decision table to output a decision instruction. This invention reduces the risk of face and gait mismatch and improves the credibility of identity authentication in complex scenarios through cross-modal binding determination and conflict record decision-making.

Description

Technical Field

[0001] This application relates to the field of biometric technology, and in particular to a multimodal identity authentication method that integrates facial and gait features. Background Technology

[0002] With the development of intelligent sensing, computer vision and biometric recognition technologies, identity authentication is evolving from contact-based verification that relies on documents, passwords or single static features to multimodal authentication that is long-distance, non-intrusive and continuous.

[0003] Facial features have the advantages of intuitive collection and high recognition efficiency, while gait features can reflect individual behavioral differences under non-contact, weak cooperation, and long-distance conditions. The integration of the two can be used in smart parks, office buildings, industrial sites, and public security scenarios to improve the reliability of identity verification in complex environments.

[0004] Existing methods often use fixed weights, distances, angles, or image quality to weight and fuse face recognition and gait recognition results. They typically assume that both types of samples naturally originate from the same authentication object and that the high-confidence modality can compensate for the low-confidence modality. This leads to two easily overlooked problems: First, when multiple people are involved, there is occlusion, trajectory loss, or reconnection, the system may incorrectly bind A's face with B's gait before fusion, resulting in cross-modal identity mismatch. Second, when the face and gait point to different identities and both have high confidence levels, simple weighting or selection can mask anomalies such as forged faces, proxy walking, and tracking drift, making it difficult to identify high-confidence conflicts between different modalities as rejection or verification signals. Summary of the Invention

[0005] To address the aforementioned problems, embodiments of the present invention provide a multimodal identity authentication method that integrates face and gait features, the method comprising:

[0006] The video frame sequence of the scene is acquired, human detection and tracking are performed on the video frame sequence, and human trajectory units are generated. The human trajectory units include human bounding box sequence, head region sequence, time stamp sequence and trajectory status identifier.

[0007] Based on the human body trajectory unit, the face image sequence and gait contour sequence are extracted, and the face features and gait features are extracted to generate the first identity candidate, the first confidence trajectory, the second identity candidate and the second confidence trajectory;

[0008] A first binding evidence chain is constructed based on the spatial attribution relationship between the head region sequence and the human body frame sequence, and a second binding evidence chain is constructed based on the phase continuity relationship between the gait contour sequence and the time-marked sequence.

[0009] Input the first and second binding evidence chains into the binding determination model, and determine the binding of the first and second binding evidence chains based on the trajectory state identifier, generating a binding state vector and a cross-modal binding identifier;

[0010] An authentication object record is established based on the cross-modal binding identifier. The authentication object record includes a first authentication object field, a second authentication object field, and a merged authentication object field.

[0011] An identity conflict record is generated based on the authentication object record, the first identity candidate, the second identity candidate, the first confidence trajectory, and the second confidence trajectory;

[0012] The authentication decision table is invoked based on the identity conflict record, and the decision instruction is output.

[0013] Furthermore, the method for generating the human trajectory unit includes: forming a first trajectory segment based on the human detection result, and forming a second trajectory segment based on the tracking result; when a disconnection marker is generated between the first trajectory segment and the second trajectory segment, generating a third trajectory segment using the directional continuation item of the head region sequence and the center continuation item of the initial human contour sequence extracted from the human detection result; and writing the trajectory status marker with the temporal relationship between the first trajectory segment, the third trajectory segment, and the second trajectory segment.

[0014] Furthermore, the first binding evidence chain includes a first evidence item, a second evidence item, and a third evidence item; the first evidence item is formed by the positional relationship of the head region sequence in the human body frame sequence, the second evidence item is formed by the following relationship of the facial key point sequence relative to the human body frame sequence, and the third evidence item is formed by the order of appearance of the head region sequence relative to the time mark sequence; the first evidence item, the second evidence item, and the third evidence item are arranged according to the time mark sequence.

[0015] Furthermore, the second binding evidence chain includes a fourth evidence item, a fifth evidence item, and a sixth evidence item; the fourth evidence item is formed by the trunk center sequence in the gait contour sequence, the fifth evidence item is formed by the foot swing area in the gait contour sequence, and the sixth evidence item is formed by the temporal continuity relationship of the phase succession relationship relative to the human trajectory unit; the fourth evidence item, the fifth evidence item, and the sixth evidence item are arranged in a time-marked sequence.

[0016] Furthermore, the binding determination model is a binary classification determination model based on the fusion of temporal coding and gating. The binding determination model includes a feature processing layer, a first coding channel, a second coding channel, a gating channel, and a determination layer. The feature processing layer organizes the first binding evidence chain into a spatial attribution input sequence according to the time-marked sequence, and organizes the second binding evidence chain into a phase inheritance input sequence. The first coding channel performs spatial attribution coding on the spatial attribution input sequence, and the second coding channel performs phase inheritance coding on the phase inheritance input sequence. The gating channel performs gating integration on the spatial attribution coding and phase inheritance coding according to the trajectory state identifier to obtain a binding state vector. The determination layer generates a cross-modal binding identifier based on the binding state vector.

[0017] Furthermore, the method for establishing the authentication object record includes: when the cross-modal binding identifier is an unbound identifier, writing the face image sequence into the first authentication object field and the gait contour sequence into the second authentication object field; when the cross-modal binding identifier is a bound identifier, writing the face image sequence and the gait contour sequence into the merged authentication object field.

[0018] Furthermore, the identity conflict record includes a candidate identity field, a confidence trajectory field, and a binding status field; the first identity candidate and the second identity candidate are written into the candidate identity field, the first confidence trajectory and the second confidence trajectory are written into the confidence trajectory field, and the binding status vector is written into the binding status field; a first conflict label is written based on the identity separation status of the candidate identity field and the trajectory stability status of the confidence trajectory field.

[0019] Furthermore, the authentication decision table includes a first decision area, a second decision area, and a third decision area; the first decision area is associated with the identity convergence status of the candidate identity field, the second decision area is associated with the first conflict label, and the third decision area is associated with the unbound identifier; the corresponding decision instruction is output according to the decision area where the identity conflict record is located.

[0020] Furthermore, the binding determination model is trained using a registration sample group, which includes a positive binding sample group and an anti-binding sample group. The positive binding sample group consists of a registered face image sequence, a registered gait contour sequence, and a registered time stamp sequence of the same registrant. The anti-binding sample group consists of a registered face image sequence of a first registrant, a registered gait contour sequence of a second registrant, and an interleaved time stamp sequence. The initial binding model is trained using the positive binding sample group and the anti-binding sample group to obtain the binding determination model.

[0021] Furthermore, the multimodal identity authentication method that integrates facial and gait features also includes: tracing back the disjoint identity index, confidence trajectory index, and binding status index based on the first conflict label, determining the conflict time marker, extracting local head attribution records and local phase continuity records, generating local misalignment verification segments by combining trajectory status identifiers, and writing the conflict source label into the identity conflict record.

[0022] The technical effects and advantages of the multimodal identity authentication method that integrates facial and gait features provided by this invention are as follows:

[0023] This invention reduces the risk of face and gait mismatch and improves the credibility of identity authentication in complex scenarios by using cross-modal binding determination and conflict recording decision-making. Before fusion, this invention constructs a first binding evidence chain and a second binding evidence chain. It determines the homology between face image sequences and gait contour sequences based on spatial attribution, follow-up relationship, order of appearance, torso connection, foot swing, and phase continuation, avoiding the direct fusion of features that do not belong to the same authentication object. It uses trajectory state identification and gating integration to handle trajectory discontinuity and reconnection scenarios, ensuring traceable data sources for human trajectory units in occlusion, intersection, and trajectory reconnection scenarios. The first identity candidate, second identity candidate, first confidence trajectory, second confidence trajectory, and binding state vector are written into the identity conflict record, and corresponding decision instructions are output through the authentication decision table. This allows for the differentiation and processing of identity consistency, identity conflict, and cross-modal non-binding situations, reducing false authentication caused by mutual cancellation of high-confidence modalities. Attached Figure Description

[0024] Figure 1 This is a flowchart of the multimodal identity authentication method that integrates face and gait features in Example 1;

[0025] Figure 2 This is a flowchart of the human trajectory completion processing method in Example 1;

[0026] Figure 3 This is a flowchart of the conflict anchor point backtracking and local misalignment verification method in Example 2. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Example 1:

[0028] Please see Figure 1As shown, embodiments of the present invention provide a multimodal identity authentication method that integrates face and gait features, the method comprising:

[0029] S1. Obtain the video frame sequence of the scene, perform human detection and tracking on the video frame sequence, and generate human trajectory units. The human trajectory units include human bounding box sequence, head region sequence, time stamp sequence and trajectory status identifier.

[0030] S2. Based on the human body trajectory unit, extract the face image sequence and gait contour sequence, extract face features and gait features, and generate the first identity candidate, the first confidence trajectory, the second identity candidate and the second confidence trajectory;

[0031] S3. Construct a first binding evidence chain based on the spatial attribution relationship between the head region sequence and the human body frame sequence, and construct a second binding evidence chain based on the phase continuity relationship between the gait contour sequence and the time marker sequence.

[0032] S4. Input the first and second binding evidence chains into the binding determination model, and determine the binding of the first and second binding evidence chains based on the trajectory state identifier, generating a binding state vector and a cross-modal binding identifier.

[0033] S5. Establish an authentication object record based on the cross-modal binding identifier. The authentication object record includes a first authentication object field, a second authentication object field, and a merged authentication object field.

[0034] S6. Generate an identity conflict record based on the authentication object record, the first identity candidate, the second identity candidate, the first confidence trajectory, and the second confidence trajectory;

[0035] S7. Based on the identity conflict record, call the authentication decision table and output the decision instruction; the decision instruction is one of three: pass instruction, review instruction, and refuse instruction.

[0036] like Figure 2 As shown, in this embodiment, the generation of the human trajectory unit is not determined directly based on the detection result of a single frame. Instead, human detection is performed on each frame of the video frame sequence to obtain the human detection result. The human detection result includes at least the human body bounding box, the head region, and the corresponding time marker. According to the order of the time markers, the detection results of adjacent frames that are consecutive in position and whose human body bounding box shapes are connected are written into the same trajectory cache. The trajectory cache forms the first trajectory segment, which is used to represent the pedestrian motion segment directly established by the human detection result.

[0037] After the first trajectory segment is formed, the pedestrian position at the next time marker is continued to be matched based on the human bounding box position, the human bounding box center displacement direction and the time marker order that have been written into the trajectory cache. The detection result of the successful match is written into the tracking cache, and the second trajectory segment is formed by the tracking cache. The second trajectory segment is used to represent the pedestrian motion segment obtained by the continuation of the tracking result. The tracking result is not an independent identity conclusion, but a position continuation record of the human bounding box in the time series. Its content includes the human bounding box that is continuation, the human bounding box before continuation, the corresponding time marker and the continuation state.

[0038] When there is no directly connectable human detection result between the end of the first trajectory segment and the beginning of the second trajectory segment, a disconnection flag is generated in the trajectory cache. The disconnection flag indicates that the same human trajectory unit is temporarily interrupted in adjacent time areas. Its generation is based on the interruption of the human frame center displacement, the interruption of the human frame shape connection, and the interruption of the time marker continuity. After the disconnection flag is generated, the first trajectory segment and the second trajectory segment are not immediately identified as two pedestrians, but trajectory reconnection processing is performed.

[0039] The trajectory completion processing includes the generation of directional continuity items and center continuity items. The directional continuity items are formed from the head region sequence. Specifically, the center position of the head region at the end of the first trajectory segment, the center position of the head region at the beginning of the second trajectory segment, and the detectable head region positions within the intermediate time region are extracted and arranged according to the time markers to obtain the displacement direction relationship of the head region. When this displacement direction relationship has a continuity relationship with the movement direction at the end of the first trajectory segment, it is written into the directional continuity item. The center continuity item is formed from the gait contour sequence. Specifically, the center position of the outer region of the human contour in the gait contour sequence is extracted and combined with the movement order of the torso region on the time markers to form the movement continuity relationship of the contour center. When this movement continuity relationship can connect the end of the first trajectory segment and the beginning of the second trajectory segment, it is written into the center continuity item.

[0040] A third trajectory segment is generated based on the directional and center connection items. This third trajectory segment is not a newly detected complete trajectory, but a supplementary trajectory segment used to connect the first and second trajectory segments. Its data includes the supplementary start point, the supplementary end point, the head region sequence involved in the supplementation, the gait contour sequence involved in the supplementation, and the corresponding time marker. Subsequently, the human trajectory unit is reassembled according to the time order of the first, third, and second trajectory segments, and this reassembly relationship is written into the trajectory status identifier. The trajectory status identifier is used to record the source state of the human trajectory unit. When the human trajectory unit is formed only by continuous detection results, a continuous state is written; when the human trajectory unit is formed by supplementation through the third trajectory segment, a discontinuous supplementation state is written. This trajectory status identifier is used as input data in the subsequent binding determination to distinguish between directly continuous trajectories and trajectories formed through supplementation.

[0041] In this embodiment, the first binding evidence chain is used to record the correspondence between the face image sequence and the human body trajectory unit before extracting face features and gait features. The first binding evidence chain does not directly represent the identity recognition result, but rather indicates whether the face image sequence originates from the head region sequence in the same human body trajectory unit.

[0042] Specifically, after obtaining the human trajectory unit, the human body frame sequence and the head region sequence are read sequentially according to the time stamp sequence. The human body frame sequence is the set of regions circumscribed in the video frame sequence of the person to be authenticated, and the head region sequence is the set of regions located within the human body frame sequence and corresponding to the head position. For the same time stamp, the head region is associated with the corresponding human body frame. When the head region is located in the upper region of the human body frame and the head region moves continuously with the human body frame between adjacent time stamps, the association is written into the first evidence item. The first evidence item is used to record the spatial source relationship of "the head region belongs to the human body frame".

[0043] Furthermore, facial keypoints are located in the image regions of the head region sequence that can form clear facial contours, resulting in a facial keypoint sequence. The facial keypoint sequence refers to the set of positions of the eyes, nose, mouth, and facial contours obtained in the head region. The facial keypoint sequence under the same time mark is matched with the human body bounding box sequence, and the directional consistency and temporal continuity between the overall positional change of the facial keypoints and the positional change of the center of the human body bounding box in adjacent time marks are compared. When the facial keypoint sequence moves synchronously with the human body bounding box sequence, this follow-up relationship is written into the second evidence item. The second evidence item is used to record the dynamic correspondence of "the facial keypoint moves with the human body bounding box".

[0044] Furthermore, the order in which the head region appears, disappears, and reappears is recorded according to the time stamp sequence to form a third piece of evidence. The third piece of evidence does not rely on single-frame repetition as the basis for judgment, but instead saves the order in which the head region appears in the continuous time stamps in correspondence with the order of continuation of the human body frame sequence. For example, before occlusion occurs, the head region corresponds to the human body frame at the end of the first trajectory segment; after occlusion is removed, the head region corresponds to the human body frame at the beginning of the second trajectory segment; the missing state of the head region in the middle time segment is written into the appearance order to maintain the complete temporal record of the head region sequence.

[0045] After forming the first, second, and third evidence items, the three types of evidence items are arranged according to the time stamp sequence to obtain the first bound evidence chain. Under the same time stamp in the first bound evidence chain, at least the human body bounding box, head region, facial key points, and evidence item type are recorded. Between adjacent time stamps, the succession relationship of each evidence item is recorded. Thus, the first bound evidence chain can serve as the input data for the subsequent bound judgment model, used to express the spatial attribution, motion follow-up, and temporal order of appearance between the face image sequence and the human body trajectory unit.

[0046] In this embodiment, the second binding evidence chain is used to record the connection between the gait contour sequence and the human trajectory unit. The second binding evidence chain does not directly output the identity result, but describes whether the gait contour sequence comes from the same human trajectory unit. The gait contour sequence is extracted from the pedestrian region corresponding to the human body frame sequence. Its content includes human body contours arranged sequentially in the time stamp sequence. The human trajectory unit is a set of the aforementioned human body frame sequence, head region sequence, time stamp sequence and trajectory status identifier.

[0047] Specifically, the gait contour sequence is read according to the time-marked sequence. The torso region is determined for the human body contour under each time mark, and the torso center is determined according to the contour distribution position of the torso region. The torso center is not the geometric center of the human body frame, but a position point formed by the stable region of the torso part in the human body contour. It is used to characterize the main body position of the pedestrian under the current time mark. The torso centers under each time mark are arranged in chronological order to form a torso center sequence. This torso center sequence is written into the fourth evidence item. The fourth evidence item is used to represent the continuity of the gait contour sequence in the main body position of the human body.

[0048] Furthermore, the foot swing region is determined in the gait contour sequence. The foot swing region is the region where the lower limb end of the human body undergoes alternating displacement in continuous time markers. Its generation process includes: first, determining the lower limb contour region based on the upper and lower structure of the human body contour; then, extracting the displacement changes of the left and right boundaries and the bottom boundary within the lower limb contour region; subsequently, writing the region showing the front and back swing order into the foot swing region; and writing the change relationship of the foot swing region in the time marker sequence into the fifth evidence item. The fifth evidence item is used to indicate the temporal source of the lower limb swing in the gait contour sequence.

[0049] Furthermore, based on the fourth and fifth evidence items, a phase continuity relationship is generated. A phase refers to the temporal state of the gait profile during walking, formed by the forward movement of the torso, foot support, foot swing, and step change. The phase continuity relationship refers to the connection between gait phases under adjacent time markers. Specifically, the movement order of the human body is determined by the torso center sequence, and the order of lower limb swing is determined by the foot swing area. The two are then matched with the human body frame sequence in the human body trajectory unit, so that the gait phase continues in the same direction as the human body trajectory unit in the time marker sequence. The temporal continuity relationship of this phase continuity relationship relative to the human body trajectory unit is written into the sixth evidence item.

[0050] After the fourth, fifth, and sixth pieces of evidence are formed, the three types of evidence are arranged according to the time stamp sequence to obtain the second binding evidence chain. Under the same time stamp in the second binding evidence chain, the human body frame, gait contour, torso center, foot swing area, and phase state are recorded. Between adjacent time stamps, the torso center continuity, foot swing continuity, and phase continuity are recorded. For example, if the torso center in the previous time stamp continues in the direction of travel, and the foot swing area in the next time stamp is in a continuous swing state, and both belong to the same human trajectory unit, the corresponding records are written into the second binding evidence chain in sequence. This second binding evidence chain serves as the data basis for subsequent binding determination and is used together with the first binding evidence chain to determine whether the face image sequence and gait contour sequence belong to the same authentication object.

[0051] In this embodiment, the binding determination model is a binary classification determination model based on temporal coding and gating fusion, which is used to perform structured processing on the first binding evidence chain and the second binding evidence chain. The binding determination model does not directly output the identity recognition result, but generates a binding state vector representing the relationship between the face image sequence and the gait contour sequence based on the first binding evidence chain, the second binding evidence chain and the trajectory state identifier, and generates a cross-modal binding identifier based on the binding state vector. The binding determination model includes a feature processing layer, a first encoding channel, a second encoding channel, a gating channel and a determination layer.

[0052] Specifically, the feature processing layer reads the first, second, and third evidence items in the first bound evidence chain according to the time-stamped sequence, and organizes the positional relationship of the head region in the human body bounding box sequence, the motion relationship of the facial key point sequence relative to the human body bounding box sequence, and the appearance order of the head region relative to the time-stamped sequence into a spatial attribution input sequence. The spatial attribution input sequence includes a head attribution field, a key point motion field, and a head appearance field. Among them, the head attribution field is used to record the attribution relationship between the head region and the human body bounding box sequence, the key point motion field is used to record the motion continuity relationship of the facial key point sequence relative to the human body bounding box sequence, and the head appearance field is used to record the appearance order of the head region in the time-stamped sequence.

[0053] Furthermore, the feature processing layer reads the fourth, fifth, and sixth evidence items in the second bound evidence chain according to the time-stamped sequence, and organizes the torso center sequence, foot swing area, and phase continuity relationship into a phase continuity input sequence. The phase continuity input sequence includes a torso continuity field, a foot swing field, and a phase continuation field. Among them, the torso continuity field is used to record the main body movement continuity relationship of the torso center sequence in the time-stamped sequence, the foot swing field is used to record the swing order of the foot swing area, and the phase continuation field is used to record the temporal continuity relationship of the phase continuity relationship relative to the human body trajectory unit.

[0054] The first encoding channel reads the spatial attribution input sequence according to the time stamp sequence and performs spatial attribution encoding on the head attribution field, key point follow-up field, and head appearance field to obtain the spatial attribution code. The spatial attribution code is used to represent the continuous relationship between the face image sequence and the human body bounding box sequence in terms of spatial origin and motion follow-up. The second encoding channel reads the phase continuity input sequence according to the time stamp sequence and performs phase continuity encoding on the torso continuity field, foot swing field, and phase continuation field to obtain the phase continuity code. The phase continuity code is used to represent the continuous relationship between the gait contour sequence and the human body trajectory unit in terms of body movement, lower limb swing, and gait phase.

[0055] After obtaining the spatial attribution code and phase continuity code, the gating channel reads the trajectory status identifier in the human trajectory unit. The trajectory status identifier includes continuous status and disconnection / reconnection status. When the trajectory status identifier is continuous, the gating channel directly aligns the spatial attribution code and phase continuity code according to the same time marker to generate the first integrated field. When the trajectory status identifier is disconnection / reconnection status, the gating channel reads the time marker corresponding to the third trajectory segment, and reconnects and aligns the head appearance field before and after the disconnection of the spatial attribution code with the phase continuation field before and after the disconnection of the phase continuity code to generate the second integrated field.

[0056] Subsequently, the first integration field, the second integration field, the trajectory status identifier, and the supplementary source information of the third trajectory segment are written into the binding status vector. The binding status vector includes a spatial attribution bit, a phase inheritance bit, a trajectory status bit, and a supplementary source bit. The spatial attribution bit comes from the spatial attribution code, the phase inheritance bit comes from the phase inheritance code, the trajectory status bit comes from the trajectory status identifier, and the supplementary source bit comes from the third trajectory segment. The decision layer reads the binding status vector and performs a binary classification mapping on the binding status vector. When the binding status vector corresponds to the same origin attribution category, the decision layer generates a bound identifier. When the binding status vector corresponds to a different origin attribution category, the decision layer generates an unbound identifier. The bound identifier and the unbound identifier constitute a cross-modal binding identifier.

[0057] In this embodiment, after the binding determination model generates a cross-modal binding identifier, an authentication object record is established based on the cross-modal binding identifier. The authentication object record is a data record corresponding to an authentication process, used to store the destination of the face image sequence and gait contour sequence. It does not directly represent the identity authentication conclusion. The authentication object record includes a first authentication object field, a second authentication object field, and a merged authentication object field. The first authentication object field is used to store object data formed by the face image sequence, the second authentication object field is used to store object data formed by the gait contour sequence, and the merged authentication object field is used to store the face image sequence and gait contour sequence that belong to the same human trajectory unit after binding determination.

[0058] Specifically, when the cross-modal binding identifier is unbound, it indicates that the first binding evidence chain and the second binding evidence chain failed to form a consistent connection under the same human trajectory unit. In this case, the face image sequence, face features, first identity candidate, and first confidence trajectory are written into the first authentication object field; the gait contour sequence, gait features, second identity candidate, and second confidence trajectory are written into the second authentication object field. The first authentication object field and the second authentication object field respectively retain their respective time stamp sequence, source trajectory status identifier, and corresponding evidence chain index, so that subsequent identity conflict records can distinguish between face source and gait source.

[0059] When the cross-modal binding identifier is "bound", it means that the face image sequence and the gait contour sequence are determined to be from the same authentication object source in terms of spatial attribution, phase continuity, and trajectory status. At this time, the face image sequence, gait contour sequence, first identity candidate, second identity candidate, first confidence trajectory, second confidence trajectory, and binding status vector are written into the merged authentication object field. The merged authentication object field saves the above data according to the time stamp sequence, so that the face side data and gait side data can be called in the same time chain.

[0060] For example: if the first pedestrian to be authenticated and the second pedestrian to be authenticated intersect in the video frame sequence, and the binding judgment model outputs an unbound identifier, then the face image sequence is written into the first authentication object field, and the gait contour sequence is written into the second authentication object field, without merging the writing; when the first pedestrian to be authenticated passes through continuously and both the face image sequence and the gait contour sequence correspond to the same human trajectory unit, the binding judgment model outputs a bound identifier, then both are written into the merged authentication object field. Thus, the authentication object record provides a clear data source for the subsequent generation of identity conflict records.

[0061] In this embodiment, after the authentication object record is established, an identity conflict record is generated based on the authentication object record. The identity conflict record is used to store the face recognition result, the gait recognition result, and the binding source status of the two. It is not directly used as the final decision instruction. The identity conflict record includes a candidate identity field, a confidence trajectory field, and a binding status field. The candidate identity field is used to record the first identity candidate and the second identity candidate, the confidence trajectory field is used to record the first confidence trajectory and the second confidence trajectory, and the binding status field is used to record the binding status vector.

[0062] Specifically, the first identity candidate is obtained by comparing the facial features with the registered facial features, and the second identity candidate is obtained by comparing the gait features with the registered gait features. Both the first and second identity candidates include a candidate identity identifier, a source sequence identifier, and a corresponding time stamp. The first identity candidate is written into the face candidate subfield of the candidate identity field, and the second identity candidate is written into the gait candidate subfield of the candidate identity field. The candidate identity identifier can be represented by the registered identity number, the registered user identifier, and the registered object index. The specific representation does not affect the generation method of this field.

[0063] The first confidence trajectory is a confidence record of the face image sequence on the time-stamped sequence, and the second confidence trajectory is a confidence record of the gait contour sequence on the time-stamped sequence. The confidence trajectory is not a single confidence value, but a sequence record composed of time stamps, candidate identity identifiers, confidence sorting status, and source segment identifiers. When writing to the confidence trajectory field, the first and second confidence trajectories are first sorted according to the time-stamped sequence, and then the sorted first confidence trajectory is written to the face confidence subfield, and the sorted second confidence trajectory is written to the gait confidence subfield, so that the retention status of the same candidate identity in continuous time can be determined later.

[0064] The binding state vector originates from the binding determination model and includes at least the spatial attribution bit, phase inheritance bit, trajectory state bit, and supplementary source bit. When writing the binding state vector into the binding state field, the spatial attribution bit is matched with the evidence item index of the first binding evidence chain, the phase inheritance bit is matched with the evidence item index of the second binding evidence chain, the trajectory state bit is matched with the trajectory state identifier in the human trajectory unit, and the supplementary source bit is matched with the time stamp of the third trajectory segment. Through the above writing method, the identity conflict record can retain the relationship between candidate identities and also retain the face source, gait source, and trajectory source when the relationship is generated.

[0065] After generating an identity conflict record, the identity separation state is determined based on the candidate identity field. The identity separation state means that the candidate identity identifier in the first identity candidate and the candidate identity identifier in the second identity candidate point to two different registered identities, and both of them originate from the same authentication object record corresponding to one authentication process. The trajectory stability state is further determined based on the confidence trajectory field. The trajectory stability state means that the first confidence trajectory continuously points to the first identity candidate in its corresponding time stamp sequence, the second confidence trajectory continuously points to the second identity candidate in its corresponding time stamp sequence, and the two are not formed by isolated recognition results under a single time stamp.

[0066] When the candidate identity field forms a disjoint identity state and the confidence trajectory field forms a stable trajectory state, a first conflict label is written into the identity conflict record. The first conflict label includes a conflict source identifier, a disjoint identity index, a confidence trajectory index, and a binding state index. The conflict source identifier is used to record that the conflict is generated by the face-side candidate identity and the gait-side candidate identity; the disjoint identity index is used to point to the first identity candidate and the second identity candidate respectively; the confidence trajectory index is used to point to the first confidence trajectory and the second confidence trajectory respectively; and the binding state index is used to point to the binding state vector in the binding state field. Thus, the first conflict label can be called by the subsequent authentication decision table, and the generation of review instructions and rejection instructions has a traceable data source.

[0067] For example: In the same authentication process, a face image sequence corresponds to registered identity A, and a gait contour sequence corresponds to registered identity B. If the first confidence trajectory continuously points to registered identity A and the second confidence trajectory continuously points to registered identity B, then an identity separation state is formed in the candidate identity field, a trajectory stable state is formed in the confidence trajectory field, and the first conflict label is written in the identity conflict record. This example is only used to illustrate the field writing relationship and does not limit the specific confidence value or the specific number of frames.

[0068] In this embodiment, the authentication decision table is invoked after the identity conflict record is generated. The authentication decision table is a pre-established data mapping table, and its entries include at least a decision area identifier, a matching field, a matching status, and a decision instruction field. The decision area identifier is used to distinguish between the first decision area, the second decision area, and the third decision area; the matching field is used to point to the candidate identity field, the confidence trajectory field, and the binding status field in the identity conflict record; the matching status is used to record the identity convergence status, the first conflict label, and the unbound identifier; and the decision instruction field is used to write the pass instruction, the review instruction, and the rejection instruction.

[0069] The first decision area is associated with the identity convergence state of the candidate identity field. The identity convergence state means that the first identity candidate and the second identity candidate point to the same registered identity, and the same registered identity comes from the same authentication object record. When reading the identity conflict record, the first identity candidate and the second identity candidate are first obtained from the candidate identity field, and then their candidate identity identifiers are compared. When their candidate identity identifiers are consistent, and the binding status vector in the binding status field corresponds to the bound identifier, the identity conflict record is assigned to the first decision area, and the decision instruction field of the first decision area outputs the pass instruction.

[0070] The second decision area is associated with the first conflict label. The first conflict label is jointly written by the identity separation status of the candidate identity field and the trajectory stability status of the confidence trajectory field. When reading an identity conflict record, if there is already a record in the candidate identity field where the first identity candidate and the second identity candidate are separate, and the first confidence trajectory and the second confidence trajectory in the confidence trajectory field are both continuous source records, then the identity conflict record is assigned to the second decision area. The second decision area does not rewrite the first identity candidate and the second identity candidate, but only writes the two and their confidence trajectory index into the decision instruction field, and the decision instruction field outputs the review instruction.

[0071] The third decision area is associated with the unbound identifier, which originates from the cross-modal binding identifier. This indicates that the face image sequence and gait contour sequence have not been written into the same merged authentication object. When reading the identity conflict record, if the binding status vector in the binding status field points to the unbound identifier, the identity conflict record is assigned to the third decision area. The third decision area reads the first authentication object field and the second authentication object field in the authentication object record, writes the source sequence identifiers of the two into the decision instruction field, and outputs the rejection instruction from the decision instruction field.

[0072] When calling the authentication decision table, matching is performed in the order of the third decision area, the second decision area, and the first decision area. That is, first it is determined whether the identity conflict record contains an unbound identifier, then it is determined whether the identity conflict record contains a first conflict label, and finally it is determined whether the candidate identity fields form an identity consistency state. The above order ensures that the authentication decision table completes the table item matching in the same authentication process in the order of binding source first, conflict record second, and identity consistency last. For example, if the first identity candidate and the second identity candidate in the candidate identity field both point to registered identity A, and the binding status field corresponds to the bound identifier, then the first decision area is matched; if they point to registered identity A and registered identity B respectively, and the identity conflict record has been written with the first conflict label, then the second decision area is matched; if the binding status field corresponds to an unbound identifier, then the third decision area is matched directly.

[0073] In this embodiment, the binding determination model is trained by the registration sample group. The registration sample group is not only used to train the face recognition model or gait recognition model, but also to train the ability to determine the binding relationship between face image sequences and gait contour sequences. The registration sample group includes a positive binding sample group and an anti-binding sample group. The positive binding sample group is used to indicate that the face image sequence and gait contour sequence of the same registrant can correspond in time. The anti-binding sample group is used to indicate the non-correspondence relationship after the face image sequences and gait contour sequences of different registrants are interleaved and combined.

[0074] Specifically, during the registration phase, a registered face image sequence, a registered gait contour sequence, and a registered timestamp sequence are collected from the same registrant. The registered face image sequence is formed by arranging the facial images visible to the registrant during walking according to the registration timestamp. The registered gait contour sequence is formed by arranging the human body contours extracted from the registrant during the same walking process according to the registration timestamp. The registration timestamp sequence is used to record the order in which the two types of sequences are collected. The registered face image sequence, registered gait contour sequence, and registered timestamp sequence of the same registrant are combined to obtain a positive binding sample group. In this positive binding sample group, the face-side data and the gait-side data have the same registrant source and the same time chain source.

[0075] Furthermore, the anti-binding sample group is formed through staggered construction. The registered face image sequence of the first registrant and the registered gait contour sequence of the second registrant are selected and combined according to the staggered time stamp sequence. The first registrant and the second registrant are different registration objects. The staggered time stamp sequence is used to make the face side data of the first registrant and the gait side data of the second registrant fall into the same sample record, but retain the original source identifiers of the two. The staggered time stamp sequence can be rearranged according to the order of collection, or it can be spliced ​​according to scene segments such as before and after occlusion, before and after trajectory discontinuity. Its function is to construct samples where the face and gait are close in surface time but have different sources.

[0076] During the initial binding model training, a first binding evidence chain and a second binding evidence chain are first generated based on the positive binding sample group. Then, a reverse first binding evidence chain and a reverse second binding evidence chain are generated based on the negative binding sample group. Both the first binding evidence chain and the reverse first binding evidence chain are generated according to the location of the head region, the movement of facial key points, and the order of head appearance. Both the second binding evidence chain and the reverse second binding evidence chain are generated according to the trunk center sequence, the foot swing region, and the phase continuity relationship. Thus, the training input received by the initial binding model is consistent with the input structure of the authentication stage.

[0077] During training, the evidence chain corresponding to the positively bound sample group is written into the bound sample record, and the evidence chain corresponding to the negatively bound sample group is written into the unbound sample record. The initial binding model learns the combined differences between spatial attribution, phase continuity, and time stamp in the two types of sample records. After training, the binding decision model is obtained. This binding decision model receives the first binding evidence chain and the second binding evidence chain during the authentication phase and outputs the binding state vector and cross-modal binding identifier.

[0078] For example: the registered face image sequence of the first registered identity and the registered gait contour sequence of the first registered identity constitute a positive binding sample; the registered face image sequence of the first registered identity and the registered gait contour sequence of the second registered identity are spliced ​​together according to adjacent collection time periods to constitute an anti-binding sample. This example is only used to illustrate the construction method of the positive binding sample group and the anti-binding sample group, and does not limit the specific number of collections and the number of samples. Example 2:

[0079] like Figure 3As shown, this embodiment further improves the design based on embodiment one. The difference is that in the actual operation of embodiment one, it was found that when the human trajectory unit is in a disconnection and reconnection state and the identity conflict record has been written into the first conflict label, although the first identity candidate and the second identity candidate come from the merged authentication object, the identity separation state may be concentrated in the local time markers near the third trajectory segment. The first binding evidence chain and the second binding evidence chain have formed a binding state as a whole, but there are still misaligned writings in some evidence items. As a result, when the authentication decision table enters the second decision area based only on the first conflict label, it is difficult to determine which category the source of the conflict belongs to: head region attribution offset, gait phase continuity offset, or third trajectory segment reconnection offset. It fails to distinguish the identity separation state formed by local binding misalignment from the stable separation state of real face features and gait features. Based on this, the multimodal identity authentication method integrating facial and gait features further includes: after generating the first conflict label, performing conflict anchor point backtracking processing based on the disjoint identity index, confidence trajectory index, and binding status index in the first conflict label to determine the conflict time marker; taking the conflict time marker as the center, extracting local head attribution records from the first binding evidence chain, extracting local phase continuity records from the second binding evidence chain, and generating a local misalignment verification segment in combination with the trajectory status identifier; generating a conflict source label based on the local misalignment verification segment, and writing the conflict source label into the identity conflict record.

[0080] In practice, the conflict anchor point backtracking process does not involve re-performing face recognition and gait recognition. Instead, it reads the first conflict label from the already generated identity conflict records. The first conflict label includes a disjointed identity index, a confidence trajectory index, and a binding state index. The disjointed identity index points to the first identity candidate and the second identity candidate, the confidence trajectory index points to the first confidence trajectory and the second confidence trajectory, and the binding state index points to the binding state vector. Based on the above indices, the time mark when the first identity candidate and the second identity candidate first form a disjointed identity state is found according to the time mark sequence, and this time mark is determined as the conflict time mark.

[0081] After determining the conflict time marker, the first, second, and third evidence items adjacent to the conflict time marker are read from the first bound evidence chain to form a local head attribution record. The local head attribution record is used to indicate whether the head region near the conflict time marker still belongs to the same human body frame, and whether the facial key point sequence still moves with the human body frame sequence. The fourth, fifth, and sixth evidence items adjacent to the conflict time marker are read from the second bound evidence chain to form a local phase continuity record. The local phase continuity record is used to indicate whether the torso center sequence, foot swing area, and phase continuity relationship near the conflict time marker still maintain temporal continuity with the same human body trajectory unit.

[0082] Subsequently, the trajectory status identifier is read. When the trajectory status identifier is in the disconnection and reconnection state, the reconnection start point, reconnection end point and corresponding time marker of the third trajectory segment are further read. The local head attribution record, local phase succession record and the reconnection information of the third trajectory segment are combined to form a local misalignment verification segment. The local misalignment verification segment is used to limit the local scope of the conflict. Its data content includes conflict time marker, head region index, gait contour index, third trajectory segment index and evidence item succession order.

[0083] Conflict source labels are generated based on the local misalignment verification segments. If the order of appearance of the head region in the local head attribution record is inconsistent with the continuation order of the human body frame sequence, a head attribution conflict label is written. If the phase order of the foot swing region in the local phase continuation record is inconsistent with the continuation order of the human body trajectory unit, a phase continuation conflict label is written. If both of the above types of records point to the supplementary start point and supplementary end point of the third trajectory segment, a supplementary source conflict label is written. After the conflict source label is written into the identity conflict record, the authentication decision table will no longer only read the first conflict label, but also the conflict source label associated with the first conflict label.

[0084] For example: In a certain authentication process, the first identity candidate points to the first registered identity, the second identity candidate points to the second registered identity, and the identity separation state only appears within a few time markers before and after the third trajectory segment is joined; at this time, the start point and end point of the third trajectory segment are read through the conflict anchor point backtracking process. If the order of appearance in the header region jumps at the join point, a header belonging conflict label is written. This example is only used to illustrate the generation method of the local misalignment verification segment and conflict source label, and does not limit the specific time length and number of frames.

[0085] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0086] The above description is merely a preferred embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present application, based on the technical solution and concept of the present application, should be covered within the scope of protection of the present application.

Claims

1. A multi-modal identity authentication method fusing facial and gait features, characterized in that, The methods include: The video frame sequence of the scene is acquired, human detection and tracking are performed on the video frame sequence, and human trajectory units are generated. The human trajectory units include human bounding box sequence, head region sequence, time stamp sequence and trajectory status identifier. Based on the human body trajectory unit, the face image sequence and gait contour sequence are extracted, and the face features and gait features are extracted to generate the first identity candidate, the first confidence trajectory, the second identity candidate and the second confidence trajectory; A first binding evidence chain is constructed based on the spatial attribution relationship between the head region sequence and the human body frame sequence, and a second binding evidence chain is constructed based on the phase continuity relationship between the gait contour sequence and the time-marked sequence. Input the first and second binding evidence chains into the binding determination model, and determine the binding of the first and second binding evidence chains based on the trajectory state identifier, generating a binding state vector and a cross-modal binding identifier; An authentication object record is established based on the cross-modal binding identifier. The authentication object record includes a first authentication object field, a second authentication object field, and a merged authentication object field. An identity conflict record is generated based on the authentication object record, the first identity candidate, the second identity candidate, the first confidence trajectory, and the second confidence trajectory; The authentication decision table is invoked based on the identity conflict record, and the decision instruction is output. 2.The multi-modal identity authentication method of fusing face and gait features according to claim 1, characterized in that, The method for generating the human trajectory unit includes: forming a first trajectory segment based on the human detection result, and forming a second trajectory segment based on the tracking result; when a disconnection marker is generated between the first trajectory segment and the second trajectory segment, generating a third trajectory segment using the directional continuation item of the head region sequence and the center continuation item of the initial human contour sequence extracted from the human detection result; and writing the trajectory status marker with the temporal relationship between the first trajectory segment, the third trajectory segment, and the second trajectory segment. 3.The multi-modal identity authentication method of fusing face and gait features according to claim 1, characterized in that, The first chain of evidence includes a first evidence item, a second evidence item, and a third evidence item; the first evidence item is formed by the positional relationship of the head region sequence in the human body frame sequence, the second evidence item is formed by the following relationship of the facial key point sequence relative to the human body frame sequence, and the third evidence item is formed by the order of appearance of the head region sequence relative to the time mark sequence; the first evidence item, the second evidence item, and the third evidence item are arranged according to the time mark sequence.

4. The multi-modal identity authentication method of fusing facial and gait features according to claim 1, characterized in that, The second binding evidence chain includes a fourth evidence item, a fifth evidence item, and a sixth evidence item; the fourth evidence item is formed by the trunk center sequence in the gait contour sequence, the fifth evidence item is formed by the foot swing area in the gait contour sequence, and the sixth evidence item is formed by the temporal continuity relationship of the phase succession relationship relative to the human trajectory unit; the fourth evidence item, the fifth evidence item, and the sixth evidence item are arranged in a time-marked sequence.

5. The multi-modal identity authentication method of claim 1, wherein, The binding determination model is a binary classification model based on the fusion of temporal coding and gating. The binding determination model includes a feature processing layer, a first coding channel, a second coding channel, a gating channel, and a determination layer. The feature processing layer organizes the first binding evidence chain into a spatial attribution input sequence according to the time-stamped sequence, and organizes the second binding evidence chain into a phase inheritance input sequence. The first coding channel performs spatial attribution coding on the spatial attribution input sequence, and the second coding channel performs phase inheritance coding on the phase inheritance input sequence. The gating channel performs gating integration on the spatial attribution coding and phase inheritance coding according to the trajectory state identifier to obtain a binding state vector. The determination layer generates a cross-modal binding identifier based on the binding state vector.

6. The multi-modal identity authentication method of fusing face and gait features according to claim 1, characterized in that, The method for establishing the authentication object record includes: when the cross-modal binding identifier is an unbound identifier, writing the face image sequence into the first authentication object field and the gait contour sequence into the second authentication object field; when the cross-modal binding identifier is a bound identifier, writing the face image sequence and the gait contour sequence into the merged authentication object field.

7. The multi-modal identity authentication method of claim 1, wherein, The identity conflict record includes a candidate identity field, a confidence trajectory field, and a binding status field; the first identity candidate and the second identity candidate are written into the candidate identity field, the first confidence trajectory and the second confidence trajectory are written into the confidence trajectory field, and the binding status vector is written into the binding status field; the first conflict label is written according to the identity separation status of the candidate identity field and the trajectory stability status of the confidence trajectory field.

8. The multi-modal identity authentication method of claim 7, wherein, The authentication decision table includes a first decision area, a second decision area, and a third decision area; the first decision area is associated with the identity convergence status of the candidate identity field, the second decision area is associated with the first conflict label, and the third decision area is associated with the unbound identifier; the corresponding decision instruction is output according to the decision area where the identity conflict record is located. 9.The multi-modal identity authentication method of fusing face and gait features according to claim 1, characterized in that, The binding determination model is trained using a registration sample group, which includes a positive binding sample group and an anti-binding sample group. The positive binding sample group consists of a registered face image sequence, a registered gait contour sequence, and a registered time stamp sequence of the same registrant. The anti-binding sample group consists of a registered face image sequence of a first registrant, a registered gait contour sequence of a second registrant, and an interleaved time stamp sequence. The initial binding model is trained using the positive binding sample group and the anti-binding sample group to obtain the binding determination model.

10. The multi-modal identity authentication method of claim 7, wherein, It also includes: based on the first conflict label, tracing back the disjointed identity index, confidence trajectory index and binding status index, determining the conflict time marker, extracting the local header attribution record and local phase succession record, combining the trajectory status identifier to generate a local misalignment verification segment, and writing the conflict source label into the identity conflict record.

Images