Key post personnel authentication system based on multi-modal biometrics

Abstract

This invention relates to the field of computer vision technology and discloses a key personnel authentication system based on multimodal biometrics. The system includes: acquiring authentication videos containing facial and walking body images; using frame timestamps as a common time reference to form facial and body trajectories; establishing candidate pairings for temporally overlapping facial and body trajectories; obtaining the face center point, hip midpoint, and ankle points at the same frame time; determining whether foot position transposition occurs in adjacent frames based on foot position order; and verifying whether the lower extension axis before and after the foot position transposition intersects the line connecting the two feet to determine if the candidate pairing meets the transposition axis condition; establishing a homogeneous binding only when the number of body trajectories and facial trajectories is one, and using the binding result for joint authentication of on-site face coding and on-site gait coding. This invention utilizes continuous visual information to complete cross-regional trajectory pairing, reducing mismatches in multi-person scenarios.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a key personnel authentication system based on multimodal biometrics. Background Technology

[0002] Authentication of key personnel often uses continuous video captured by camera equipment as input. The video may simultaneously show a person's face and walking body. Current processing workflows typically first detect face and body regions frame by frame, then generate face and body trajectories based on the positional continuity of adjacent frames. Subsequently, a live face code is generated from the face trajectory, and a live gait code is generated from the body trajectory. These are then compared with the corresponding codes in a registration database to obtain candidate identities and authentication results.

[0003] In the above process, face trajectories and body trajectories are generated independently from two types of detection and tracking results. The trajectories themselves usually only retain the temporal continuity of similar regions and may not necessarily carry directly corresponding personnel identification identifiers. Especially when multiple people are in the same frame, overlapping, occluding each other, or adjacent people entering and leaving at the same time, it is impossible to determine which face trajectory belongs to which body trajectory, or which person is adjacent to the same person, based solely on the existence of the same frame, the proximity of image positions, or the continuous changes of a single detection area.

[0004] If pairing is done directly based on single-frame distance, detection area overlap, trajectory duration, or nearest neighbor relationship, temporary proximity, intersection, and separation may occur during walking, turning, or crossing, leading to incorrect association between face trajectory and body trajectory. Improving the continuity of face tracking or body tracking alone can only improve the formation of their respective trajectories, but cannot compensate for the cross-regional correspondence between the two types of trajectories.

[0005] If cross-regional pairing is inaccurate, subsequent face and gait encoding may originate from different individuals. Even if both encodings yield candidate identities, joint authentication still lacks visual evidence of a shared individual origin. For scenarios requiring authentication of key personnel within continuous video, a trajectory pairing method oriented towards visual temporal information is needed. This method simultaneously examines the continuous changes in face and walking body images at common frame moments, using the preceding and following states of body points and the relative positional relationship between the face and body to filter candidate pairings. A bidirectional unique relationship is then used to determine a common-origin binding, providing corresponding face and body trajectories for subsequent joint authentication. Summary of the Invention

[0006] This invention provides a key personnel authentication system based on multimodal biometrics, which solves the technical problems mentioned in the background art.

[0007] To achieve the above-mentioned objectives, the present invention provides the following technical solution.

[0008] A key personnel authentication system based on multimodal biometrics includes a video acquisition module, a trajectory formation module, a transposition and clamping axis determination module, and a homology binding module. The video acquisition module acquires certified videos containing facial images and walking human images, using frame moments as a common time reference; The trajectory forming module receives the certified video, forms the face trajectory and the body trajectory, and forms candidate pairs of face trajectories and body trajectories that overlap in the frame time intervals. The foot position swapping module receives candidate pairs and obtains the face center point, hip midpoint, and double ankle point pair composed of the left and right ankle points at the same frame time. It determines the walking direction based on the changes in the spatial position of the human body trajectory in adjacent frames. It determines the foot position order according to the left and right ankle points relative to the hip midpoint along the walking direction. When the foot position order is reversed in adjacent frames, it determines that the foot position is swapped. At each frame time before and after the foot position swapping, a downward extension axis extending from the face center point through the hip midpoint to the feet and a double foot line connecting the left and right ankle points are formed respectively. When the downward extension axis at each frame time intersects the double foot line at that frame time, the candidate pairing is determined to meet the foot position swapping condition. The homo-source binding module establishes a homo-source binding and outputs a homo-source binding result when the candidate paired face trajectory satisfies the transposition axis condition only with one human trajectory and the human trajectory satisfies the transposition axis condition only with its face trajectory.

[0009] Compared with the prior art, the present invention has the following substantial features and significant progress: This invention uses frame moments in the authentication video as a common time reference, forming face trajectories and body trajectories from face images and walking body images respectively. Candidate pairings are established for temporally overlapping trajectories, and face center point, hip midpoint, and ankle point pairs are obtained at the same frame moment. Using foot position transposition as the before and after state, it is determined whether the lower extension axis before and after the foot position transposition intersects with the line connecting the two feet, forming a transposition clamping axis condition. Then, a one-to-one homogeneous binding is established based on the bidirectional unique relationship between the number of body trajectories and the number of face trajectories, thereby expanding cross-regional trajectory pairing from single-frame proximity judgment to continuous visual temporal judgment, reducing face and body trajectory mismatches in multi-person parallel, interleaved, and separate acquisition scenarios, and providing homogeneous face and body trajectories for joint authentication of on-site face coding and on-site gait coding. Attached Figure Description

[0010] Figure 1 This is a diagram illustrating the architecture and homogeneous binding mechanism of this invention; Figure 2 This is a flowchart of the authentication trajectory pairing process of the present invention; Figure 3 This is a diagram showing the data for determining the shift clamping shaft in this invention. Detailed Implementation

[0011] The following description is provided in conjunction with the accompanying drawings. It should be understood that the following embodiments are for illustrative purposes only and are not intended to limit the invention; those skilled in the art can make equivalent substitutions or combinations for the specific implementations.

[0012] The following is combined with Figures 1 to 3 This paper describes the specific implementation of a key personnel authentication system based on multimodal biometrics. In this implementation, the authentication video includes both facial and walking body images. The system uses the frame timestamps of the same authentication video as a common time reference, first generating facial and body trajectories separately, then performing candidate pairing and transposition axis determination on the two types of trajectories, and finally establishing a homogeneous binding based on bidirectional uniqueness.

[0013] This system is used for identity authentication of key personnel before they enter controlled areas. The video acquisition module 111 obtains the authentication video, in which the same person can simultaneously display both their face and walking body image. Because the face and walking body images are located in different visual regions of the same video, the face trajectory and body trajectory are formed by different detection results, and they do not automatically carry a correspondence to the same person. When multiple people enter the frame simultaneously, in staggered sequences, or adjacent individuals, mismatches can easily occur based solely on the proximity of a single detection area.

[0014] like Figure 1 As shown, the main processing chain of the system includes a video acquisition module 111, a trajectory formation module 112, a candidate pairing module 113, a transposition clamping axis determination module 114, and a homogeneous binding module 115. Figure 1 In this module, number 111 corresponds to the video acquisition module, number 112 to the trajectory formation module, number 113 to the candidate pairing module, number 114 to the transposition axis determination module, and number 115 to the same-source binding module; number 116 corresponds to the registration database, number 117 to the face encoding network, number 118 to the gait encoding network, and number 119 to the identity verification module. The candidate pairing module 113 is set after the trajectory formation module 112 forms two types of trajectories and is used to perform candidate pairing processing; the transposition axis determination module 114 receives the candidate pairs output by the candidate pairing module 113; and the same-source binding module 115 establishes one-to-one same-source bindings based on the candidate pairs that meet the transposition axis determination conditions output by the transposition axis determination module 114. The registration database 116, face encoding network 117, gait encoding network 118, and identity verification module 119 are located in a joint authentication chain after the one-to-one same-source bindings.

[0015] Figure 1The data stream shown enters the video acquisition module 111 from the authenticated video, and is then processed by the trajectory forming module 112 to form two types of trajectories. The candidate pairing module 113 then filters for trajectory pairs with overlapping frame time intervals. The transposition axis determination module 114 extracts the face center point, hip midpoint, and ankle point pairs from the common frame time of the candidate pairs, determines the foot position transposition based on adjacent frame time, and judges the intersection relationship between the lower extension axis and the line connecting the feet. The homogeneous binding module 115 performs bidirectional uniqueness verification on the candidate pairs that pass the transposition axis condition, and provides the homogeneous binding results to the face encoding network 117 and the gait encoding network 118 respectively.

[0016] Figure 2 The above processing relationship is represented by a continuous flow from obtaining the authentication video to obtaining the authentication result. S211 corresponds to video acquisition and establishing a common time reference; S212 corresponds to frame-by-frame detection and formation of two types of trajectories; S213 and S214 correspond to time overlap candidate pairing and organization of time points in the same frame; S215 corresponds to determining the walking direction; S216 corresponds to judging the reverse order of foot positions in adjacent frames; S217 corresponds to determining foot position swapping; S218 corresponds to forming the lower extension axis and the line connecting the two feet; S219 corresponds to judging the intersection of the two positions before and after the swapping; S220 corresponds to forming the result of the swapping clamping axis condition judgment; S221 and S222 correspond to bidirectional uniqueness and one-to-one homogeneous binding; S223 to S225 correspond to on-site code generation, registration code comparison, and determination of the authentication result.

[0017] This embodiment takes the fixed field of view of the camera device during the authentication video capture as an example, and all the following geometric determinations are performed in this fixed image coordinate system. The center point of the face, the midpoint of the hip, the left ankle point, and the right ankle point in each frame are all represented in the same image coordinate system, with the horizontal coordinate pointing to the right side of the image and the vertical coordinate pointing to the bottom of the image. The coordinate unit is the image coordinate unit. In this embodiment, the image coordinate determination tolerance is fixed at 1 image coordinate unit and remains unchanged throughout all frames of the same authentication video; when determining whether changes in two points can be distinguished, the condition that the changes in both coordinate components do not exceed the image coordinate determination tolerance is used as the condition that they cannot be distinguished.

[0018] The video acquisition module 111 obtains the authentication video from the camera device or the already acquired video data. The authentication video consists of frames arranged in chronological order, and each frame can simultaneously contain facial images and walking human images. Facial images serve as the visual source for forming facial trajectories, and walking human images serve as the visual source for forming human trajectories; both types of images are retained in the same authentication video in chronological order. Figure 2 S211 first reads the authentication video and establishes a frame time for each frame in the authentication video.

[0019] The frame time is the time index of the frame image in the authentication video. The frame times are arranged according to the acquisition order of the authentication video and serve as a common time reference for face trajectory and human body trajectory. The video acquisition module 111 transmits the authentication video and the frame time corresponding to each frame to the trajectory forming module 112, and the trajectory forming module 112 records the two types of detection results on the same time axis accordingly.

[0020] Within the same frame, the facial image and the walking body image correspond to the visual state of the same person at that moment. Subsequently acquired pairs of facial center points, hip midpoints, and ankle points all carry the same frame time identifier. Only points with the same frame time and belonging to the same candidate pair are organized into a set of geometric data. This avoids stitching together points from different frames to create a non-existent person pose.

[0021] The trajectory formation module 112 marks the validity of frame moments. A valid frame moment simultaneously meets two conditions: the corresponding detection result and the required human body position are obtained, and the left and right labels of the left and right ankle points can be determined. Frame moments that do not simultaneously meet the above conditions are marked as invalid frame moments, and subsequent point organization and geometric determination only read valid frame moments. The foot position order and the transposition clamping axis condition are determined on the adjacent temporal states of two valid frame moments.

[0022] Figure 2 S212, performed frame-by-frame face and body detection on the authentication video by the trajectory forming module 112. Face detection yields face detection regions, and body detection yields body detection regions. The face detection regions correspond to face images in the authentication video, and the body detection regions correspond to walking body images in the authentication video. The two types of detection are recorded sequentially along the frame timeline, and the face detection regions and body detection regions are not directly merged into the same detection object.

[0023] For face detection regions, the trajectory forming module 112 spatially and continuously associates the face detection region of the current frame with existing face trajectories according to frame time sequence; for human body detection regions, the trajectory forming module 112 spatially and continuously associates them with existing human body trajectories in the same way. Spatial continuity association is determined based on the image coordinate distance between the geometric center of the current detection region and the geometric center recorded at a valid position on the corresponding trajectory. The trajectory continuity association range is fixed before the start of the authentication video processing, representing the maximum allowable geometric center displacement of the same detected object in adjacent frames; only combinations with distances not exceeding the trajectory continuity association range are included in the association candidate set. The trajectory continuity association range remains unchanged during processing.

[0024] In the specific implementation, all candidate combinations of associations within the same frame are arranged in ascending order of geometric center distance. The combination with the smallest distance is selected sequentially, and the detected region and its existing trajectory are removed from subsequent combinations to form a one-to-one trajectory association. When combinations with the smallest distances are in tandem, the current detected region is retained as an undetermined association, and the corresponding existing trajectory is maintained until the next valid detection result. When the distance between the current detected region and all existing trajectories exceeds the range of continuous trajectory association, a new trajectory is started with the current detected region. This process generates multiple face trajectories and multiple human body trajectories respectively.

[0025] When a detection area corresponding to a trajectory is not obtained at a certain frame time, that frame time is retained as a position record gap in that trajectory. When a detection area is obtained in the next frame, the most recent valid position record of the trajectory is used to participate in spatial continuity association, and whether to connect to the original trajectory is determined according to the trajectory continuity association range and the one-to-one allocation rule. For subsequent adjacent frame times used for geometric determination, frame times with complete valid points before and after are selected from the same trajectory. The start and end range of the frame times of the trajectory is determined by the actual valid position records of the trajectory, and the trajectory forming module 112 provides this range along with the position records to the candidate pairing module 113.

[0026] The face trajectory formed by the above processing is a continuous positional record of the face image across multiple frames, while the human body trajectory is a continuous positional record of the walking human body image across multiple frames. Face and human body trajectories respectively complete the temporal continuous recording of the same type of visual region, and the cross-regional correspondence is determined by... Figure 2 S213 and subsequent candidate pairing and geometric determination are completed.

[0027] The candidate pairing module 113 receives the face trajectory and body trajectory output by the trajectory forming module 112, and reads the effective frame time range of each trajectory. For a face trajectory and a body trajectory with a common interval in the effective frame time range, the candidate pairing module 113 establishes a candidate pairing. Figure 2 S213 uses this to filter out face trajectories and human body trajectories that overlap in frame time intervals, and only trajectories with common intervals are included in the transposition clamping axis determination.

[0028] Candidate pairings represent the correspondence between the face trajectory and the human trajectory to be judged. They are established one-to-one based on common origin after the candidate pairings complete the transposition and bidirectional uniqueness determination. When multiple face trajectories and multiple human trajectories exist in an authentication video, the candidate pairing module 113 establishes candidate pairs for trajectory combinations with a common frame time interval. Each candidate pair retains its constituent face trajectory, human trajectory, and common frame time interval.

[0029] Figure 2S214 organizes point data within the common frame time of each candidate pairing. For the same frame time... The candidate pairing module 113 obtains the face center point from the face trajectory in the candidate pair, and obtains the hip midpoint and ankle point pairs from the human body trajectory in the candidate pair, and matches the four types of points with the frame time. Record them together. The face center point, hip midpoint, and ankle points must come from the same candidate pair and must correspond to the same frame time.

[0030] The face center point represents the spatial reference point of the face trajectory at a given frame time. In this embodiment, the face center point is obtained from the geometric center of the face detection region. (Frame time) The center point of the face is denoted as This point is used together with the midpoint of the hip to construct the downward axis.

[0031] The trajectory formation module 112 performs human posture point localization on the human trajectory, obtaining the left hip point, right hip point, left ankle point, and right ankle point from the human posture points. The midpoint between the left hip point and the right hip point is determined as the hip midpoint, at frame time... The midpoint of the hip is denoted as The left and right ankle points are respectively denoted as... and The two together constitute the frame time. The two ankle points .

[0032] The left and right ankle points are identified by the left and right labels output from the human posture point localization and are maintained in the frame time sequence along the same human trajectory. A frame time only forms a hip midpoint and double ankle point pair when the left ankle point, right ankle point, left hip point, and right hip point are acquired simultaneously; the foot position order comparison only begins at that frame time when the labels of the left and right ankle points can be determined.

[0033] Candidate pairing module 113 , , and The system organizes the point data into points at the same frame time and sends the point data to the transposition and alignment determination module 114. Thus, within a candidate pair, the transposition and alignment determination module 114 uses the same face trajectory, the same human body trajectory, and the corresponding points at the same frame time. Figure 1 Data streams 113 to 114 are thus formed.

[0034] The transposition and clamping axis determination module 114 selects two adjacent valid frame moments from the same human body trajectory. and Here, "adjacent" refers to frames that are adjacent in sequence according to the human body's trajectory, and both frames contain complete human body positions. The module determines the walking direction based on the spatial position change of the hip midpoint between two frames, and uses this walking direction as a comparison benchmark for foot position order between the two frames. Since this implementation uses a fixed image coordinate system, the change of the hip midpoint directly represents the walking direction of the human body in the image coordinate system.

[0035] In the same image coordinate system, frame time and frame time The midpoints of the hips are denoted as follows: and The walking direction is determined based on the displacement of the midpoint of the hip from the previous valid frame to the next valid frame.

[0036] Specifically, walking direction vector Hip midpoint based on the next valid frame time Subtract the midpoint of the hip at the previous valid frame time. It is obtained in this way.

[0037] in, and Representing the times of adjacent valid frames respectively and Midpoint of the hip Indicates the time from frame Pointing to frame time The walking direction vector. The above processing yields the displacement direction used for comparison; subsequent foot position order comparisons use this directional relationship.

[0038] In this embodiment, when and When both the horizontal and vertical coordinate changes do not exceed the image coordinate judgment tolerance, the two frame times are considered as indistinguishable valid positional changes and do not form a coordinate change. When the change in at least one coordinate component exceeds the image coordinate determination tolerance, the above procedure applies. minus Formed in this way Not formed. Meanwhile, the transposition clamping axis determination module 114 continues to select the next set of adjacent valid frame times within the common frame time interval of the same candidate pair. In this way, zero displacement and displacement within the image coordinate determination tolerance will not be sent as walking direction into the foot position order determination.

[0039] At frame time The transposition clamping shaft determination module 114 is based on Using this as a baseline, observe the left ankle point separately. and right ankle point along The order of events. Specifically, the displacement of the left ankle relative to the midpoint of the hip is based on... minus The displacement of the right ankle relative to the midpoint of the hip is obtained by means of... minus This is obtained in the way described above, and the two displacement vectors are respectively compared with... Perform dot product operation.

[0040] The pin order is determined according to the following calculation and sign determination process: Specifically, the displacement of the left ankle relative to the midpoint of the hip is compared with... Perform a dot product to obtain a signed score for the left ankle point, and then compare the displacement of the right ankle point relative to the midpoint of the hip with... Perform a dot product to obtain the signed score of the right ankle point; then subtract the signed score of the right ankle point from the signed score of the left ankle point, and apply the result using a sign function. The pin order is determined based on whether the difference is positive, negative, or zero. .

[0041] in, and They are the frame times. The left ankle point and the right ankle point, It is the midpoint of the hip at the same frame time. The walking direction is obtained from the displacement of the midpoint of the hip in adjacent frames. Representing frame time The foot position order. The dot product operation described above is used to evaluate the relative order of the two ankle points along the walking direction. Determine the sign of the difference between the two scores.

[0042] When the marked score of the left ankle point is greater than the marked score of the right ankle point, the foot position order is... A +1 indicates that the left ankle point precedes the right ankle point along the walking direction; when the score of the left ankle point is less than the score of the right ankle point, the foot position order is adjusted. A value of -1 indicates that the right ankle point precedes the left ankle point along the walking direction. In this implementation, no independent threshold is set for foot position scoring. When two scores are equal, the foot position order is not output at that frame time. When two scores are unequal, the foot position order is determined according to their magnitude relationship.

[0043] At the selected adjacent valid frame times and In this process, the order of two foot positions is compared using the same set of walking directions. A foot position swap is only performed when both foot positions are in a definite positive or negative state. Whether the foot positions are reversed in adjacent frames is determined by the following product relationship.

[0044] Specifically, the frame time Foot position order With frame time Foot position order Perform a product operation and determine if the product is less than zero; if the product is less than zero, determine that the pin order of adjacent frames is reversed and that a pin swap has occurred.

[0045] in, and Representing frame time respectively and frame time The pin order is determined. When the two pin order values ​​are +1 and -1 respectively, the product is less than zero, and the module determines that the pin order of adjacent frames is reversed and determines the pin transposition. If either pin order is not determined, the frame will not enter the pin transposition determination process. Figure 2 After determining the reversal, S216 proceeds to S217. If no reversal occurs, the next adjacent valid frame time is selected.

[0046] For the frame moment before the foot position transposition Frame time after foot position swap The transposition clamping axis determination module 114 constructs a downward extension axis using the face center point and hip midpoint at each frame time. The downward extension axis starts from the hip midpoint and extends towards the foot along the direction from the face center point to the hip midpoint; this direction causes the downward extension axis to extend from the side where the face is located, through the hip midpoint, to the side where the foot is located.

[0047] Frame time The lower extension axis is denoted as And it is determined using the following parameterized construction method: Specifically, first based on the midpoint of the hip Subtract the center point of the face The direction vector from the center point of the face to the midpoint of the hip is obtained by means of this method, and then the direction vector is combined with the lower extension axis parameter. Multiply them, and then multiply the resulting vector by the midpoint of the hip. Perform vector summation to obtain Corresponding points on; parameters Take a value that is not less than zero.

[0048] in, Representing frame time The lower extension axis, and These represent the center point of the face and the midpoint of the hip at the same moment in the same frame. This indicates the parameters of the lower extension axis. The value of is limited to not less than zero, so that the lower extension axis extends from the midpoint of the hip along the... minus The direction obtained extends, therefore This represents a ray, not a bidirectional straight line centered at the midpoint of the hip. The values ​​of the lower extension axis parameters are not preset manually, but are determined by the solution obtained in subsequent intersection judgments where the coordinates of the corresponding points on the lower extension axis and the line connecting the two feet are equal.

[0049] when and When the difference between the horizontal and vertical coordinates does not exceed the image coordinate judgment tolerance, the face center point is treated as coinciding with the hip midpoint, and no downward extension axis is formed at that frame time; when the difference of any coordinate component exceeds the image coordinate judgment tolerance, the method is used... minus The obtained direction vector determines the direction of the lower extension axis. The transposition and clamping axis determination module 114 re-acquires the face center point and hip midpoint in other adjacent valid frame moments of the same candidate pair; only the frame moment that can determine the direction of the lower extension axis is used for transposition and clamping axis condition determination.

[0050] For frame time A line connecting the two ankles connects the left and right ankle points within the same pair of ankle points. This line represents a finite line segment between the two ankle points and cannot be extended into an infinite straight line beyond those points. (Frame time) The line connecting the two feet is denoted as And it is determined using the following parameterized construction method: Specifically, starting with the right ankle point Subtract the left ankle point The direction vector from the left ankle to the right ankle is obtained by means of this method, and then the direction vector is combined with the line parameters connecting the two feet. Multiply them, and then multiply the resulting vector with the left ankle point. Perform vector summation to obtain Corresponding points on; parameters Take a value between 0 and 1, inclusive.

[0051] in, Representing frame time The line connecting the two feet, and These represent the left and right ankle points at the same moment in the same frame. This indicates the parameters for the connection between the two feet. The value of is limited to between 0 and 1, including 0 and 1, so that the corresponding point obtained is limited to the left ankle point and the right ankle point, and both endpoints belong to the line connecting the two feet. The value of the line connecting the two feet is not preset manually, but is determined by the solution result of the lower extension axis being equal to the coordinates of the corresponding point of the line connecting the two feet in the subsequent intersection judgment.

[0052] When the difference between the x-coordinate and y-coordinate of the left and right ankle points does not exceed the image coordinate judgment tolerance, the two ankle points are treated as overlapping, and no line connecting the two feet is formed at that frame. When the difference between any coordinate component exceeds the image coordinate judgment tolerance, the left and right ankle points corresponding to that frame are used to form a line connecting the two feet. The coordinates of the left and right ankle points can change at different frame times, but the line connecting the two feet at each frame time only uses the left and right ankle points corresponding to that frame time. Figure 2 The S218 forms a lower extension axis and a line connecting the two feet before and after the foot position is changed.

[0053] The transposition clamping axis determination module 114 first performs an intersection determination on the lower extension axis and the line connecting the two feet at a given frame time. Under ideal continuous coordinates, this intersection relationship is determined according to the following parameter conditions: Specifically, determine whether there is a lower axis parameter. Parameters of the connection between the two feet ,make Take a value that is not less than zero. Take a value between 0 and 1, inclusive, and based on and The corresponding point coordinates obtained are the same. When there are parameter values ​​that satisfy the above conditions, it is determined that the lower extension axis intersects with the double-leg connection line. The lower extension axis parameter and the double-leg connection line parameter are not pre-set empirical parameters, but unknowns determined by the condition that the corresponding point coordinates are equal. Let the corresponding points represented by the parameterized construction formulas of the lower extension axis and the double-leg connection line have equal abscissa components and equal ordinate components in the same image coordinate system, thus obtaining a system of equations consisting of two linear equations. The unknowns in the system of equations are the lower extension axis parameter and the double-leg connection line parameter, totaling two. Both the number of equations and the number of unknowns are two. Those skilled in the art can obtain the values ​​of the lower extension axis parameter and the double-leg connection parameter by substitution and elimination. When the obtained lower extension axis parameter is not less than zero and the double-leg connection parameter is between 0 and 1 (including 0 and 1), it indicates that the corresponding point on the lower extension axis is simultaneously located on the double-leg connection line, determining that the two types of geometric objects intersect. When the system of equations has no solution or the obtained parameter values ​​do not satisfy the above range, it is determined that the two types of geometric objects do not intersect.

[0054] in, Is a frame time The lower extension axis, It is the line connecting the two feet at the same moment in the same frame. These are the parameters of the lower extension axis. These are the parameters for the connection between the two legs. They must satisfy the above range constraints. and A point on the lower extension axis is simultaneously part of the line connecting the two feet, thus causing the two types of geometric objects to intersect.

[0055] The actual points are derived from discrete images and human pose point localization results. To ensure that intersection judgment applies to image coordinates, the transposition axis judgment module 114 uses a fixed image coordinate judgment tolerance within the same certified video; in this embodiment, the image coordinate judgment tolerance is 1 image coordinate unit. For each frame, the corresponding points are first obtained according to the above-described downward extension axis parameterization construction method and double-leg connection parameterization construction method. It is then checked whether there are parameter values ​​that make the two coordinate components of the two types of corresponding points completely equal, while requiring... Not less than zero The coordinates are between 0 and 1, including 0 and 1; if they exist, they are treated as exact intersections. If no exact intersection exists, the nearest point is obtained on the support line of the lower extension axis and the support line of the double-leg connection, respectively. Projection points falling outside the ray range are changed to ray endpoints, and projection points falling outside the line segment range are changed to the corresponding line segment endpoints, thus obtaining the nearest point pair that meets the above parameter value range. If the Euclidean distance of the nearest point pair does not exceed the image coordinate judgment tolerance, the frame time is treated as an intersection relationship; if it exceeds the image coordinate judgment tolerance, it is treated as a non-intersection. When two objects are collinear, it is directly checked whether there is a common corresponding point between the two objects defined by the value ranges of the lower extension axis parameter and the double-leg connection parameter; if there is a common corresponding point, it is treated as an intersection. The above judgment order remains unchanged in each frame time of the same certified video, and the unit of image coordinate judgment tolerance is image coordinate units. When there is no exact intersection in discrete image coordinates, the parameters of the lower extension axis and the double-leg connection are taken to minimize the sum of the squared errors of the horizontal and vertical coordinates of the two corresponding points. This value is obtained by minimizing the sum of the squared errors with respect to the lower extension axis and the double-leg connection parameters respectively. When the obtained analytical value falls outside the range of the ray or the range of the line segment, the endpoint of the ray or the endpoint of the corresponding line segment is taken instead, and then the tolerance is determined by comparing the Euclidean distance of the nearest point pair with the image coordinates.

[0056] Intersection relationships include the case where the endpoint of the line connecting the two legs intersects with the lower extension axis, with the endpoints according to the aforementioned parameters for the line connecting the two legs. The value range is used for judgment. If the lower extension axis is collinear with the line connecting the two feet, directly check the lower extension axis parameter. The value range and parameters of the two-foot connection The range of values ​​for each value determines whether two objects have a common corresponding point; if a common corresponding point exists, they are treated as intersecting. If the difference between the two coordinate components of the center point of the face and the midpoint of the hip does not exceed the image coordinate judgment tolerance, or the difference between the two coordinate components of the left ankle point and the right ankle point does not exceed the image coordinate judgment tolerance, the geometric objects are treated as either direction degradation or length degradation, and that frame moment is treated as an invalid frame moment.

[0057] The foot-positioning clamping axis determination module 114 checks the frame time before and after the foot position swap. In the previous frame, the lower extension axis is formed by the center point of the face and the midpoint of the hip in the previous frame, and its intersection with the line connecting the two feet in the previous frame is determined. In the next frame, the lower extension axis is formed by the center point of the face and the midpoint of the hip in the next frame, and its intersection with the line connecting the two feet in the next frame is determined. When the lower extension axis of each frame intersects the line connecting the two feet in that frame, the candidate pairing satisfies the foot-positioning clamping axis condition.

[0058] Figure 2 S219 judges the two intersection results before and after the pin position transposition. If both intersections are valid, the process proceeds to S220, and the transposition clamping axis determination module 114 forms a determination result that satisfies the transposition clamping axis condition for the current candidate pair; if either intersection is invalid, the current candidate pair does not form this determination result, and the process returns to S213 to process the next candidate pair. Thus, Figure 2 The foot position transposition, front and rear geometric states, and transposition clamping axis conditions are naturally connected according to the frame time order of the same candidate pair.

[0059] Specifically, in the same candidate pair, the line from the center of the face and the midpoint of the hip, extending from the upper body to the feet, passes through the line connecting the left and right ankles in both states before and after the foot position order is reversed. Adjacent frames provide dynamic state changes, while the intersection of two consecutive frames provides geometric constraints; together, they form the basis for determining candidate pairs.

[0060] The same-source binding module 115 receives the judgment result formed by the transposition clamping shaft judgment module 114 for all candidate pairs. Figure 2 After obtaining the result that the current candidate pairing meets the transposition clamping axis condition, S220 statistically verifies the candidate pairing relationship in the video; the statistical object is all candidate pairs, rather than only observing the local adjacent frames of the current candidate pairing.

[0061] For a given face trajectory, the same-source binding module 115 counts the number of human body trajectories that form candidate pairs with that face trajectory and satisfy the transposition axis condition. This number is called the number of human body trajectories, representing the number of human body trajectories that meet the condition when viewed from the direction of that face trajectory.

[0062] When the number of human body trajectories corresponding to the face trajectory in the candidate pair is one, and the number of face trajectories corresponding to the human body trajectory in the candidate pair is one, only the current candidate pair is retained in both directions. Based on this, the homologous binding module 115 establishes homologous bindings between the face trajectory and the human body trajectory, and outputs homologous binding results.

[0063] The one-to-one homogeneous binding result indicates that the bound face trajectory and body trajectory belong to two visual sources of the same person. The homogeneous binding module 115 outputs the face trajectory associated with the one-to-one homogeneous binding to the face coding network 117, and outputs the body trajectory associated with the one-to-one homogeneous binding to the gait coding network 118.

[0064] The homologous binding module 115 outputs a homologous binding result only when the number of human body trajectories and the number of face trajectories are both one. Other combinations of numbers are left unbound, and the system continues to process other candidate pairings. This process maintains the requirement of bidirectional uniqueness, avoiding the determination of homologous relationship based solely on the matching result of one side of a face trajectory.

[0065] like Figure 1 As shown, the registration database 116 is used to store multiple identity records. Each identity record includes a corresponding personnel identification identifier, a registered face code, and a registered gait code. The registered face code and registered gait code correspond to the personnel identification identifier in the same identity record; therefore, the two types of registration codes belong to different biometric data of the same person.

[0066] The face encoding network 117 receives face trajectories associated with a single source and acquires the corresponding face images according to the frame time sequence of the face trajectory. The face images are fed into the face encoding network 117 with a fixed image size and the same input processing method, and the network outputs the on-site face code. The registered face codes in the registration database 116 are generated or stored using the same face encoding network and the same input processing method to ensure that the on-site face codes and registered face codes are under the same encoding representation.

[0067] The gait coding network 118 receives human body trajectories associated with a single source and obtains corresponding walking human images according to the frame time sequence of the human body trajectory. The walking human images are sent to the gait coding network 118 according to a fixed frame time sequence and the same input processing method, and the network outputs on-site gait codes. The registered gait codes in the registration library 116 are generated or stored using the same gait coding network and the same input processing method to ensure that the on-site gait codes and registered gait codes have a comparable source relationship.

[0068] Face coding network 117 and gait coding network 118 respectively read the face trajectory and body trajectory associated with the one-to-one binding result, and retain the generated on-site codes as two types of codes for the same authentication object. Identity determination module 119 reads the two types of on-site codes and their corresponding registration codes according to the one-to-one binding result.

[0069] This embodiment can employ a face encoding network 117 and a gait encoding network 118 capable of outputting fixed encoding representations. In practice, the face encoding network 117 uses the same face encoding network and the same input processing method for both the on-site face encoding and the registered face encoding, and the gait encoding network 118 uses the same gait encoding network and the same input processing method for both the on-site gait encoding and the registered gait encoding, thereby ensuring that the two types of on-site encodings are represented under the same encoding representation and can be compared with their corresponding registered encodings.

[0070] The identity verification module 119 reads identity records from the registration database 116 and receives on-site face codes and on-site gait codes. The identity verification module 119 uses a fixed face code comparison method to compare the on-site face code with the registered face code in each identity record to obtain face code similarity; it also uses a fixed gait code comparison method to compare the on-site gait code with the registered gait code in the same identity record to obtain gait code similarity.

[0071] In one specific implementation, both the face encoding comparison method and the gait encoding comparison method use the cosine similarity of the encoding vectors. The identity determination module 119 pre-sets face encoding similarity thresholds and gait encoding similarity thresholds, which remain unchanged throughout the same authentication process. The two thresholds are determined based on the verification results of samples from the same person and samples from different persons formed during the registration phase; identity records whose face encoding similarity reaches the face encoding similarity threshold generate face candidate identities, and identity records whose gait encoding similarity reaches the gait encoding similarity threshold generate gait candidate identities.

[0072] The identity determination module 119 matches the two types of candidate identities according to the personnel identity identifier. When both the face candidate identity and the gait candidate identity point to the same personnel identity identifier, the identity determination module 119 determines the authentication success result and uses this result as the joint authentication output. When the two types of candidate identities cannot point to the same personnel identity identifier at the same time, the authentication result remains in the failed state; when multiple records simultaneously reach the corresponding threshold, the identity determination module 119 retains the personnel identity identifier with the highest similarity, and cases with the same highest similarity do not form a confirmed candidate identity.

[0073] Figure 2S223 sends the face trajectory and body trajectory associated with the homogeneous binding results into the face coding network 117 and the gait coding network 118 respectively. S224 compares the two types of on-site codes with the registration codes in the same identity record respectively. S225 determines the authentication result based on whether the face candidate identity and the gait candidate identity point to the same person identity identifier.

[0074] The following is combined with Figure 3 This describes the complete process of a candidate pairing, from obtaining the point location, determining the direction, judging the foot position, judging the geometric intersection, to binding them as homologous.

[0075] Figure 3 The data relationship before and after the transposition is represented by the change in foot position status between adjacent frames: the left ankle point comes first in the frame before the transposition, and the right ankle point comes first in the frame after the transposition.

[0076] The authentication video contains a first face trajectory, a second face trajectory, a first human body trajectory, and a second human body trajectory. The candidate pairing module 113 establishes candidate pairs for trajectories whose frame time intervals overlap. The following example uses a candidate pair formed by the first face trajectory and the first human body trajectory, selecting two adjacent valid frame times. and Please provide an explanation.

[0077] At frame time The center point of the face in the first face trajectory is The midpoint of the hip of the first human body trajectory is The left ankle point is The right ankle point is At frame time The center point of the face is The midpoint of the hip is The left ankle point is The right ankle point is These four types of points belong to the corresponding trajectories of the same candidate pair, and the points in each frame belong to the same frame time.

[0078] The hip rotation determination module 114 first determines the walking direction based on the midpoint of the hip at two frame times. Specifically, it uses the frame times... midpoint of the hip Subtract frame time midpoint of the hip The walking direction vector is obtained. Its value is Therefore, in this embodiment, the walking direction is along the positive horizontal axis of the image, starting from frame time... The midpoint of the hip points to the frame time. The midpoint of the hip.

[0079] At frame time Use the left ankle point Subtract the midpoint of the hip The displacement of the left ankle relative to the midpoint of the hip is obtained. ; compare the displacement with Perform a dot product operation to obtain the signed score for the left ankle point. Use the right ankle point. Subtract the midpoint of the hip The displacement of the right ankle relative to the midpoint of the hip is obtained. ; compare the displacement with Perform a dot product operation to obtain the signed score for the right ankle point. Since the marked score for the left ankle is greater than that for the right ankle, the difference between the two is positive; therefore, the foot position order is determined accordingly. Adding +1 indicates that the left ankle point precedes the right ankle point along the walking direction.

[0080] At frame time Use the left ankle point Subtract the midpoint of the hip The displacement of the left ankle relative to the midpoint of the hip is obtained. ; compare the displacement with Perform a dot product operation to obtain the signed score for the left ankle point. Use the right ankle point. Subtract the midpoint of the hip The displacement of the right ankle relative to the midpoint of the hip is obtained. ; compare the displacement with Perform a dot product operation to obtain the signed score for the right ankle point. Because the marked score of the left ankle point is less than that of the right ankle point, the difference between the two is negative; therefore, the foot position order is... The value is -1, indicating that the right ankle point precedes the left ankle point along the walking direction.

[0081] because Take +1 and If the product of the two is less than zero, the pin position swapping determination module 114 determines that the candidate pairing has undergone pin position swapping. Figure 3 The state line shown is a diagram of the state change before and after: the state before the position change corresponds to the left ankle point first, and the state after the position change corresponds to the right ankle point first.

[0082] At frame time From the center point of the face and midpoint of the hip The resulting downward axis extends positively along the vertical axis of the image, passing through the midpoint of the hip and then extending towards the foot. It starts from the left ankle point. and right ankle point The line connecting the two feet is located at the ordinate of 300 in the image coordinate system. The lower axis intersects the line connecting the two feet at... Parameters of the lower extension axis when intersecting Take 1.25, the parameter for connecting both feet. Approximately 0.4857, both satisfy the corresponding value range. The above parameter values ​​are obtained by solving the two parameterized constructs with equal coordinate components: In the lower extension axis parameterized construct, the ordinate of the midpoint of the hip is 200, and the ordinate component of the direction vector is 80. Let the ordinate of the corresponding point be equal to the ordinate of the line connecting the two feet, 300. That is, the product of 200 plus 80 and the lower extension axis parameter equals 300, and the lower extension axis parameter is obtained as 100 divided by 80, which is 1.25. In the line connecting the two feet parameterized construct, the abscissa of the left ankle is 134, and the abscissa component of the direction vector from the left ankle to the right ankle is −70. Let the abscissa of the corresponding point be equal to the abscissa of the lower extension axis, 100. That is, the product of 134 plus −70 and the line connecting the two feet parameter equals 100, and the line connecting the two feet parameter is obtained as 34 divided by 70, which is approximately 0.4857.

[0083] At frame time From the center point of the face and midpoint of the hip The resulting lower extension axis extends along the positive direction of the image's vertical axis, and the line connecting the two feet is formed by... Connected to The lower axis intersects the line connecting the two feet at... Parameters of the lower extension axis when intersecting Take 1.25, the parameter for connecting both feet. Approximately 0.4857. In this numerical embodiment, the image coordinate determination tolerance is taken as one image coordinate unit; the two intersection points mentioned above are intersection points with consistent coordinates, so there is no need to use approximate distance for adjustment. Similarly, at this frame time, the ordinate of the midpoint of the hip is 200, and the ordinate component of the direction vector is 80. Let the ordinate of the corresponding point be equal to the ordinate of the line connecting the two feet, 300. That is, 200 plus 80 and the product of the lower extension axis parameter equals 300, and the lower extension axis parameter is obtained as 1.25; at this frame time, the abscissa of the left ankle point is 70, and the abscissa component of the direction vector from the left ankle point to the right ankle point is 70. Let the abscissa of the corresponding point be equal to the abscissa of the lower extension axis, 104. That is, 70 plus 70 and the product of the line connecting the two feet equals 104, and the line connecting the two feet parameter is obtained as 34 divided by 70, approximately 0.4857.

[0084] The two frames before and after the foot position swap satisfy the intersection relationship between the lower extension axis and the line connecting the two feet at the same frame time. Based on this, the swapping axis determination module 114 determines that the candidate pairing formed by the first face trajectory and the first human body trajectory satisfies the swapping axis condition.

[0085] The candidate pairing module 113 continues to perform the same valid frame, foot position order, foot position transposition, and transposition clamping axis condition determination on other candidate pairs formed by the first face trajectory and the second human body trajectory, the second face trajectory and the first human body trajectory, and the second face trajectory and the second human body trajectory. It also retains a record for each candidate pair, including the foot position order at the previous frame, the foot position order at the next frame, and the results of the two intersections. In this embodiment, the two intersections of the first face trajectory and the second human body trajectory are respectively intersecting and non-intersecting; the foot position order records for the second face trajectory and the first human body trajectory are respectively... and The first face trajectory and the second human body trajectory did not form a reverse direction; the two intersections of the second face trajectory and the second human body trajectory were non-intersecting and intersecting, respectively. Therefore, only the candidate pairings corresponding to the first face trajectory and the first human body trajectory satisfy the transposition clamping axis condition.

[0086] Based on this, the same-source binding module 115 obtains: for the first face trajectory, the human body trajectory that satisfies the transposition axis condition is the first human body trajectory, and the number of human body trajectories is one; for the first human body trajectory, the face trajectory that satisfies the transposition axis condition is the first face trajectory, and the number of face trajectories is one. When both directions are one, the same-source binding module 115 establishes a one-to-one same-source binding between the first face trajectory and the first human body trajectory, and outputs the one-to-one same-source binding result.

[0087] The face encoding network 117 generates a live face code from the face image corresponding to the first face trajectory, and the gait encoding network 118 generates a live gait code from the walking body image corresponding to the first body trajectory. The identity determination module 119 retrieves the registered face code and registered gait code from the registration database 116 for the same identity record and compares them with the two types of live codes. If the candidate face identity and candidate gait identity obtained from the two comparisons both point to the same person's identity identifier, then the identity determination module 119 determines that the authentication has passed.

[0088] Although embodiments of this application have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of this application, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A key personnel authentication system based on multimodal biometrics, comprising a video acquisition module, a trajectory formation module, a transposition and axis determination module, and a homologous binding module, characterized in that: The video acquisition module acquires certified videos containing facial images and walking human images, using frame moments as a common time reference; The trajectory forming module receives the certified video, forms the face trajectory and the body trajectory, and forms candidate pairs of face trajectories and body trajectories that overlap in the frame time intervals. The foot position swapping module receives candidate pairs and obtains the face center point, hip midpoint, and double ankle point pair composed of the left and right ankle points at the same frame time. It determines the walking direction based on the changes in the spatial position of the human body trajectory in adjacent frames. It determines the foot position order according to the left and right ankle points relative to the hip midpoint along the walking direction. When the foot position order is reversed in adjacent frames, it determines that the foot position is swapped. At each frame time before and after the foot position swapping, a downward extension axis extending from the face center point through the hip midpoint to the feet and a double foot line connecting the left and right ankle points are formed respectively. When the downward extension axis at each frame time intersects the double foot line at that frame time, the candidate pairing is determined to meet the foot position swapping condition. The homo-source binding module establishes a homo-source binding and outputs a homo-source binding result when the candidate paired face trajectory satisfies the transposition axis condition only with one human trajectory and the human trajectory satisfies the transposition axis condition only with its face trajectory.

2. The multi-modal biometric based critical post personnel authentication system as claimed in claim 1, wherein, The trajectory forming module performs face detection and human body detection frame by frame on the certified video, obtains face detection regions and human body detection regions respectively, and forms face trajectory and human body trajectory respectively based on the continuous spatial correlation between the face detection regions and the human body detection regions in adjacent frames.

3. The multi-modal biometric based critical post personnel authentication system as claimed in claim 1, wherein, The trajectory forming module obtains the left hip point, right hip point, left ankle point, and right ankle point from the human body trajectory through human posture point positioning. The midpoint of the hip is the midpoint of the line connecting the left hip point and the right hip point, and the left ankle point and the right ankle point form a pair of double ankle points.

4. The multi-modal biometric based critical post personnel authentication system as claimed in claim 3, wherein, The foot position shift determination module determines the walking direction based on the displacement of the midpoint of the hip in adjacent frames, determines the order of the left and right ankle points relative to the midpoint of the hip along the walking direction as the foot position order, and determines the foot position shift when the foot position order is reversed in adjacent frames.

5. The multi-modal biometric based critical post personnel authentication system as claimed in claim 3, wherein, The lower extension axis is a ray extending from the midpoint of the hip along the direction from the center of the face to the midpoint of the hip towards the foot, and the line connecting the two feet is the line segment between the left ankle point and the right ankle point.

6. The multi-modal biometric based critical post personnel authentication system as claimed in claim 1, wherein, For each candidate pair formed by the authentication video, the same-source binding module counts the number of human body trajectories that satisfy the transposition axis condition with the face trajectory in the candidate pair, and the number of face trajectories that satisfy the transposition axis condition with the human body trajectory in the candidate pair. When the number of human body trajectories is one and the number of face trajectories is one, a one-to-one same-source binding is established between the face trajectory and the human body trajectory in the candidate pair.

7. The key personnel authentication system based on multimodal biometrics according to claim 1, characterized in that, It includes a registration database, which contains multiple identity records, each of which includes a corresponding person identity identifier, a registered face code, and a registered gait code.

8. The key personnel authentication system based on multimodal biometrics according to claim 7, characterized in that, It includes a face encoding network and a gait encoding network; the face encoding network generates on-site face codes based on face images in face trajectories that are bound together with the face images in the ...

9. The key personnel authentication system based on multimodal biometrics according to claim 8, characterized in that, The system includes an identity determination module; the identity determination module receives the on-site face code and the on-site gait code, compares the on-site face code and the on-site gait code with the registered face code and registered gait code of the same identity record in the registration database, respectively, to obtain face candidate identity and gait candidate identity, and determines the authentication success result when the face candidate identity and the gait candidate identity both point to the same person identity identifier.

Images