A multi-modal biometric dynamic weight object recognition method and system

Abstract

The application relates to a multi-modal biological feature dynamic weight object recognition method and system, wherein the method comprises the following steps: collecting multiple biological feature data of a user and time stamping; according to the time interval of the first-arriving data and the delayed-arriving data, attenuating the basic weight of the first-arriving data to obtain an initial weight; according to the feature difference degree of the update data, adjusting the initial weight to obtain an update weight; after the delayed-arriving data arrives, aligning the multiple biological feature data on the same time reference, and determining the final weight of the delayed-arriving data according to the quality of the delayed-arriving data, a temporary recognition result and the average value of the update weight; and obtaining an object recognition result according to the update weight and confidence of the first-arriving data and the final weight and confidence of the delayed-arriving data.

Description

Technical Field

[0001] This application relates to the field of biometrics, specifically to an object recognition method and system with dynamic weights of multimodal biometric features. Background Technology

[0002] The rapid development of information technology has placed higher demands on the security, accuracy, and reliability of identity authentication. Biometric identification technology, as an important means of identity authentication, has been widely used due to its advantages such as being less prone to forgetting, having good anti-counterfeiting performance, and being difficult to forge or steal. Single-modal biometric identification systems rely on a single biometric feature (such as face, fingerprint, iris, etc.) for identity authentication; however, in practical applications, they are easily affected by factors such as data quality, environmental changes, and sensor noise, resulting in problems such as limited recognition rate and insufficient robustness.

[0003] To overcome the limitations of unimodal recognition, multimodal biometric recognition technology has emerged. This technology integrates information from different biometric sources and leverages information complementarity to effectively improve the accuracy and reliability of the recognition system. Traditional multimodal fusion methods mainly include feature-level fusion, score-level fusion, and decision-level fusion. Among them, score-level fusion is widely used due to its simplicity and stable performance; its core is to assign weights to the matching scores of each modality and then perform a weighted combination.

[0004] However, existing multimodal fusion methods, especially those based on fixed weights, have significant shortcomings. First, the acquisition and processing of biometric data inherently differ in time scale. For example, fingerprint recognition can be completed in milliseconds, facial recognition requires tens of milliseconds of inference time, while voiceprint recognition requires accumulating hundreds of milliseconds of speech signals. Second, users provide biometric data in a specific order. For instance, facial data is immediately acquired when a user stands in front of a camera, followed by fingerprint acquisition by pressing a finger on a fingerprint sensor, and finally, voiceprint data is acquired by speaking into a microphone. This asynchronicity means that during dynamic recognition, some modal data arrive first, while others arrive later. If the system simply waits for all data to be ready before fusion, it will result in response delays and a degraded user experience. Furthermore, if decisions are made solely based on the first arriving data, the complementary advantages of multimodal information cannot be fully utilized, making it difficult to guarantee recognition accuracy.

[0005] Secondly, existing methods fail to adequately consider the impact of data timeliness on weight allocation. In continuous authentication or dynamic interaction scenarios, the user's state and environment are constantly changing, and data collected earlier may not accurately reflect the current state. Fixed weight allocation strategies cannot adaptively adjust the contribution of each modality, making it difficult to strike a balance between rapid response and accurate identification. Furthermore, when delayed data arrives, existing technologies lack effective asynchronous fusion and weight adjustment mechanisms to effectively align it with earlier arriving data and dynamically correct preliminary identification results.

[0006] Therefore, there is an urgent need for an object recognition method that can adapt to multimodal asynchronous data streams and dynamically adjust weight allocation to solve the problems of existing technologies in terms of real-time performance, accuracy, and robustness. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and system for object recognition with dynamic weights based on multimodal biometric features. The weight allocation is dynamically adjusted according to the acquisition time of multimodal biometric data to improve the real-time performance and accuracy of object recognition.

[0008] This application first discloses an object recognition method with dynamic weights based on multimodal biometric features, the specific steps of which are as follows:

[0009] Step 1: Collect multiple biometric data of the user and timestamp the multiple biometric data. The multiple biometric data includes multiple data that arrive earlier and data that arrive later within a time window.

[0010] Step 2: Determine the time interval between the first-arriving data and the delayed-arriving data based on the timestamps of the first-arriving data and the delayed-arriving data. Then, based on the time interval, reduce the base weight of the first-arriving data to obtain the initial weight.

[0011] The first-arriving data includes multiple data updated within the time window. The initial weight is adjusted according to the feature difference of the updated data to obtain the updated weight, and a temporary identification result is obtained based on the first-arriving data.

[0012] Step 3: After the delayed data arrives, the multiple biometric data are aligned on the same time reference according to the timestamp, and the final weight of the delayed data is determined based on the quality of the delayed data, the provisional identification result and the average of the update weight.

[0013] Step 4: Obtain the object recognition result based on the update weight and confidence of the first arriving data and the weight and confidence of the delayed arriving data.

[0014] Preferably, in step 1, the plurality of biometric data includes face data, fingerprint data, voiceprint data, and iris data. The number of first-arriving data is plurality, the number of delayed-arriving data is one, and the delayed-arriving data is the last biometric data to arrive.

[0015] Preferably, in step 1, the time window starts from the arrival time of the first arriving data and slides backward for a preset time. If the delayed arriving data does not arrive within the time window, the identification is determined to have failed.

[0016] Preferably, in step 2, the initial weights are:

[0017]

[0018] Among them, W o,i W is the initial weight for the i-th data point that arrives first. b,i Let λ be the basic weight of the i-th data point that arrives first. i Let Δ be the attenuation coefficient of the i-th data arriving first. t,i Let be the time interval between the i-th data that arrives first and the data that arrives late.

[0019] Preferably, in step 2, the multiple data updated within the time window are multiple data of the same biometric feature collected at preset time intervals, or multiple data of the same biometric feature input by the user multiple times within the time window.

[0020] Preferably, in step 2, feature vectors of the first-arriving data and the updated data are extracted, and the feature difference is the cosine distance between the feature vectors of the first-arriving data and the updated data;

[0021] The updated weight is:

[0022]

[0023] Among them, W n,i W is the update weight for the i-th data that arrives first. o,i Let η be the initial weight of the i-th data point that arrives first, η be the global base learning rate, and Δ be the initial weight. f This represents the feature difference degree.

[0024] Preferably, in step 3, the quality of the delayed arrival data is a weighted average of the signal clarity, signal integrity, and matching degree with the expected model of the delayed arrival data.

[0025] The base weight of the delayed arrival data is obtained based on the quality of the delayed arrival data and the average of the updated weights:

[0026]

[0027] Among them, W d Q is the base weight for delayed arrival data. d To assess the quality of delayed arrival data, Avg is the average of the updated weights for all first-arriving data, and α and β are hyperparameters.

[0028] Preferably, in step 3, if the provisional identification result of the delayed arrival data is consistent with the provisional identification result of the first arrival data, then the base weight of the delayed arrival data is increased to obtain the final weight of the delayed arrival data.

[0029]

[0030] Among them, W f W represents the final weight of the delayed arrival data. d P is the base weight for delayed arrival data. f P represents the confidence level of the provisional identification result based on the first arriving data. d γ represents the confidence level of the provisional identification results for delayed data, and γ is the reward coefficient.

[0031] If the provisional identification result of the delayed arrival data is inconsistent with the provisional identification result of the first arrival data, then the basic weight of the delayed arrival data will be used as the final weight of the delayed arrival data.

[0032] Preferably, in step 4, the updated weights and confidence levels of the first-arriving data and the weights and confidence levels of the delayed-arriving data are weighted and summed to obtain a fusion score, and the object recognition result is output based on the fusion score.

[0033] On the other hand, this application also provides an object recognition system with dynamic weighted multimodal biometric features, comprising:

[0034] The acquisition unit collects multiple biometric data of the user and timestamps the multiple biometric data. The multiple biometric data includes multiple data that arrive earlier and data that arrive later within a time window.

[0035] First-arriving data weight unit: The first-arriving data weight unit determines the time interval between the first-arriving data and the delayed-arriving data based on the timestamps of the first-arriving data and the delayed-arriving data, and attenuates the basic weight of the first-arriving data according to the time interval to obtain the initial weight;

[0036] The first-arriving data includes multiple data updated within the time window. The initial weight is adjusted according to the feature difference of the updated data to obtain the updated weight, and a temporary identification result is obtained based on the first-arriving data.

[0037] The delayed arrival data weighting unit aligns the multiple biometric data on the same time reference according to the timestamp after the delayed arrival data arrives, and determines the final weight of the delayed arrival data based on the quality of the delayed arrival data, the provisional identification result and the average of the updated weights.

[0038] The output unit obtains the object recognition result based on the update weight and confidence of the first-arriving data and the weight and confidence of the delayed-arriving data.

[0039] This application proposes a multimodal biometric dynamic weighting object recognition method and system. By timestamping all biometric data, a precise time reference is established for subsequent asynchronous data processing and dynamic weight allocation, solving the inherent temporal inconsistency problem of multimodal data streams. By exponentially decaying the basic weights over time intervals, the system can dynamically reflect the timeliness value of the data, avoiding the undue influence of outdated data on the recognition results. When the delayed data matches the temporary recognition result of the earlier data, its weight is increased, strengthening the effect of collaborative verification and improving the system's recognition confidence when the evidence is consistent. By comprehensively considering the quality of delayed data, the average value of the updated weights, and the consistency with the temporary recognition results, appropriate weights are assigned to delayed data, taking into account both the quality of the data itself and the overall state of the system. Attached Figure Description

[0040] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0041] Figure 1 A flowchart of an object recognition method with dynamic weighted multimodal biometric features provided as an exemplary embodiment of the present invention;

[0042] Figure 2 A schematic diagram of the structure of an object recognition system with dynamic weighted multimodal biometric features provided as an exemplary embodiment of the present invention. Detailed Implementation

[0043] The detailed description that follows, taken in conjunction with the accompanying drawings, is intended to describe various exemplary embodiments of the invention and is not intended to represent only embodiments in which the invention can be practiced. The detailed description includes specific details for the purpose of providing a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without these specific details. In some instances, well-known structures and components are shown in block diagram form to avoid obscuring the concept of the invention.

[0044] Example 1

[0045] This embodiment provides a method for object recognition using dynamic weights of multimodal biometric features, such as... Figure 1 As shown, the method provided in this embodiment includes the following steps:

[0046] Step 1: Collect multiple biometric data of the user and timestamp them. The multiple biometric data include multiple early-arriving data and late-arriving data within the time window.

[0047] Biometric data refers to the digital representation of measurable physiological or behavioral characteristics extracted from a user. Biometric data includes facial data, fingerprint data, iris data, palm print data, voiceprint data, gait data, etc. Collecting multiple biometric data points from a user involves acquiring various biometric features, such as facial images, voiceprint data, and fingerprint images, through different sensors on the identification device (e.g., camera, microphone, fingerprint sensor). This leverages the advantages of different biometric features to compensate for the limitations of a single feature, thereby enhancing the system's anti-spoofing capabilities.

[0048] Timestamping multiple biometric data involves the system recording a high-precision, uniform time stamp for each biometric data point at the moment it is acquired by the sensor. This clearly defines the order and specific location of each data point on the timeline. This is crucial for processing multimodal data streams with different processing speeds, enabling the system to know which data came first and which came later.

[0049] The order in which biometric data is collected results in some data arriving first and others arriving later. There are inherent differences in the time scale of biometric data collection and processing. For example, fingerprint recognition can be completed in milliseconds, facial recognition requires tens of milliseconds of inference time, while voiceprint recognition requires accumulating hundreds of milliseconds of voice signals. Secondly, the order in which users provide biometric data also matters. For instance, facial data is collected immediately when a user stands in front of the camera, followed by fingerprint collection by pressing their finger on the fingerprint sensor, and finally, voiceprint data is collected by speaking into the microphone.

[0050] A time window divides a continuous data stream into discrete, finite processing units. The system doesn't need to wait indefinitely; instead, it processes data in units of the window. Within the time window, the system allows the first arriving data to produce preliminary results, enabling rapid response. Simultaneously, it reserves a certain waiting time for delayed data to ensure the accuracy of the final decision.

[0051] Step 2: Determine the time interval between the first-arriving data and the delayed-arriving data based on the timestamps of the first-arriving data and the delayed-arriving data. Based on the time interval, reduce the base weight of the first-arriving data to obtain the initial weight.

[0052] The time interval is calculated by subtracting the arrival timestamp of each first-arriving data point from the arrival timestamp of the delayed-arriving data, measuring the waiting time of each first-arriving data point. The decay of the base weight is a penalty mechanism because the user state represented by the first-arriving data may have changed during the waiting period, or its importance may have naturally decreased over time, thus requiring a reduction in its importance. For example, an attacker might display a high-resolution photo of an authorized user in front of the camera at the start of verification, and then appear in front of the camera themselves. If the weight is not decayed, the forged, outdated photo will have equal importance in the decision-making process as the real, current face, potentially deceiving the system. Similarly, if user A leaves the recognition device and user B uses it, the system may still retain some biometric data from user A. If the weight of this old data is not decayed, it will interfere with the correct identification of the current user B. Therefore, linking the weight to the freshness of the data and proactively reducing the weight of first-arriving data can improve the robustness, accuracy, and security of the system.

[0053] The first arriving data includes multiple data updates within the time window. The initial weights are adjusted based on the feature differences of the updated data to obtain the updated weights, and a temporary identification result is obtained based on the first arriving data.

[0054] Instead of using only the first data from each modality, the system continuously receives and utilizes new data from that modality throughout the entire time window. This could be face frames captured continuously by the camera or multiple attempts by the user to press the fingerprint sensor.

[0055] Adjusting the initial weights based on feature differences is to reward stable data. The more stable the data, the greater the weight it is assigned. Conversely, the more drastic the data changes, the less weight it receives.

[0056] Before the delayed data arrives, the system performs a weighted fusion based on the current updated weights of all first-arriving modalities and their identification results to arrive at a provisional identification result. This provisional identification result is then used for subsequent comparison with the identification results of the delayed data.

[0057] Step 3: After the delayed data arrives, align multiple biometric data on the same time reference according to the timestamp, and determine the final weight of the delayed data based on the quality of the delayed data, the provisional identification results, and the average of the update weights.

[0058] In this process, once the delayed data arrives, all biometric data are collected, thus resolving the time misalignment issue caused by different collection and processing speeds of multimodal data. By using the data timestamp as a reference, all data are aligned on the same time base, eliminating the difference in arrival time.

[0059] The quality of delayed data indicates its reliability; provisional quality assessment results indicate whether the information in delayed data supports the judgment of earlier data; and the average value of updated weights indicates the system's dependence on earlier data. These indicators collectively determine the final weight of delayed data.

[0060] Step 4: Obtain the object recognition result based on the update weight and confidence of the first arriving data and the weight and confidence of the delayed arriving data.

[0061] In step 1, multiple biometric data include face data, fingerprint data, voiceprint data, and iris data. There are multiple data that arrive first, and one data that arrives late. The late data is the last biometric data to arrive.

[0062] For example, a user's facial data is collected at time t0, fingerprint data at time t1, iris data at time t2, and voiceprint data at time t3. Among these, facial data, fingerprint data, and iris data arrive first, while voiceprint data arrives later.

[0063] In step 1, the time window starts from the arrival time of the first arriving data and slides backward for a preset time. If the delayed arriving data does not arrive within the time window, the identification is determined to have failed.

[0064] If not all biometric data are collected within the time window, it indicates that the user has not provided complete data or that there is a system problem such as sensor malfunction. This judgment and identification is intended to prevent the system from getting stuck in an infinite waiting deadlock state because data for a certain modality can never be obtained.

[0065] In step 2, the initial weights are:

[0066]

[0067] Among them, W o,i W is the initial weight for the i-th data point that arrives first. b,iLet λ be the basic weight of the i-th data point that arrives first. i Let Δ be the attenuation coefficient of the i-th data arriving first. t,i Let be the time interval between the i-th data that arrives first and the data that arrives late.

[0068] The base weight is a pre-set weight for the i-th first arriving data point, derived from prior knowledge or offline training. It reflects the inherent reliability of the data under ideal, synchronous conditions. For example, the base weight for iris is 0.3, for fingerprint is 0.3, for face is 0.2, and for voiceprint is 0.2. The attenuation coefficient is an adjustable hyperparameter that controls the rate at which the weight of the i-th first arriving data point decays over time. For example, a smaller attenuation coefficient for relatively stable modalities (fingerprint, iris) indicates that the data remains reliable even after a period of waiting. For volatile modalities (face), whose features change rapidly with factors such as face angle and lighting, a larger attenuation coefficient indicates that the data value decreases rapidly over time.

[0069] Exponential decay function exp(-λ) i Δ t,i The decay factor gradually approaches 0 from 1, ensuring that the change in weight is continuous and smooth, avoiding system instability caused by abrupt changes. Moreover, the decay rate is fast at first and then slows down, which is consistent with the intuitive understanding that "data has the highest value when it is first generated, and then the value drops rapidly, but it will not disappear completely".

[0070] In step 2, the multiple data updated within the time window are multiple data of the same biometric feature collected at preset time intervals, or multiple data of the same biometric feature entered by the user multiple times within the time window.

[0071] Extract the feature vectors of the first-arriving data and the updated data. The feature difference is the cosine distance between the feature vectors of the first-arriving data and the updated data.

[0072] In this embodiment, a deep learning model is used to extract feature vectors from the first arriving data and the updated data. The feature vectors represent the essential features of the data. Cosine distance measures the difference in direction between two feature vectors, which is suitable for measuring feature similarity and indicates the degree of deviation in content between the old and new data. The larger the cosine distance or feature difference, the greater the content change; the smaller the cosine distance or feature difference, the more stable the content.

[0073] When the content of the new and old data is stable, it indicates that the users are cooperating well, the conditions are stable, and the data is highly reliable. In this case, the data weight should be increased to make the decision more dependent on the reliable data. On the other hand, when the content of the new and old data changes greatly, it indicates that the data is unstable and changing rapidly, and the data is unreliable. The weight should be maintained or reduced to provide a security barrier.

[0074] The updated weights are:

[0075]

[0076] Among them, W n,i W is the update weight for the i-th data that arrives first. o,i Let η be the initial weight of the i-th data point that arrives first, η be the global base learning rate, and Δ be the initial weight. f This represents the feature difference degree.

[0077] The global base learning rate is a preset small constant, such as 0.05, which controls the overall magnitude of each adjustment. Based on new evidence, the weights are adjusted in small, smooth steps to prevent overfitting.

[0078] The feature dissimilarity is the cosine distance, which ranges from [0,2]. When the feature dissimilarity is very small, i.e., Δ... f When η approaches 0, η(1-Δ f As the value of Δ increases, the adjustment range for the initial weights is significantly improved; when the feature heterogeneity is large, i.e., Δ... f When η approaches 1, η(1-Δ f When the feature difference is close to 0, the adjustment range is small, and the weight remains basically unchanged; when the feature difference is large, i.e., Δ f When η approaches 2, η(1-Δ f If the value becomes negative, it indicates that the old and new data are completely opposite, which may be a serious interference or attack, and the weight should be reduced.

[0079] In step 3, the quality of the delayed arrival data is a weighted average of the signal clarity, signal integrity, and matching degree with the expected model.

[0080] Among these, signal clarity refers to the signal-to-noise ratio of the voiceprint data, detecting whether the audio is distorted, such as the blurriness and contrast of the face image, and whether the image is clear. Signal integrity analyzes whether the effective speech duration of the voiceprint data meets the recognition requirements, which can be done by checking whether the gait data is a complete and continuous gait cycle. Matching degree with the expected model involves performing activity detection on the voiceprint data to confirm whether it is a valid human voice, which can be done by checking whether the face data is occluded.

[0081] The quality of delayed data is:

[0082] Q d = W1*Q1 + W2*Q2 + W3*Q3

[0083] In the formula, Q dTo determine the quality of delayed arrival data, Q1 represents the signal clarity of the delayed arrival data, Q2 represents the signal integrity, Q3 represents the degree of matching with the expected model, W1 represents the weight of signal clarity, W2 represents the weight of signal integrity, and W3 represents the weight of matching with the expected model.

[0084] The base weights of the delayed-arriving data are derived based on the average of the quality and update weights of the delayed-arriving data.

[0085]

[0086] Among them, W d Q is the base weight for delayed arrival data. d To assess the quality of delayed arrival data, Avg is the average of the updated weights for all first-arriving data, and α and β are hyperparameters.

[0087] The average of the update weights for all first-arriving data is calculated as the average of the update weights for all aligned first-arriving data. It represents the overall confidence level of the system before the arrival of delayed data. A low Avg indicates high system uncertainty, and a large (1-Avg) value indicates a stronger compensation effect; a high Avg indicates high system certainty, and a small (1-Avg) value indicates a weaker compensation effect.

[0088] In step 3, if the provisional identification result of the delayed data is consistent with the provisional identification result of the first-arriving data, then the base weight of the delayed data is increased:

[0089]

[0090] Among them, W f W represents the final weight of the delayed arrival data. d P is the base weight for delayed arrival data. f P represents the confidence level of the provisional identification result based on the first arriving data. d γ represents the confidence level of the provisional identification results for delayed data, and γ is the reward coefficient.

[0091] If the provisional identification result of the delayed data is inconsistent with the provisional identification result of the first data, the basic weight of the delayed data will be used as the final weight of the delayed data.

[0092] In this embodiment, step 3 determines whether the judgment result of the first-arriving data is certain based on the information brought by the delayed-arriving data. For example, the first-arriving data may be facial data, fingerprint data, or iris data. The confidence level of each object is obtained by analyzing the data of the three modalities through the model. After weighted summation with the update weights, the object with the highest confidence level is obtained as the temporary identification result of the first-arriving data. This confidence level is the confidence level P of the temporary identification result of the first-arriving data. fBy analyzing the delayed arrival data using a model, the confidence level of each object is obtained. The object with the highest confidence level is the provisional identification result of the delayed arrival data, and this confidence level is the confidence level P of the provisional identification result of the delayed arrival data. d If the provisional identification result of the delayed data is consistent with the provisional identification result of the earlier data, for example, both are identified as user A, then it means that the delayed data has consolidated and confirmed the identification result, and (P) can be used. f +P d ) / 2 indicates the consistency score; when the results are consistent, the higher the confidence level between the two parties, the higher the confidence level. f +P d The larger the ) / 2 value, the more the consistency score is used to reward the base weight of delayed data. Ultimately, delayed data receives a high weight, the overall confidence of the system is enhanced, and the output recognition result is more explicit.

[0093] If the provisional identification results of delayed data and the provisional identification results of first-arriving data are inconsistent, the base weight of the delayed data will not be processed. In this situation, the two parties are in dispute, and the allocation of weights needs to be more cautious and objective. Without any other evaluation criteria, the adjustment of weights should not be interfered with to favor either side. Instead, the status quo should be maintained, and the existing weights should be used to make a comprehensive judgment, allowing both parties to play their due roles in the final assessment.

[0094] In step 4, the updated weights and confidence levels of the first-arriving data and the weights and confidence levels of the delayed-arriving data are weighted and summed to obtain a fusion score, and the object recognition result is output based on the fusion score.

[0095] In this embodiment, all candidate results mentioned by data from each modality are listed, for example, (User A, User B, User C, ...). For each candidate result, its fusion score is calculated:

[0096]

[0097] In the formula, S k W is the fusion score of the k-th candidate result. n,i W is the update weight for the i-th data that arrives first. f P represents the final weight of the delayed arrival data. k,f P represents the confidence level of the k-th candidate result that arrives with the first data. k,d Let N be the confidence level of the k-th candidate result of the delayed arrival data, and N be the number of data that arrive first.

[0098] The candidate result with the highest fusion score is selected as the object recognition result.

[0099] Example 2

[0100] like Figure 2 As shown, this embodiment provides an object recognition system with dynamic weighted multimodal biometric features, including:

[0101] The acquisition unit 100 collects multiple biometric data of the user and timestamps the multiple biometric data. The multiple biometric data includes multiple data that arrive first and data that arrive late within a time window.

[0102] First-arriving data weight unit 200: The first-arriving data weight unit determines the time interval between the first-arriving data and the delayed-arriving data based on the timestamps of the first-arriving data and the delayed-arriving data, and attenuates the basic weight of the first-arriving data according to the time interval to obtain the initial weight;

[0103] The first-arriving data includes multiple data updated within the time window. The initial weight is adjusted according to the feature difference of the updated data to obtain the updated weight, and a temporary identification result is obtained based on the first-arriving data.

[0104] The delayed arrival data weighting unit 300 aligns the multiple biometric data on the same time reference according to the timestamp after the delayed arrival data arrives, and determines the final weight of the delayed arrival data based on the quality of the delayed arrival data, the provisional identification result and the average of the updated weights.

[0105] Output unit 400 obtains object recognition results based on the update weight and confidence of the first-arriving data and the weight and confidence of the delayed-arriving data.

[0106] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.

Claims

1. A multi-modal biometric dynamic weight object recognition method, comprising the following steps: Step 1, collecting a plurality of biometric data of a user, and time stamping the plurality of biometric data, wherein the plurality of biometric data comprises a plurality of first-arrival data and delayed-arrival data within a time window; Step 2, determining a time interval of the first-arrival data and the delayed-arrival data according to time stamps of the first-arrival data and the delayed-arrival data, and attenuating a base weight of the first-arrival data according to the time interval to obtain an initial weight; the first-arrival data comprises a plurality of updated data within the time window, and the initial weight is adjusted according to a feature difference degree of the updated data to obtain an updated weight, and a temporary recognition result is obtained according to the first-arrival data; Step 3, after the delayed-arrival data arrives, aligning the plurality of biometric data on a same time reference according to the time stamps, and determining a final weight of the delayed-arrival data according to a quality of the delayed-arrival data, a temporary recognition result, and an average value of the updated weight; Step 4, obtaining an object recognition result according to the updated weight and confidence of the first-arrival data, and the final weight and confidence of the delayed-arrival data; in the step 3, the quality of the delayed-arrival data is a weighted average of signal clarity, signal integrity, and matching degree with an expected model of the delayed-arrival data; the base weight of the delayed-arrival data is obtained according to the quality of the delayed-arrival data and the average value of the updated weight: where W d is the base weight of the late-arriving data, Q d is the quality of the late-arriving data, Avg is the average of all the pre-arriving data update weights, and a and β are hyperparameters; if the temporary recognition result of the delayed-arrival data is consistent with the temporary recognition result of the first-arrival data, the base weight of the delayed-arrival data is increased to obtain the final weight of the delayed-arrival data: wherein W f is the maximum weight of the late-arrival data, W d is the base weight of the late-arrival data, P f is the confidence of the temporary recognition result of the early-arrival data, P d is the confidence of the temporary recognition result of the late-arrival data, and γ is a reward coefficient. if the temporary recognition result of the delayed-arrival data is inconsistent with the temporary recognition result of the first-arrival data, the base weight of the delayed-arrival data is taken as the final weight of the delayed-arrival data.

2. The method of claim 1, wherein, in the step 1, the plurality of biometric data comprises face data, fingerprint data, voiceprint data, and iris data, the number of the first-arrival data is a plurality, and the number of the delayed-arrival data is one, wherein the delayed-arrival data is the last-arrival biometric data.

3. The method of claim 1, wherein, in the step 1, the time window starts at an arrival time of the first-arrival data and slides backward by a preset time length, and if the delayed-arrival data does not arrive within the time window, it is determined that the recognition fails.

4. The method of claim 1, wherein, in the step 2, the initial weight is: where W o,i is the initial weight of the ith early-arriving data, W b,i is the base weight of the ith early-arriving data, λ i is the decay coefficient of the ith early-arriving data, Δ t,i is the time interval between the ith early-arriving data and the delayed-arriving data.

5. The method of claim 1, wherein, in the step 2, the plurality of updated data within the time window is a plurality of data of the same biometric feature collected at a preset time interval, or a plurality of data of the same biometric feature input by the user within the time window.

6. The method of claim 1, wherein, in the step 2, a feature vector of the first-arrival data and the updated data is extracted, and the feature difference degree is a cosine distance of the feature vectors of the first-arrival data and the updated data; the updated weight is: where W n,i is the update weight of the i-th arriving data, W o,i is the initial weight of the i-th arriving data, η is the global base learning rate, Δ f is the feature difference degree.

7. The method of claim 1, wherein, In the step 4, the latest weight and confidence of the early-arrival data and the latest weight and confidence of the late-arrival data are used to perform weighted summation to obtain a fusion score, and the object recognition result is output according to the fusion score.

8. A multi-modal biometric dynamic weight object recognition system, characterized by, Comprise: A collection unit collects a plurality of biological feature data of a user, timestamps the plurality of biological feature data, and the plurality of biological feature data comprises a plurality of early-arrival data and late-arrival data within a time window; An early-arrival data weight unit determines a time interval of the early-arrival data and the late-arrival data according to the timestamps of the early-arrival data and the late-arrival data, and attenuates a basic weight of the early-arrival data according to the time interval to obtain an initial weight; The early-arrival data comprises a plurality of updated data within the time window, the initial weight is adjusted according to a feature difference degree of the updated data to obtain an updated weight, and a temporary recognition result is obtained according to the early-arrival data; A late-arrival data weight unit aligns the plurality of biological feature data on the same time reference according to the timestamps after the late-arrival data arrives, and determines a latest weight of the late-arrival data according to a quality of the late-arrival data, a temporary recognition result, and an average value of the updated weight; An output unit obtains an object recognition result according to the updated weight and confidence of the early-arrival data and the latest weight and confidence of the late-arrival data; The quality of the late-arrival data is a weighted average value of signal clarity, signal integrity, and matching degree with an expected model of the late-arrival data; The basic weight of the late-arrival data is obtained according to the quality of the late-arrival data and the average value of the updated weight: where W d is the base weight of the late-arriving data, Q d is the quality of the late-arriving data, Avg is the average of all the pre-arriving data update weights, and a and b are hyperparameters; If the temporary recognition result of the late-arrival data is consistent with the temporary recognition result of the early-arrival data, the basic weight of the late-arrival data is increased to obtain the latest weight of the late-arrival data: wherein W f is the maximum weight of the late-arrival data, W d is the base weight of the late-arrival data, P f is the confidence of the temporary recognition result of the early-arrival data, P d is the confidence of the temporary recognition result of the late-arrival data, and γ is a reward coefficient. If the temporary recognition result of the late-arrival data is inconsistent with the temporary recognition result of the early-arrival data, the basic weight of the late-arrival data is taken as the latest weight of the late-arrival data.

Images