An attendance management system and method based on multi-modal recognition

Abstract

This invention relates to the field of data recognition technology, and more particularly to an attendance management system and method based on multimodal recognition. It includes: collecting and synchronizing multimodal biometric data and environmental data via timestamps; preprocessing the timestamped environmental data; obtaining preprocessed multimodal biometric data; extracting features from the preprocessed multimodal biometric data to obtain biometric features of different modalities; aligning and mapping the biometric features of different modalities; generating a comprehensive feature representation based on the timestamped environmental data; matching the comprehensive feature representation with features stored in the attendance database, outputting the highest matching score, obtaining the recognition result based on the highest matching score, and generating an abnormal behavior report. This addresses the technical problems of existing attendance management systems having limited user feature data and inaccurate processing of user feature data.

Description

Technical Field

[0001] This invention relates to the field of data recognition technology, and in particular to an attendance management system and method based on multimodal recognition. Background Technology

[0002] In modern society, attendance management, as an important component of enterprise management and personnel attendance records, is widely used in various enterprises, schools, and government agencies. However, traditional attendance management methods, such as paper sign-in sheets and access control card swipes, have many shortcomings and cannot effectively meet complex and ever-changing attendance needs. With the development of biometric recognition technology, attendance management systems based on biometrics such as fingerprints, faces, irises, and voice are gradually being promoted and applied. These biometric recognition technologies have improved the security and convenience of attendance management systems to a certain extent, but some problems still need to be solved in practical applications.

[0003] In summary, at least the following technical problems exist: the user characteristic data in the attendance management system is too limited and the processing of user characteristic data is not accurate enough. Summary of the Invention

[0004] This invention provides an attendance management system and method based on multimodal recognition to solve the technical problems of existing attendance management systems having limited user feature data and inaccurate processing of user feature data.

[0005] The present invention provides an attendance management system and method based on multimodal recognition, specifically including the following technical solutions:

[0006] An attendance management method based on multimodal recognition includes the following steps:

[0007] S1. Collect and time-stamp synchronized multimodal biometric data and environmental data to obtain time-stamp synchronized multimodal biometric data and environmental data, and preprocess the time-stamp synchronized environmental data to obtain preprocessed environmental data; based on the preprocessed environmental data, preprocess the time-stamp synchronized multimodal biometric data to obtain preprocessed multimodal biometric data.

[0008] S2. Feature extraction is performed on the preprocessed multimodal biometric data to obtain biometric features of different modalities. The biometric features of different modalities are aligned and mapped to obtain an aligned multimodal feature vector set. Then, based on the environmental data synchronized with timestamps, the aligned multimodal feature vector set is weighted through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation.

[0009] S3. Perform matching calculations between the comprehensive feature representation and the stored feature vectors in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate abnormal behavior reports.

[0010] Preferably, S1 specifically includes:

[0011] The image and speech modalities in the multimodal biometric data after time-stamp synchronization are denoised. Then, the pixel values ​​and amplitudes of the denoised image and speech modalities of the multimodal biometric data are normalized respectively to obtain the normalized image and speech modalities of the multimodal biometric data.

[0012] Preferably, S1 specifically includes:

[0013] Based on the preprocessed environmental data, image cropping and contrast adjustment processing, as well as adaptive adjustment processing of speech signals, are performed on the image and speech modalities of the normalized multimodal biometric data, and an adaptive contrast enhancement algorithm is introduced.

[0014] Preferably, S1 specifically includes:

[0015] In the implementation of the adaptive contrast enhancement algorithm, the fingerprint image of the normalized multimodal biometric data is subjected to multi-level logarithmic-exponential transformation to obtain the fingerprint image after logarithmic-exponential transformation; and local adaptive detail enhancement processing is performed on the fingerprint image after logarithmic-exponential transformation.

[0016] Preferably, S1 specifically includes:

[0017] In the implementation of the adaptive contrast enhancement algorithm, a local detail enhancement factor and an adaptive gain control factor are introduced to perform local adaptive detail enhancement on the fingerprint image after multi-level logarithmic-exponential transformation, so as to obtain the fingerprint image after local adaptive detail enhancement. Then, multi-scale local mapping correction is performed on the fingerprint image after local adaptive detail enhancement to obtain the final corrected enhanced fingerprint image.

[0018] Preferably, S2 specifically includes:

[0019] A joint texture-structure tensor feature extraction and nonlinear mapping algorithm is used to extract features from fingerprint image data of preprocessed multimodal biometric data.

[0020] Preferably, S2 specifically includes:

[0021] In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, fingerprint texture features are extracted to obtain local texture information of the fingerprint; the preprocessed fingerprint image data is normalized to obtain normalized fingerprint image data, and the horizontal and vertical gradients of the normalized fingerprint image data are calculated to obtain the local gradient magnitude. A structure tensor is constructed using the local gradient information; and based on the structure tensor, a joint feature tensor is introduced, and feature matrix mapping is performed on the joint feature tensor.

[0022] Preferably, S2 specifically includes:

[0023] In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, adaptive nonlinear mapping is performed. The mapped feature matrix is ​​adaptively transformed by the nonlinear mapping function. After the adaptive nonlinear mapping is completed, the values ​​of the mapped feature matrix are arranged as column vectors to form the final fingerprint feature vector.

[0024] An attendance management system based on multimodal recognition includes the following components:

[0025] The system includes a data acquisition module, a data preprocessing module, a modal feature extraction and alignment module, a multimodal feature fusion and decision-making module, a data storage and management module, and a behavior analysis and anomaly detection module.

[0026] The data acquisition module acquires the user's multimodal biometric data and environmental data, and timestamps the acquired multimodal biometric data and environmental data. The timestamp-synchronized multimodal biometric data and environmental data are then sent to the data preprocessing module, while the timestamp-synchronized environmental data is sent to the multimodal feature fusion and decision module.

[0027] The data preprocessing module preprocesses the timestamped multimodal biometric data based on the timestamped environmental data to obtain preprocessed multimodal biometric data, and then sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module.

[0028] The modality feature extraction and alignment module extracts the feature vector of each modality from the preprocessed multimodal biometric data, then aligns and maps the features of different modalities to obtain an aligned multimodal feature vector set, and sends the aligned multimodal feature vector set to the multimodal feature fusion and decision module.

[0029] The multimodal feature fusion and decision module, based on the time-stamped synchronized environmental data, dynamically adjusts the fusion weights of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism, and performs weighted processing on the multimodal feature vector set to generate a comprehensive feature representation. Based on the comprehensive feature representation, it performs matching calculations with the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module.

[0030] The data storage and management module uses a distributed storage system to store attendance data, and also uses blockchain technology to store attendance data records and data access logs. The stored attendance data records and data access logs are then sent to the behavior analysis and anomaly detection module.

[0031] The behavior analysis and anomaly detection module analyzes attendance data in real time based on the attendance data records and data access logs stored in the data storage and management module, detects abnormal behavior in the attendance data, provides early warnings and prompts for detected abnormal behavior, and generates anomaly behavior reports.

[0032] The beneficial effects of the technical solution of the present invention are:

[0033] 1. The introduced dynamic weight adaptive adjustment mechanism can dynamically adjust the weight allocation of biometrics of different modalities according to changes in environmental data (such as light intensity, noise level, etc.), ensuring that the importance of each modality can be automatically adjusted under different environmental conditions (such as excessively bright or dark light, high noise, etc.).

[0034] 2. Multi-level logarithmic-exponential transformation and multi-scale local mapping techniques are introduced to enhance the details of different levels of the fingerprint image, especially the details of ridges and minutiae. After multi-level logarithmic-exponential transformation, local adaptive detail enhancement processing is performed on the transformed fingerprint image, so that the fingerprint image can achieve effective contrast improvement under different brightness conditions.

[0035] 3. By employing a joint texture-structure tensor feature extraction and nonlinear mapping algorithm to extract features from fingerprint images, the local details and global structural information of the fingerprint image can be effectively combined. Furthermore, the discriminative power of the features is enhanced through adaptive nonlinear mapping, enabling the extracted fingerprint features to maintain high accuracy under various complex conditions. Attached Figure Description

[0036] Figure 1 This is a structural diagram of an attendance management system based on multimodal recognition as described in this invention;

[0037] Figure 2This is a flowchart of an attendance management method based on multimodal recognition as described in this invention. Detailed Implementation

[0038] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0040] The following description, in conjunction with the accompanying drawings, details the specific solution of the attendance management system and method based on multimodal recognition provided by this invention.

[0041] See attached document Figure 1 The diagram illustrates a multimodal recognition-based attendance management system according to an embodiment of the present invention. The system includes the following components:

[0042] The system includes a data acquisition module, a data preprocessing module, a modal feature extraction and alignment module, a multimodal feature fusion and decision-making module, a data storage and management module, and a behavior analysis and anomaly detection module.

[0043] The data acquisition module acquires the user's multimodal biometric data and environmental data, and timestamps the acquired multimodal biometric data and environmental data to ensure the consistency of the data's time sequence. The timestamp-synchronized multimodal biometric data and environmental data are then sent to the data preprocessing module, while the timestamp-synchronized environmental data is sent to the multimodal feature fusion and decision module.

[0044] The multimodal biometric data includes facial images, iris images, fingerprint images, and voice signals; the environmental data includes light intensity, noise level, etc.

[0045] The data preprocessing module preprocesses the timestamped multimodal biometric data based on the timestamped environmental data to obtain preprocessed multimodal biometric data, and then sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module.

[0046] The modality feature extraction and alignment module extracts the feature vector of each modality from the preprocessed multimodal biometric data, and then aligns and maps the features of different modalities to ensure that the feature vectors have the same dimension and similar distribution in the same feature space, thereby obtaining an aligned multimodal feature vector set, and sending the aligned multimodal feature vector set to the multimodal feature fusion and decision module;

[0047] The feature vectors of the modality include face feature vectors, iris feature vectors, fingerprint feature vectors, and voice feature vectors, forming a multimodal feature vector set;

[0048] The multimodal feature fusion and decision module, based on the time-stamped synchronized environmental data, dynamically adjusts the modal weights of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism. It further weights the multimodal feature vector set to generate a comprehensive feature representation. Based on the comprehensive feature representation, it performs matching calculations with the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module.

[0049] The attendance database is used to store user characteristics;

[0050] The data storage and management module uses a distributed storage system to store attendance data, and also uses blockchain technology to store attendance data records and data access logs. The stored attendance data records and data access logs are then sent to the behavior analysis and anomaly detection module.

[0051] The attendance data may include attendance timestamps, matching scores, recognition results, etc.

[0052] The behavior analysis and anomaly detection module, based on the attendance data records and data access logs stored in the data storage and management module, uses an aggregation analysis algorithm to analyze attendance data in real time, detect abnormal behavior in attendance data, provide early warnings and prompts for detected abnormal behavior, and generate abnormal behavior reports.

[0053] When managing attendance, one type of biometric feature can be selected for identification, such as fingerprint, face, iris, or voice, depending on the specific application scenario and requirements.

[0054] See attached document Figure 2 The diagram illustrates a flowchart of an attendance management method based on multimodal recognition, provided by an embodiment of the present invention. The method includes the following steps:

[0055] S1. Collect and time-stamp synchronized multimodal biometric data and environmental data to obtain time-stamp synchronized multimodal biometric data and environmental data, and preprocess the time-stamp synchronized environmental data to obtain preprocessed environmental data; based on the preprocessed environmental data, preprocess the time-stamp synchronized multimodal biometric data to obtain preprocessed multimodal biometric data.

[0056] Multimodal sensor devices are used to collect data, obtaining the user's multimodal biometric data and environmental data. These multimodal sensor devices include: face image acquisition devices, iris image acquisition devices, fingerprint image acquisition devices, voice signal acquisition devices, environmental data sensors, etc. Each multimodal sensor device records a precise timestamp while collecting data. All timestamps of the multimodal sensor devices use a unified clock source, such as synchronization through a Network Time Protocol (NTP) server, to obtain multimodal biometric data and environmental data after timestamp synchronization.

[0057] The multimodal biometric data includes facial images, iris images, fingerprint images, and voice signals; the environmental data includes light intensity and noise level.

[0058] The environmental data after timestamp synchronization is preprocessed using a method familiar to those skilled in the art to obtain preprocessed environmental data.

[0059] For image modalities (such as faces, irises, and fingerprints) in the multimodal biometric data after time-stamp synchronization, filtering methods are used for denoising. For mild noise interference (such as random noise), 3×3 or 5×5 median filters are used for image denoising; for images with edge details (such as fingerprints and irises), bilateral filtering is used for smoothing to preserve the edge features of the image. For speech modalities in the multimodal biometric data after time-stamp synchronization, Short Time Fourier Transform (STFT) is used to convert the speech signal to the frequency domain, spectral subtraction is used to remove noise, and then the denoised speech signal is converted back to the time domain.

[0060] Furthermore, pixel values ​​and amplitudes of the image and speech modalities of the denoised multimodal biometric data are normalized respectively to obtain the normalized image and speech modalities of the multimodal biometric data.

[0061] Furthermore, based on the preprocessed environmental data, image cropping and contrast adjustment processing, as well as adaptive adjustment processing of speech signals, are performed on the image and speech modalities of the normalized multimodal biometric data.

[0062] Image cropping and contrast adjustment processing for image modalities of normalized multimodal biometric data:

[0063] Face images: Face regions are located using face detection algorithms (such as MTCNN) and cropped to a fixed size (e.g., 112×112 pixels). If the lighting is too dark or too bright, brightness equalization is performed based on the ambient light intensity in the preprocessed environmental data, and adaptive histogram equalization (CLAHE) is used to enhance contrast. Iris images: Iris regions are extracted using iris segmentation algorithms (such as the Daugman algorithm), non-iris regions (such as the white of the eye and pupil) are masked, and contrast stretching algorithms are used to enhance the details of the iris. Fingerprint images: For fingerprint images with incomplete edges, masking is used to remove invalid fingerprint regions, and Gabor filters are used to enhance the fingerprint texture to ensure the clarity of fingerprint ridge features.

[0064] To address the issue of unclear fingerprint ridge features caused by variations in multimodal sensor devices and ambient lighting, as well as uneven contrast and brightness in fingerprint images, an adaptive contrast enhancement algorithm is introduced. This algorithm enhances the details at different levels of the fingerprint image layer by layer through multi-layer nonlinear transformations and multi-scale local mapping, and controls the flexibility and balance of the enhancement process through an adaptive mechanism. The specific implementation process is as follows:

[0065] First, a multi-level logarithmic-exponential transform is performed on the fingerprint images of the normalized multimodal biometric data. The form of the multi-level logarithmic-exponential transform is as follows:

[0066]

[0067] in, It is the fingerprint image of pixel (x,y) after the log-exponential transformation of the l-th layer, representing the image pixel value after the nonlinear transformation of the l-th layer; It is the fingerprint image of pixel (x,y) after the log-exponential transformation of the (l-1)th layer, representing the image pixel value after the (l-1)th layer nonlinear transformation; It is a fingerprint image of the normalized multimodal biometric data at pixel (x,y); α l β is the logarithmic amplification factor, which controls the overall contrast enhancement of the logarithmic-exponential transform in the l-th layer; l It is the exponential amplification factor, which adjusts the enhancement level of the image brightness region in the log-exponential transform of the l-th layer; γ l These are the control coefficients for the l-th level nonlinear transformation, affecting the degree of nonlinear transformation in the l-th level log-exponential transformation and controlling the enhancement intensity of the image brightness region. Finally, based on a preset number L according to the specific scene, the fingerprint image of pixel (x,y) after the last level log-exponential transformation is obtained.

[0068] Furthermore, after multiple logarithmic-exponential transformations, local adaptive detail enhancement processing is performed on the transformed fingerprint image. The enhancement amplitude of different regions is adaptively adjusted using local brightness features, so that the fingerprint image can achieve effective contrast enhancement under different brightness conditions.

[0069] To achieve adaptive detail enhancement, a local detail enhancement factor E(x,y) is introduced. This factor measures the ratio of the local brightness features of the current pixel to the global brightness mean, and is used to dynamically adjust the intensity of detail enhancement. The definition of the local detail enhancement factor is:

[0070]

[0071] Where E(x,y) is the local detail enhancement factor at pixel (x,y); B(x,y) is the local brightness feature at pixel (x,y). Ω is a local neighborhood window used to calculate the average brightness around a pixel. It is the pixel value at position (i,j) of the fingerprint image of the Lth layer after multi-layer logarithmic-exponential transformation; μ B It is the global brightness average, representing the average brightness of the entire fingerprint image. M and N are the number of rows and columns of the fingerprint image, respectively; ∈ is a very small number to avoid numerator and denominator being 0.

[0072] Furthermore, after calculating the local detail enhancement factor, an adaptive gain control factor G(x,y) is introduced to finely control the enhancement amplitude based on the ambient light intensity and ambient noise level. The adaptive gain control factor is defined as:

[0073]

[0074] in, V is the ambient light intensity, representing the lighting conditions of the fingerprint acquisition environment; V is the ambient noise level, representing the noise interference in the fingerprint acquisition environment; G(x,y) is the adaptive gain control factor, used to dynamically adjust the magnitude of detail enhancement under different lighting and noise conditions. As the lighting and noise levels increase, the adaptive gain factor tends to saturate to avoid over-enhancement.

[0075] Combining local detail enhancement factors and adaptive gain control factors, local adaptive detail enhancement processing is performed on the fingerprint image after multi-level logarithmic-exponential transformation. The formula is as follows:

[0076]

[0077] Among them, I enh(x,y) is the fingerprint image after local adaptive detail enhancement of pixel (x,y). Through joint adjustment of local detail enhancement factor and adaptive gain control factor, adaptive contrast enhancement is achieved. It adaptively adjusts the brightness of the fingerprint image through a local detail enhancement factor; It adjusts the detail enhancement amplitude through an adaptive gain control factor to ensure that image details can be fully magnified under different lighting conditions.

[0078] To avoid uneven transitions in the enhanced region of the fingerprint image after local adaptive detail enhancement, multi-scale local mapping correction is performed on the fingerprint image after local adaptive detail enhancement. Multi-scale analysis is used to eliminate artifacts caused by excessive enhancement of local adaptive details, making the detail features of the fingerprint image more natural at different scales.

[0079] Fingerprint image I with locally adaptive detail enhancement using wavelet decomposition enh (x,y) is decomposed into S detail components W at different scales. s (x,y) and an approximation component A S (x,y), the formula is

[0080]

[0081] Among them, W s (x,y) is the detail component of pixel (x,y) at scale s, representing the detail information at different scales;

[0082] For each detail component W s Local mapping correction is performed on (x,y) to suppress excessive enhancement of local adaptive details. The multi-scale local mapping correction formula is as follows:

[0083]

[0084] in, It is the detail component of the pixel (x,y) after local mapping correction at scale s, which eliminates artifacts caused by excessive local adaptive detail enhancement and preserves the original detail features of the fingerprint image; κ s It is the detail component correction coefficient, which controls the correction strength; λ s These are nonlinear mapping control coefficients that affect the degree of nonlinearity in the local mapping correction process.

[0085] Finally, the detail components after local mapping correction are... With approximation component A S Reconstructing (x,y) yields the final corrected enhanced fingerprint image:

[0086]

[0087] Among them, I final (x,y) is the enhanced fingerprint image after the final correction of pixel (x,y), that is, the pre-processed fingerprint image data. After multi-scale local mapping correction and reconstruction, it not only retains the improvement of global contrast, but also achieves adaptive detail enhancement and artifact suppression in local details.

[0088] For scenes with illumination intensity below 200 lux, an image brightness enhancement algorithm (such as gamma correction) is used to enhance brightness; when the illumination intensity is above 5000 lux, a local contrast enhancement algorithm is used to reduce the exposure of overly bright areas; when the noise level is above 80 dB, a strong noise suppression algorithm based on spectral subtraction is used for the speech signal to reduce the interference of background noise on speech features.

[0089] The above processing yields preprocessed multimodal biometric data.

[0090] S2. Feature extraction is performed on the preprocessed multimodal biometric data to obtain biometric features of different modalities. The biometric features of different modalities are aligned and mapped to obtain an aligned multimodal feature vector set. Then, based on the environmental data synchronized with timestamps, the aligned multimodal feature vector set is weighted through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation.

[0091] Feature extraction is performed on the preprocessed multimodal biometric data using existing feature engineering techniques to obtain biometric features of different modalities. For example, existing deep convolutional neural networks are used to extract features from face and iris images, and short-time Fourier transform and long short-time memory networks are used to extract features from speech signals. For fingerprint images, a joint texture-structure tensor feature extraction and nonlinear mapping algorithm is used for feature extraction. By introducing a joint texture-structure tensor to represent the local details and global structural information of the fingerprint image, and using an adaptive nonlinear mapping method, highly discriminative fingerprint feature vectors are generated. The specific implementation process of the joint texture-structure tensor feature extraction and nonlinear mapping algorithm is as follows:

[0092] The first step is fingerprint texture feature extraction; fingerprint texture features are extracted from the preprocessed multimodal biometric data, including local texture information such as ridges and valleys. This involves extracting local texture information from the fingerprint image data, specifically the ridge and valley features. final Normalization of (x,y) yields the normalized fingerprint image data I. norm (x,y) provides a good foundation for subsequent fingerprint texture feature extraction. The edge information of the horizontal and vertical gradients in the orientation domain of the normalized fingerprint image data is calculated: horizontal gradient... The grayscale change rate of the fingerprint image along the x-axis is used to capture horizontal edge information in the fingerprint image; the vertical gradient... The grayscale change rate of the fingerprint image along the y-axis is used to capture vertical edge information in the fingerprint image; simultaneously, the local gradient magnitude is calculated using the horizontal and vertical gradients. The intensity of the local gradient reflects the saliency of edge information in the fingerprint image, and θ(i,j) represents the direction of the local gradient, used to describe the directional characteristics of the ridges and valleys of the fingerprint.

[0093]

[0094] Furthermore, to quantify the texture features of fingerprint images, a directional gradient histogram is introduced, dividing the local gradient direction θ(i,j) into... Divide the region into 8 directional intervals (e.g., 45 degrees each), and then calculate the local neighborhood window within each directional interval. The sum of the local gradient magnitudes of all pixels within the area. The calculation formula is:

[0095]

[0096] in, It is a local neighborhood window (e.g., a 3×3 or 5×5 rectangular region) used to collect gradient information within a local region; It is the local gradient magnitude, representing the local neighborhood window. The gradient intensity at the local position (i,j); θ(i,j) is the local gradient direction, representing the local neighborhood window. The gradient direction at the interior position (i,j); δ k (·) is the indicator function; when the local gradient direction θ(i,j) falls within the k-th direction interval, δ k (θ(i,j))=1, otherwise δ k (θ(i,j))=0;H k (x,y) is the value of the gradient histogram in the k-th direction interval, representing the gradient intensity in the k-th direction interval within the local region. Finally, the gradient distribution across all direction intervals within the local region is obtained, resulting in the comprehensive feature vector H(x,y). k (x,y)∈H(x,y).

[0097] The second step is to construct the structure tensor and fuse features; the structure tensor is constructed using local gradient information and is defined as follows:

[0098]

[0099] Where J(x,y) is the structure tensor, which reflects the distribution characteristics of the local gradient direction of the fingerprint image at pixel (x,y); It is the average of the squared horizontal gradients within the local neighborhood, used to describe the consistency of the horizontal direction in the local region; It is the average of the product of the horizontal and vertical gradients in the local neighborhood, used to describe the cross characteristics of the local region; It is the average of the squared vertical gradients within the local neighborhood, used to describe the consistency of the vertical direction in the local region.

[0100] To achieve a joint description of fingerprint texture and structural information, a joint feature tensor F(x,y) is introduced as follows:

[0101]

[0102] Furthermore, to enhance the discriminative power of the features, a feature matrix mapping is performed on the joint feature tensor F(x,y) using a mapping function. This maps the joint feature tensor F(x,y) to a new feature space, maximizing the discriminative power of the features. The mapping function is defined as follows:

[0103]

[0104] Where |F(x,y)| is the magnitude of the joint feature tensor, representing the intensity of fingerprint texture and structural information in the local region; The smoothing parameter controls the degree of influence of the structure tensor on the feature matrix; trace(J(x,y)) is the trace of the structure tensor, which is the sum of the diagonal elements of the structure tensor and is used to measure the intensity of local structural information; M(x,y) is the feature matrix, which is the feature representation after mapping and integrates fingerprint texture and structural information.

[0105] The third step is adaptive nonlinear mapping; the mapped feature matrix is ​​adaptively transformed using a nonlinear mapping function to further improve the discriminative power of the features. First, the adaptive mapping weights of the feature matrix M(x,y) are calculated. Its definition is:

[0106]

[0107] in, These are adaptive mapping weights that control the intensity of the nonlinear mapping of the feature matrix; ||M(x,y)|| F σ is the Frobenius norm of the feature matrix, which is the square root of the sum of the squares of all elements, representing the overall strength of the feature matrix; μ is the mean of the feature strength, used to calculate the average strength of the feature matrix; σ is the standard deviation of the feature strength, used to control the sensitivity of the adaptive mapping weights.

[0108] Furthermore, an adaptive nonlinear mapping is applied to the feature matrix M(x,y). This maps the feature matrix to a higher feature space, maximizing the discriminative power of the features. The nonlinear mapping function is defined as follows:

[0109]

[0110] Among them, M mapped (x,y) is the mapped feature matrix, which is a feature representation after nonlinear transformation and has higher discriminative power; tanh(·) is the hyperbolic tangent function, which restricts the nonlinear mapping result to the range of [-1,1] to prevent the feature data after mapping from overflowing. This is an adjustment parameter for the nonlinear mapping, used to control the transformation magnitude of the characteristic matrix in the nonlinear mapping; log(1+||M(x,y)|| F ) is the logarithmic transformation term of the feature matrix strength, used to further enhance the discriminative power of the features.

[0111] After completing the adaptive nonlinear mapping, all the mapped feature matrix values ​​are arranged as column vectors to form the final fingerprint feature vector f. final By transforming local feature information into a global feature representation, subsequent fingerprint matching and recognition can be performed. The final fingerprint feature vector generation process is as follows:

[0112]

[0113] Among them, f final The final fingerprint feature vector integrates the local fingerprint texture and structural features of the fingerprint image and has high discriminative power; M is the number of rows in the fingerprint image, representing the height of the fingerprint image; N is the number of columns in the fingerprint image, representing the width of the fingerprint image; flatten(·) flattens the mapped feature matrix values ​​into a vector form, which is used to transform a two-dimensional feature matrix into a one-dimensional feature vector.

[0114] After extracting features from different modalities through the above process, existing canonical correlation analysis algorithms are used for alignment and mapping to obtain an aligned set of multimodal feature vectors.

[0115] Furthermore, based on the time-stamped synchronized environmental data, a dynamic weight adaptive adjustment mechanism is used to weight the aligned multimodal feature vector set to generate a comprehensive feature representation;

[0116] Based on the timestamped environmental data, a dynamic weight adaptive adjustment mechanism is introduced. By dynamically adjusting the weights, adaptive adjustment is achieved. The weight formulas for different modalities are as follows:

[0117]

[0118] Among them, w i It is the weight of the i-th mode; Indicates the number of modal types; It is the first Each environmental factor (such as light intensity, noise level, etc.) represents the environmental data after being timestamped and synchronized. Indicates the relationship between the i-th mode and the i-th mode. The interaction angle parameter between environmental factors is used to adjust the influence of environmental factors on each mode, and the range is [0, 2π]. Indicates the relationship between the i-th mode and the i-th mode. The smoothing coefficient between environmental factors is used to control the smoothing effect of environmental factors on each mode; m represents the total number of environmental factors.

[0119] Finally, the aligned multimodal feature vector set is weighted according to the weights of different modalities to generate a comprehensive feature representation.

[0120] S3. Perform matching calculations between the comprehensive feature representation and the stored feature vectors in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate abnormal behavior reports.

[0121] First, the comprehensive feature representation is matched with the stored feature vectors (i.e., all user features) in the attendance database. The identity of the current user is determined by the similarity measure between the comprehensive feature representation and the stored features in the attendance database. The similarity measure is calculated using cosine similarity (or other distance measurement methods). T represents the similarity measure with the q-th stored feature in the attendance database. q It is the qth stored feature in the attendance database.

[0122] Then collect all similarity metric scores. The data is sorted in descending order, where Q represents the number of features stored in the attendance database. The top K highest-scoring matching results are selected as candidate matching results, which will be used to determine the candidate list of recognition results.

[0123] Furthermore, the recognition result is generated based on the candidate matching results. First, the recognition result is generated based on the candidate matching results and a preset threshold. Determine the user identity corresponding to the current comprehensive feature representation. If the highest matching score is SIM... max Exceeding the preset threshold If a match is found, the corresponding user ID is output as the identification result; otherwise, "Not identified" or "Identity unknown" is output. Threshold Based on a preset similarity threshold using expert experience, the system further derives the recognition results and matching scores (similarity scores) for subsequent storage and management.

[0124] Furthermore, the identification results and matching scores are stored and managed. This ensures that the information for each identification event is completely recorded, facilitating subsequent data analysis and attendance management. Each identification event includes the following information: timestamp, employee ID, matching score, feature vector, and environmental data. These identification events are packaged into a single record unit and stored using a distributed database or blockchain technology.

[0125] In data storage, to prevent data loss and enable rapid recovery in the event of failure or attack, the data in the attendance database is regularly backed up to the cloud or local disk and stored in an encrypted manner to ensure data security and integrity.

[0126] Regularly analyze and detect abnormal behavior in the attendance database. Use existing aggregation analysis algorithms to aggregate and analyze the stored data, identify abnormal situations in attendance behavior patterns (such as proxy clocking in, long periods of no clocking in, etc.), and generate abnormal behavior reports.

[0127] In summary, an attendance management system and method based on multimodal recognition have been developed.

[0128] The order of the embodiments is for illustrative purposes only and does not represent the superiority or inferiority of the embodiments. The processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0129] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A multi-modal recognition based attendance management method, characterized in that, Includes the following steps: S1. Collect and time-stamp-synchronize multimodal biometric data and environmental data, and preprocess the time-stamp-synchronized environmental data to obtain preprocessed environmental data; denoise the image and speech modalities in the time-stamp-synchronized multimodal biometric data, and normalize the pixel values ​​and amplitude of the denoised image and speech modalities respectively to obtain normalized image and speech modalities of the multimodal biometric data; based on the preprocessed environmental data, perform image cropping and contrast adjustment processing and adaptive speech signal adjustment processing on the normalized image and speech modalities of the multimodal biometric data, and introduce an adaptive contrast enhancement algorithm; in the implementation of the adaptive contrast enhancement algorithm, perform multi-level logarithmic-exponential transform on the fingerprint image of the normalized multimodal biometric data to obtain the logarithmic-exponential transformed fingerprint image; introduce a local detail enhancement factor and an adaptive gain control factor to perform local adaptive detail enhancement on the multi-level logarithmic-exponential transformed fingerprint image to obtain the locally adaptive detail enhanced fingerprint image, the formula is: in, It is a pixel. Fingerprint image with local adaptive detail enhancement; It is a pixel. Local detail enhancement factor; ,in, It is a pixel. Local brightness characteristics; It is the global brightness average, representing the average brightness of the entire fingerprint image; It is used to avoid numbers where the numerator and denominator are both zero; It is a pixel. On the last floor Fingerprint image after layer logarithmic-exponential transformation; It is an adaptive gain control factor; ,in, It refers to ambient light intensity; It refers to the level of ambient noise. The fingerprint image after local adaptive detail enhancement is then subjected to multi-scale local mapping correction to obtain the final corrected enhanced fingerprint image. The above processing yields preprocessed multimodal biometric data; S2. Feature extraction is performed on the preprocessed multimodal biometric data to obtain biometric features of different modalities. The biometric features of different modalities are aligned and mapped to obtain an aligned multimodal feature vector set. Then, based on the environmental data synchronized with timestamps, the aligned multimodal feature vector set is weighted through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation. S3. Perform matching calculations between the comprehensive feature representation and the stored feature vectors in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate abnormal behavior reports.

2. The attendance management method based on multi-modal recognition according to claim 1, characterized in that, S2 specifically includes: A joint texture-structure tensor feature extraction and nonlinear mapping algorithm is used to extract features from fingerprint image data of preprocessed multimodal biometric data. 3.The attendance management method based on multi-modal recognition of claim 2, wherein, S2 specifically includes: In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, fingerprint texture features are extracted to obtain local texture information of the fingerprint; the preprocessed fingerprint image data is normalized to obtain normalized fingerprint image data, and the horizontal and vertical gradients of the normalized fingerprint image data are calculated to obtain the local gradient magnitude. A structure tensor is constructed using the local gradient information; and based on the structure tensor, a joint feature tensor is introduced, and feature matrix mapping is performed on the joint feature tensor.

4. The attendance management method based on multimodal recognition according to claim 3, characterized in that, S2 specifically includes: In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, adaptive nonlinear mapping is performed. The mapped feature matrix is ​​adaptively transformed by the nonlinear mapping function. After the adaptive nonlinear mapping is completed, the values ​​of the mapped feature matrix are arranged as column vectors to form the final fingerprint feature vector.

5. An attendance management system based on multimodal recognition, applied to the attendance management method based on multimodal recognition as described in claim 1, characterized in that, Includes the following parts: The system includes a data acquisition module, a data preprocessing module, a modal feature extraction and alignment module, a multimodal feature fusion and decision-making module, a data storage and management module, and a behavior analysis and anomaly detection module. The data acquisition module acquires the user's multimodal biometric data and environmental data, and timestamps the acquired multimodal biometric data and environmental data. The timestamp-synchronized multimodal biometric data and environmental data are then sent to the data preprocessing module, while the timestamp-synchronized environmental data is sent to the multimodal feature fusion and decision module. The data preprocessing module preprocesses the timestamped multimodal biometric data based on the timestamped environmental data to obtain preprocessed multimodal biometric data, and then sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module. The modality feature extraction and alignment module extracts the feature vector of each modality from the preprocessed multimodal biometric data, then aligns and maps the features of different modalities to obtain an aligned multimodal feature vector set, and sends the aligned multimodal feature vector set to the multimodal feature fusion and decision module. The multimodal feature fusion and decision module, based on the environmental data synchronized with timestamps, dynamically adjusts the fusion weights of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism, and performs weighted processing on the multimodal feature vector set to generate a comprehensive feature representation. Based on the comprehensive feature representation, it performs matching calculations with the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module. The data storage and management module uses a distributed storage system to store attendance data, and also uses blockchain technology to store attendance data records and data access logs. The stored attendance data records and data access logs are then sent to the behavior analysis and anomaly detection module. The behavior analysis and anomaly detection module analyzes attendance data in real time based on the attendance data records and data access logs stored in the data storage and management module, detects abnormal behavior in the attendance data, provides early warnings and prompts for detected abnormal behavior, and generates anomaly behavior reports.

Images