Attendance management system and method based on multi-modal identification

Abstract

The invention relates to the technical field of data identification, in particular to an attendance management system and method based on multi-modal identification. Comprising the following steps: collecting and synchronously processing multi-modal biological characteristic data and environment data through a timestamp, and preprocessing the environment data after timestamp synchronization; pre-processed multi-modal biological characteristic data is obtained; performing feature extraction on the preprocessed multi-modal biological feature data to obtain biological features of different modalities, and performing alignment and mapping processing on the biological features of different modalities; generating comprehensive feature representation based on the environment data after timestamp synchronization; and carrying out matching calculation on the comprehensive feature representation and storage features in an attendance database, outputting a highest matching score, obtaining an identification result according to the highest matching score, and generating an abnormal behavior report. The technical problems that in an existing attendance management system, user feature data is single, and user feature data processing is not accurate enough are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data recognition, and in particular to an attendance management system and method based on multimodal recognition. Background Art

[0002] In modern society, attendance management, as an important part of enterprise management and personnel attendance records, is widely used in various enterprises, schools and government agencies. However, traditional attendance management methods, such as paper sign-in and access card swiping, have many defects and cannot effectively meet the complex and changing attendance needs. With the development of biometric recognition technology, attendance management systems based on biometrics such as fingerprints, faces, irises, and voices have gradually been promoted and applied. These biometric recognition technologies have improved the security and convenience of attendance management systems to a certain extent, but there are still some problems that need to be solved in practical applications.

[0003] In summary, there are at least the following technical problems: the user feature data in the attendance management system is single and the user feature data is not processed accurately enough. Summary of the invention

[0004] The present invention provides an attendance management system and method based on multimodal recognition, so as to solve the technical problems that user characteristic data in the existing attendance management system is single and the processing of user characteristic data is not accurate enough.

[0005] The present invention provides an attendance management system and method based on multimodal recognition, which specifically includes the following technical solutions:

[0006] An attendance management method based on multimodal recognition comprises the following steps:

[0007] S1. Collect and process multimodal biometric data and environmental data through timestamp synchronization to obtain multimodal biometric data and environmental data after timestamp synchronization, and preprocess the environmental data after timestamp synchronization to obtain preprocessed environmental data; preprocess the multimodal biometric data after timestamp synchronization based on the preprocessed environmental data to obtain preprocessed multimodal biometric data;

[0008] S2. Extract features from the preprocessed multimodal biometric data to obtain biometric features of different modalities, align and map the biometric features of different modalities to obtain an aligned multimodal feature vector set; then, based on the environmental data synchronized with the timestamp, perform weighted processing on the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation;

[0009] S3. Perform matching calculation on the comprehensive feature representation and the stored feature vector in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate an abnormal behavior report.

[0010] Preferably, the S1 specifically includes:

[0011] The image modality and speech modality in the multimodal biometric data after timestamp synchronization are denoised, and the image modality and speech modality of the denoised multimodal biometric data are respectively pixel value normalized and amplitude normalized to obtain the normalized image modality and speech modality of the multimodal biometric data.

[0012] Preferably, the S1 specifically includes:

[0013] Based on the preprocessed environmental data, image cropping and contrast adjustment are performed on the image modality and voice modality of the normalized multimodal biometric data, as well as voice signal adaptive adjustment processing, and an adaptive contrast enhancement algorithm is introduced.

[0014] Preferably, the S1 specifically includes:

[0015] In the process of implementing the adaptive contrast enhancement algorithm, the fingerprint image of the normalized multimodal biometric data is subjected to multi-layer logarithmic-exponential transformation to obtain the fingerprint image after logarithmic-exponential transformation; and the fingerprint image after logarithmic-exponential transformation is subjected to local adaptive detail enhancement processing.

[0016] Preferably, the S1 specifically includes:

[0017] In the process of implementing the adaptive contrast enhancement algorithm, a local detail enhancement factor and an adaptive gain control factor are introduced, and the fingerprint image after multi-layer logarithmic-exponential transformation is locally adaptively enhanced to obtain a fingerprint image after local adaptive detail enhancement. The fingerprint image after local adaptive detail enhancement is then subjected to multi-scale local mapping correction to obtain the final corrected enhanced fingerprint image.

[0018] Preferably, the S2 specifically includes:

[0019] The joint texture-structure tensor feature extraction and nonlinear mapping algorithm are used to extract features from the fingerprint image data of the preprocessed multimodal biometric data.

[0020] Preferably, the S2 specifically includes:

[0021] In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, fingerprint texture feature extraction is performed to obtain local texture information of the fingerprint; the preprocessed fingerprint image data is normalized to obtain normalized fingerprint image data, and the horizontal gradient and vertical gradient of the normalized fingerprint image data are calculated to obtain the local gradient amplitude, and the structure tensor is constructed through the local gradient information; and based on the structure tensor, a joint feature tensor is introduced, and the joint feature tensor is mapped to a feature matrix.

[0022] Preferably, the S2 specifically includes:

[0023] In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, adaptive nonlinear mapping is performed, and the mapped feature matrix is ​​adaptively transformed through the nonlinear mapping function; after completing the adaptive nonlinear mapping, the values ​​of the mapped feature matrix are arranged as column vectors to form the final fingerprint feature vector.

[0024] An attendance management system based on multimodal recognition includes the following parts:

[0025] Data acquisition module, data preprocessing module, modal feature extraction and alignment module, multimodal feature fusion and decision module, data storage and management module, behavior analysis and anomaly detection module;

[0026] The data acquisition module acquires the user's multimodal biometric data and environmental data, and performs timestamp synchronization on the collected multimodal biometric data and environmental data, and sends the timestamp-synchronized multimodal biometric data and environmental data to the data preprocessing module, and at the same time sends the timestamp-synchronized environmental data to the multimodal feature fusion and decision module;

[0027] A data preprocessing module, which preprocesses the multimodal biometric data after timestamp synchronization based on the environmental data after timestamp synchronization to obtain the preprocessed multimodal biometric data, and sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module;

[0028] The modal feature extraction and alignment module extracts the feature vector of each modality in the preprocessed multimodal biometric data, aligns and maps the features of different modalities, obtains an aligned multimodal feature vector set, and sends the aligned multimodal feature vector set to the multimodal feature fusion and decision module;

[0029] The multimodal feature fusion and decision module dynamically adjusts the fusion weight of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism based on the environmental data after timestamp synchronization, and performs weighted processing on the multimodal feature vector set to generate a comprehensive feature representation. The module performs matching calculation based on the comprehensive feature representation and the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module;

[0030] The data storage and management module uses a distributed storage system to store attendance data, and uses blockchain technology to store attendance data records and data access logs, and sends the stored attendance data records and data access logs to the behavior analysis and anomaly detection module;

[0031] The behavior analysis and anomaly detection module analyzes attendance data in real time based on the attendance data records and data access logs stored in the data storage and management module, detects abnormal behaviors in the attendance data, issues early warnings and prompts for detected abnormal behaviors, and generates abnormal behavior reports.

[0032] The beneficial effects of the technical solution of the present invention are:

[0033] 1. The dynamic weight adaptive adjustment mechanism introduced can dynamically adjust the weight distribution of biometric features of different modalities according to changes in environmental data (such as light intensity, noise level, etc.), ensuring that the importance of each modality can be automatically adjusted under different environmental conditions (such as too bright or too dark light, high noise, etc.).

[0034] 2. Multi-layer logarithmic-exponential transformation and multi-scale local mapping technology are introduced to enhance the details of different levels of fingerprint images layer by layer, especially the details of lines and detail points. After multi-layer logarithmic-exponential transformation, the fingerprint image after logarithmic-exponential transformation is subjected to local adaptive detail enhancement processing, so that the fingerprint image can obtain effective contrast improvement under different brightness conditions.

[0035] 3. The joint texture-structure tensor feature extraction and nonlinear mapping algorithm are used to extract features from fingerprint images, which can effectively combine the local details of the fingerprint image with the global structural information, and improve the distinctiveness of the features through adaptive nonlinear mapping, so that the extracted fingerprint features can still maintain high-accuracy recognition performance under various complex conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a structural diagram of an attendance management system based on multimodal recognition according to the present invention;

[0037] Figure 2This is a flow chart of an attendance management method based on multimodal recognition described in the present invention. DETAILED DESCRIPTION

[0038] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0040] The specific scheme of an attendance management system and method based on multimodal recognition provided by the present invention is described in detail below with reference to the accompanying drawings.

[0041] See attached Figure 1 , which shows a structural diagram of an attendance management system based on multimodal recognition provided by an embodiment of the present invention, the system includes the following parts:

[0042] Data acquisition module, data preprocessing module, modal feature extraction and alignment module, multimodal feature fusion and decision module, data storage and management module, behavior analysis and anomaly detection module;

[0043] The data acquisition module obtains the user's multimodal biometric data and environmental data, and synchronizes the collected multimodal biometric data and environmental data with timestamps to ensure the temporal consistency of the data. The multimodal biometric data and environmental data after timestamp synchronization are sent to the data preprocessing module, and the environmental data after timestamp synchronization is sent to the multimodal feature fusion and decision module.

[0044] The multimodal biometric data includes face image, iris image, fingerprint image and voice signal; the environmental data includes light intensity, noise level, etc.;

[0045] A data preprocessing module, based on the environmental data after timestamp synchronization, preprocesses the multimodal biometric data after timestamp synchronization to obtain the preprocessed multimodal biometric data, and sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module;

[0046] The modal feature extraction and alignment module extracts the feature vector of each modality in the preprocessed multimodal biometric data, and then aligns and maps the features of different modalities to ensure that the feature vectors have the same dimension and similar distribution in the same feature space, obtains the aligned multimodal feature vector set, and sends the aligned multimodal feature vector set to the multimodal feature fusion and decision module;

[0047] The feature vectors of the modalities include a face feature vector, an iris feature vector, a fingerprint feature vector and a voice feature vector, forming a multi-modal feature vector set;

[0048] The multimodal feature fusion and decision module dynamically adjusts the modal weights of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism based on the environmental data after timestamp synchronization, further weights the multimodal feature vector set, generates a comprehensive feature representation, performs matching calculation based on the comprehensive feature representation and the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module;

[0049] The attendance database is used to store user characteristics;

[0050] The data storage and management module uses a distributed storage system to store attendance data, and uses blockchain technology to store attendance data records and data access logs, and sends the stored attendance data records and data access logs to the behavior analysis and anomaly detection module;

[0051] The attendance data may include attendance timestamp, matching score, recognition result, etc.;

[0052] The behavior analysis and anomaly detection module uses an aggregation analysis algorithm to analyze attendance data in real time based on the records of attendance data and data access logs stored in the data storage and management module, detects abnormal behaviors in attendance data, issues early warnings and prompts for detected abnormal behaviors, and generates abnormal behavior reports;

[0053] When conducting attendance management, you can select one of the modal biometric features for recognition, such as fingerprint, face, iris, or voice, depending on the specific application scenario and needs.

[0054] See attached Figure 2 , which shows a flow chart of an attendance management method based on multimodal recognition provided by an embodiment of the present invention, the method comprising the following steps:

[0055] S1. Collect and process multimodal biometric data and environmental data through timestamp synchronization to obtain multimodal biometric data and environmental data after timestamp synchronization, and preprocess the environmental data after timestamp synchronization to obtain preprocessed environmental data; preprocess the multimodal biometric data after timestamp synchronization based on the preprocessed environmental data to obtain preprocessed multimodal biometric data;

[0056] Multimodal sensor devices are used to collect data to obtain the user's multimodal biometric data and environmental data. The multimodal sensor devices include: face image collection devices, iris image collection devices, fingerprint image collection devices, voice signal collection devices, environmental data sensors, etc.; each multimodal sensor device records the current precise timestamp while collecting data. The timestamps of all multimodal sensor devices use a unified clock source, such as synchronization through a network time protocol (NTP) server, to obtain multimodal biometric data and environmental data after timestamp synchronization.

[0057] The multimodal biometric data includes face image, iris image, fingerprint image, and voice signal; the environmental data includes light intensity and noise level;

[0058] A preprocessing method familiar to those skilled in the art is used on the environmental data after time stamp synchronization to obtain preprocessed environmental data;

[0059] The image modality (such as face, iris, and fingerprint) in the multimodal biometric data after timestamp synchronization is denoised using filtering. For mild noise interference (such as random noise), a 3×3 or 5×5 median filter is used for image denoising; for images with edge details (such as fingerprints and irises), bilateral filtering is used for smoothing to retain the edge features of the image. For the voice modality in the multimodal biometric data after timestamp synchronization, the short-time Fourier transform (STFT) is used to convert the voice signal to the frequency domain, and the spectral subtraction method is used to remove the noise, and then the denoised voice signal is converted back to the time domain;

[0060] Furthermore, pixel value normalization and amplitude normalization are performed on the image modality and speech modality of the denoised multimodal biometric data, respectively, to obtain the normalized image modality and speech modality of the multimodal biometric data.

[0061] Furthermore, based on the preprocessed environmental data, image cropping and contrast adjustment processing are performed on the image modality and voice modality of the normalized multimodal biometric data, and voice signal adaptive adjustment processing is performed.

[0062] Image cropping and contrast adjustment processing for the image modality of normalized multimodal biometric data:

[0063] Face image: The face area is located by face detection algorithm (such as MTCNN) and cropped into an image of fixed size (such as 112×112 pixels). If the light is too dark or too bright, the brightness is balanced according to the ambient light intensity in the preprocessed environmental data, and the contrast is improved by using adaptive histogram equalization (CLAHE) technology; Iris image: The iris area is extracted by using iris segmentation algorithm (such as Daugman algorithm), the non-iris area (such as white of the eye, pupil) is masked, and the contrast stretching algorithm is used to enhance the iris details; Fingerprint image: For fingerprint images with incomplete edges, masking is used to remove invalid fingerprint areas at the edge, and the Gabor filter is used to enhance the fingerprint texture to ensure the clarity of the fingerprint line features.

[0064] In order to solve the problem of unclear fingerprint line features due to the differences in multimodal sensor devices and ambient lighting, as well as the uneven contrast and brightness of fingerprint images, an adaptive contrast enhancement algorithm is introduced. The adaptive contrast enhancement algorithm enhances the details of different levels of fingerprint images layer by layer through multi-layer nonlinear transformation and multi-scale local mapping, and controls the flexibility and balance of the enhancement process through an adaptive mechanism. The specific implementation process is as follows:

[0065] First, the fingerprint image of the normalized multimodal biometric data is subjected to a multi-layer logarithmic-exponential transformation. The form of the multi-layer logarithmic-exponential transformation is:

[0066]

[0067] in, is the fingerprint image of the pixel point (x, y) after the logarithmic-exponential transformation at the lth layer, indicating the image pixel value after the lth layer of nonlinear transformation; is the fingerprint image of the pixel point (x, y) after the l-1th layer logarithmic-exponential transformation, which represents the image pixel value after the l-1th layer nonlinear transformation; is the fingerprint image of the normalized multimodal biometric data at the pixel point (x, y); α l is the logarithmic magnification factor, which controls the overall contrast improvement of the logarithmic-exponential transformation of the first layer; β l is the exponential amplification factor, which adjusts the enhancement degree of the image brightness area of ​​the lth layer logarithmic-exponential transformation; γ l It is the control coefficient of the lth level nonlinear transformation, which affects the nonlinear transformation degree of the lth level logarithmic-exponential transformation and controls the enhancement intensity of the image brightness area. Finally, the specified number of layers L preset according to the specific scene is reached, and the fingerprint image of the pixel (x, y) after the last level logarithmic-exponential transformation is obtained.

[0068] Furthermore, after multiple layers of logarithmic-exponential transformation, the fingerprint image after logarithmic-exponential transformation is subjected to local adaptive detail enhancement processing, and the enhancement amplitude of different regions is adaptively adjusted using local brightness features, so that the fingerprint image can obtain effective contrast enhancement under different brightness conditions;

[0069] In order to achieve adaptive detail enhancement processing, the local detail enhancement factor E(x,y) is introduced to dynamically adjust the intensity of detail enhancement by measuring the ratio of the local brightness feature of the current pixel to the global brightness mean. The local detail enhancement factor is defined as:

[0070]

[0071] Among them, E(x,y) is the local detail enhancement factor at the pixel point (x,y); B(x,y) is the local brightness feature of the pixel point (x,y). Ω is the local neighborhood window, which is used to calculate the average brightness around the pixel. is the pixel value of the fingerprint image at position (i, j) of the Lth layer after multi-layer logarithmic-exponential transformation; μ B is the global brightness mean, which represents the average brightness of the entire fingerprint image. M, N are the number of rows and columns of the fingerprint image respectively; ∈ is a very small number to avoid the numerator and denominator being 0.

[0072] Furthermore, after calculating the local detail enhancement factor, the adaptive gain control factor G(x,y) is introduced to finely control the enhancement amplitude according to the ambient light intensity and ambient noise level. The adaptive gain control factor is defined as:

[0073]

[0074] in, is the ambient light intensity, indicating the lighting conditions of the fingerprint collection environment; V is the ambient noise level, indicating the noise interference of the fingerprint collection environment; G(x,y) is the adaptive gain control factor, which is used to dynamically adjust the amplitude of detail enhancement under different lighting and noise conditions. As the lighting and noise levels increase, the adaptive gain factor tends to saturation to avoid over-enhancement.

[0075] Combining the local detail enhancement factor and the adaptive gain control factor, the fingerprint image after multi-layer logarithmic-exponential transformation is subjected to local adaptive detail enhancement. The formula is:

[0076]

[0077] Among them, I enh(x, y) is the fingerprint image after local adaptive detail enhancement of pixel point (x, y). The adaptive contrast is improved by the joint adjustment of the local detail enhancement factor and the adaptive gain control factor. It is to adaptively adjust the brightness of fingerprint image through local detail enhancement factor; The detail enhancement amplitude is adjusted through an adaptive gain control factor to ensure that image details can be fully amplified under different lighting conditions.

[0078] In order to avoid uneven transition in the enhanced area of ​​the fingerprint image after local adaptive detail enhancement, multi-scale local mapping correction is performed on the fingerprint image after local adaptive detail enhancement. The artifacts of over-enhancement of local adaptive details are eliminated through multi-scale analysis, making the detail features of the fingerprint image at different scales more natural.

[0079] Fingerprint image I after local adaptive detail enhancement using wavelet decomposition enh (x, y) is decomposed into S scale detail components W s (x,y) and an approximation component A S (x,y), the formula is

[0080]

[0081] Among them, W s (x, y) is the detail component of the pixel (x, y) at scale s, indicating detail information at different scales;

[0082] For each detail component W s (x, y) is subjected to local mapping correction to suppress excessive enhancement of local adaptive details. The multi-scale local mapping correction formula is:

[0083]

[0084] in, It is the detail component of the pixel (x, y) after local mapping correction at scale s, which eliminates the artifacts caused by excessive enhancement of local adaptive details and retains the original detail features of the fingerprint image; κ s is the detail component correction coefficient, which controls the correction strength; s It is the nonlinear mapping control coefficient, which affects the degree of nonlinearity in the local mapping correction process.

[0085] Finally, the detail component after local mapping correction is and the approximation component A S (x, y) is reconstructed to obtain the final corrected enhanced fingerprint image:

[0086]

[0087] Among them, I final (x, y) is the enhanced fingerprint image after the final correction of the pixel point (x, y), that is, the preprocessed fingerprint image data. After multi-scale local mapping correction and reconstruction, it not only retains the improvement of global contrast, but also realizes adaptive detail enhancement and artifact suppression in local details.

[0088] For scenes with light intensity below 200 lux, an image brightness enhancement algorithm (such as gamma correction) is used to enhance the brightness. When the light intensity is higher than 5000 lux, a local contrast enhancement algorithm is used to reduce the exposure of overly bright areas. When the noise level is higher than 80 dB, a strong noise suppression algorithm based on spectral subtraction is used for speech signals to reduce the interference of background noise on speech features.

[0089] After the above processing, the preprocessed multimodal biometric data is obtained.

[0090] S2. Extract features from the preprocessed multimodal biometric data to obtain biometric features of different modalities, align and map the biometric features of different modalities to obtain an aligned multimodal feature vector set; then, based on the environmental data synchronized with the timestamp, perform weighted processing on the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation;

[0091] The preprocessed multimodal biometric data is subjected to feature extraction using existing feature engineering techniques to obtain biometric features of different modalities. For example, the existing deep convolutional neural network is used to extract features from face images and iris images, and the speech signal is subjected to feature extraction using a short-time Fourier transform and long short-term memory network. For fingerprint images, the joint texture-structure tensor feature extraction and nonlinear mapping algorithm are used for feature extraction. The joint texture-structure tensor is introduced to represent the local details and global structural information of the fingerprint image, and a highly discriminative fingerprint feature vector is generated through an adaptive nonlinear mapping method. The specific implementation process of the joint texture-structure tensor feature extraction and nonlinear mapping algorithm is as follows:

[0092] The first step is to extract fingerprint texture features. The fingerprint image data in the pre-processed multimodal biometric data is subjected to fingerprint texture feature extraction to extract the local texture information of the fingerprint, including the ridge and valley features of the fingerprint. final (x, y) is normalized to obtain the normalized fingerprint image data I norm (x, y), which provides a good basis for the subsequent fingerprint texture feature extraction. Calculate the edge information of the horizontal gradient and vertical gradient of the normalized fingerprint image data in the direction domain: horizontal gradient The grayscale change rate of the fingerprint image in the x-axis direction is used to capture the horizontal edge information of the fingerprint image; the vertical gradient The grayscale change rate of the fingerprint image in the y-axis direction is used to capture the vertical edge information of the fingerprint image; at the same time, the horizontal gradient and vertical gradient are used to calculate the local gradient amplitude It represents the strength of the local gradient, reflecting the significance of the edge information in the fingerprint image. θ(i,j) represents the direction of the local gradient, which is used to describe the directional characteristics of the ridges and valleys of the fingerprint:

[0093]

[0094] Furthermore, in order to quantify the texture features of the fingerprint image, the directional domain gradient histogram is introduced to divide the local gradient direction θ(i,j) into direction intervals (for example, divided into 8 directions, each direction interval is 45 degrees), and then count the local neighborhood windows in each direction interval The sum of the local gradient amplitudes of all pixels in the image. The calculation formula is:

[0095]

[0096] in, It is a local neighborhood window (such as a 3×3 or 5×5 rectangular area) used to count the gradient information in the local area; is the local gradient magnitude, representing the local neighborhood window The gradient strength at position (i, j) in the inner image; θ(i, j) is the local gradient direction, indicating the local neighborhood window The gradient direction at the inner position (i, j); δ k (·) is the indicator function. When the local gradient direction θ(i,j) falls in the kth direction interval, δ k (θ(i,j))=1, otherwise δ k (θ(i,j))=0;H k (x, y) is the value of the directional domain gradient histogram in the kth directional interval, which indicates the gradient strength of the kth directional interval in the local area. Finally, the gradient distribution of all directional intervals in the local area is obtained, and the comprehensive feature vector H(x, y) is obtained. k (x,y)∈H(x,y).

[0097] The second step is to build the structure tensor and feature fusion; the structure tensor is built through local gradient information, which is defined as:

[0098]

[0099] Among them, J(x,y) is the structural tensor, which reflects the distribution characteristics of the local gradient direction of the fingerprint image at the pixel point (x,y); It is the average value of the square of the horizontal gradient in the local neighborhood, which is used to describe the consistency of the horizontal direction of the local area; It is the average of the product of the horizontal gradient and the vertical gradient in the local neighborhood, which is used to describe the cross-characteristics of the local area; It is the average value of the square of the vertical gradient in the local neighborhood and is used to describe the consistency of the vertical direction of the local area.

[0100] In order to realize the joint description of fingerprint texture and structural information, the joint feature tensor F(x,y) is introduced as:

[0101]

[0102] Furthermore, in order to improve the distinguishability of features, the joint feature tensor F(x,y) is mapped to a new feature space by a mapping function to perform feature matrix mapping, so as to maximize the distinguishability of features. The mapping function is defined as:

[0103]

[0104] Among them, |F(x,y)| is the modulus of the joint feature tensor, which represents the strength of fingerprint texture and structural information in the local area; is a smoothing parameter that controls the influence of the structure tensor on the feature matrix; trace(J(x,y)) is the trace of the structure tensor, that is, the sum of the diagonal elements of the structure tensor, which is used to measure the strength of local structural information; M(x,y) is the feature matrix, which is the feature representation after mapping and integrates the fingerprint texture and structure information.

[0105] The third step is adaptive nonlinear mapping. The mapped feature matrix is ​​adaptively transformed through the nonlinear mapping function to further improve the distinguishability of the features. First, the adaptive mapping weight of the feature matrix M(x, y) is calculated. It is defined as:

[0106]

[0107] in, is the adaptive mapping weight, which controls the nonlinear mapping strength of the feature matrix; ||M(x,y)|| F is the Frobenius norm of the feature matrix, that is, the square root of the sum of squares of all elements, which represents the overall strength of the feature matrix; μ is the mean of the feature strength, which is used to calculate the average strength of the feature matrix; σ is the standard deviation of the feature strength, which is used to control the sensitivity of the adaptive mapping weight.

[0108] Furthermore, the feature matrix M(x,y) is adaptively nonlinearly mapped. The feature matrix is ​​mapped to a higher feature space to maximize the distinguishability of the features. The nonlinear mapping function is defined as:

[0109]

[0110] Among them, M mapped (x, y) is the feature matrix after mapping. The feature representation after nonlinear transformation has higher discrimination. Tanh(·) is the hyperbolic tangent function, which limits the nonlinear mapping result to the range of [-1, 1] to prevent the overflow of feature data after mapping. It is the adjustment parameter of nonlinear mapping, which is used to control the transformation amplitude of feature matrix in nonlinear mapping; log(1+||M(x,y)|| F ) is the logarithmic transformation term of the feature matrix strength, which is used to further improve the distinguishability of the features.

[0111] After completing the adaptive nonlinear mapping, all mapped feature matrix values ​​are arranged as column vectors to form the final fingerprint feature vector f final . By converting local feature information into global feature representation, it is convenient for subsequent fingerprint matching and recognition. The final fingerprint feature vector generation process is as follows:

[0112]

[0113] Among them, f final is the final fingerprint feature vector, which integrates the local fingerprint texture features and structural features of the fingerprint image and has high discrimination; M is the number of rows of the fingerprint image, indicating the height of the fingerprint image; N is the number of columns of the fingerprint image, indicating the width of the fingerprint image; flatten(·) flattens the mapped feature matrix values ​​by column into a vector form, which is used to convert the two-dimensional feature matrix into a one-dimensional feature vector.

[0114] After the above process, the features of different modes are extracted, and the existing canonical correlation analysis algorithm is used for alignment and mapping to obtain the aligned multi-modal feature vector set.

[0115] Furthermore, based on the environmental data after timestamp synchronization, the aligned multimodal feature vector set is weighted through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation;

[0116] Based on the environmental data after timestamp synchronization, a dynamic weight adaptive adjustment mechanism is introduced to achieve adaptive adjustment by dynamically adjusting the weight. The weight formulas of different modes are as follows:

[0117]

[0118] Among them, w i is the weight of the i-th mode; Indicates the number of modes; It is Environmental factors (such as light intensity, noise level, etc.), which represent environmental data after timestamp synchronization; Represents the relationship between the i-th mode and the The interaction angle parameter between environmental factors is used to adjust the impact of environmental factors on each mode, and the range is [0,2π]; Represents the relationship between the i-th mode and the The smoothing coefficient between environmental factors is used to control the smoothing regulation effect of environmental factors on each mode; m represents the total number of environmental factors;

[0119] Finally, the aligned multimodal feature vector set is weighted by the weights of different modalities to generate a comprehensive feature representation.

[0120] S3. Perform matching calculation on the comprehensive feature representation and the stored feature vector in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate an abnormal behavior report.

[0121] First, the comprehensive feature representation is matched with the stored feature vectors in the attendance database (i.e., all user features). The identity of the current user is determined by the similarity measurement between the comprehensive feature representation and the features stored in the attendance database. The similarity measurement is calculated by cosine similarity (or other distance measurement methods) to obtain the similarity measurement represents the similarity measure with the qth stored feature in the attendance database, T q It is the qth stored feature in the attendance database.

[0122] Then all the similarity measurement scores are combined Sort in descending order, Q represents the number of features stored in the attendance database, and select the top K matching results with the highest scores as candidate matching results, which will be used to determine the candidate list of recognition results.

[0123] Further, the recognition result is generated based on the candidate matching result. First, based on the candidate matching result and the preset threshold Determine the user identity corresponding to the current comprehensive feature representation. If the highest matching score SIM max Exceeding the preset threshold If the match is successful, the corresponding user ID is output as the recognition result; otherwise, "unrecognized" or "unknown identity" is output. It is based on the similarity threshold preset by the expert experience method to further obtain the recognition result and matching score (similarity score) for subsequent storage and management.

[0124] Furthermore, the recognition results and matching scores are stored and managed. Ensure that the information of each recognition event is fully recorded to facilitate subsequent data analysis and attendance management. Each recognition event contains the following information: timestamp, employee ID, matching score, feature vector, environmental data, etc. The recognition event is packaged into a record unit and stored through a distributed database or blockchain technology.

[0125] In data storage, in order to prevent data loss and to quickly recover in the event of failure or attack, the data in the attendance database is regularly backed up to the cloud or local disk and stored in encrypted form to ensure data security and integrity.

[0126] Regularly analyze the data in the attendance database and detect abnormal behavior. Use the existing aggregation analysis algorithm to aggregate and analyze the stored data, find out the abnormalities in the attendance behavior pattern (such as punching in for someone else, not punching in for a long time, etc.), and generate an abnormal behavior report.

[0127] In summary, an attendance management system and method based on multimodal recognition has been completed.

[0128] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0129] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0130] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. An attendance management method based on multimodal recognition, characterized in that: The following steps are involved: S1. Collect and process multimodal biometric data and environmental data through timestamp synchronization to obtain multimodal biometric data and environmental data after timestamp synchronization, and preprocess the environmental data after timestamp synchronization to obtain preprocessed environmental data; Preprocessing the multimodal biometric data after time stamp synchronization based on the preprocessed environmental data to obtain preprocessed multimodal biometric data; S2. Extract features from the preprocessed multimodal biometric data to obtain biometric features of different modalities, align and map the biometric features of different modalities to obtain an aligned multimodal feature vector set; then, based on the environmental data synchronized with the timestamp, perform weighted processing on the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism to generate a comprehensive feature representation; S3. Perform matching calculation on the comprehensive feature representation and the stored feature vector in the attendance database, output the highest matching score, obtain the recognition result based on the highest matching score, store the recognition result and the highest matching score in the attendance database and manage them; analyze the data in the attendance database and detect abnormal behavior, and generate an abnormal behavior report.

2. The attendance management method based on multimodal recognition according to claim 1, characterized in that: The S1 specifically includes: The image modality and speech modality in the multimodal biometric data after timestamp synchronization are denoised, and the image modality and speech modality of the denoised multimodal biometric data are respectively pixel value normalized and amplitude normalized to obtain the normalized image modality and speech modality of the multimodal biometric data.

3. The attendance management method based on multimodal recognition according to claim 2 is characterized in that: The S1 specifically includes: Based on the preprocessed environmental data, image cropping and contrast adjustment are performed on the image modality and voice modality of the normalized multimodal biometric data, as well as voice signal adaptive adjustment processing, and an adaptive contrast enhancement algorithm is introduced.

4. The attendance management method based on multimodal recognition according to claim 3 is characterized in that: The S1 specifically includes: In the process of implementing the adaptive contrast enhancement algorithm, the fingerprint image of the normalized multimodal biometric data is subjected to multi-layer logarithmic-exponential transformation to obtain the fingerprint image after logarithmic-exponential transformation; and the fingerprint image after logarithmic-exponential transformation is subjected to local adaptive detail enhancement processing.

5. The attendance management method based on multimodal recognition according to claim 4 is characterized in that: The S1 specifically includes: In the process of implementing the adaptive contrast enhancement algorithm, a local detail enhancement factor and an adaptive gain control factor are introduced, and the fingerprint image after multi-layer logarithmic-exponential transformation is locally adaptively enhanced to obtain a fingerprint image after local adaptive detail enhancement. The fingerprint image after local adaptive detail enhancement is then subjected to multi-scale local mapping correction to obtain the final corrected enhanced fingerprint image.

6. The attendance management method based on multimodal recognition according to claim 1, characterized in that: The S2 specifically includes: The joint texture-structure tensor feature extraction and nonlinear mapping algorithm are used to extract features from the fingerprint image data of the preprocessed multimodal biometric data.

7. The attendance management method based on multimodal recognition according to claim 6 is characterized in that: The S2 specifically includes: In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, fingerprint texture feature extraction is performed to obtain local texture information of the fingerprint; the preprocessed fingerprint image data is normalized to obtain normalized fingerprint image data, and the horizontal gradient and vertical gradient of the normalized fingerprint image data are calculated to obtain the local gradient amplitude, and the structure tensor is constructed through the local gradient information; and based on the structure tensor, a joint feature tensor is introduced, and the joint feature tensor is mapped to a feature matrix.

8. The attendance management method based on multimodal recognition according to claim 7 is characterized in that: The S2 specifically includes: In the joint texture-structure tensor feature extraction and nonlinear mapping algorithm, adaptive nonlinear mapping is performed, and the mapped feature matrix is ​​adaptively transformed through the nonlinear mapping function; after completing the adaptive nonlinear mapping, the values ​​of the mapped feature matrix are arranged as column vectors to form the final fingerprint feature vector.

9. An attendance management system based on multimodal recognition, applied to the attendance management method based on multimodal recognition as claimed in claim 1, characterized in that: Includes the following parts: Data acquisition module, data preprocessing module, modal feature extraction and alignment module, multimodal feature fusion and decision module, data storage and management module, behavior analysis and anomaly detection module; The data acquisition module acquires the user's multimodal biometric data and environmental data, and performs timestamp synchronization on the collected multimodal biometric data and environmental data, and sends the timestamp-synchronized multimodal biometric data and environmental data to the data preprocessing module, and at the same time sends the timestamp-synchronized environmental data to the multimodal feature fusion and decision module; A data preprocessing module, which preprocesses the multimodal biometric data after timestamp synchronization based on the environmental data after timestamp synchronization to obtain the preprocessed multimodal biometric data, and sends the preprocessed multimodal biometric data to the modal feature extraction and alignment module; The modal feature extraction and alignment module extracts the feature vector of each modality in the preprocessed multimodal biometric data, aligns and maps the features of different modalities, obtains an aligned multimodal feature vector set, and sends the aligned multimodal feature vector set to the multimodal feature fusion and decision module; The multimodal feature fusion and decision module dynamically adjusts the fusion weight of the aligned multimodal feature vector set through a dynamic weight adaptive adjustment mechanism based on the environmental data after timestamp synchronization, and performs weighted processing on the multimodal feature vector set to generate a comprehensive feature representation. The module performs matching calculation based on the comprehensive feature representation and the stored features in the attendance database, outputs the highest matching score, obtains the recognition result based on the highest matching score, and sends the recognition result and the highest matching score to the data storage and management module; The data storage and management module uses a distributed storage system to store attendance data, and uses blockchain technology to store attendance data records and data access logs, and sends the stored attendance data records and data access logs to the behavior analysis and anomaly detection module; The behavior analysis and anomaly detection module analyzes attendance data in real time based on the attendance data records and data access logs stored in the data storage and management module, detects abnormal behaviors in the attendance data, issues early warnings and prompts for detected abnormal behaviors, and generates abnormal behavior reports.

Images