A face recognition method and system based on cross-modal face feature fusion

Abstract

The application relates to a face recognition method, system, electronic device and storage medium based on cross-modal face feature fusion. Through the cooperative work of an IR terminal device and a cloud server, efficient and accurate face recognition is realized. The method first uses a small model on the IR terminal device for preliminary recognition. When the recognition fails to reach a preset threshold, the optimal IR face image is selected and uploaded to the cloud server. The cloud server extracts the IR face feature using a large model, and based on a preset fusion rule combining an environmental parameter compensation factor and a feature increment, fuses the IR face feature with a pre-stored RGB face feature to generate a fusion feature and return the fusion feature to the terminal device. The terminal device uses the fusion feature for subsequent recognition, greatly improving the recognition accuracy.

Description

Technical Field

[0001] This application relates to the field of face recognition, and in particular to a cross-modal face feature fusion method, system, computer device, and computer-readable storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, facial recognition technology has been widely applied in various scenarios, such as security monitoring, access control systems, and mobile payments. Traditional facial recognition is mainly based on visible light (RGB) images, but the recognition effect of visible light cameras drops significantly in low light or dark environments. To solve this problem, infrared (IR) facial recognition technology has emerged, which can achieve effective recognition in low light or even complete darkness.

[0003] The existing technology still has the following problems: First, when facial features extracted based on RGB images are sent to devices with only IR cameras, the similarity between the facial features extracted locally by the IR camera and those extracted by a small model is generally low due to factors such as modal differences, photo quality, ambient light, and pose. It is usually below 80%, and the similarity of some difficult-to-identify samples is even between 40% and 60%, resulting in a low recognition success rate.

[0004] Secondly, most existing methods rely on complex pre-trained models or feature mapping networks, which are usually computationally intensive and difficult to run efficiently on resource-constrained terminal devices. Summary of the Invention

[0005] This application provides a face recognition method, system, computer device, and computer-readable storage medium based on cross-modal face feature fusion, to at least solve the problem of low face recognition success rate of IR terminal devices in related technologies.

[0006] In a first aspect, embodiments of this application provide a face recognition method based on cross-modal face feature fusion, implemented through an IR terminal device and a cloud server, the method comprising:

[0007] The IR terminal device receives multiple sets of IR face images, extracts the IR face features of each IR face image using a small model, and performs face recognition based on the IR face features.

[0008] After the number of recognition failures reaches a preset failure threshold, the optimal IR face image is obtained from multiple sets of IR face images and uploaded to the cloud server.

[0009] The cloud server extracts the IR face features of the optimal IR face image through a large model, and based on a preset fusion strategy, fuses the IR face features with the pre-stored RGB face features of the same user to obtain the fused features and sends them to the terminal device.

[0010] The terminal device receives the updated IR face image, extracts the features of the updated IR face image in real time using the small model, compares the extracted features with the fused features, and outputs the recognition result.

[0011] In some embodiments, before the IR terminal device receives multiple sets of IR face images, the cloud server stores the RGB features in a local library and sends the RGB features to the terminal device for persistent storage. Face recognition based on the IR face features includes:

[0012] The terminal device performs face recognition by comparing each IR face feature with the RGB features stored locally.

[0013] In some embodiments, fusing the IR facial features with pre-stored RGB facial features of the same user based on a preset fusion strategy includes:

[0014] According to the preset evaluation criteria, the image quality scores corresponding to the IR face features and the RGB face features are obtained respectively;

[0015] Based on the image quality score and the pre-configured modal weights corresponding to the IR face features and the RGB face features, the dynamic weights of the features are calculated.

[0016] The fused features are obtained by performing an initial weighting operation based on the IR face features and the RGB face features and their corresponding dynamic feature weights.

[0017] In some embodiments, the method further includes:

[0018] By using the light sensor deployed in the IR terminal device, the actual light intensity corresponding to the acquisition of the optimal IR image is obtained. Based on the actual light intensity and the preset reference light intensity, an environmental compensation factor is obtained according to the first preset weighted calculation rule.

[0019] In cases where feature fusion is not performed for the first time, the latest fused feature value generated by the current fusion and the historical fused feature value of the previous fusion are obtained by the feature storage module deployed on the server.

[0020] Based on the latest fused feature value, the historical fused feature value, and the stored historical feature increments, the current feature increment is obtained through the second preset weighted operation rule;

[0021] The result of the initial weighting operation is combined with the product of the environmental compensation factor and the current feature increment to obtain the fused feature.

[0022] In some embodiments, the method further includes, during the calculation of the environmental compensation factor:

[0023] The image detection and processing module deployed in the IR terminal device identifies abnormal infrared spots in the optimal IR image and determines the intensity of infrared interference based on the abnormal infrared spots.

[0024] The first environmental compensation factor is calculated based on the actual light intensity and the preset reference light intensity, and the second environmental compensation factor is calculated based on the infrared interference intensity and the reference interference threshold.

[0025] Based on the first environmental compensation factor and the second environmental compensation factor, a weighted calculation is performed to obtain an environmental compensation factor that comprehensively considers the influence of multiple factors.

[0026] In some embodiments, obtaining the optimal IR face image includes:

[0027] For IR face images that fail to be recognized, count the number of times the face ID with the highest score appears each time recognition fails, and calculate the average score of each face ID;

[0028] Obtain the target face ID that has the highest number of occurrences and the highest average score;

[0029] If the ratio of the occurrence frequency of the target face ID to all failed face images is greater than a preset ratio, and its corresponding average score exceeds a preset score, the image with the best quality is selected from the IR images corresponding to the target face ID to obtain the optimal IR image.

[0030] In some embodiments, before the cloud server extracts the IR features of the optimal IR face image using a large model, the method further includes:

[0031] The quality of the optimal IR face image is assessed.

[0032] If the quality assessment is satisfactory, the optimal IR face image is used to perform a cross-modal search in the full RGB database using a large model. Based on the cross-modal search results, the RGB face features belonging to the same user as the IR face image are determined.

[0033] Secondly, embodiments of this application provide a face recognition system based on cross-modal face feature fusion, the system being implemented through an IR terminal device and a cloud server, including:

[0034] An IR terminal device is used to receive multiple sets of IR face images, extract IR face features from each IR face image using a small model, and perform face recognition based on the IR face features.

[0035] Additionally, after the number of recognition failures reaches a preset failure threshold, the optimal IR face image is obtained from multiple sets of IR face images and uploaded to the cloud server.

[0036] The cloud server is used to extract the IR face features of the optimal IR face image through a large model, and based on a preset fusion strategy, fuse the IR face features with the RGB face features of the same user that are stored in advance to obtain the fused features and send them to the terminal device.

[0037] The terminal device is also used to receive the updated IR face image, extract the features of the updated IR face image in real time through the small model, compare the features extracted in real time with the fused features, and output the recognition result.

[0038] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0039] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0040] Compared to related technologies, the face recognition method based on cross-modal face feature fusion provided in this application effectively solves the problem of low recognition success rate in existing technologies where face features extracted from RGB images are compared with face features collected locally by an IR camera. Compared with existing technologies, the solution in this application has the following significant advantages:

[0041] 1. In response to the issue of feature shifts in RGB original images caused by makeup, Photoshop, etc., the average pass rate increased from 43% to 98%, an improvement of 55 percentage points, significantly enhancing recognition accuracy;

[0042] 2. In scenarios where the original RGB image is of poor quality due to poor photo acquisition, the recognition pass rate increased from 63% to 99%, an improvement of 36 percentage points, effectively solving the problem of image quality affecting the recognition results;

[0043] 3. In scenarios where the imaging quality of the terminal device's IR camera is poor, the recognition pass rate increased from 56% to 98%, an improvement of 42 percentage points, significantly improving the recognition effect under poor imaging conditions.

[0044] This application's solution achieves effective fusion of cross-modal facial features through the collaborative work of terminal devices and cloud platforms, combining the complementary advantages of small and large models. It addresses the impact of factors such as modal differences, image quality, ambient lighting, and pose on facial recognition success rates. Furthermore, this technology is also applicable to non-cross-modal feature fusion techniques, combining images captured by the terminal for secondary feature fusion, significantly optimizing facial recognition success rates. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 This is a flowchart of a face recognition method based on cross-modal face feature fusion according to an embodiment of this application;

[0047] Figure 2 This is a flowchart of another face recognition method based on cross-modal face feature fusion according to an embodiment of this application;

[0048] Figure 3 This is a structural block diagram of a face recognition system based on cross-modal face feature fusion according to an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0051] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0052] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0053] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0054] This application provides a face recognition method based on cross-modal face feature fusion, implemented through an IR terminal device and a cloud server. Figure 1This is a flowchart of a face recognition method based on cross-modal face feature fusion according to an embodiment of this application, such as... Figure 1 As shown, the process includes the following steps:

[0055] S101, the IR terminal device receives multiple sets of IR face images, extracts the IR face features of each IR face image through a small model, and performs face recognition based on the IR face features.

[0056] Before the IR terminal device receives multiple sets of IR face images, the cloud server stores RGB features in a local library and sends the RGB features to the terminal device for persistent storage.

[0057] Specifically, face recognition based on IR face features includes: the terminal device performing face recognition by comparing each IR face feature with the locally persistently stored RGB features.

[0058] An IR terminal device can be an infrared camera installed on a device such as an access control system, time attendance machine, or smartphone. This terminal device captures the user's facial images using an infrared sensor, forming multiple sets of IR facial images. Typically, these images are captured continuously at a rate of 5-30 frames per second, with each set containing 3-10 facial images from different angles or with different expressions.

[0059] Furthermore, the small model used in IR terminal devices is a lightweight neural network model, such as MobileNetV3 or ShuffleNetV2, with a model size typically between 5-20MB, making it suitable for running on terminal devices. This small model can quickly extract feature vectors from IR face images, with the feature vectors typically having a dimension of 128-512. The inference time of the small model is generally controlled within 50-200 milliseconds to ensure real-time performance.

[0060] In the specific recognition process, the IR terminal device performs face recognition based on IR facial features by comparing the extracted feature vector with a locally stored feature library. The comparison method typically uses cosine similarity or Euclidean distance calculations, with a threshold (such as 0.7 or 0.8) set for judgment. If the similarity exceeds the threshold, the recognition is considered successful; otherwise, it is recorded as a recognition failure.

[0061] S102, after the number of recognition failures reaches the preset failure threshold, the optimal IR face image is obtained from multiple sets of IR face images and uploaded to the cloud server;

[0062] The preset failure threshold refers to the number of consecutive recognition failures, which can typically be set to 3-5 times. When the number of consecutive recognition failures reaches this threshold, the process of acquiring the optimal IR face image is triggered.

[0063] Specifically, obtaining the optimal IR face image includes the following subdivision steps:

[0064] S1, For IR face images that fail to be recognized, count the number of times the face ID with the TOP score appears each time the recognition fails, and calculate the average score of each face ID;

[0065] S2, obtain the target face ID with the most TOP1 occurrences and the highest average score;

[0066] S3. If the ratio of the occurrence frequency of the target face ID to all failed face images is greater than a preset ratio, and its corresponding average score exceeds a preset score, select the image with the best quality from the IR images corresponding to the target face ID to obtain the optimal IR image.

[0067] Specifically, the process of obtaining the optimal IR face image involves selecting the highest quality IR face image from multiple sets of failed recognition attempts. The quality evaluation criteria include factors such as face sharpness, illumination uniformity, and pose angle.

[0068] The specific implementation method includes: for IR face images that failed to be recognized, counting the number of times the top-scoring face ID appeared in each failed recognition, and calculating the average score of each face ID. The target face ID with the most occurrences and the highest average score is obtained; if the ratio of the occurrences of the target face ID to all failed face images is greater than a preset ratio (e.g., 0.6), and its corresponding average score exceeds a preset score (e.g., 0.5), the image with the best quality is selected from the IR images corresponding to that target face ID to obtain the optimal IR image.

[0069] Optionally, during the process of uploading the optimal IR face image to the cloud server, the IR terminal device compresses the image (usually to 100-300KB) and sends it to the cloud server via a secure encrypted channel (such as HTTPS or TLS). Additionally, metadata such as the device ID and timestamp are included during the upload process for processing by the cloud server.

[0070] S103, the cloud server extracts the IR face features of the optimal IR face image through a large model, and based on a preset fusion strategy, fuses the IR face features with the RGB face features of the same user that are stored in advance to obtain the fused features and sends them to the terminal device.

[0071] It should be noted that before the cloud server extracts the IR features of the optimal IR face image through the large model, it needs to perform a quality assessment on the optimal IR face image. If the quality assessment is satisfactory, the optimal IR face image is searched across the entire RGB library, and based on the cross-modal search results, the RGB face features belonging to the same user as the IR face image are determined.

[0072] Specifically, the quality assessment includes detecting whether the face is complete, whether it is obstructed, and whether the lighting is appropriate. If the quality assessment fails, the cloud server will return an error message to the terminal device; if the quality assessment passes, subsequent processing will continue.

[0073] In this embodiment, the cloud server uses a large model to perform cross-modal retrieval of the optimal IR face image within the full RGB database to determine the RGB face features belonging to the same user as the IR face image. The full RGB database refers to the RGB face feature library of all users pre-stored by the cloud server, typically containing feature data from tens of thousands to millions of users. Cross-modal retrieval refers to the process of feature matching between different modalities (IR and RGB), usually employing specialized cross-modal matching algorithms such as HFR (Heterogeneous Face Recognition) or CMFH (Cross-Modal Face Hallucination).

[0074] The large models used on cloud servers are computationally powerful, parameter-intensive deep neural network models, such as ResNet-101 or SENet-154, with model sizes typically ranging from 100 to 500 MB. These large models can extract more refined and robust in-line facial features, with feature vectors typically ranging from 512 to 2048 dimensions. The inference time for these large models is generally within 200-1000 milliseconds, and because they run on the server side, they have lower performance requirements for the terminal devices.

[0075] Furthermore, based on a preset fusion strategy, the fusion of IR facial features with pre-stored RGB facial features of the same user includes:

[0076] S1. Based on the preset evaluation criteria, obtain the image quality scores corresponding to IR face features and RGB face features respectively;

[0077] S2, calculate the dynamic weights of features based on the image quality score and the modal weights corresponding to the pre-configured IR and RGB face features;

[0078] S3 performs an initial weighted operation based on IR and RGB face features and their corresponding dynamic feature weights to obtain the fused features.

[0079] The process involves fusing IR and RGB face features based on a preset fusion strategy to generate a fused feature that can adapt to both IR and RGB modalities. Specifically, the method involves obtaining image quality scores for IR and RGB face features according to preset evaluation criteria. These criteria include sharpness (0-1), pose (0-1), and illumination (0-1), which are combined to obtain a final quality score (0-1). Based on the image quality score and the pre-configured modal weights for IR and RGB face features, dynamic feature weights are calculated. Modal weights refer to the basic weights of different modal features during the fusion process; typically, the IR modal weight is 0.3-0.5, and the RGB modal weight is 0.5-0.7. Finally, an initial weighted calculation is performed on the IR and RGB face features and their corresponding dynamic feature weights to obtain the fused feature.

[0080] The specific fusion process can be represented by the following formula:

[0081] fts_update=[w1·fts_RGB+w1·fts_IR]

[0082] Among them, fts_update is the fused feature, fts_RGB and fts_IR are RGB face features and IR face features respectively, and w1 and w2 are the weights of RGB features and IR features calculated based on image quality and preset modality weights respectively.

[0083] Specifically, the dynamic weight calculation is expressed by the following formula:

[0084] w1 = θ·Q_RGB / (Q_RGB + Q_IR + ε)

[0085] w2=(1-θ)·Q_IR / (Q_RGB+Q_IR+ε)

[0086] Where Q_RGB / Q_IR is the RGB / IR image quality score (0-1), θ is the preset modal weight (0.5-0.7), and ε is a minimal constant.

[0087] Optionally, to further improve the quality of the fused features, this method also considers the influence of environmental factors and historical features. The actual illumination intensity corresponding to the acquisition of the optimal IR image is obtained using a light sensor deployed in the IR terminal device. Based on the actual illumination intensity and a preset reference illumination intensity, an environmental compensation factor is obtained according to a first preset weighting rule.

[0088] It is understandable that, in cases where feature fusion is not performed for the first time, the latest fused feature value generated by the current fusion and the historical fused feature values ​​from the previous fusion are obtained through a feature storage module deployed on the server. The feature storage module is a high-performance database used to store the user's historical feature data, including fused feature values ​​and feature increments. Based on the latest fused feature value, historical fused feature values, and stored historical feature increments, the current feature increment is obtained through a second preset weighted calculation rule.

[0089] The result of the initial weighted calculation is combined with the product of the environmental compensation factor and the current feature increment to obtain the final fused feature. The dimension of the final fused feature is the same as that of the initial weighted calculation result, which is usually 512-2048 dimensions.

[0090] It is understandable that, considering environmental compensation and the impact of historical characteristics, the feature fusion process can be represented by the following formula:

[0091] fts_update=[w1·fts_RGB+w2·fts_IR]+λ·Δfts

[0092] Where λ is the environmental compensation factor, λ=1+0.5·tanh((E-E0) / 1000), E represents the current light intensity (lux), E0 is the reference light intensity, and Δfts is the special increment, Δfts=0.3·(fts_current-fts_previous)+0.7·Δfts_old.

[0093] Wherein, λ reflects the degree of interference of the environment, such as lighting, on feature extraction. The greater the interference, the greater the deviation of λ from 1 (e.g., λ>1 under strong light, λ<1 under weak light). Δfts reflects the dynamic trend of feature updates (e.g., the amount of change of the current feature relative to the historical features).

[0094] It is understandable that the degree of environmental interference directly affects the "effective amplitude" of feature updates. For example, under strong light conditions, the IR feature error is large, and the feature increment Δfts needs to be amplified by λ to more aggressively correct the feature offset. Conversely, when the illumination is suitable, λ≈1, and Δfts is updated according to the original amplitude to avoid over-adjustment.

[0095] Specifically, when the light intensity (E) deviates from the reference value (E0), the λ-pass tanh function generates a compensation coefficient. For example, when the light is strong (E > E0), λ > 1, making the fused feature more biased towards the RGB feature (w1 increases), reducing the error caused by overexposure of the IR feature; when the light is weak (E < E0), λ < 1, moderately enhancing the weight compensation of the IR feature and utilizing the imaging stability of infrared light in low light. Additionally, the non-linear characteristic of the tanh function enables λ to be smoothly adjusted when the ambient light changes slowly, avoiding feature mutations caused by small fluctuations in the environment. For example, when the light intensity changes by 100 lux, the change in λ is much smaller than when it changes by 1000 lux, which conforms to the human eye's perception characteristics of ambient light.

[0096] Furthermore, Δfts uses the weighted formula 0.3·(current difference) + 0.7·historical increment to make the feature update have "memory": the weight of 0.3 emphasizes the current feature change and quickly responds to new data; the weight of 0.7 retains the historical increment trend, avoiding feature oscillations caused by single noise (such as an IR image being distorted by sudden strong light in a certain frame, and the historical increment can smooth this error).

[0097] For example, when the user's makeup causes the RGB feature to shift, after the IR feature uploaded by the terminal is fused by the platform's large model, Δfts will accumulate the "feature difference after makeup", making the subsequent fused feature AB gradually adapt to the change and avoiding recognition jumps caused by one-time updates.

[0098] It can be understood that λ adjusts the compensation intensity according to the environment, making the model more "aggressive" in correcting features under harsh conditions; Δfts maintains the "smoothness" of feature updates through historical weighting, avoiding model oscillations caused by aggressive adjustments. The multiplication operation between the two essentially couples the "degree of environmental disturbance" and the "feature change requirement", enabling the update strategy of the fused feature to have "environmental perception ability", which conforms to the physical logic of "environmental impact on feature quality" in cross-modal recognition.

[0099] Optionally, in order to further improve the robustness of feature fusion, in this embodiment, the influence of infrared interference is also considered during the calculation of the environmental compensation factor. Through the image detection and processing module deployed in the IR terminal device, abnormal infrared light spots in the optimal IR image are identified, and the infrared interference intensity is determined based on the abnormal infrared light spots. The infrared interference intensity is usually represented by a numerical value from 0 to 1, where 0 indicates no interference and 1 indicates the strongest interference.

[0100] Calculate the first environmental compensation factor and the second environmental compensation factor respectively based on the light illumination factor and the infrared interference factor, and perform a weighted operation based on the first environmental compensation factor and the second environmental compensation factor to obtain the environmental compensation factor that comprehensively considers the influence of various factors.

[0101] At this point, the formula for calculating λ can be expressed as λ=1+w1·f(E)+w2·f(T), where w1~w2 represent the weights of each factor (e.g., illumination weight 0.6, infrared interference weight 0.4), which can be determined through training with historical data.

[0102] It is understandable that the above methods integrate multiple factors such as illumination and infrared interference into a comprehensive compensation factor according to weights. By determining the weights through training, the influence of each factor can be quantified, avoiding the one-sidedness of single compensation. When multiple factors cause concurrent interference, the compensation intensity is allocated according to weights to adapt to the characteristics of different scene environments and improve the stability of cross-environment recognition.

[0103] Step S103: After receiving the IR face photo uploaded by the terminal on the platform and successfully matching it with the large model, the RGB features are fused with the re-extracted IR features to form a new feature. After being sent to the terminal, the success rate of face recognition in cross-modal (such as IR camera and RGB registered photo) and low-quality image (such as blurry or abnormal lighting) scenarios can be optimized.

[0104] S104, the terminal device receives the updated IR face image, extracts the features of the updated IR face image in real time through a small model, compares the features extracted in real time with the fused features, and outputs the recognition result.

[0105] During the process of distributing the fused features to the terminal device, the cloud server compresses the feature vector (usually to 10-50KB) and sends it to the terminal device through a secure encrypted channel. Metadata such as feature ID and timestamp are attached during the distribution process to facilitate processing by the terminal device.

[0106] The terminal device compares the extracted features with the fused features in real time and outputs the recognition result. Feature comparison methods typically use cosine similarity or Euclidean distance calculations, with a threshold (such as 0.7 or 0.8) set for judgment. If the similarity exceeds the threshold, a successful recognition result is output, including user ID, name, and other information; otherwise, a recognition failure result is output.

[0107] also, Figure 2 This is a flowchart of another face recognition method based on cross-modal face feature fusion according to an embodiment of this application.

[0108] Through steps S101 to S104 above, the face recognition method based on cross-modal face feature fusion provided in this embodiment achieves efficient and accurate face recognition through the collaborative work of the IR terminal device and the cloud server. The method first performs preliminary recognition on the IR terminal device using a small model. When recognition failures reach a preset threshold, the optimal IR face image is selected and uploaded to the cloud server. The cloud server extracts IR face features using a large model and, based on a preset fusion rule combining environmental parameter compensation factors and feature increments, fuses these features with pre-stored RGB face features to generate a fused feature, which is then returned to the terminal device. The terminal device uses this fused feature for subsequent recognition, greatly improving the recognition accuracy.

[0109] This embodiment also provides a face recognition system based on cross-modal face feature fusion. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0110] Figure 3 This is a structural block diagram of a face recognition system based on cross-modal face feature fusion according to an embodiment of this application, such as... Figure 3 As shown, the system is implemented through an IR terminal device and a cloud server, including: an IR terminal device 30 and a cloud server 31, wherein,

[0111] IR terminal device 30 is used to receive multiple sets of IR face images, extract the IR face features of each IR face image through a small model, and perform face recognition based on the IR face features.

[0112] Additionally, after the number of recognition failures reaches a preset failure threshold, the optimal IR face image is obtained from multiple sets of IR face images and uploaded to the cloud server.

[0113] The cloud server 31 is used to extract the IR face features of the optimal IR face image through a large model, and based on a preset fusion strategy, fuse the IR face features with the RGB face features of the same user that are stored in advance to obtain the fused features and send them to the terminal device.

[0114] Terminal device 30 is also used to receive updated IR face images, extract features of updated IR face images in real time using a small model, compare features based on real-time extracted features and fused features, and output recognition results.

[0115] The system described above achieves efficient and accurate face recognition through the collaborative work of IR terminal devices and cloud servers. The method first uses a small model on the IR terminal device for preliminary recognition. When recognition failures reach a preset threshold, the optimal IR face image is selected and uploaded to the cloud server. The cloud server uses a large model to extract IR face features and, based on a preset fusion rule incorporating environmental parameter compensation factors and feature increments, fuses these features with pre-stored RGB face features to generate a fused feature, which is then returned to the terminal device. The terminal device uses this fused feature for subsequent recognition, significantly improving recognition accuracy.

[0116] In one embodiment, Figure 4 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 4 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 4 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system and computer programs to run, the computer programs are executed by the processor to implement a face recognition method based on cross-modal face feature fusion, and the database stores data.

[0117] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0118] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0119] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A face recognition method based on cross-modal face feature fusion, characterized in that, The method is implemented through an IR terminal device and a cloud server, and comprises the following steps: The IR terminal device receives a plurality of groups of IR face images, extracts IR face features of each IR face image through a small model, and performs face recognition based on the IR face features; After the number of recognition failures reaches a preset failure threshold, an optimal IR face image is obtained from the plurality of groups of IR face images and uploaded to the cloud server; The cloud server extracts IR face features of the optimal IR face image through a large model, fuses the IR face features with pre-stored RGB face features of the same user based on a preset fusion strategy, obtains fused features, and sends the fused features to the IR terminal device; The IR terminal device receives updated IR face images, extracts features of the updated IR face images in real time through the small model, compares the real-time extracted features with the fused features, and outputs a recognition result; The method further comprises the following steps: In the case of non-first feature fusion, the latest fused feature value generated by current fusion and the historical fused feature value obtained by previous fusion are obtained through a feature storage module deployed in the cloud server; Based on the latest fused feature value, the historical fused feature value, and the stored historical feature increment, a current feature increment is obtained through a second preset weighting operation rule; The product of the result of the initial weighting operation, the environmental compensation factor, and the current feature increment is combined to obtain the fused features, wherein the result of the initial weighting operation is obtained by weighting operation based on the IR face features and the RGB face features and their corresponding feature dynamic weights, and the feature dynamic weights are calculated based on image quality scores corresponding to the IR face features and the RGB face features and pre-configured modal weights.

2. The method of claim 1, wherein, Before the IR terminal device receives the plurality of groups of IR face images, the cloud server stores the RGB face features in a local library and sends the RGB face features to the IR terminal device for persistent storage, and the face recognition based on the IR face features comprises the following steps: The IR terminal device performs face recognition by comparing each IR face feature with the locally stored RGB face features.

3. The method of claim 2, wherein, Based on the preset fusion strategy, the IR face features are fused with the pre-stored RGB face features of the same user, which comprises the following steps: According to a preset evaluation standard, image quality scores corresponding to the IR face features and the RGB face features are obtained respectively; According to the image quality scores and pre-configured modal weights corresponding to the IR face features and the RGB face features, feature dynamic weights are calculated; The initial weighting operation is performed based on the IR face features, the RGB face features, and their corresponding feature dynamic weights to obtain the fused features.

4. The method of claim 3, wherein, In the process of calculating the environmental compensation factor, the method further comprises the following steps: An image detection processing module deployed in the IR terminal device is used to identify abnormal infrared light spots in the optimal IR face image, and determine the infrared interference intensity according to the abnormal infrared light spots; A first environment compensation factor is calculated according to the actual illumination intensity and a preset reference illumination intensity, and a second environment compensation factor is calculated according to the infrared interference intensity and a reference interference threshold value; A weighting operation is performed based on the first environment compensation factor and the second environment compensation factor to obtain an environment compensation factor that comprehensively considers the influence of multiple factors.

5. The method of claim 1, wherein, Obtaining the optimal IR face image includes: For the IR face image for which the recognition fails, the number of times that the face ID with the highest score (TOP1) appears each time the recognition fails is counted, and the average score of each face ID is calculated; A target face ID with the highest number of times and the highest average score is obtained. If the ratio of the number of times that the target face ID appears to the number of images of all failed faces is greater than a preset ratio, and the average score corresponding to the target face ID exceeds a preset score, the optimal image is selected from the IR image corresponding to the target face ID, and the optimal IR face image is obtained.

6. The method of claim 1, wherein, Before the cloud server extracts the IR features of the optimal IR face image by using a large model, the method further includes: Performing quality judgment on the optimal IR face image; In the case of passing the quality judgment, the optimal IR face image is searched in a full-quantity RGB library by using a large model, and RGB face features of the same user as the IR face image are determined according to the cross-modal search result.

7. A face recognition system based on cross-modal face feature fusion, characterized in that, The system is implemented by using an IR terminal device and a cloud server, and includes: The IR terminal device is configured to receive multiple groups of IR face images, extract IR face features of each IR face image by using a small model, and perform face recognition based on the IR face features; After the number of failed recognitions reaches a preset failure threshold, the optimal IR face image is obtained from the multiple groups of IR face images and uploaded to the cloud server; The cloud server is configured to extract the IR face features of the optimal IR face image by using a large model, fuse the IR face features with pre-stored RGB face features of the same user based on a preset fusion strategy, obtain a fused feature, and send the fused feature to the IR terminal device; The IR terminal device is further configured to receive an updated IR face image, extract features of the updated IR face image in real time by using the small model, perform feature comparison based on the real-time extracted features and the fused feature, and output a recognition result; The actual illumination intensity corresponding to the optimal IR face image is obtained when the optimal IR face image is collected, and an environment compensation factor is obtained according to a first preset weighting operation rule based on the actual illumination intensity and a preset reference illumination intensity; In the case of non-first feature fusion, the latest fused feature value generated by current fusion and the historical fused feature value obtained by completing the fusion last time are obtained by using a feature storage module deployed in the cloud server; Based on the latest fusion feature value, the historical fusion feature value, and the stored historical feature increment, a current feature increment is obtained through a second preset weighting operation rule; The product of the environmental compensation factor and the current feature increment is combined with the result of the initial weighting operation to obtain the fusion feature, wherein the result of the initial weighting operation is obtained according to the IR face feature and the RGB face feature and the corresponding feature dynamic weight weighting operation, and the feature dynamic weight is calculated according to the image quality scores corresponding to the IR face feature and the RGB face feature and the pre-configured modal weight.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method in any one of claims 1 to 6.

9. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the method in any one of claims 1 to 6.

Images