Image recognition method, image recognition model training method and related device

Abstract

The present invention provides an image recognition method, a training method for an image recognition model, and related devices, wherein the image recognition method comprises: the present invention divides the image to be recognized into a first area, a second area, and a third area; extracts the texture features of the first area, the high-dimensional features of the second area, and the frequency domain features of the third area; fuses the texture features, the high-dimensional features, and the frequency domain features, and obtains the recognition result of the image to be recognized based on the fusion result. The method divides the image to be processed into multiple areas, and extracts different types of features for the multiple areas respectively. When the application scenario changes and causes some features to fail, other features can still ensure the effectiveness of the entire image recognition method, thereby improving the robustness of image recognition.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to an image recognition method, an image recognition model training method and related devices. Background Art

[0002] The development of digital technology has brought human society into the era of artificial intelligence, and various biometric technologies have been widely used in people's daily lives. Subsequently, biometric technologies based on human faces have been widely used in user identity authentication for systems such as gate access, mobile phone unlocking, and attendance check-in. However, these face recognition authentication systems have huge security issues. For example, when a user's face information is stolen, it is projected onto a mobile phone screen, printed as a photo, or made into a mask to disguise the user's face for identification and authentication, attacking the identity authentication system. If the face authentication system does not accurately detect the disguised face information, it will cause immeasurable losses to the user, so it is necessary to accurately detect the face to be authenticated. Summary of the invention

[0003] The present invention provides an image recognition method, an image recognition model training method and related devices, and the method can improve the robustness of image recognition results.

[0004] To solve the above technical problems, the first technical solution provided by the present invention is: to provide an image recognition method, comprising: dividing the image to be recognized into a first area, a second area and a third area; extracting the texture features of the first area, the high-dimensional features of the second area, and the frequency domain features of the third area; fusing the texture features, the high-dimensional features and the frequency domain features, and obtaining the recognition result of the image to be recognized based on the fusion result.

[0005] Among them, the step of extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region includes: processing the image to be identified to obtain a texture map and a spectrum map of the image to be identified; performing feature extraction on the first region based on the texture map to obtain the texture features, performing feature extraction on the third region based on the spectrum map to obtain the frequency domain features, and performing feature extraction on the second region to obtain the high-dimensional features.

[0006] Among them, the step of dividing the image to be identified into a first area, a second area and a third area includes: obtaining a category activation heat map corresponding to the image to be identified; and dividing the image to be identified into the first area, the second area and the third area based on the category activation heat map.

[0007] Among them, the step of dividing the image to be identified into the first area, the second area and the third area based on the category activation heat map includes: determining the key area based on the intensity of the pixel points in the category activation heat map; dividing the key area based on preset rules to obtain the first area, the second area and the third area.

[0008] The first area at least includes a mouth area and a nose area of ​​the target in the image to be identified; the second area at least includes an eye area of ​​the target; and the third area at least includes a face area of ​​the target.

[0009] Among them, the step of fusing the texture features, the high-dimensional features and the frequency domain features and obtaining the recognition result of the image to be recognized based on the fusion result includes: fusing the texture features, the high-dimensional features and the frequency domain features to obtain a fusion result; obtaining the liveness confidence of the image to be recognized based on the fusion result, and in response to the liveness confidence being greater than a confidence threshold, the recognition result of the image to be recognized is a liveness image.

[0010] The step of obtaining the image to be identified includes: performing quality detection on the image to be processed, removing the image to be processed whose quality score is less than a threshold, and using the remaining image to be processed as the image to be identified.

[0011] To solve the above technical problems, the second technical solution provided by the present invention is: to provide an image recognition device, comprising: a region division module, used to divide the image to be recognized into a first region, a second region and a third region; a feature extraction module, used to extract the texture features of the first region, the high-dimensional features of the second region, and the frequency domain features of the third region; and a recognition module, used to fuse the texture features, the high-dimensional features and the frequency domain features, and obtain the recognition result of the image to be recognized based on the fusion result.

[0012] To solve the above technical problems, the third technical solution provided by the present invention is: to provide a training method for an image recognition model, the image recognition model is used to implement any of the above image recognition methods; the training method includes: using an initial model to divide each sample image in a training sample set into a first area, a second area and a third area; extracting texture features of the first area, high-dimensional features of the second area, and frequency domain features of the third area; using the texture features, the high-dimensional features and the frequency domain features to train the initial model, thereby obtaining the image recognition model.

[0013] Among them, the step of extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region includes: processing the sample image to obtain a texture map and a spectrum map of the sample image; performing feature extraction on the first region based on the texture map to obtain the texture features, performing feature extraction on the third region based on the spectrum map to obtain the frequency domain features, and performing feature extraction on the second region to obtain the high-dimensional features.

[0014] Among them, the step of using the texture features, the high-dimensional features and the frequency domain features to train the initial model and then obtain the image recognition model includes: using the mean square error function to calculate the first loss amount based on the texture features and the texture map, using the mean square error function to calculate the second loss amount based on the frequency domain features and the spectrum map, using the mean square error function to calculate the third loss amount of the high-dimensional features based on the high-dimensional features and the preset labels of the sample image; fusing the first loss amount, the second loss amount and the third loss amount, and obtaining the fourth loss amount based on the fusion result; performing weighted calculation on the first loss amount, the second loss amount, the third loss amount and the fourth loss amount to obtain the fifth loss amount; using the fifth loss amount to adjust the parameters of the initial model, and then obtain the image recognition model.

[0015] To solve the above technical problems, the fourth technical solution provided by the present invention is: to provide a training device for an image recognition model, comprising: a division module, used to divide each sample image in a training sample set into a first region, a second region and a third region using an initial model; an extraction module, used to extract texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region; a training module, used to train the initial model using the texture features, the high-dimensional features and the frequency domain features, and thereby obtain the image recognition model.

[0016] To solve the above technical problems, the fifth technical solution provided by the present invention is: to provide an electronic device, comprising: a memory and a processor, wherein the memory stores program instructions, and the processor calls the program instructions from the memory to execute any of the above methods.

[0017] In order to solve the above technical problems, the sixth technical solution provided by the present invention is: providing a computer-readable storage medium storing a program file, and the program file can be executed to implement any of the above methods.

[0018] The beneficial effect of the present invention is different from the prior art. The present invention divides the image to be identified into a first area, a second area and a third area; extracts the texture features of the first area, the high-dimensional features of the second area, and the frequency domain features of the third area; fuses the texture features, the high-dimensional features and the frequency domain features, and obtains the recognition result of the image to be identified based on the fusion result. The method divides the image to be processed into multiple areas, and extracts different types of features for the multiple areas respectively. When the application scenario changes and causes some features to fail, other features can still ensure the effectiveness of the entire image recognition method, thereby improving the robustness of image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0020] Figure 1 A schematic diagram of a flow chart of an embodiment of an image recognition method of the present invention;

[0021] Figure 2 A schematic diagram of the structure of an embodiment of an image recognition device of the present invention;

[0022] Figure 3 A flowchart of an embodiment of a method for training an image recognition model of the present invention;

[0023] Figure 4 A schematic diagram of the structure of an embodiment of a training device for an image recognition model of the present invention;

[0024] Figure 5 It is a structural schematic diagram of an embodiment of an electronic device of the present invention;

[0025] Figure 6 It is a schematic diagram of the structure of an embodiment of a computer-readable storage medium of the present invention. DETAILED DESCRIPTION

[0026] The development of digital technology has brought human society into the era of artificial intelligence, and various biometric technologies have been widely used in people's daily lives. Subsequently, biometric technologies based on human faces have been widely used in user identity authentication for gate access, mobile phone unlocking, attendance check-in and other systems. However, these face recognition authentication systems have huge security issues. For example, when the user's face information is stolen, it is projected onto the mobile phone screen, printed as a photo, or made into a mask to disguise the user's face for identification and authentication, attacking the identity authentication system. If the face authentication system does not accurately detect the disguised face information, it will cause immeasurable losses to the user. Therefore, it is necessary to perform liveness detection on the face to be authenticated.

[0027] Face liveness detection technology is a technology that uses facial image information analysis to determine whether the face captured by the camera is a disguised face (mainly including printed face pictures, faces on electronic device screens, and face masks) or a real live face. The current mainstream liveness verification methods include lip shape interactive verification, coordinated action verification, face heart rate judgment, image optical flow judgment, and texture analysis of face images. In addition, there are also methods for judging by using a depth camera to obtain a face depth map, or using an infrared camera to obtain the infrared reflection feature analysis method of the face.

[0028] The traditional liveness detection method is to extract different artificially designed features from the face to be verified, such as local binary patterns, scale-invariant features, etc., and fuse the above features or classify them separately through support vector machines. However, with the changes in application scenarios, the designed artificial features are difficult to express the liveness information of all scenarios. Due to the effectiveness of deep learning in extracting facial features, it is widely used in the field of liveness detection. Compared with traditional methods, deep learning methods based on convolutional neural networks can be trained with a large amount of data to extract effective information that can describe all liveness, greatly improving the security of liveness detection algorithms. Therefore, the ability to realize the extraction and fusion of traditional features and different features on a lightweight network can improve the generalization ability of the network and its robustness in actual application scenarios.

[0029] The prior art proposes a liveness detection method, the design key points of which are to obtain pixel features of the image to be processed based on the image to be processed including the detection object, perform edge detection on the image to be processed to extract the edge of the detection object as the first image, perform normalization processing on the image to be processed to obtain the second image; input the image to be processed, the first image and the second image into a deep neural network model, so that the deep neural network model outputs a first judgment result of whether the detection object is alive; input the first judgment result and the pixel features into a first classifier, so that the first classifier outputs a second judgment result of whether the detection object is alive, and determines whether the detection object is alive according to the second judgment result. The main disadvantage of its design is that the algorithm depends on the edge detection effect of the image to be processed, and whether the edge detection result of the image to be detected is robust directly affects the liveness detection result. In addition, more importantly, in liveness detection, the edge features in the image to be detected are not the features of all non-liveness attacks, and similar edges also exist in the live image to be detected under complex backgrounds, so this method is easy to fail in other attack types and complex scenarios.

[0030] The prior art also proposes a liveness detection method, the design key points of which are to process the image to be processed through a neural network to obtain the probability that multiple pixels of the image to be processed correspond to the prosthesis; determine the predicted face area in the image to be processed; based on the probability that multiple pixels of the image to be processed correspond to the prosthesis and the predicted face area, obtain the liveness detection result of the image to be processed. The main disadvantage of its design is that it is necessary to separately design a fully convolutional neural network based on pixel-level output. Compared with the network that obtains multi-dimensional features after downsampling and connects the fully connected layer, it has a large number of parameters and is relatively time-consuming. In the liveness detection task, in normal use scenarios, the pixel-by-pixel characteristics of live and non-live images are not obvious. The entire image is labeled and trained pixel by pixel, which depends on the training data and has poor generalization to the data of actual application scenarios.

[0031] Therefore, the liveness detection methods in the prior art need to be improved. The present application proposes an image recognition method and a training method for an image recognition model. The application scenarios of the image recognition method are more extensive and the results are more stable. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] See also Figure 1 , is a flow chart of a first embodiment of the image recognition method of the present invention, which specifically includes:

[0033] Step S11: Divide the image to be identified into a first area, a second area and a third area.

[0034] In one embodiment, the image to be identified can be randomly divided into at least two areas, and the specific division method is not limited. For example, if the image to be identified only includes a face image, it can be divided evenly or non-evenly from left to right, and then divided into at least two areas; or, it can also be divided evenly or non-evenly from top to bottom, and then divided into at least two areas. Specifically, the image to be identified can be divided into 2 areas, or it can be divided into 3 or 4 areas, and there is no specific limitation. For example, in a specific embodiment, the image to be identified is divided into a first area, a second area, and a third area.

[0035] In one embodiment, in order to improve the robustness of the image recognition result, the image to be recognized is processed to obtain a class activation heat map corresponding to the image to be recognized, and the image to be recognized is divided into a first area, a second area, and a third area based on the class activation heat map. Specifically, if the image recognition method of the present application is applied to liveness detection, the trained liveness detection model is used to process the image to be recognized to obtain a class activation heat map of the distribution of facial liveness features.

[0036] Furthermore, in one embodiment, the key area is determined based on the intensity of the pixel points in the class activation heat map, and then the key area is divided based on a preset rule to obtain a first area, a second area, and a third area. Specifically, by performing statistics on a large number of images, the more sensitive key areas are determined based on the class thermal activation map. For each image, the image key area S h、w、s . Calculate the pixel intensity of each pixel in the category thermal activation map and get the intensity mean I mean , the intensity is greater than I mean Pixels of intensity greater than I mean The pixels of the key area are formed.

[0037] After obtaining the key area, the key area is divided based on the preset rules to obtain the first area, the second area and the third area. In one embodiment, the divided first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target. In another embodiment, the first area, the second area and the third area may also include other areas of the target, but are not limited thereto. This embodiment is only described by taking the example that the first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target.

[0038] In one embodiment of the present application, in order to reduce the amount of calculation, the quality detection of the images to be processed is performed first, and the images to be processed whose quality scores are less than the threshold are removed, and the remaining images to be processed are used as images to be identified. It can be understood that the images to be processed whose quality scores are less than the threshold are directly determined as non-living images, thereby reducing the number of images to be identified and reducing the amount of calculation.

[0039] Step S12: extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region.

[0040] Specifically, different types of features are extracted from the first region, the second region, and the third region, respectively. For example, texture features, frequency domain features, high-dimensional features, low-dimensional features, semantic features, spatial features, etc. can be extracted from at least the first region, the second region, and the third region, without specific limitation.

[0041] This embodiment is described by taking the first region, the second region and the third region as examples. Specifically, the texture features of the first region are extracted, the high-dimensional features of the second region are extracted, and the frequency domain features of the third region are extracted.

[0042] Specifically, the image to be identified is processed to obtain a texture map and a spectrum map of the image to be identified. Feature extraction is performed on the first region based on the texture map to obtain texture features of the first region. In a specific embodiment, the texture map can be used to supervise the convolutional neural network CNN to extract features from the first region to obtain texture features of the first region. Feature extraction is performed on the third region based on the spectrum map to obtain frequency domain features. In a specific embodiment, the spectrum map can be used to supervise the convolutional neural network CNN to extract features from the third region to obtain frequency domain features. Feature extraction is performed on the second region to obtain high-dimensional features. Specifically, feature extraction is performed on the second region using a convolutional neural network CNN to obtain high-dimensional features.

[0043] In one embodiment, a local binary pattern (LBP) of the image to be identified is obtained, and the local binary pattern represents a texture map of the image to be identified. A Fourier spectrum (FS) of the image to be identified is obtained, and the Fourier spectrum represents a spectrum map of the image to be identified.

[0044] Step S13: Fusing the texture features, high-dimensional features, and frequency domain features, and obtaining a recognition result of the image to be recognized based on the fusion result.

[0045] In this embodiment, different types of features are fused, and a recognition result of the image to be recognized is obtained based on the fusion result.

[0046] Specifically, the texture features, frequency domain features and high-dimensional features are fused, and the recognition result of the image to be recognized is obtained based on the fused result. For example, in one embodiment, target detection can be performed based on the fused result, or the image to be recognized can be classified based on the fused result.

[0047] In a specific embodiment of the present application, liveness detection is performed based on the fused result to determine whether the image to be identified is a liveness image. Specifically, texture features, high-dimensional features, and frequency domain features are fused to obtain a fusion result, and the liveness confidence of the image to be identified is obtained based on the fusion result. In response to the liveness confidence being greater than a confidence threshold, the identification result of the image to be identified is a liveness image.

[0048] This application can not only judge living and non-living objects based on texture features, but also combines feature information in the frequency domain and higher-dimensional semantic information to enhance the robustness of the algorithm. Therefore, the application scenarios of this proposal are more extensive and the results are more stable.

[0049] The image recognition method of this embodiment divides the image to be processed into a first area, a second area and a third area, and extracts different types of features for the first area, the second area and the third area, for example, extracting texture features of the first area, high-dimensional features of the second area, and frequency domain features of the third area; when the application scenario changes and causes some features to fail, other features can still ensure the effectiveness of the entire image recognition method, thereby improving the robustness of image recognition. In addition, in this embodiment, a convolutional neural network is used to learn low-level texture information and activity information in the frequency domain in a targeted manner through auxiliary supervised learning, extract deeper features in the network high-dimensional space, and then fuse low-level features (texture features, frequency domain features) and high-level features (high-dimensional features), which are more descriptive and more robust than the features extracted by other methods.

[0050] See also Figure 2 , is a schematic diagram of the structure of an embodiment of the image recognition device of the present invention, which specifically includes: a region division module 21, a feature extraction module 22 and a recognition module 23.

[0051] The region division module 21 is used to divide the image to be identified into at least two regions.

[0052] In one embodiment, the region division module 21 may randomly divide the image to be identified into a first region, a second region, and a third region.

[0053] In one embodiment, in order to improve the robustness of the image recognition result, the region division module 21 processes the image to be recognized to obtain a category activation heat map corresponding to the image to be recognized, and divides the image to be recognized into a first region, a second region, and a third region based on the category activation heat map. Specifically, if the image recognition method of the present application is applied to liveness detection, the trained liveness detection model is used to process the image to be recognized to obtain a category activation heat map of the distribution of facial liveness features.

[0054] Further, in one embodiment, the region division module 21 determines the key region based on the intensity of the pixel points in the category activation heat map, and then divides the key region based on a preset rule to obtain a first region, a second region, and a third region. Specifically, by performing statistics on a large number of images, the more sensitive key region is determined based on the category heat activation map. For each image, the image key region S is obtained. h、w、s . Calculate the pixel intensity of each pixel in the category thermal activation map and get the intensity mean I mean , the intensity is greater than I mean Pixels of intensity greater than I mean The pixels of the key area are formed.

[0055] After obtaining the key area, the area division module 21 divides the key area based on a preset rule to obtain a first area, a second area, and a third area. In one embodiment, the divided first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target. In another embodiment, the first area, the second area, and the third area may also include other areas of the target, but are not limited thereto. This embodiment is only described by taking the example that the first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target.

[0056] In one embodiment of the present application, in order to reduce the amount of calculation, the quality detection of the images to be processed is performed first, and the images to be processed whose quality scores are less than the threshold are removed, and the remaining images to be processed are used as images to be identified. It can be understood that the images to be processed whose quality scores are less than the threshold are directly determined as non-living images, thereby reducing the number of images to be identified and reducing the amount of calculation.

[0057] The feature extraction module 22 is used to extract texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region.

[0058] Specifically, the feature extraction module 22 extracts different types of features from at least two regions respectively. For example, texture features, frequency domain features, high-dimensional features, low-dimensional features, semantic features, spatial features, etc. can be extracted from the first region, the second region, and the third region, without specific limitation.

[0059] This embodiment is described by taking the first region, the second region and the third region as examples. Specifically, the feature extraction module 22 extracts texture features of the first region, extracts high-dimensional features of the second region and extracts frequency domain features of the third region.

[0060] Specifically, the feature extraction module 22 processes the image to be identified to obtain a texture map and a spectrum map of the image to be identified. The feature extraction module 22 performs feature extraction on the first region based on the texture map, and then obtains the texture features of the first region. In a specific embodiment, the texture map can be used to supervise the convolutional neural network CNN to extract features from the first region, and then obtain the texture features of the first region. Based on the spectrum map, feature extraction is performed on the third region to obtain frequency domain features. In a specific embodiment, the spectrum map can be used to supervise the convolutional neural network CNN to extract features from the third region to obtain frequency domain features. Feature extraction is performed on the second region to obtain high-dimensional features. Specifically, the convolutional neural network CNN is used to extract features from the second region to obtain high-dimensional features.

[0061] In one embodiment, the feature extraction module 22 obtains a local binary pattern (LBP) of the image to be identified, which represents the texture map of the image to be identified, and obtains a Fourier spectrum (FS) of the image to be identified, which represents the spectrum of the image to be identified.

[0062] The recognition module 23 is used to fuse the texture features, the high-dimensional features and the frequency domain features, and obtain a recognition result of the image to be recognized based on the fusion result.

[0063] Specifically, the recognition module 23 fuses the texture features, frequency domain features and high-dimensional features, and obtains the recognition result of the image to be recognized based on the fused result. For example, in one embodiment, target detection can be performed based on the fused result, or the image to be recognized can be classified based on the fused result.

[0064] In a specific embodiment of the present application, the recognition module 23 performs liveness detection based on the fused result to determine whether the image to be recognized is a liveness image. Specifically, the texture features, high-dimensional features, and frequency domain features are fused to obtain a fusion result, and the liveness confidence of the image to be recognized is obtained based on the fusion result. In response to the liveness confidence being greater than the confidence threshold, the recognition result of the image to be recognized is a liveness image.

[0065] This application can not only judge living and non-living objects based on texture features, but also combines feature information in the frequency domain and higher-dimensional semantic information to enhance the robustness of the algorithm. Therefore, the application scenarios of this proposal are more extensive and the results are more stable.

[0066] The image recognition device of this embodiment divides the image to be processed into a first area, a second area and a third area, and extracts different types of features for the first area, the second area and the third area, for example, extracts texture features of the first area, high-dimensional features of the second area, and frequency domain features of the third area; when the application scenario changes and causes some features to fail, other features can still ensure the effectiveness of the entire image recognition method, thereby improving the robustness of image recognition. In addition, in this embodiment, a convolutional neural network is used to learn low-level texture information and activity information in the frequency domain in a targeted manner through auxiliary supervised learning, extract deeper features in the network high-dimensional space, and then fuse low-level features (texture features, frequency domain features) and high-level features (high-dimensional features), which are more descriptive and more robust than the features extracted by other methods.

[0067] See also Figure 3 , is a flow chart of an embodiment of a method for training an image recognition model of the present invention. The image recognition model of this embodiment is used to implement the above Figure 1 Specifically, the training method of the image recognition model includes:

[0068] Step S31: using the initial model to divide each sample image in the training sample set into a first region, a second region and a third region.

[0069] For example, each sample image in the training sample set is divided using the initial model to obtain a first area, a second area, and a third area. Specifically, the initial model is integrated with an algorithm module that can obtain a category activation heat map. The sample image is processed using the initial model to obtain a category activation heat map of the sample image, and the key area of ​​each sample image is determined based on the intensity of the pixel points in the category activation heat map, and then the key area is divided based on preset rules to obtain a first area, a second area, and a third area. Divided into a first area, a second area, and a third area. In one embodiment, the divided first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target.

[0070] Step S32: extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region.

[0071] The initial model has a module capable of acquiring a texture map and a module capable of acquiring a spectrum map. The initial model also has a convolutional neural network. The sample image is processed using the initial model to obtain a texture map and a spectrum map of the sample image. Feature extraction is performed on the first region based on the texture map to obtain texture features, feature extraction is performed on the third region based on the spectrum map to obtain frequency domain features, and feature extraction is performed on the second region to obtain high-dimensional features. Specifically, the texture map is used as supervision so that the convolutional neural network extracts texture features from the first region, the spectrum map is used as supervision so that the convolutional neural network extracts frequency domain features from the third region, and the sample image is used as supervision so that the convolutional neural network extracts high-dimensional features from the second region. The texture map is a local binary pattern map, and the spectrum map is a Fourier spectrum map.

[0072] Because of the difference between reflection and secondary imaging, texture features are generally considered to be the main way to distinguish between some living and non-living objects in simple attack scenarios. By processing the sample image to obtain a texture map, such as a local binary pattern map, the obtained texture map is used for supervised learning in the lower-dimensional convolutional neural network feature layer, so that the layer can retain the ability to extract texture features of living or non-living objects at a lower temperature. Frequency domain features such as Fourier frequency domain features can characterize the unique characteristics of living or non-living objects in the frequency domain space. The sample image is Fourier transformed and converted to the frequency domain. The spectrum map is input at a position slightly higher than the dimension of the texture network branch of the convolutional neural network for supervised learning, so that the branch network can retain the characteristics of frequency domain feature extraction. In relatively complex attack scenarios, the texture features of non-living and living objects are relatively difficult to distinguish, but they are relatively easier to distinguish in the frequency domain. By allowing the branch network to extract spectrum features, it can effectively cope with the changes in facial imaging under different scene attacks and ensure the stable output of the network.

[0073] Specifically, the initial model has three output branches, one is a texture feature network branch, one is a frequency domain feature network branch, and the other is a high-dimensional feature network branch. The texture feature network branch is used to obtain the texture map of the sample image, and use the texture map to supervise the convolutional neural network to extract texture features from the first region. The frequency domain feature network branch is used to obtain the spectrum map of the sample image, and use the spectrum map to supervise the convolutional neural network to extract frequency domain features from the third region. The high-dimensional feature network branch is used to use the sample image to supervise the convolutional neural network to extract high-dimensional features from the second region.

[0074] Specifically, in the low-level feature map, the texture map of the first region, such as the local binary pattern map, is used to supervise the network learning. In the next level, the spectrum map is used to supervise the learning of the third region features. In the higher-dimensional space of the network, a convolutional neural network is used to extract the higher-dimensional hidden active features of the second region of the face. As the network depth increases, the residual module is used to extract the features of the face image. As the dimension of the features increases, the extracted features become more and more abstract. By extracting deeper network features from the face images of the sample images, the neural network can learn deeper subtle features that are difficult to visualize.

[0075] Step S33: training the initial model using the texture features, the high-dimensional features, and the frequency domain features, thereby obtaining the image recognition model.

[0076] Specifically, the initial model is trained using texture features, high-dimensional features, and frequency domain features to obtain an image recognition model.

[0077] In one embodiment, a first loss amount loss1 is calculated based on texture features and a texture map using a mean square error function, a second loss amount loss2 is calculated based on frequency domain features and a spectrum map using a mean square error function, and a third loss amount loss3 of high-dimensional features is calculated based on high-dimensional features and preset labels of sample images using a mean square error function.

[0078] The mean square error function can detect the deviation between the true value and the predicted value. The texture map is used as the true value, and the texture feature is used as the predicted value, and then the first loss amount Loss1 is calculated. It can be understood that the spectrum map is used as the true value, and the frequency domain feature is used as the predicted value, and then the second loss amount Loss2 is calculated. The preset label of the sample image is used as the true value, and the high-dimensional feature is used as the predicted value to calculate the third loss amount Loss3. The first loss amount, the second loss amount and the third loss amount are fused, and the fourth loss amount is obtained based on the fusion result. Specifically, a 1×1 convolution layer can be used to exchange channel features, and the first loss amount Loss1, the second loss amount Lss2 and the third loss amount Loss3 can be fused into a more robust fourth loss amount Loss4.

[0079] The first loss, the second loss, the third loss, and the fourth loss are weighted to obtain a fifth loss, and the fifth loss is used to adjust the parameters of the initial model to obtain an image recognition model. Specifically, a cross entropy classification loss Loss is applied to the living and non-living discrimination information. highdim This is because if the classification loss constraint is not applied to each network, the final output information of each network will no longer have its original meaning, and the entire network will no longer have the above-mentioned interpretability. This application uses a multi-task loss function, including the classification loss of the three network branches and the fused classification loss, specifically:

[0080] Loss = αLoss0 + γLoss highdim +βLosS1+βLoss2+βLoss3;

[0081] Among them, Loss0 is the fourth loss, Loss highdim is the cross entropy classification loss, Loss is the fifth loss, and α, β, and γ are weighted coefficients.

[0082] The back propagation method is used to adjust the parameters of the initial model based on the fifth loss amount, thereby obtaining an image recognition model.

[0083] The training method of the image recognition model of this application uses the designed convolutional neural network to learn low-level texture information and activity information in the frequency domain through auxiliary supervised learning when extracting features from sample images, extracting deeper features in the network high-dimensional space, and then fusing low-level features with high-level features. Compared with other methods, the features extracted are more descriptive and more robust. In addition, the number of samples is relatively reduced, and it can adapt to different scenarios and has stronger generalization ability.

[0084] See also Figure 4 , is a structural diagram of an embodiment of a training device for an image recognition model of the present invention, which specifically includes: a division module 41, an extraction module 42 and a training module 43.

[0085] The division module 41 is used to divide each sample image in the training sample set into a first region, a second region and a third region by using the initial model.

[0086] For example, the segmentation module 41 uses the initial model to segment each sample image in the training sample set to obtain a first area, a second area, and a third area. Specifically, the initial model is integrated with an algorithm module that can obtain a category activation heat map. The sample image is processed using the initial model to obtain a category activation heat map of the sample image, and the key area of ​​each sample image is determined based on the intensity of the pixel points in the category activation heat map, and then the key area is divided based on preset rules to obtain a first area, a second area, and a third area. Divided into a first area, a second area, and a third area. In one embodiment, the divided first area includes the mouth area and the nose area of ​​the target in the image to be identified; the second area at least includes the eye area of ​​the target; and the third area at least includes the face area of ​​the target.

[0087] The extraction module 42 is used to extract the texture features of the first region, the high-dimensional features of the second region, and the frequency domain features of the third region.

[0088] The initial model has a module capable of acquiring a texture map and a module capable of acquiring a spectrum map. And the initial model has a convolutional neural network. The extraction module 42 processes the sample image using the initial model to obtain a texture map and a spectrum map of the sample image. Feature extraction is performed on the first region based on the texture map to obtain texture features, feature extraction is performed on the third region based on the spectrum map to obtain frequency domain features, and feature extraction is performed on the second region to obtain high-dimensional features. Specifically, the texture map is used as supervision so that the convolutional neural network extracts texture features from the first region, the spectrum map is used as supervision so that the convolutional neural network extracts frequency domain features from the third region, and the sample image is used as supervision so that the convolutional neural network extracts high-dimensional features from the second region. The texture map is a local binary pattern map, and the spectrum map is a Fourier spectrum map.

[0089] Because of the difference between reflection and secondary imaging, texture features are generally considered to be the main way to distinguish between some living and non-living objects in simple attack scenarios. By processing the sample image to obtain a texture map, such as a local binary pattern map, the obtained texture map is used for supervised learning in the lower-dimensional convolutional neural network feature layer, so that the layer can retain the ability to extract texture features of living or non-living objects at a lower temperature. Frequency domain features such as Fourier frequency domain features can characterize the unique characteristics of living or non-living objects in the frequency domain space. The sample image is Fourier transformed and converted to the frequency domain. The spectrum map is input at a position slightly higher than the dimension of the texture network branch of the convolutional neural network for supervised learning, so that the branch network can retain the characteristics of frequency domain feature extraction. In relatively complex attack scenarios, the texture features of non-living and living objects are relatively difficult to distinguish, but they are relatively easier to distinguish in the frequency domain. By allowing the branch network to extract spectrum features, it can effectively cope with the changes in facial imaging under different scene attacks and ensure the stable output of the network.

[0090] Specifically, the initial model has three output branches, one is a texture feature network branch, one is a frequency domain feature network branch, and the other is a high-dimensional feature network branch. The texture feature network branch is used to obtain the texture map of the sample image, and use the texture map to supervise the convolutional neural network to extract texture features from the first region. The frequency domain feature network branch is used to obtain the spectrum map of the sample image, and use the spectrum map to supervise the convolutional neural network to extract frequency domain features from the third region. The high-dimensional feature network branch is used to use the sample image to supervise the convolutional neural network to extract high-dimensional features from the second region.

[0091] Specifically, the extraction module 42 uses a texture map of the first region, such as a local binary pattern map, to supervise network learning in a low-level feature map, uses a spectrum map to supervise learning of features in the third region in the next layer, and uses a convolutional neural network to extract higher-dimensional hidden active features of the second region of the face in a higher-dimensional space of the network. As the network depth increases, a residual module is used to extract features of the face image, and as the dimension of the feature increases, the extracted features become more and more abstract. By extracting deeper network features of the face image of the sample image, the neural network can learn deeper subtle features that are difficult to visualize.

[0092] The training module 43 is used to train the initial model using the texture features, the high-dimensional features and the frequency domain features, so as to obtain the image recognition model.

[0093] Specifically, the training module 43 trains the initial model using texture features, high-dimensional features, and frequency domain features to obtain an image recognition model.

[0094] In one embodiment, the training module 43 uses a mean square error function to calculate a first loss amount loss1 based on texture features and a texture map, uses a mean square error function to calculate a second loss amount loss2 based on frequency domain features and a spectrum map, and uses a mean square error function to calculate a third loss amount loss3 of high-dimensional features based on high-dimensional features and preset labels of sample images.

[0095] The mean square error function can detect the deviation between the true value and the predicted value. The texture map is used as the true value, and the texture feature is used as the predicted value, and then the first loss amount Loss1 is calculated. It can be understood that the spectrum map is used as the true value, and the frequency domain feature is used as the predicted value, and then the second loss amount Loss2 is calculated. The preset label of the sample image is used as the true value, and the high-dimensional feature is used as the predicted value to calculate the third loss amount Loss3. The first loss amount, the second loss amount and the third loss amount are fused, and the fourth loss amount is obtained based on the fusion result. Specifically, a 1×1 convolution layer can be used to exchange channel features, and the first loss amount Loss1, the second loss amount Lss2 and the third loss amount Loss3 can be fused into a more robust fourth loss amount Loss4.

[0096] The training module 43 performs a weighted calculation on the first loss, the second loss, the third loss, and the fourth loss to obtain a fifth loss, and uses the fifth loss to adjust the parameters of the initial model to obtain an image recognition model. Specifically, a cross entropy classification loss Loss is applied to the living and non-living discrimination information. highdim This is because if the classification loss constraint is not applied to each network, the final output information of each network will no longer have its original meaning, and the entire network will no longer have the above-mentioned interpretability. This application uses a multi-task loss function, including the classification loss of the three network branches and the fused classification loss, specifically:

[0097] Loss = αLoss0 + γLoss highdim +βLoss1+βLoss2+βLoss3;

[0098] Among them, Loss0 is the fourth loss, Loss highdim is the cross entropy classification loss, Loss is the fifth loss, and α, β, and γ are weighted coefficients.

[0099] The training module 43 uses a back propagation method to adjust the parameters of the initial model based on the fifth loss amount, thereby obtaining an image recognition model.

[0100] The training device of the image recognition model of this application uses the designed convolutional neural network to learn low-level texture information and activity information in the frequency domain through auxiliary supervised learning when extracting features from sample images, extracting deeper features in the network high-dimensional space, and then fusing low-level features with high-level features. Compared with other methods, the features extracted are more descriptive and more robust. In addition, the number of samples is relatively reduced, and it can adapt to different scenarios and has stronger generalization ability.

[0101] See also Figure 5 , is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention, the electronic device includes a memory 202 and a processor 201 connected to each other.

[0102] The memory 202 is used to store program instructions for implementing any of the above-mentioned device methods.

[0103] The processor 201 is used to execute program instructions stored in the memory 202 .

[0104] The processor 201 may also be referred to as a CPU (Central Processing Unit). The processor 201 may be an integrated circuit chip having signal processing capabilities. The processor 201 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0105] The memory 202 can be a memory stick, a TF card, etc., which can store all the information in the electronic device of the device, including the input raw data, computer programs, intermediate operation results and final operation results are all stored in the memory. It stores and retrieves information according to the location specified by the controller. With the memory, the electronic device has a memory function to ensure normal operation. The memory of the electronic device can be divided into main memory (internal memory) and auxiliary memory (external memory) according to its use, and there is also a classification method of dividing it into external memory and internal memory. External memory is usually a magnetic medium or an optical disk, etc., which can store information for a long time. Memory refers to the storage component on the motherboard, which is used to store the data and programs currently being executed, but is only used to temporarily store programs and data. If the power is turned off or the power is cut off, the data will be lost.

[0106] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a system server, or a network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application.

[0110] See also Figure 6, which is a schematic diagram of the structure of the computer-readable storage medium of the present invention. The storage medium of the present application stores a program file 203 that can implement all the above methods, wherein the program file 203 can be stored in the above storage medium in the form of a software product, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage device includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, etc., which can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0111] The above are only implementation modes of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. An image recognition method, characterized in that: include: Obtain a category activation heat map of the distribution of facial activity features corresponding to the image to be recognized, calculate the pixel intensity of each pixel point in the category activation heat map, obtain the intensity mean, select pixels whose intensity is greater than the intensity mean, and obtain the key area; Dividing the key area based on a preset rule to obtain a first area, a second area, and a third area; Extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region; The texture features, the high-dimensional features and the frequency domain features are fused, and a liveness confidence of the image to be identified is obtained based on the fusion result. In response to the liveness confidence being greater than a confidence threshold, the identification result of the image to be identified is a liveness image.

2. The method according to claim 1, characterized in that The step of extracting the texture features of the first region, the high-dimensional features of the second region, and the frequency domain features of the third region comprises: Processing the image to be identified to obtain a texture map and a spectrum map of the image to be identified; Feature extraction is performed on the first region based on the texture map to obtain the texture feature, feature extraction is performed on the third region based on the spectrum map to obtain the frequency domain feature, and feature extraction is performed on the second region to obtain the high-dimensional feature.

3. The method according to claim 1, characterized in that The first area includes at least a mouth area and a nose area of ​​the target in the image to be identified; The second area includes at least the eye area of ​​the target; The third area includes at least a face area of ​​the target.

4. The image recognition method according to claim 1, characterized in that: Get the image to be recognized, including: Perform quality detection on the images to be processed, remove the images to be processed whose quality scores are less than a threshold, and use the remaining images to be processed as the images to be identified.

5. An image recognition device, characterized in that: include: A region division module is used to obtain a category activation heat map of the distribution of facial active features corresponding to the image to be recognized, calculate the pixel intensity of each pixel point in the category activation heat map, obtain the intensity mean, select the pixel points whose intensity is greater than the intensity mean, and obtain the key area; Dividing the key area based on a preset rule to obtain a first area, a second area, and a third area; A feature extraction module, used to extract texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region; The recognition module is used to fuse the texture features, the high-dimensional features and the frequency domain features, and obtain the liveness confidence of the image to be recognized based on the fusion result. In response to the liveness confidence being greater than the confidence threshold, the recognition result of the image to be recognized is a liveness image.

6. A method for training an image recognition model, characterized in that: The image recognition model is used to implement the image recognition method described in any one of claims 1 to 4 above; the training method comprises: Using the initial model to obtain a class activation heat map of the distribution of facial activity features corresponding to each sample image in the training sample set, calculating the pixel intensity of each pixel point in the class activation heat map to obtain an intensity mean, selecting pixels with intensities greater than the intensity mean to obtain key areas; dividing the key areas based on preset rules to obtain a first area, a second area, and a third area; Extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region; The initial model is trained using the texture features, the high-dimensional features, and the frequency domain features to obtain the image recognition model.

7. The method according to claim 6, characterized in that The step of extracting texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region comprises: Processing the sample image to obtain a texture map and a spectrum map of the sample image; Feature extraction is performed on the first region based on the texture map to obtain the texture feature, feature extraction is performed on the third region based on the spectrum map to obtain the frequency domain feature, and feature extraction is performed on the second region to obtain the high-dimensional feature.

8. The method according to claim 6, characterized in that The step of training the initial model using the texture features, the high-dimensional features, and the frequency domain features to obtain the image recognition model includes: A first loss amount is calculated based on the texture feature and the texture map of the sample image using a mean square error function, a second loss amount is calculated based on the frequency domain feature and the spectrum map of the sample image using a mean square error function, and a third loss amount of the high-dimensional feature is calculated based on the high-dimensional feature and a preset label of the sample image using a mean square error function; fusing the first loss amount, the second loss amount, and the third loss amount, and obtaining a fourth loss amount based on the fusion result; Performing weighted calculation on the first loss amount, the second loss amount, the third loss amount, and the fourth loss amount to obtain a fifth loss amount; The fifth loss amount is used to adjust the parameters of the initial model to obtain the image recognition model.

9. A training device for an image recognition model, characterized in that: include: A partitioning module is used to obtain a category activation heat map of the distribution of facial active features corresponding to each sample image in the training sample set using the initial model, calculate the pixel intensity of each pixel point in the category activation heat map, obtain the intensity mean, select the pixel points whose intensity is greater than the intensity mean, and obtain the key area; Dividing the key area based on a preset rule to obtain a first area, a second area, and a third area; An extraction module, used to extract texture features of the first region, high-dimensional features of the second region, and frequency domain features of the third region; A training module is used to train the initial model using the texture features, the high-dimensional features and the frequency domain features, so as to obtain the image recognition model.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the memory stores program instructions, and the processor retrieves the program instructions from the memory to execute the image recognition method as described in any one of claims 1 to 4 and / or the training method of the image recognition model as described in any one of claims 6 to 8.

11. A computer-readable storage medium, characterized in that: A program file is stored, and the program file can be executed to implement the image recognition method described in any one of claims 1 to 4 and / or the training method of the image recognition model described in any one of claims 6 to 8.

Images