Face recognition device, method, equipment and storage medium

Abstract

The present invention provides a face recognition device, method, equipment and storage medium, belonging to the field of image processing technology. The method specifically includes analyzing the picture modality of the target processing image and outputting it to the shallow separation network corresponding to the picture modality; extracting the underlying feature data corresponding to the picture modality, the underlying feature data at least including the shallow edge and texture underlying features of the face data; using a deep shared network to respectively receive the underlying feature data extracted by at least two different shallow feature extraction modules, and outputting the target face feature vector corresponding to the target processing image; analyzing the similarity value between the target face feature vector and the registered face feature vector; judging whether the target face image in the target processing image is consistent with the registered face image according to the similarity value. Through the processing scheme of the present application, different recognition strategies are adopted according to different scenes to maximize the universality of the model scene application.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a face recognition device, method, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology and intelligent computing chips, face recognition technology has been widely deployed on AIOT terminal devices. Source data of different modalities (RGB / IR / depth) has been widely adopted as input for face recognition models. However, existing methods generally use single modality data to train corresponding model parameters.

[0003] Since the model needs to run on the terminal device, the memory of the terminal device is much lower than the memory of each server, so the model cannot be too large. Therefore, the terminal device cannot train the model parameters by fusing different modal data like the server to make the model have the ability to represent two modal data. This will inevitably increase the model size or make it difficult for network training to converge, and the model parameters and size may exceed the computing power or storage capacity of the AIOT device.

[0004] If multiple models trained based on face data of different modalities are set on the terminal device, the usage scenarios of each modality need to be set in advance. For example, in outdoor access control application scenarios, the RGB model is set for face recognition because the IR image will be interfered by ambient light and the face imaging quality is poor when the sunlight is strong during the day; at night or in low light scenes, the IR model is set for recognition because the RGB face imaging quality is poor. However, in changing scenarios, the AIOT system cannot switch models in real time according to the needs of the scene, resulting in poor user experience. In addition, the system is unavailable during the switching process, and two model parameter data need to be stored, which takes up a lot of storage space. Summary of the invention

[0005] Therefore, in order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a face recognition device, method, equipment and storage medium that adopt different recognition strategies according to different scenarios to maximize the universality of model scenario application.

[0006] In order to achieve the above-mentioned purpose, the present invention provides a face recognition device, comprising: a data recognition selector, used to analyze the picture modality of a target processing image and output it to a shallow separation network corresponding to the picture modality; at least two different shallow feature extraction modules, used to extract underlying feature data corresponding to the picture modality, wherein the underlying feature data at least includes shallow edges and underlying texture features of face data; a deep shared network operation module, used to respectively receive underlying feature data extracted by at least two different shallow feature extraction modules, and output a target face feature vector corresponding to the target processing image; a feature analysis module, used to analyze the similarity value between the target face feature vector and a registered face feature vector, wherein the registered face feature vector is obtained by extracting the registered face image through the shallow feature extraction module and the deep shared network operation module; and an image recognition judgment module, used to judge whether the target face image in the target processing image is consistent with the registered face image according to the similarity value.

[0007] In one embodiment, the data identification selector includes: a channel information extraction unit, used to extract channel information of a target processed image; a modality judgment unit, used to determine the image modality of the target processed image based on the channel information; and a modality output unit, used to output the target processed image to a shallow feature extraction module of the corresponding modality.

[0008] In one embodiment, the shallow feature extraction module includes: a convolution operation unit, which is used to perform convolution processing on the target face image using a trained convolutional network, wherein the number of each layer in the network structure matches the processing resources of the terminal device, and the parameters of each layer correspond to the image modality; a bottom-level feature extraction unit, which is used to determine the bottom-level feature information of the face through the trained convolutional network, and perform robust processing on the target processed image, so that the convolutional network extracts bottom-level feature data corresponding to the target face image, and the bottom-level feature data includes at least shallow edges and bottom-level texture features of the face data.

[0009] In one of the embodiments, the deep shared network operation module includes: a data selection unit, used to respectively receive the underlying feature data transmitted by at least two shallow feature extraction modules, and use the image modality and the underlying feature data as input data of the deep shared network unit; a deep shared network unit, used to use a trained deep convolutional network to perform convolution processing on the target face image, and generate a target face feature vector in combination with the image modality and the underlying feature data.

[0010] In one embodiment, the deep shared network operation module includes: a data integration unit, which is used to respectively receive the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrate the underlying feature data of each output channel as input data of the deep shared network; a deep shared network unit, which is used to use a trained deep convolutional network to perform convolution processing on the target face image, and generate a target face feature vector in combination with the integrated underlying feature data.

[0011] In one embodiment, the deep shared network operation module includes: a deep shared network unit, used to perform convolution processing on the target face image according to the picture modality using a trained deep convolutional network to generate deep feature data; a parameter sharing unit, used to generate a target face feature vector corresponding to the picture modality by combining the deep feature data and the underlying feature data through shared parameters.

[0012] In one of the embodiments, the feature analysis module includes: a similarity calculation unit, configured to calculate a similarity value between a target face feature vector of the target face image and a registered face feature vector based on a similarity function.

[0013] In one of the embodiments, the similarity calculation unit calculates the first similarity values ​​between the target face feature vector of the target face image and the registered face feature vector under different image modalities based on the similarity function, and obtains the final similarity value between the target face feature vector of the target face image and the registered face feature vector based on multiple first similarity values.

[0014] The present invention also provides a face recognition method, which includes: analyzing the picture modality of a target processed image and outputting it to a shallow separation network corresponding to the picture modality; extracting underlying feature data corresponding to the picture modality, wherein the underlying feature data at least includes shallow edges and underlying texture features of face data; respectively receiving underlying feature data extracted by at least two different shallow feature extraction modules, and outputting a target face feature vector corresponding to the target processed image; analyzing a similarity value between the target face feature vector and a registered face feature vector, wherein the registered face feature vector is extracted from a registered face image through the shallow feature extraction module and the deep shared network operation module; and judging whether the target face image in the target processed image is consistent with the registered face image according to the similarity value.

[0015] In one of the embodiments, the analysis of the image modality of the target processed image and outputting it to a shallow separation network of the corresponding image modality includes: extracting channel information of the target processed image; determining the image modality of the target processed image based on the channel information; and outputting the target processed image to a shallow feature extraction module of the corresponding modality.

[0016] In one of the embodiments, the extracting the underlying feature data corresponding to the image modality includes: using a trained convolutional network to perform convolution processing on the target face image, wherein the number of each type of layers in the network structure matches the processing resources of the terminal device, and the parameters of each type of layer correspond to the picture modality; determining the underlying feature information of the face through the trained convolutional network, and performing robust processing on the target processed image so that the convolutional network extracts the underlying feature data corresponding to the target face image, and the underlying feature data at least includes shallow edges of the face data and underlying texture features.

[0017] In one of the embodiments, the method of respectively receiving the underlying feature data extracted by at least two different shallow feature extraction modules and outputting a target face feature vector corresponding to the target processed image includes: respectively receiving the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrating the underlying feature data of each output channel as input data of the deep shared network; using a trained deep convolutional network to perform convolution processing on the target face image, and generating a target face feature vector in combination with the integrated underlying feature data.

[0018] In one of the embodiments, the method of using a trained deep convolutional network to perform convolution processing on the target facial image and generating a target facial feature vector by combining the integrated underlying feature data includes: using a trained deep convolutional network to perform convolution processing on the target facial image according to the image modality to generate deep feature data; and combining the deep feature data and the underlying feature data through shared parameters to generate a target facial feature vector corresponding to the image modality.

[0019] The present invention also provides a device, comprising a memory and a processor, wherein the memory stores a terminal program, and is characterized in that the processor implements the steps of the above method when executing the terminal program.

[0020] The present invention also provides a computer-readable storage medium on which a terminal program is stored, characterized in that the terminal program implements the steps of the above method when executed by a processor.

[0021] Compared with the prior art, the advantages of the present invention are: a multimodal input separation-sharing network model is adopted to integrate the expression characteristics of multimodal data; and a branch cross network parameter training method is adopted to achieve fast model training convergence; based on the separation-sharing network model, different recognition strategies can be adopted according to different scenarios to maximize the universality of model scenario application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 is a structural block diagram of a face recognition device in an embodiment of the present invention;

[0024] Figure 2 is a schematic flow chart of a face recognition method in an embodiment of the present invention;

[0025] Figure 3 is a schematic flow chart of a face recognition method in an embodiment of the present invention;

[0026] Figure 4 is a schematic flow chart of a face recognition method in an embodiment of the present invention;

[0027] Figure 5 is a schematic diagram of steps of a face recognition method in an embodiment of the present invention; and

[0028] Figure 6 1 is an internal structure diagram of a device in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0031] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0032] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0033] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, one skilled in the art will appreciate that aspects may be practiced without these specific details.

[0034] like Figure 1 As shown, the embodiment of the present application provides a face recognition device, which can be applied to an AIOT terminal. The face recognition method provided by the embodiment of the present invention is applied to a small AIOT terminal device with a face recognition function, wherein the small AIOT terminal device includes: a mobile phone, a tablet computer, a vehicle-mounted device, a security inspection device, etc. The face recognition function depends on the specific terminal device and the application environment. For example, the face unlocking function on the mobile phone verifies the identity of the face in the image taken by the mobile phone, and then decides whether to unlock the mobile phone screen according to the identity recognition result. Due to the limitations of power consumption and cost, the computing power and storage space of small terminal devices are not enough to carry multiple high-precision face recognition algorithms based on deep convolutional networks. AIOT integrates AI technology and IoT technology, generates and collects massive data from different dimensions through the Internet of Things, and stores it in the cloud and edge, and then realizes the digitization and intelligent connection of all things through big data analysis and higher forms of artificial intelligence. The face recognition device includes a data recognition selector 101, at least two different shallow feature extraction modules 102, a deep shared network operation module 103, a feature analysis module 104 and an image recognition judgment module 105.

[0035] The data identification selector 101 is used to analyze the image modality of the target processed image and output it to the shallow separation network of the corresponding image modality. The data identification selector 101 can identify the image modality of the target processed image according to the source data (RGB / IR / depth) of different modalities, so as to output the target processed image to the shallow separation network of the corresponding image modality. For example, when the image modality is RGB, the source data has three-channel data, namely R, G, and B, and each channel tends to a certain color; when the image modality is IR, the original data of the infrared image does not have three channels, and only one channel is the mapped infrared data, and the three-channel data is obtained by mapping the infrared data to a pseudo color table; when the image modality is depth, the original data of the depth image does not have three channels, and the depth image directly reflects the three-dimensional information of the geometric shape of the visible surface of the scene. The depth image is an image that uses the distance (depth) from the image collector to each point in the scene as the pixel value, so it has only one channel, which is the mapped depth data, and the three-channel data is obtained by mapping the depth data to a pseudo color table.

[0036] The shallow feature extraction module 102 is used to extract the underlying feature data corresponding to the image modality, and the underlying feature data at least includes the shallow edges of the face data and the underlying texture features. Different shallow feature extraction modules 102 correspond to different image modalities. Each shallow feature extraction module can be a residual network composed of several layers of convolution-pooling-activation layers, which is used to extract the shallow edges of the face data, the underlying texture features, etc.; each shallow feature extraction module is used to extract the underlying feature data of the target processed image corresponding to the image modality. Multiple shallow feature extraction modules 102 can be trained in parallel, and the terminal can input the sample data of the two image modalities into the training models of the two parallel modules in sequence according to the Mini-Batch size, using a cross-training strategy, and respectively train the shallow separation networks of different branches, wherein each sample data can contain multiple sample face images with different image modalities.

[0037] The deep shared network operation module 103 is used to receive the underlying feature data extracted by at least two different shallow feature extraction modules, and output a target face feature vector corresponding to the target processed image. Figure 2 As shown, the deep shared network operation module 103 can receive the underlying feature data extracted by at least two different shallow feature extraction modules, obtain the adjustment coefficient corresponding to the image modality, and output the target face feature vector in combination with the adjustment coefficient and the received underlying feature data; or Figure 3 As shown, the underlying feature data extracted by at least two different shallow feature extraction modules can be received, and the received underlying feature data can be fused to finally output the target face feature vector; or, as shown Figure 4As shown in the figure, different deep convolutional networks can be used to receive the underlying feature data extracted by at least two different shallow feature extraction modules, and determine the shared parameters through the image modality, analyze the received underlying feature data, and finally classify and output the target face feature vector. The deep convolutional network not only further analyzes and extracts the detailed features such as the shallow edges and underlying texture features of the face data, but also extracts the larger local or overall structural information of the image, and then encodes it into high-dimensional feature information for face feature recognition.

[0038] In one embodiment, when the target processed image is the first modality data, the current modality face recognition data is loaded into the first shallow separation network, and the target face feature vector E1 corresponding to the first shallow separation network is extracted through the deep shared network. 1 _tar; When the target processed image is the second modality data, the current modality face recognition data is loaded into the second shallow separation network, and the target face feature vector E2 corresponding to the second shallow separation network is extracted through the deep shared network 2 _tar.

[0039] In another embodiment, when the target processed image is the first modality data, the current modality face recognition data is loaded into the first shallow separation network, and the face feature vector E1 corresponding to the first shallow separation network is extracted through the deep shared network. 1 _tar, and then load the current first modality data into the second shallow separation network, and extract E1 through the deep shared network 2 _tar, and finally get the target face feature vector [E1 1 _tar, E1 2 _tar]; when the target processed image is the second modality data, the current modality face recognition data is loaded into the second shallow separation network, and the face feature vector E2 corresponding to the second shallow separation network is extracted through the deep shared network 1 _tar, and then load the current second modality data into the first shallow separation network, and extract E2 through the deep shared network 2 _tar, and finally get the target face feature vector [E2 2 _tar, E2 1 _tar].

[0040] The feature analysis module 104 is used to analyze the similarity between the target face feature vector and the registered face feature vector. The registered face feature vector is extracted from the registered face image through the shallow feature extraction module and the deep shared network operation module. The feature analysis module 104 obtains the trained registered face feature vector E1 1 _reg and / or E2 2 _reg, where E1 1_reg is the registered face target face feature vector obtained by the first modality data through the first separation network, E2 2 _reg represents the target face feature vector of the registered face obtained by the second modal data through the second separation network. The terminal calculates the similarity value T between the target face feature vector and the registered face feature vector based on the similarity function f(E_reg, E_tar).

[0041] The image recognition judgment module 105 is used to judge whether the target face image in the target processing image is consistent with the registered face image according to the similarity value. The terminal can store a preset similarity value T in advance. h , when T ≥ T h The terminal determines that the recognition is successful and the target face image in the target processed image is consistent with the registered face image. In one embodiment, if the registered face feature vector contains E1 1 _reg and / or E2 2 _reg, then based on the similarity function f(E1 1 _reg,E1 1 _tar) and / or f(E2 2 _reg,E2 2 _tar)Calculate T1 1 and / or T2 2 , if T1 1 and / or T2 2 ≥T h , the authentication is successful; when "and" is used, it means that face recognition authentication is performed simultaneously through two modal data; when "or" is used, it means that face recognition authentication is performed using a single modal data.

[0042] In one embodiment, if the registered face feature vector includes [E2 2 _reg, E1 2 _reg] and / or [E2 2 _reg, E2 1 _reg], then based on the similarity function f(E1 1 _reg,E1 1 _tar)and f(E1 2 _reg,E1 2 _tar) and / or f(E2 2 _reg,E2 2 _tar)and f(E2 1 _reg,E2 1 _tar)Calculate T1 1 and T1 2 and / or T2 2 and T2 1 , if T1 1≥T h And T1 2 ≥T h ' and / or T2 2 ≥T h And T2 1 ≥T h ', then the authentication is successful, where T h ≥T h '. When "and" is used, it means that the two modal data are obtained by two shallow separation networks respectively [E1 1 _tar, E1 2 _tar] and [E2 2 _tar, E2 1 _tar] performs face recognition authentication at the same time. When "or" is used, it means that a single modality data is obtained by two shallow separation networks respectively [E1 1 _tar, E1 2 _tar] or [E2 2 _tar, E2 1 _tar] for face recognition authentication.

[0043] The above-mentioned face recognition device adopts a multimodal input separation-sharing network model to integrate the expression characteristics of multimodal data; and adopts a branch cross network parameter training method to achieve fast model training convergence; based on the separation-sharing network model, different recognition strategies can be adopted according to different scenarios to maximize the universality of model scenario application.

[0044] In one embodiment, the data identification selector includes: a channel information extraction unit, used to extract channel information of a target processed image; a modality judgment unit, used to determine the image modality of the target processed image based on the channel information; and a modality output unit, used to output the target processed image to a shallow feature extraction module of the corresponding modality.

[0045] The channel information extraction unit extracts channel information from the source data. When the image modality is RGB, the source data has three-channel data, namely R, G, and B, and each channel tends to a certain color; when the image modality is IR, the source data has only one channel, which is the mapped temperature data; when the image modality is depth, the source data has one channel, which is the mapped depth data.

[0046] The modality determination unit determines the image modality according to the extracted channel information.

[0047] In one of the embodiments, the shallow feature extraction module includes a convolution operation unit and a bottom-level feature extraction unit.

[0048] The convolution operation unit is used to perform convolution processing on the target face image using a trained convolutional network, wherein the number of various layers in the network structure matches the processing resources of the terminal device, and the parameters of various layers correspond to the image modality.

[0049] After receiving the face image to be processed, the convolution operation unit uses a trained convolutional network to perform convolution processing on the face image. It should be emphasized that the number of various layers in the network structure of the convolutional network involved in each embodiment of the present invention matches the processing resources of the terminal device, and the parameters of various layers in the network structure can be pre-trained to have image modality robustness.

[0050] Convolutional network is a multi-layer feedforward neural network, whose neuron responses are only related to the local area of ​​the input layer. It is widely used in image and video analysis. Common layer modules in convolutional networks are: convolutional layer, local connection unshared weight layer, and fully connected layer, as follows:

[0051] The convolutional layer uses multiple filters to perform convolution operations on the input signal and output a multi-channel signal.

[0052] The local connection non-shared weight layer (Local layer), referred to as the local connection layer, has a similar structure to the convolutional layer, but the filters do not share weights, that is, different filters are used at different positions of the input signal.

[0053] The fully connected layer multiplies the input vector by the weight matrix to obtain the output vector.

[0054] The input and output of the convolution layer and the local connection layer are both multi-channel two-dimensional data, and the processing process will maintain the two-dimensional structure of the image. The input and output of the fully connected layer are both vectors, regardless of the image structure. Convolutional networks generally include: at least one convolution layer and / or at least one local connection layer, and also include at least one fully connected layer. Among them, it should be noted that the order of convolutional layers and / or local connection layers can be swapped. If all three layers are included: convolutional layers are generally used at the head close to the input, local connection layers are generally used in the middle, and fully connected layers are generally used at the tail. The number of each layer can be any number, and the specific number of layers is determined by the specific application.

[0055] The more layers in the convolutional network, the higher the recognition accuracy of the face image, but the computational complexity and processing speed are also slower. Therefore, in order to enable small terminal devices to achieve efficient computing and processing, the number of various layers in the network structure of the convolutional network involved in each embodiment of the present invention matches the processing resources of the terminal device. For example, VGG16 includes 13 convolutional layers and 3 fully connected layers, so the first 6 layers of the convolutional layer can be used as a shallow convolutional network, and the 6 layers of shallow convolutional networks constitute a shallow feature extraction module 102, and the last 7 layers are used as a deep shared network, and the 7 layers of deep shared networks constitute a deep shared network operation module 103, wherein the shallow convolutional network is used to extract the target image texture, edge and other detailed features, and the deep shared network is used to further reorganize the above-mentioned texture, edge and other detailed features, while extracting the larger local or overall structural information of the image through the characteristics of the larger receptive field of the deep network, and then encoding it into high-dimensional feature information for face feature recognition; at the same time, due to the use of deep sharing, especially the fully connected layer, the amount of model parameters is greatly reduced.

[0056] After the terminal determines the various layer structures in the convolutional network, it also needs to determine the unknown parameters in each layer before using the convolutional network to process the face image to be processed, which is the testing process. Among them, the parameters of each layer in the network structure specifically include: the parameters of the convolutional layer and the local connection layer are called filters, and the parameters of the fully connected layer are called projection matrices.

[0057] Compared with the number of layers in a deep convolutional network running on a high-performance PC or server, the number of layers in the convolutional network in the embodiment of the present invention is greatly reduced. In order to maintain the recognition performance of the deep convolutional network and reduce its computational and storage complexity to be suitable for small terminal devices, the convolutional network provided in this embodiment can be pre-trained for image modality robustness so that the parameters of each layer have image modality robustness.

[0058] The training set of convolutional networks generally consists of two parts: face images and face category labels. Each face image corresponds to a face category label, and the category label indicates the identity of the person. By comparing the category labels, it can be determined which images are from the same person and which images are from different people.

[0059] In the training process with picture modality robustness, the face image training set and the picture modality category label training set are input from the beginning and the end of the network, and the picture modality robust processing method for face images is embedded in the convolutional network, and the preset objective function is optimized by forward propagation and back propagation. Among them, the objective function is the engine in the network training process. The present invention does not limit the specific form of the objective function, so as to determine the parameters of various layers in the convolutional network, and these parameters have picture modality robustness. It should be emphasized that the network structure and the embedded picture modality robust processing method used by the convolutional network during the training process are consistent with the network structure and the embedded picture modality robust processing method used in the subsequent testing process.

[0060] The underlying feature extraction unit is used to determine the underlying feature information of the face through a trained convolutional network, and perform robust processing on the target processing image so that the convolutional network extracts the underlying feature data corresponding to the target face image, and the underlying feature data at least includes the shallow edge and texture underlying features of the face data.

[0061] After the underlying feature extraction unit uses the above-described convolutional network that has been trained for image modality robustness to perform convolution processing on the face image to be processed, it first obtains the first feature map that has been convolution processed from the first node in the convolutional network; then uses the pooling layer in the convolutional network to perform pooling processing on the image data of the first feature map. Pooling is a process of performing integration processing such as averaging on each convolutional feature on the basis of convolutional feature extraction, and continuing to reduce the feature dimension, thereby generating a second feature map; finally, the second feature map is sent back to the convolutional network through the second node adjacent to the first node, so that subsequent nodes in the convolutional network continue to process the second feature map, thereby outputting a vector corresponding to the face image after processing by the fully connected layer in the convolutional network, that is, extracting the underlying feature data corresponding to the face image, and the underlying feature data at least includes shallow edges of face data and underlying texture features.

[0062] Compared with the deep convolutional network running on a high-performance PC or server, the above-mentioned face recognition device has a convolutional network structure that matches the processing resources of the terminal device, and embeds the robust processing process of the image modality into the convolutional network, which greatly reduces the pressure on the convolutional network in terms of image modality robustness, thereby being able to use a shallower convolutional network structure to achieve the required recognition rate, reducing the convolutional network's demand for computing and storage resources, and enabling the convolutional network to run quickly and with high performance on resource-constrained small terminal devices, such as mobile phones, tablet computers, and vehicle-mounted equipment.

[0063] In one embodiment, the deep shared network operation module includes a data selection unit and a deep shared network unit.

[0064] The data selection unit is used to receive the underlying feature data transmitted by at least two shallow feature extraction modules respectively, and use the image modality and the underlying feature data as input data of the deep shared network unit. The data selection unit can obtain an adjustment coefficient corresponding to the image modality, and output the adjustment coefficient and the received underlying feature data to the deep shared network unit.

[0065] The deep shared network unit is used to perform convolution processing on the target face image using a trained deep convolutional network, and generate a target face feature vector by combining the image modality and the underlying feature data. The deep shared network unit can output the target face feature vector by combining the adjustment coefficient and the received underlying feature data.

[0066] In one embodiment, the deep shared network operation module includes a data integration unit and a deep shared network unit.

[0067] The data integration unit is used to respectively receive the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrate the underlying feature data of the respective output channels as the input data of the deep shared network.

[0068] The deep shared network unit is used to perform convolution processing on the target face image using a trained deep convolutional network, and generate a target face feature vector by combining the integrated underlying feature data. The deep shared network unit fuses the received underlying feature data and finally outputs the target face feature vector, that is, it receives the shallow feature data extracted by the shallow separation network of the two modal data, and integrates the shallow feature data of each output channel as the input data of the deep shared network.

[0069] In one embodiment, the deep shared network operation module includes a deep shared network unit and a parameter sharing unit.

[0070] The deep shared network unit is used to perform convolution processing on the target face image using a trained deep convolutional network according to the image modality to generate deep feature data. The terminal can use different deep convolutional networks to respectively receive the underlying feature data extracted by at least two different shallow feature extraction modules.

[0071] The parameter sharing unit is used to generate a target face feature vector corresponding to the image modality by combining the deep feature data and the underlying feature data through shared parameters. The parameter sharing unit determines the shared parameters through the image modality, analyzes the received underlying feature data, and finally classifies and outputs the target face feature vector.

[0072] In one of the embodiments, the feature analysis module includes: a similarity calculation unit, which is used to calculate the similarity value between the target face feature vector of the target face image and the registered face feature vector based on the similarity function.

[0073] In one embodiment, the similarity function may be λT 1 +(1-λ)T 2 ≥T h , when λT1 1 +(1-λ)T1 2 ≥T h and / or λT2 2 +(1-λ)T2 1 ≥T h , the terminal determines that the recognition is successful and the target face image in the target processed image is consistent with the registered face image.

[0074] In one embodiment, when T<T h , the terminal can determine that the registered face information is inconsistent with the target face information modality. At this time, when the registered information is E1 1 _reg or E2 2 _reg, the terminal can load the current modality data into another shallow separation network, extract them separately through the deep shared network, and obtain the target face feature vector E2 1 _tar or E1 2 _tar, and then based on the similarity function f(E1 1 _reg,E2 1 _tar) or f(E2 2 _reg,E1 2 _tar), calculate T 2-1 1 or T 1-2 2 , if T 2-1 1 ≥T h 2-1 or T 1-2 2 ≥T h 2-1 , the target face is authenticated successfully.

[0075] In one of the embodiments, the similarity calculation unit calculates the first similarity values ​​between the target face feature vector of the target face image and the registered face feature vector under different image modalities based on the similarity function, and obtains the final similarity value between the target face feature vector of the target face image and the registered face feature vector based on multiple first similarity values.

[0076] In one embodiment, if the registered face feature vector includes [E1 1 _tar, E1 2 _tar] and [E2 2 _tar, E2 1_tar], then based on the similarity function f(E1 1 _reg,E1 1 _tar)and f(E1 2 _reg,E1 2 _tar) and f(E2 2 _reg,E2 2 _tar)and f(E2 1 _reg,E2 1 _tar)Calculate T1 1 and T1 2 and T2 2 and T2 1 , if T1 1 ≥T h And T1 2 ≥T h ' and T2 2 ≥T h And T2 1 ≥T h ', then the authentication is successful, where T h ≥T h '. When "and" is used, it means that the two modal data are obtained by two shallow separation networks respectively [E1 1 _tar, E1 2 _tar] and [E2 2 _tar, E2 1 _tar]Face recognition authentication is also performed.

[0077] In one embodiment, when T1 1 <T h And T1 2 <T h ' and / or T2 2 <T h And T2 1 <T h ', the terminal can determine that the registered face information is inconsistent with the target face information modality. At this time, the registered information is [E1 1 _reg, E1 2 _reg] or [E2 2 _reg, E2 1 _reg], the terminal can load the current modality data into another shallow separation network, extract them separately through the deep shared network, obtain the target face feature vector, and then based on the similarity function f(E1 1 _reg,E2 1 _tar)and f(E1 2 _reg,E2 2 _tar) or f(E2 1_reg,E1 1 _tar)and f(E2 2 _reg,E1 2 _tar)Calculate T 2-1 1 and T 2-1 2 or T 1-2 1 and T 2-1 2 , if (T 2-1 1 +T 2-1 2 ) / 2≥Th 2-1 or (T 1-2 1 +T 1-2 2 ) / 2≥Th 2-1 , the authentication is successful.

[0078] In one embodiment, Figure 5 As shown, a face recognition method is provided, comprising the following steps:

[0079] Step 201, analyzing the image modality of the target processed image and outputting it to a shallow separation network corresponding to the image modality;

[0080] Step 202, extracting underlying feature data corresponding to the image modality, the underlying feature data at least including shallow edges of face data and underlying texture features;

[0081] Step 203, receiving the underlying feature data extracted by at least two different shallow feature extraction modules respectively, and outputting a target face feature vector corresponding to the target processed image;

[0082] Step 204, analyzing the similarity between the target face feature vector and the registered face feature vector, where the registered face feature vector is extracted from the registered face image through a shallow feature extraction module and a deep shared network operation module;

[0083] Step 205: determine whether the target face image in the target processed image is consistent with the registered face image based on the similarity value.

[0084] In one embodiment, the data identification step 201 further includes the following sub-steps:

[0085] Extract channel information of the target processed image;

[0086] Determine the image modality of the target processed image according to the channel information;

[0087] The target processed image is output to the shallow feature extraction module of the corresponding modality.

[0088] In one embodiment, the shallow feature extraction step 202 further includes the following sub-steps:

[0089] A trained convolutional network is used to perform convolution processing on the target face image, where the number of layers in the network structure matches the processing resources of the terminal device, and the parameters of each layer correspond to the image modality;

[0090] The underlying feature information of the face is determined through the trained convolutional network, and the target processing image is robustly processed so that the convolutional network can extract the underlying feature data corresponding to the target face image, and the underlying feature data at least includes the shallow edge and texture underlying features of the face data.

[0091] In one embodiment, the deep shared network computing step 203 further includes the following sub-steps:

[0092] Respectively receive the underlying feature data transmitted by at least two shallow feature extraction modules, and use the image modality and the underlying feature data as input data of the deep shared network unit;

[0093] A trained deep convolutional network is used to perform convolution processing on the target face image, and the target face feature vector is generated by combining the image modality and the underlying feature data.

[0094] In one embodiment, the deep shared network computing step 203 further includes the following sub-steps:

[0095] Respectively receive the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrate the underlying feature data of the respective output channels as input data of the deep shared network;

[0096] The trained deep convolutional network is used to perform convolution processing on the target face image, and the target face feature vector is generated by combining the integrated underlying feature data.

[0097] In one embodiment, the deep shared network computing step 203 further includes the following sub-steps:

[0098] According to the image modality, a trained deep convolutional network is used to perform convolution processing on the target face image to generate deep feature data;

[0099] The target face feature vector corresponding to the image modality is generated by combining the deep feature data and the underlying feature data through shared parameters.

[0100] In one embodiment, a device is provided. The device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the device is used to provide computing and control capabilities. The memory of the device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a terminal program. The internal memory provides an environment for the operation of the operating system and the terminal program in the non-volatile storage medium. The network interface of the device is used to communicate with an external terminal through a network connection. When the terminal program is executed by the processor, a face recognition method is implemented. The display screen of the device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the device housing, or an external keyboard, touchpad or mouse, etc.

[0101] Those skilled in the art will understand that Figure 6 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the device to which the scheme of the present application is applied. The specific device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0102] A device includes a memory and one or more processors, wherein a terminal program is stored in the memory, and when the terminal program is executed by the processor, the steps of the face recognition method provided in any embodiment of the present application are implemented.

[0103] One or more non-volatile storage media storing computer-readable instructions, which, when executed by one or more processors, enable the one or more processors to implement the steps of the face recognition method provided in any embodiment of the present application.

[0104] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A face recognition device, characterized in that: include: The data recognition selector is used to analyze the image modality of the target processed image and output it to the shallow separation network corresponding to the image modality; At least two different shallow feature extraction modules, respectively obtained by training the shallow separation network, for extracting underlying feature data corresponding to the image modality, wherein the underlying feature data at least includes shallow edge and texture underlying features of face data; A deep shared network operation module, used to respectively receive the underlying feature data extracted by at least two different shallow feature extraction modules, and output a target face feature vector corresponding to the target processed image; A feature analysis module, used for analyzing the similarity between the target face feature vector and the registered face feature vector, wherein the registered face feature vector is extracted from the registered face image through the shallow feature extraction module and the deep shared network operation module; An image recognition and judgment module is used to judge whether the target face image in the target processed image is consistent with the registered face image according to the similarity value, Wherein, the data identification selector includes: A channel information extraction unit, used for extracting channel information of a target processed image; A mode determination unit, configured to determine a picture mode of the target processed image according to the channel information; A modality output unit, used to output the target processed image to a shallow feature extraction module of a corresponding modality, The shallow feature extraction module includes: A convolution operation unit, configured to perform convolution processing on the target face image using a trained convolutional network, wherein the number of various layers in the network structure matches the processing resources of the terminal device, and the parameters of the various layers correspond to the image modality; The underlying feature extraction unit is used to determine the underlying feature information of the face through the trained convolutional network, and perform robust processing on the target processed image so that the convolutional network extracts the underlying feature data corresponding to the target face image, and the underlying feature data at least includes the shallow edge and texture underlying features of the face data.

2. The face recognition device according to claim 1, characterized in that: The deep shared network computing module includes: A data selection unit, configured to respectively receive the underlying feature data transmitted by at least two shallow feature extraction modules, and use the image modality and the underlying feature data as input data of a deep shared network unit; The deep shared network unit is used to perform convolution processing on the target face image using a trained deep convolutional network, and generate a target face feature vector by combining the image modality and the underlying feature data.

3. The face recognition device according to claim 1, characterized in that: The deep shared network computing module includes: A data integration unit, configured to respectively receive the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrate the underlying feature data of the respective output channels as input data of the deep shared network; The deep shared network unit is used to use a trained deep convolutional network to perform convolution processing on the target face image, and generate a target face feature vector in combination with the integrated underlying feature data.

4. The face recognition device according to claim 1, characterized in that: The deep shared network computing module includes: A deep shared network unit, configured to perform convolution processing on the target face image using a trained deep convolutional network according to the image modality to generate deep feature data; A parameter sharing unit is used to generate a target face feature vector corresponding to the image modality by combining the deep feature data and the bottom feature data through shared parameters.

5. The face recognition device according to claim 1, characterized in that: The feature analysis module comprises: The similarity calculation unit is used to calculate the similarity value between the target face feature vector of the target face image and the registered face feature vector based on the similarity function.

6. The face recognition device according to claim 5, characterized in that: The similarity calculation unit calculates the first similarity values ​​between the target face feature vector of the target face image and the registered face feature vector under different picture modalities based on the similarity function, and obtains the final similarity value between the target face feature vector of the target face image and the registered face feature vector according to multiple first similarity values.

7. A face recognition method, characterized in that: The method comprises: Analyze the image modality of the target processed image and output it to the shallow separation network corresponding to the image modality; At least two different shallow feature extraction modules are respectively obtained by training the shallow separation network to extract underlying feature data corresponding to the image modality, wherein the underlying feature data at least includes shallow edges of face data and underlying texture features; Respectively receiving the underlying feature data extracted by at least two different shallow feature extraction modules, and outputting a target face feature vector corresponding to the target processed image; Analyzing the similarity value between the target face feature vector and the registered face feature vector, wherein the registered face feature vector is extracted from the registered face image through the shallow feature extraction module and the deep shared network operation module; Determine whether the target face image in the target processed image is consistent with the registered face image according to the similarity value, The image modality of the analysis target processing image and outputting it to the shallow separation network corresponding to the image modality includes: Extract channel information of the target processed image; Determine the picture modality of the target processed image according to the channel information; Outputting the target processed image to a shallow feature extraction module of the corresponding modality, The extracting of underlying feature data corresponding to the image modality includes: Using a trained convolutional network to perform convolution processing on the target face image, wherein the number of layers in the network structure matches the processing resources of the terminal device, and the parameters of the layers correspond to the image modality; The underlying feature information of the face is determined through a trained convolutional network, and the target processing image is robustly processed so that the convolutional network extracts underlying feature data corresponding to the target face image, and the underlying feature data at least includes shallow edges of face data and underlying texture features.

8. The method according to claim 7, characterized in that The step of respectively receiving the underlying feature data extracted by at least two different shallow feature extraction modules and outputting a target face feature vector corresponding to the target processed image comprises: Respectively receive the underlying feature data carrying the image modality transmitted by at least two shallow feature extraction modules, and integrate the underlying feature data of the respective output channels as input data of the deep shared network; The trained deep convolutional network is used to perform convolution processing on the target face image, and the target face feature vector is generated by combining the integrated underlying feature data.

9. The method according to claim 8, characterized in that The method of using a trained deep convolutional network to perform convolution processing on the target face image and combining the integrated underlying feature data to generate a target face feature vector includes: Performing convolution processing on the target face image using a trained deep convolutional network according to the image modality to generate deep feature data; The target face feature vector corresponding to the image modality is generated by combining the deep feature data and the underlying feature data through shared parameters.

10. A device comprising a memory and a processor, wherein the memory stores a terminal program, characterized in that: When the processor executes the terminal program, the steps of the method described in any one of claims 7 to 9 are implemented.

11. A computer-readable storage medium having a terminal program stored thereon, characterized in that: When the terminal program is executed by a processor, the steps of the method described in any one of claims 7 to 9 are implemented.

Images