Visual analysis method and device for human face image comparison, equipment and storage medium

Abstract

The invention relates to the field of image detection and the field of financial science and technology, and relates to a visual analysis method and device for face image comparison, equipment and a storage medium. The method comprises the steps of generating a first shielding picture set and a second shielding picture set by fusing a first comparison image and a second comparison image with a preset mask picture set, and extracting a first vector, a second vector, a first shielding vector set and a second shielding vector set by using an image feature extraction network, the method comprises the following steps: calculating a first cosine value set, a second cosine value set and a third cosine value set, calculating a fourth correlation coefficient set and a fifth correlation coefficient set by using a Pearson's correlation coefficient algorithm, generating a corresponding thermodynamic diagram according to the correlation coefficient sets, and returning the thermodynamic diagram to a user, thereby generating a visual thermodynamic diagram. By generating the visual thermodynamic diagram, the same and different positions of the face pictures after comparison can be intuitively reflected.

Description

Technical Field

[0001] The present invention relates to the field of image detection and financial technology, and to a visual analysis method, device, equipment and storage medium for comparing facial images. Background Art

[0002] As a form of biometric identification, face recognition is currently widely used in the field of financial technology, such as identity authentication, payment systems, bank card opening and withdrawal, personal financial management, etc. The face recognition system extracts and compares key features in face images to determine the similarity between two faces to achieve identity authentication and recognition. However, the existing comparison system also has the following problems:

[0003] 1) The face recognition system is opaque. As a black box neural network model, the face recognition model is very complex and difficult to understand. The system directly uses the final similarity score to represent the similarity between two faces, which reduces the credibility of the system;

[0004] 2) Some existing interpretability methods, such as activation heat maps and gradient methods, are mainly aimed at classification networks and feature outputs of a single image, and cannot intuitively show which parts of the two compared images are similar or different;

[0005] 3) In the case of recognition model comparison failure (the same person with a score below the threshold is identified as a different person, and different people with scores too high are identified as the same person), there is nothing we can do and we cannot make correct improvements.

[0006] Therefore, how to intuitively reflect the similarities and differences after comparing facial images has become a technical problem that needs to be solved urgently. Summary of the invention

[0007] In view of the above, it is necessary to provide a visual analysis method for face image comparison to solve the problem that the similarities and differences between the face images after comparison cannot be intuitively reflected during the face image comparison process.

[0008] To achieve the above object, the present invention provides a visual analysis method for face image comparison, the method comprising:

[0009] After receiving a face image comparison request sent by a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0010] Using a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extracting the feature vector of the second contrast image to obtain a second vector, extracting the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extracting the feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0011] Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0012] Calculating a first correlation coefficient set according to the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set according to the second cosine value set and the pixel value set, calculating a third correlation coefficient set according to the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set;

[0013] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0014] In addition, to achieve the above-mentioned purpose, the present invention also provides a visual analysis device for face image comparison, the device comprising:

[0015] The occlusion image generation module is used to obtain a first comparison image and a second comparison image after receiving a face image comparison request sent by a user, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set;

[0016] Image feature extraction module: used to use a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extract the feature vector of the second contrast image to obtain a second vector, extract the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract the feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0017] A cosine value calculation module: used for calculating the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0018] A correlation coefficient calculation module: used to calculate a first correlation coefficient set according to the first cosine value set and the pixel value sets of the corresponding coordinates of all mask images, calculate a second correlation coefficient set according to the second cosine value set and the pixel value set, calculate a third correlation coefficient set according to the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set;

[0019] Heat map return module: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat map to the user.

[0020] In addition, to achieve the above object, the present invention further provides a device, comprising:

[0021] a memory storing at least one computer program; and

[0022] The processor executes the program stored in the memory to implement the above-mentioned visual analysis method for facial image comparison.

[0023] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is executed by a processor in the device to implement the above-mentioned visual analysis method for facial image comparison.

[0024] In the above technical solution provided by the present invention, a first occlusion picture set and a second occlusion picture set are generated by fusing the first comparison image and the second comparison image with a preset mask picture set respectively, so as to provide for subsequent comparison and analysis of the original picture and the occlusion picture; the first vector, the second vector, the first occlusion vector set and the second occlusion vector set are extracted by using an image feature extraction network; the first cosine value set, the second cosine value set and the third cosine value set are calculated, and the similarity value between the comparison image and the occlusion image is calculated by the cosine algorithm; then the fourth correlation coefficient set of the second comparison image compared with the first comparison image and the fifth correlation coefficient set of the first comparison image compared with the second comparison image are calculated by using the Pearson correlation coefficient algorithm, and then a corresponding heat map is generated according to the above correlation coefficient sets and returned to the user. By generating a visualized heat map, the similarities and differences between the face pictures after comparison can be intuitively reflected, and it is found out which areas of the face recognition model need to improve the similarity, and the area is optimized in a targeted manner, thereby improving the efficiency of model optimization and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a schematic diagram of an application environment of a visual analysis method for face image comparison in one embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a flow chart of a visual analysis method for face image comparison provided by an embodiment of the present invention;

[0027] Figure 3 A schematic diagram of a structure of a visual analysis device for comparing facial images provided by an embodiment of the present invention;

[0028] Figure 4 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;

[0029] Figure 5 It is another structural schematic diagram of a computer device in one embodiment of the present invention.

[0030] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0033] The visual analysis method for face image comparison provided by the embodiment of the present invention can be applied in the following aspects: Figure 1In an application environment, the client communicates with the server through a network. After receiving a face image comparison request sent by a user, the server obtains a first comparison image and a second comparison image, fuses the first comparison image with a preset mask image set to generate a first occlusion image set, and fuses the second comparison image with the mask image set to generate a second occlusion image set; uses a preset image feature extraction network to extract the feature vector of the first comparison image to obtain a first vector, extracts the feature vector of the second comparison image to obtain a second vector, extracts the feature vector of the first occlusion image set to obtain a first occlusion vector set, and extracts the feature vector of the second occlusion image set to obtain a second occlusion vector set; calculates the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculates the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, calculates the cosine value of each occlusion vector in the first occlusion vector set and the cosine value of the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set; according to the first A first correlation coefficient set is calculated based on a cosine value set and a pixel value set of the corresponding coordinates of all mask images, a second correlation coefficient set is calculated based on the second cosine value set and the pixel value set, a third correlation coefficient set is calculated based on the third cosine value set and the pixel value set, and a fourth correlation coefficient set is obtained based on the first correlation coefficient set and the second correlation coefficient set, and a fifth correlation coefficient set is obtained based on the third correlation coefficient set and the second correlation coefficient set; a first positive correlation heat map and a first negative correlation heat map are generated based on the fourth correlation coefficient set, and a second positive correlation heat map and a second negative correlation heat map are generated based on the fifth correlation coefficient set, and the generated heat map is returned to the user. In the present invention, by generating a visual heat map of face image comparison, the similarities and differences after face image comparison can be intuitively reflected, and the face recognition model needs to improve the similarity of which areas of the face recognition, and the area is optimized in a targeted manner to improve the efficiency of model optimization and development. Among them, the client can be but not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0034] Reference Figure 2 The flowchart of the visual analysis method for face image comparison provided by an embodiment of the present invention is shown. In the embodiment of the present invention, the visual analysis method for face image comparison includes the following steps S1-S5:

[0035] S1. After receiving a face image comparison request sent by a user, obtain a first comparison image and a second comparison image, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set.

[0036] In one embodiment, after receiving a facial image comparison request sent by a user, the comparison request is parsed to obtain a first comparison image and a second comparison image for mutual comparison, and then the first comparison image and the second comparison image are respectively fused with a preset mask image set to generate a corresponding first occlusion image set and a second occlusion image set.

[0037] In one embodiment, the step of fusing the first contrast image with a preset mask picture set to generate a first occlusion picture set, and fusing the second contrast image with the mask picture set to generate a second occlusion picture set includes:

[0038] Extracting mask images from the mask image set one by one;

[0039] Calling a preset image fusion algorithm to fuse the mask image and the first comparison image to obtain a first occlusion image, and combining all the first occlusion images into the first occlusion image set;

[0040] The mask image and the second contrast image are fused by using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

[0041] Specifically, the mask image is a grayscale image with pixel values ​​between 0 and 1. The essence of the mask image is a grayscale image, in which black represents hiding, white represents display, and gray represents semi-transparency or partial display. Each mask image in the mask image set is different. The mask image is fused with the contrast image using an image fusion algorithm in order to generate an occlusion image with random occluders.

[0042] In one embodiment, the generation of the mask picture set includes:

[0043] Randomly generate a preset number of grayscale images of a first size;

[0044] Using a preset difference algorithm to change the first size of the grayscale image into a second size to obtain a mask image;

[0045] The preset number of mask images are combined into the mask image set.

[0046] Specifically, the preset number is set according to demand, which can be 500, the first size can be 8*8, and the random generation refers to a random value of 0 or 1 for each pixel in the grayscale image, 0 represents transparency, and 1 represents black. The second size is the same as the size of the comparison image sent by the above user, that is, the grayscale image of the first size is converted into a grayscale image of the same size as the comparison image using a preset difference algorithm. The difference algorithm is an algorithm for filling blank pixels in the enlarged image when the image is enlarged, estimating the value of unknown pixels through known pixels, thereby filling the gaps between images. It can be understood that the pixel value in the grayscale image of the first size is 0 or 1. When the difference algorithm converts the grayscale image into the second size, the pixel value in the image not only includes 0 and 1, but also includes values ​​between 0 and 1, for example, 0.5 or 0.4.

[0047] In one embodiment, the generation of the mask picture set includes:

[0048] Creating a new folder containing a preset number of blank images, extracting the blank images from the folder using a preset script, obtaining a third size of the blank images, and determining a value range according to the third size;

[0049] Using a preset random number generation function to generate a set of random numbers in the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank picture according to the random array to obtain an initial picture;

[0050] Using a preset difference algorithm to change the third size of the initial image into a fourth size to obtain a mask image;

[0051] Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

[0052] Specifically, the size of the blank image is the same as the size of the comparison image, and the preset number can be 500. The blank images are extracted from the folder in sequence through the preset script. The value interval is determined according to the third size of the blank image. For example, the blank image is composed of 8*8, and the value interval is [0, 8]. After the value interval is determined, a random array is generated using a random number generation function. It can be understood that the random array is in the form of coordinates, for example, (7, 8) (3, 5) (8, 3), and the values ​​of X and Y in the coordinates are within the value interval. The occlusion area drawing function is used to draw the occlusion area in the blank image according to the random array. Specifically, the values ​​of the coordinate points corresponding to the random array in the blank image are set to 1, and the remaining coordinate points are set to 0, thereby generating an initial image with a value of 0 or 1, wherein 0 represents a transparent area and 1 represents a black area. Then, the initial image is enlarged using a difference algorithm to generate a mask image of the same size as the comparison image sent by the user. It can be understood that the values ​​of the pixels in the mask image enlarged by the difference algorithm are between 0 and 1, that is, it includes black, transparent and gray areas. Then repeat the above steps to turn all blank images into mask images and generate a mask image set.

[0053] In this embodiment, a corresponding occluded image is generated by fusing a randomly generated mask image with two original images to be compared, so as to facilitate subsequent comparative analysis of the original image and the occluded image.

[0054] S2. Use a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extract the feature vector of the second contrast image to obtain a second vector, extract the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract the feature vector of the second occlusion picture set to obtain a second occlusion vector set.

[0055] In one embodiment, the image feature extraction network is used to extract feature vectors of an image. The image feature extraction network may be a convolutional neural network (CNN), a graph neural network (GNN), a DenseNet, a Res2Net, a multi-scale backbone network, etc.

[0056] S3. Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set.

[0057] The first cosine value set, the second cosine value set and the third cosine value set are calculated using a preset cosine algorithm. In one embodiment, the formula of the preset cosine algorithm is:

[0058]

[0059] Among them, S(A,B) represents the similarity between feature vector A and feature vector B, A·B represents the dot product of feature vector A and B, and M A represents the value corresponding to the eigenvector A, M B Represents the value corresponding to the eigenvector B.

[0060] For example, the cosine value of each occlusion vector in the first occlusion vector set is calculated using a preset similarity algorithm:

[0061]

[0062] Where S1(A1,Bi) represents the cosine value of the first vector A1 and the i-th eigenvector Bi in the first occlusion vector set, A1·Bi represents the dot product of the first vector A1 and the i-th eigenvector Bi in the first occlusion vector set, M A1 represents the value corresponding to the first vector A1, M Bi Represents the value corresponding to the i-th eigenvector Bi in the first occlusion vector set.

[0063] The method of calculating the cosine value of each occlusion vector in the second occlusion vector set and the second occlusion vector set by using a preset similarity algorithm is as follows:

[0064]

[0065] Where S2(A2,bi) represents the cosine value of the second vector A2 and the i-th eigenvector bi in the second occlusion vector set, A2·bi represents the dot product of the second vector A2 and the i-th eigenvector bi in the second occlusion vector set, M A2 Indicates the value corresponding to the second vector A2, M bi Represents the value corresponding to the i-th eigenvector bi in the second occlusion vector set.

[0066] The method uses a preset similarity algorithm to calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set:

[0067]

[0068] Among them, the i-th occlusion vector in the first occlusion vector set and the i-th occlusion vector in the second occlusion vector set correspond to the same mask image. S3(Bi,bi) represents the cosine value of the i-th feature vector Bi in the first occlusion vector set and the i-th feature vector bi in the second occlusion vector set, Bi·bi represents the dot product of the i-th feature vector Bi in the first occlusion vector set and the i-th feature vector bi in the second occlusion vector set, MBi represents the value corresponding to the i-th eigenvector Bi in the first occlusion vector set, M bi Represents the value corresponding to the i-th eigenvector bi in the second occlusion vector set.

[0069] S4. Calculate a first correlation coefficient set based on the first cosine value set and the pixel value set of the corresponding coordinates of all mask images, calculate a second correlation coefficient set based on the second cosine value set and the pixel value set, calculate a third correlation coefficient set based on the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set, and each correlation coefficient set is calculated using a preset correlation coefficient algorithm.

[0070] In one embodiment, the preset correlation coefficient algorithm is the Pearson correlation coefficient, which is a statistic used to measure the degree of linear correlation between two variables X and Y (two cosine values), and its value is between -1 and 1. When the two variables are completely positively correlated, the correlation coefficient is 1; when the two variables are completely negatively correlated, the correlation coefficient is -1; when there is no linear relationship between the two variables, the correlation coefficient is 0.

[0071] Specifically, the formula of the Pearson correlation coefficient is:

[0072]

[0073] Among them, x is the cosine value set, y is the pixel value set composed of the pixel values ​​of the corresponding coordinates of all mask images, cov(x,y) refers to the covariance of the cosine set and the pixel value set, and σ refers to the standard deviation of the cosine set and the pixel value set.

[0074] Taking s1 as an example, x is the first cosine value set (S10, S11, ..., S1 n ), y is the pixel value set corresponding to a certain coordinate (for example, the coordinate at (0,0)) in all mask images (if there are 500 mask images, there are 500 pixel values). The first cosine value set is calculated with the pixel value set corresponding to each coordinate in all mask images in turn to obtain the correlation coefficient corresponding to each coordinate position, and then the correlation coefficient corresponding to each coordinate position is combined into the first correlation coefficient set.

[0075] In one embodiment, obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set, comprises:

[0076] Sequentially extract the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set;

[0077] Calculate the average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and form the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set;

[0078] Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set;

[0079] The fifth correlation coefficient is obtained by calculating the average value of the third correlation coefficient and the second correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

[0080] Specifically, the correlation coefficients corresponding to the same coordinate position in the first correlation coefficient set and the second correlation coefficient set are added and averaged to obtain the Pearson correlation coefficient set of the second comparison image relative to the first comparison image. Similarly, the correlation coefficients corresponding to the same coordinate position in the third correlation coefficient set and the second correlation coefficient set are added and averaged to obtain the Pearson correlation coefficient set of the first comparison image relative to the second comparison image.

[0081] This embodiment calculates the Pearson correlation coefficient of the comparison image based on the cosine scores of the comparison image and the occlusion image. According to the value of the Pearson correlation coefficient, the positive correlation and negative correlation between the two comparison images and the corresponding correlation degree can be directly obtained.

[0082] S5. Generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

[0083] In one embodiment, generating a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, and generating a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, includes:

[0084] Extracting all positive correlation coefficients in the fourth correlation coefficient set, and overlaying the positive correlation coefficients into the second comparison image to obtain a first positive correlation heat map;

[0085] Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map;

[0086] Extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map;

[0087] All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid on the first comparison image to obtain a second negative correlation heat map.

[0088] Specifically, the positive / negative correlation coefficient is covered to the first / second comparison image by marking the image coordinate position corresponding to the correlation coefficient with different colors. The first positive correlation heat map can intuitively reflect the positive correlation position of the second comparison image compared with the first comparison image, that is, the position with high similarity.

[0089] In one embodiment, returning the generated heat map to the user may be:

[0090] Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map;

[0091] Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map;

[0092] The first heat map and the second heat map are returned to the user.

[0093] In this embodiment, when analyzing multiple groups of comparative pictures at the same time, in order to reduce the pressure of picture transmission and improve the efficiency of transmission, the positive correlation heat map and the negative correlation heat map corresponding to the same picture can be combined to obtain two heat maps, thereby improving the efficiency of picture transmission. Two different colors are used in the first heat map to mark the positive correlation area and the negative correlation area respectively, and the same is true for the second heat map.

[0094] In this embodiment, by constructing a visualized heat map based on the Pearson correlation coefficient, the shortcomings of the recognition model can be clearly seen, that is, which parts of the face need to be improved in similarity, and targeted optimization can be performed on the area to improve the efficiency of model optimization and development.

[0095] In the above technical solution provided by the present embodiment, the first contrast image and the second contrast image are respectively fused with the preset mask image set to generate the first occlusion image set and the second occlusion image set for subsequent comparative analysis between the original image and the occlusion image; the first vector, the second vector, the first occlusion vector set and the second occlusion vector set are extracted using the image feature extraction network; the first cosine value set, the second cosine value set and the third cosine value set are calculated using the preset cosine algorithm, and the similarity value between the contrast image and the occlusion image is calculated using the cosine algorithm; and the Pearson correlation coefficient algorithm is then used to calculate the similarity between the second contrast image and the occlusion image. By calculating the Pearson correlation coefficient of the fourth correlation coefficient set of the first comparison image and the fifth correlation coefficient set of the first comparison image compared with the second comparison image, the positive correlation degree and the negative correlation degree between the first comparison image and the second comparison image can be directly obtained, and then the corresponding heat map is generated according to the above correlation coefficient set and returned to the user. By generating a visual heat map, the similarities and differences between the face images after comparison can be intuitively reflected, and it is found out which areas of the face recognition model need to improve the similarity, and the area can be optimized in a targeted manner to improve the efficiency of model optimization and development.

[0096] like Figure 3 FIG. 1 is a schematic diagram of the structure of the visual analysis device for comparing facial images of the present invention.

[0097] The visual analysis device 100 for face image comparison of the present invention can be installed in a device. According to the functions to be implemented, the visual analysis device for face image comparison can include an occlusion picture generation module 101, an image feature extraction module 102, a cosine value calculation module 103, a correlation coefficient calculation module 104, and a heat map return module 105. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a device processor and can complete fixed functions, which are stored in the memory of the device.

[0098] In this embodiment, the functions of each module / unit are as follows:

[0099] The occlusion image generation module 101 is used to obtain a first comparison image and a second comparison image after receiving a face image comparison request sent by a user, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set;

[0100] Image feature extraction module 102: used to use a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extract the feature vector of the second contrast image to obtain a second vector, extract the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract the feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0101] The cosine value calculation module 103 is used to calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0102] Correlation coefficient calculation module 104: used to calculate a first correlation coefficient set according to the first cosine value set and the pixel value sets of the corresponding coordinates of all mask images, calculate a second correlation coefficient set according to the second cosine value set and the pixel value set, calculate a third correlation coefficient set according to the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set;

[0103] Heat map returning module 105: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

[0104] In one embodiment, the occlusion picture generation module 101 is specifically used to:

[0105] Extracting mask images from the mask image set one by one;

[0106] Calling a preset image fusion algorithm to fuse the mask image and the first comparison image to obtain a first occlusion image, and combining all the first occlusion images into the first occlusion image set;

[0107] The mask image and the second contrast image are fused by using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

[0108] In one embodiment, the occlusion picture generation module 101 is specifically used to:

[0109] Randomly generate a preset number of grayscale images of a first size;

[0110] Using a preset difference algorithm to change the first size of the grayscale image into a second size to obtain a mask image;

[0111] The preset number of mask images are combined into the mask image set.

[0112] In one embodiment, the occlusion picture generation module 101 is further used for:

[0113] Creating a new folder containing a preset number of blank images, extracting the blank images from the folder using a preset script, obtaining a third size of the blank images, and determining a value range according to the third size;

[0114] Using a preset random number generation function to generate a set of random numbers in the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank picture according to the random array to obtain an initial picture;

[0115] Using a preset difference algorithm to change the third size of the initial image into a fourth size to obtain a mask image;

[0116] Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

[0117] In one embodiment, the correlation coefficient calculation module 104 is specifically used to:

[0118] Sequentially extract the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set;

[0119] Calculate the average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and form the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set;

[0120] Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set;

[0121] The fifth correlation coefficient is obtained by calculating the average value of the third correlation coefficient and the second correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

[0122] In one embodiment, the heat map return module 105 is specifically used to:

[0123] Extracting all positive correlation coefficients in the fourth correlation coefficient set, and overlaying the positive correlation coefficients into the second comparison image to obtain a first positive correlation heat map;

[0124] Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map;

[0125] Extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map;

[0126] All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid on the first comparison image to obtain a second negative correlation heat map.

[0127] In one embodiment, the heat map return module 105 is specifically used to:

[0128] Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map;

[0129] Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map;

[0130] The first heat map and the second heat map are returned to the user.

[0131] The present invention provides a visual analysis device for comparing facial images. The device first generates a first occluded image set and a second occluded image set by fusing a first comparison image and a second comparison image with a preset mask image set respectively, then extracts corresponding feature adjacencies, calculates a corresponding correlation coefficient set by calculating a cosine value and a Pearson correlation coefficient, and finally generates a corresponding heat map based on the correlation coefficient set and returns it to a user. By generating a visual heat map, the similarities and differences between facial images after comparison can be intuitively reflected, and it is found out which areas of a facial recognition model need to be improved in terms of similarity in recognizing a face, and the area is optimized in a targeted manner, thereby improving the efficiency of model optimization and development.

[0132] For the specific definition of the visual analysis device for facial image comparison, please refer to the definition of the intelligent question and answer method above, which will not be repeated here. Each module in the above-mentioned visual analysis device for facial image comparison can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, the functions or steps on the server side of a visual analysis method for facial image comparison are implemented.

[0134] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements a function or step on the client side of a visual analysis of facial image comparison

[0135] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0136] After receiving a face image comparison request sent by a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0137] Using a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extracting the feature vector of the second contrast image to obtain a second vector, extracting the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extracting the feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0138] Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0139] Calculating a first correlation coefficient set according to the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set according to the second cosine value set and the pixel value set, calculating a third correlation coefficient set according to the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set;

[0140] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0141] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0142] After receiving a face image comparison request sent by a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0143] Using a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extracting the feature vector of the second contrast image to obtain a second vector, extracting the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extracting the feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0144] Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0145] Calculating a first correlation coefficient set according to the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set according to the second cosine value set and the pixel value set, calculating a third correlation coefficient set according to the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set;

[0146] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0147] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0148] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0149] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0150] The above-described embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should be included in the protection scope of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for example introduction and do not represent actual use.

Claims

1. A visual analysis method for face image comparison, characterized in that: The method comprises: After receiving a face image comparison request sent by a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set; Using a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extracting the feature vector of the second contrast image to obtain a second vector, extracting the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extracting the feature vector of the second occlusion picture set to obtain a second occlusion vector set; Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set; Calculating a first correlation coefficient set according to the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set according to the second cosine value set and the pixel value set, calculating a third correlation coefficient set according to the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set; A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

2. The visual analysis method for facial image comparison according to claim 1, characterized in that: The step of fusing the first contrast image with a preset mask picture set to generate a first occlusion picture set, and fusing the second contrast image with the mask picture set to generate a second occlusion picture set includes: Extracting mask images from the mask image set one by one; Calling a preset image fusion algorithm to fuse the mask image and the first comparison image to obtain a first occlusion image, and combining all the first occlusion images into the first occlusion image set; The mask image and the second contrast image are fused by using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

3. The visual analysis method for facial image comparison according to claim 1, characterized in that: The generation of the mask image set includes: Randomly generate a preset number of grayscale images of a first size; Using a preset difference algorithm to change the first size of the grayscale image into a second size to obtain a mask image; The preset number of mask images are combined into the mask image set.

4. The visual analysis method for facial image comparison according to claim 1, characterized in that: The generation of the mask image set includes: Create a new folder containing a preset number of blank pictures, extract the blank pictures from the folder using a preset script, obtain the third size of the blank pictures, and determine a value range according to the third size; Using a preset random number generation function to generate a set of random numbers in the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank picture according to the random array to obtain an initial picture; Using a preset difference algorithm to change the third size of the initial image into a fourth size to obtain a mask image; Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

5. The visual analysis method for facial image comparison according to claim 1, characterized in that: The step of obtaining a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set, comprises: Sequentially extract the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set; Calculate the average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and form the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set; Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set; The fifth correlation coefficient is obtained by calculating the average value of the third correlation coefficient and the second correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

6. The visual analysis method for facial image comparison according to claim 1, characterized in that: The generating of a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, and generating a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, comprises: Extracting all positive correlation coefficients in the fourth correlation coefficient set, and overlaying the positive correlation coefficients into the second comparison image to obtain a first positive correlation heat map; Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map; Extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map; All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid on the first comparison image to obtain a second negative correlation heat map.

7. The visual analysis method for facial image comparison according to claim 1, characterized in that: The step of returning the generated heat map to the user includes: Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map; Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map; The first heat map and the second heat map are returned to the user.

8. A visual analysis device for comparing facial images, characterized in that: The device comprises: The occlusion image generation module is used to obtain a first comparison image and a second comparison image after receiving a face image comparison request sent by a user, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set; Image feature extraction module: used to use a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extract the feature vector of the second contrast image to obtain a second vector, extract the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract the feature vector of the second occlusion picture set to obtain a second occlusion vector set; A cosine value calculation module: used for calculating the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set; A correlation coefficient calculation module: used to calculate a first correlation coefficient set according to the first cosine value set and the pixel value sets of the corresponding coordinates of all mask images, calculate a second correlation coefficient set according to the second cosine value set and the pixel value set, calculate a third correlation coefficient set according to the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set; Heat map return module: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat map to the user.

9. A device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the visual analysis method for facial image comparison as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the visual analysis method for facial image comparison as described in any one of claims 1 to 7 is implemented.

Images