Visual analysis method, device, equipment and storage medium for facial image comparison

Abstract

The present invention relates to the fields of image detection and financial technology, and relates to a visual analysis method, device, equipment and storage medium for facial image comparison. The method comprises: generating a first occlusion image set and a second occlusion image set by fusing a first comparison image and a second comparison image with a preset mask image set, extracting a first vector, a second vector, a first occlusion vector set and a second occlusion vector set using an image feature extraction network, calculating a first cosine value set, a second cosine value set and a third cosine value set, calculating a fourth correlation coefficient set and a fifth correlation coefficient set using a Pearson correlation coefficient algorithm, and then generating a corresponding heat map based on the above correlation coefficient set and returning it to the user, thereby generating a visual heat map. By generating a visual heat map, the present invention can intuitively reflect the similarities and differences after comparing facial images.

Description

Technical Field

[0001] The present invention relates to the fields of image detection and financial technology, and relates to a visual analysis method, device, equipment and storage medium for facial image comparison. Background Art

[0002] Facial recognition, a form of biometric identification, is currently widely used in the financial technology sector, including identity verification, payment systems, bank card activation and withdrawals, and personal financial management. Facial recognition systems extract and compare key features in facial images to determine the similarity between two faces for authentication and identification. However, existing comparison systems also have the following issues:

[0003] 1) Face recognition systems are opaque. As a black-box neural network model, the face recognition model is very complex and difficult to understand. The system directly uses the final similarity score to represent the similarity between two faces, which reduces the system's credibility.

[0004] 2) Existing interpretability methods, such as activation heatmaps and gradient-based methods, primarily target classification networks and the feature output of a single image. They cannot intuitively show which parts of the two compared images differ from each other.

[0005] 3) When the recognition model fails to match (the same person is identified as different people because the score is below the threshold, and different people are identified as the same person because the score is too high), there is nothing we can do and we cannot make correct improvements.

[0006] Therefore, how to intuitively reflect the similarities and differences after comparing facial images has become a technical problem that needs to be solved urgently. Summary of the Invention

[0007] In view of the above, it is necessary to provide a visual analysis method for face image comparison to solve the problem that the similarities and differences between the face images after comparison cannot be intuitively reflected during the face image comparison process.

[0008] To achieve the above object, the present invention provides a visual analysis method for facial image comparison, the method comprising:

[0009] After receiving a face image comparison request from a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0010] Using a preset image feature extraction network, extracting a feature vector of the first contrast image to obtain a first vector, extracting a feature vector of the second contrast image to obtain a second vector, extracting feature vectors of the first occlusion picture set to obtain a first occlusion vector set, and extracting feature vectors of the second occlusion picture set to obtain a second occlusion vector set;

[0011] Calculating the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0012] Calculating a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set based on the second cosine value set and the pixel value set, calculating a third correlation coefficient set based on the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set;

[0013] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0014] In addition, to achieve the above-mentioned purpose, the present invention further provides a visual analysis device for comparing facial images, the device comprising:

[0015] The occlusion image generation module is configured to, upon receiving a face image comparison request from a user, obtain a first comparison image and a second comparison image, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set;

[0016] An image feature extraction module: configured to use a preset image feature extraction network to extract a feature vector of the first contrast image to obtain a first vector, extract a feature vector of the second contrast image to obtain a second vector, extract a feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract a feature vector of the second occlusion picture set to obtain a second occlusion vector set;

[0017] a cosine value calculation module configured to calculate the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0018] a correlation coefficient calculation module configured to calculate a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculate a second correlation coefficient set based on the second cosine value set and the pixel value set, calculate a third correlation coefficient set based on the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set;

[0019] Heat map return module: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

[0020] In addition, to achieve the above object, the present invention further provides a device, comprising:

[0021] a memory storing at least one computer program; and

[0022] The processor executes the program stored in the memory to implement the above-mentioned visual analysis method for facial image comparison.

[0023] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in the device to implement the above-mentioned visual analysis method for facial image comparison.

[0024] In the above-mentioned technical solution provided by the present invention, a first occlusion picture set and a second occlusion picture set are generated by respectively fusing the first comparison image and the second comparison image with a preset mask picture set for subsequent comparison analysis of the original image and the occlusion picture; the first vector, the second vector, the first occlusion vector set and the second occlusion vector set are extracted using an image feature extraction network; the first cosine value set, the second cosine value set and the third cosine value set are calculated, and the similarity value between the comparison image and the occlusion image is calculated by the cosine algorithm; the fourth correlation coefficient set of the second comparison image compared with the first comparison image and the fifth correlation coefficient set of the first comparison image compared with the second comparison image are calculated using the Pearson correlation coefficient algorithm, and then a corresponding heat map is generated according to the above-mentioned correlation coefficient set and returned to the user. By generating a visual heat map, the similarities and differences between the face images after comparison can be intuitively reflected, and it is possible to find out which areas of the face recognition model need to improve their similarity, and perform targeted optimization on these areas to improve the efficiency of model optimization and development. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 1 is a schematic diagram of an application environment of a visual analysis method for facial image comparison according to an embodiment of the present invention;

[0026] Figure 2 A schematic diagram of a flow chart of a visual analysis method for facial image comparison provided by one embodiment of the present invention;

[0027] Figure 3 A schematic structural diagram of a visual analysis device for facial image comparison provided by one embodiment of the present invention;

[0028] Figure 4 is a structural diagram of a computer device in one embodiment of the present invention;

[0029] Figure 5 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention.

[0030] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0032] It should be noted that the descriptions of "first", "second", etc. in the present invention are for descriptive purposes only and should not be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of such features. In addition, the technical solutions between the various embodiments can be combined with each other, but this must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0033] The visual analysis method for face image comparison provided by the embodiment of the present invention can be applied in the following fields: Figure 1In an application environment, the client communicates with the server through a network. After receiving the face image comparison request sent by the user, the server obtains the first comparison image and the second comparison image, fuses the first comparison image with the preset mask image set to generate the first occlusion image set, and fuses the second comparison image with the mask image set to generate the second occlusion image set; uses a preset image feature extraction network to extract the feature vector of the first comparison image to obtain the first vector, extracts the feature vector of the second comparison image to obtain the second vector, extracts the feature vector of the first occlusion image set to obtain the first occlusion vector set, and extracts the feature vector of the second occlusion image set to obtain the second occlusion vector set; calculates the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain the first cosine value set, calculates the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain the second cosine value set, calculates the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain the third cosine value set; according to the first A first correlation coefficient set is calculated based on a cosine value set and a pixel value set corresponding to the coordinates of all mask images; a second correlation coefficient set is calculated based on the second cosine value set and the pixel value set; a third correlation coefficient set is calculated based on the third cosine value set and the pixel value set; a fourth correlation coefficient set is obtained based on the first correlation coefficient set and the second correlation coefficient set; and a fifth correlation coefficient set is obtained based on the third correlation coefficient set and the second correlation coefficient set; a first positive correlation heat map and a first negative correlation heat map are generated based on the fourth correlation coefficient set, and a second positive correlation heat map and a second negative correlation heat map are generated based on the fifth correlation coefficient set, and the generated heat maps are returned to the user. In the present invention, by generating a visual heat map for comparing facial images, the similarities and differences between the facial images after comparison can be intuitively reflected, and the face recognition model can identify which areas of the face need to be improved in terms of similarity, and perform targeted optimization on these areas, thereby improving the efficiency of model optimization and development. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. The server can be implemented as an independent server or a server cluster consisting of multiple servers. The present invention is described in detail below through specific examples.

[0034] Reference Figure 2 FIG. 1 is a flow chart of a visual analysis method for comparing facial images according to an embodiment of the present invention. In the embodiment of the present invention, the visual analysis method for comparing facial images includes the following steps S1-S5:

[0035] S1. After receiving a face image comparison request sent by a user, obtain a first comparison image and a second comparison image, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set.

[0036] In one embodiment, after receiving a facial image comparison request sent by a user, the comparison request is parsed to obtain a first comparison image and a second comparison image for mutual comparison, and then the first comparison image and the second comparison image are respectively fused with a preset mask image set to generate a corresponding first occlusion image set and a second occlusion image set.

[0037] In one embodiment, fusing the first contrast image with a preset mask picture set to generate a first occlusion picture set, and fusing the second contrast image with the mask picture set to generate a second occlusion picture set, includes:

[0038] Extracting mask images from the mask image set one by one;

[0039] Invoking a preset image fusion algorithm to fuse the mask image and the first contrast image to obtain a first occlusion image, and combining all the first occlusion images into a first occlusion image set;

[0040] The mask image and the second contrast image are fused using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

[0041] Specifically, the mask image is a grayscale image with pixel values ​​between 0 and 1. Essentially, a mask image is a grayscale image, where black indicates concealment, white indicates visibility, and gray indicates semi-transparency or partial visibility. Each mask image in the mask image set is unique. The mask image is fused with the comparison image using an image fusion algorithm to generate an occlusion image with random occluders.

[0042] In one embodiment, the generation of the mask picture set includes:

[0043] Randomly generate a preset number of grayscale images of the first size;

[0044] Using a preset difference algorithm to convert the first size of the grayscale image into a second size to obtain a mask image;

[0045] The preset number of mask images are combined into the mask image set.

[0046] Specifically, the preset number is set according to demand and can be 500. The first size can be 8*8. The random generation refers to a random value of 0 or 1 for each pixel in the grayscale image, 0 represents transparency, and 1 represents black. The second size is the same as the size of the comparison image sent by the above-mentioned user, that is, the preset difference algorithm is used to convert the grayscale image of the first size into a grayscale image of the same size as the comparison image. The difference algorithm is an algorithm for filling blank pixels in the enlarged image when the image is enlarged, and the value of unknown pixels is estimated by known pixels to fill the gaps between the images. It can be understood that the pixel value in the grayscale image of the first size is 0 or 1. When the difference algorithm converts the grayscale image into the second size, the pixel values ​​in the image include not only 0 and 1, but also values ​​between 0 and 1, for example, 0.5 or 0.4.

[0047] In one embodiment, the generation of the mask picture set includes:

[0048] Creating a new folder containing a preset number of blank images, extracting the blank images from the folder using a preset script, obtaining a third size of the blank images, and determining a value range based on the third size;

[0049] Using a preset random number generation function to generate a set of random numbers within the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank image according to the random array to obtain an initial image;

[0050] Using a preset difference algorithm to change the third size of the initial image to a fourth size to obtain a mask image;

[0051] Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

[0052] Specifically, the size of the blank image is the same as the size of the comparison image, and the preset number can be 500. The blank images are extracted from the folder in sequence through a preset script. The value interval is determined according to the third size of the blank image. For example, if the blank image is composed of 8*8, the value interval is [0, 8]. After determining the value interval, a random array is generated using a random number generation function. It can be understood that the random array is in the form of coordinates, for example, (7, 8) (3, 5) (8, 3), and the X and Y values ​​in the coordinates are within the value interval. The occlusion area drawing function is used to draw the occlusion area in the blank image according to the random array. Specifically, the values ​​of the coordinate points corresponding to the random array in the blank image are set to 1, and the remaining coordinate points are set to 0, thereby generating an initial image with a value of 0 or 1, where 0 represents a transparent area and 1 represents a black area. The initial image is then enlarged using a difference algorithm to generate a mask image of the same size as the comparison image sent by the user. It can be understood that the pixel values ​​in the mask image enlarged by the difference algorithm are between 0 and 1, that is, including black, transparent and gray areas. Then repeat the above steps to turn all blank images into mask images and generate a mask image set.

[0053] In this embodiment, a corresponding occlusion image is generated by fusing a randomly generated mask image with two original images to be compared, so as to facilitate subsequent comparative analysis of the original image and the occlusion image.

[0054] S2. Use a preset image feature extraction network to extract the feature vector of the first contrast image to obtain a first vector, extract the feature vector of the second contrast image to obtain a second vector, extract the feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract the feature vector of the second occlusion picture set to obtain a second occlusion vector set.

[0055] In one embodiment, the image feature extraction network is used to extract feature vectors of an image. The image feature extraction network can be a convolutional neural network (CNN), a graph neural network (GNN), a DenseNet, a Res2Net, a multi-scale backbone network, etc.

[0056] S3. Calculate the cosine value of the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value of the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set.

[0057] The first cosine value set, the second cosine value set, and the third cosine value set are calculated using a preset cosine algorithm. In one embodiment, the formula of the preset cosine algorithm is:

[0058]

[0059] Among them, S(A,B) represents the similarity between feature vector A and feature vector B, A·B represents the dot product of feature vector A and B, and M A Represents the value corresponding to the eigenvector A, M B Represents the value corresponding to the eigenvector B.

[0060] For example, the cosine value of the first vector and each occlusion vector in the first occlusion vector set is calculated using a preset similarity algorithm:

[0061]

[0062] Among them, S1(A1,Bi) represents the cosine value of the first vector A1 and the i-th eigenvector Bi in the first occlusion vector set, A1·Bi represents the dot product of the first vector A1 and the i-th eigenvector Bi in the first occlusion vector set, M A1 Indicates the value corresponding to the first vector A1, M Bi Represents the value corresponding to the i-th eigenvector Bi in the first occlusion vector set.

[0063] The method further comprises calculating the cosine value of each occlusion vector in the second occlusion vector set by using a preset similarity algorithm:

[0064]

[0065] Among them, S2(A2,bi) represents the cosine value of the second vector A2 and the i-th eigenvector bi in the second occlusion vector set, A2·bi represents the dot product of the second vector A2 and the i-th eigenvector bi in the second occlusion vector set, M A2 Indicates the value corresponding to the second vector A2, M bi Represents the value corresponding to the i-th eigenvector bi in the second occlusion vector set.

[0066] The preset similarity algorithm is used to calculate the cosine value of each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set:

[0067]

[0068] Among them, the i-th occlusion vector in the first occlusion vector set and the i-th occlusion vector in the second occlusion vector set correspond to the same mask image. S3(Bi,bi) represents the cosine value of the i-th eigenvector Bi in the first occlusion vector set and the i-th eigenvector bi in the second occlusion vector set, Bi·bi represents the dot product of the i-th eigenvector Bi in the first occlusion vector set and the i-th eigenvector bi in the second occlusion vector set, MBi Represents the value corresponding to the i-th eigenvector Bi in the first occlusion vector set, M bi Represents the value corresponding to the i-th eigenvector bi in the second occlusion vector set.

[0069] S4. Calculate a first correlation coefficient set based on the first cosine value set and the pixel value set of the corresponding coordinates of all mask images, calculate a second correlation coefficient set based on the second cosine value set and the pixel value set, calculate a third correlation coefficient set based on the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set, and each correlation coefficient set is calculated using a preset correlation coefficient algorithm.

[0070] In one embodiment, the preset correlation coefficient algorithm is the Pearson correlation coefficient. The Pearson correlation coefficient is a statistic used to measure the degree of linear correlation between two variables X and Y (two cosine values), and its value is between -1 and 1. When the two variables are completely positively correlated, the correlation coefficient is 1; when the two variables are completely negatively correlated, the correlation coefficient is -1; when there is no linear relationship between the two variables, the correlation coefficient is 0.

[0071] Specifically, the formula for the Pearson correlation coefficient is:

[0072]

[0073] Among them, x is the cosine value set, y is the pixel value set composed of the pixel values ​​of the corresponding coordinates of all mask images, cov(x,y) refers to the covariance of the cosine set and the pixel value set, and σ refers to the standard deviation of the cosine set and the pixel value set.

[0074] Taking s1 as an example, x is the first cosine value set (S10, S11, ..., S1 n ), y is the set of pixel values ​​corresponding to a certain coordinate (for example, the coordinate at (0,0)) in all mask images (if there are 500 mask images, there are 500 pixel values). After sequentially calculating the first cosine value set with the set of pixel values ​​corresponding to each coordinate in all mask images, a correlation coefficient corresponding to each coordinate position is obtained, and then the correlation coefficient corresponding to each coordinate position is combined into the first correlation coefficient set.

[0075] In one embodiment, obtaining a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set, includes:

[0076] Sequentially extracting the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set;

[0077] Calculating an average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and forming the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set;

[0078] Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set;

[0079] The average value of the third correlation coefficient and the second correlation coefficient is calculated to obtain a fifth correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

[0080] Specifically, the correlation coefficients corresponding to the same coordinate position in the first correlation coefficient set and the second correlation coefficient set are added and averaged to obtain a Pearson correlation coefficient set of the second comparison image relative to the first comparison image. Similarly, the correlation coefficients corresponding to the same coordinate position in the third correlation coefficient set and the second correlation coefficient set are added and averaged to obtain a Pearson correlation coefficient set of the first comparison image relative to the second comparison image.

[0081] This embodiment calculates the Pearson correlation coefficient of the comparison image based on the cosine scores of the comparison image and the occlusion image. According to the value of the Pearson correlation coefficient, the positive correlation and negative correlation between the two comparison images and the corresponding correlation degree can be directly obtained.

[0082] S5. Generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

[0083] In one embodiment, generating a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, and generating a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, includes:

[0084] Extracting all positive correlation coefficients from the fourth correlation coefficient set, and overlaying the positive correlation coefficients onto the second comparison image to obtain a first positive correlation heat map;

[0085] Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map;

[0086] extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map;

[0087] All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid onto the first comparison image to obtain a second negative correlation heat map.

[0088] Specifically, overlaying the positive / negative correlation coefficients onto the first / second comparison images is specifically to mark the image coordinate positions corresponding to the correlation coefficients with different colors. The first positive correlation heat map can intuitively reflect the positive correlation positions of the second comparison image compared to the first comparison image, i.e., positions with high similarity.

[0089] In one embodiment, returning the generated heat map to the user may be:

[0090] Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map;

[0091] Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map;

[0092] The first heat map and the second heat map are returned to the user.

[0093] In this embodiment, when analyzing multiple sets of comparative images simultaneously, in order to reduce the pressure of image transmission and improve transmission efficiency, the positive correlation heat map and negative correlation heat map corresponding to the same image can be combined to obtain two heat maps, thereby improving the efficiency of image transmission. In the first heat map, two different colors are used to mark the positive correlation area and the negative correlation area respectively, and the same is true for the second heat map.

[0094] In this embodiment, by constructing a visual heat map based on the Pearson correlation coefficient, the shortcomings of the recognition model can be clearly seen, that is, which parts of the face need to be improved in similarity, and targeted optimization can be performed on these areas to improve the efficiency of model optimization and development.

[0095] In the above technical solution provided by this embodiment, the first contrast image and the second contrast image are respectively fused with the preset mask image set to generate the first occlusion image set and the second occlusion image set for subsequent comparative analysis between the original image and the occlusion image; the first vector, the second vector, the first occlusion vector set and the second occlusion vector set are extracted using the image feature extraction network; the first cosine value set, the second cosine value set and the third cosine value set are calculated using the preset cosine algorithm, and the similarity value between the contrast image and the occlusion image is calculated using the cosine algorithm; and the Pearson correlation coefficient algorithm is then used to calculate the similarity between the second contrast image and the occlusion image. By calculating the Pearson correlation coefficient of the fourth correlation coefficient set of the first comparison image and the fifth correlation coefficient set of the first comparison image compared with the second comparison image, the positive correlation degree and negative correlation degree between the first comparison image and the second comparison image can be directly obtained, and then the corresponding heat map is generated according to the above correlation coefficient set and returned to the user. By generating a visual heat map, the similarities and differences after the face images are compared can be intuitively reflected, and it is possible to find out which areas of the face recognition model need to improve the similarity, and perform targeted optimization on the area, thereby improving the efficiency of model optimization and development.

[0096] like Figure 3 FIG. 1 is a schematic diagram of the structure of the visual analysis device for comparing facial images of the present invention.

[0097] The visual analysis device 100 for facial image comparison described in the present invention can be installed in a device. Depending on the functionality implemented, the visual analysis device for facial image comparison can include an occlusion image generation module 101, an image feature extraction module 102, a cosine value calculation module 103, a correlation coefficient calculation module 104, and a heat map return module 105. A module, also referred to as a unit, is a series of computer program segments that can be executed by a device processor and perform a fixed function and are stored in the device's memory.

[0098] In this embodiment, the functions of each module / unit are as follows:

[0099] The occlusion image generation module 101 is configured to, upon receiving a face image comparison request from a user, obtain a first comparison image and a second comparison image, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set;

[0100] Image feature extraction module 102: configured to use a preset image feature extraction network to extract feature vectors of the first comparison image to obtain a first vector, extract feature vectors of the second comparison image to obtain a second vector, extract feature vectors of the first occlusion picture set to obtain a first occlusion vector set, and extract feature vectors of the second occlusion picture set to obtain a second occlusion vector set;

[0101] Cosine value calculation module 103: used to calculate the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0102] Correlation coefficient calculation module 104: used to calculate a first correlation coefficient set based on the first cosine value set and the pixel value sets of the corresponding coordinates of all mask images, calculate a second correlation coefficient set based on the second cosine value set and the pixel value set, calculate a third correlation coefficient set based on the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set;

[0103] Heat map return module 105: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

[0104] In one embodiment, the occlusion image generation module 101 is specifically configured to:

[0105] Extracting mask images from the mask image set one by one;

[0106] Invoking a preset image fusion algorithm to fuse the mask image and the first contrast image to obtain a first occlusion image, and combining all the first occlusion images into a first occlusion image set;

[0107] The mask image and the second contrast image are fused using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

[0108] In one embodiment, the occlusion image generation module 101 is specifically configured to:

[0109] Randomly generate a preset number of grayscale images of the first size;

[0110] Using a preset difference algorithm to convert the first size of the grayscale image into a second size to obtain a mask image;

[0111] The preset number of mask images are combined into the mask image set.

[0112] In one embodiment, the occlusion image generation module 101 is further configured to:

[0113] Creating a new folder containing a preset number of blank images, extracting the blank images from the folder using a preset script, obtaining a third size of the blank images, and determining a value range based on the third size;

[0114] Using a preset random number generation function to generate a set of random numbers within the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank image according to the random array to obtain an initial image;

[0115] Using a preset difference algorithm to change the third size of the initial image to a fourth size to obtain a mask image;

[0116] Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

[0117] In one embodiment, the correlation coefficient calculation module 104 is specifically configured to:

[0118] Sequentially extracting the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set;

[0119] Calculating an average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and forming the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set;

[0120] Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set;

[0121] The average value of the third correlation coefficient and the second correlation coefficient is calculated to obtain a fifth correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

[0122] In one embodiment, the heat map return module 105 is specifically configured to:

[0123] Extracting all positive correlation coefficients from the fourth correlation coefficient set, and overlaying the positive correlation coefficients onto the second comparison image to obtain a first positive correlation heat map;

[0124] Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map;

[0125] extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map;

[0126] All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid onto the first comparison image to obtain a second negative correlation heat map.

[0127] In one embodiment, the heat map return module 105 is specifically configured to:

[0128] Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map;

[0129] Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map;

[0130] The first heat map and the second heat map are returned to the user.

[0131] The present invention provides a visual analysis device for comparing facial images. The device first generates a first occlusion image set and a second occlusion image set by fusing a first comparison image and a second comparison image with a preset mask image set respectively. Then, corresponding feature adjacencies are extracted. A corresponding correlation coefficient set is calculated by calculating cosine values ​​and Pearson correlation coefficients. Finally, a corresponding heat map is generated based on the correlation coefficient set and returned to the user. By generating a visual heat map, the similarities and differences between the facial images after comparison can be intuitively reflected, and it is found out which areas of the face recognition model need to improve their similarity, and these areas are optimized in a targeted manner, thereby improving the efficiency of model optimization and development.

[0132] The specific definition of the visual analysis device for facial image comparison can be found in the definition of the intelligent question-answering method above and will not be repeated here. The various modules in the above-mentioned visual analysis device for facial image comparison can be implemented in whole or in part through software, hardware, or a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of the above modules.

[0133] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a visual analysis method for facial image comparison.

[0134] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it realizes a function or step on the client side of a visual analysis of facial image comparison.

[0135] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0136] After receiving a face image comparison request from a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0137] Using a preset image feature extraction network, extracting a feature vector of the first contrast image to obtain a first vector, extracting a feature vector of the second contrast image to obtain a second vector, extracting feature vectors of the first occlusion picture set to obtain a first occlusion vector set, and extracting feature vectors of the second occlusion picture set to obtain a second occlusion vector set;

[0138] Calculating the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0139] Calculating a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set based on the second cosine value set and the pixel value set, calculating a third correlation coefficient set based on the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set;

[0140] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0141] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0142] After receiving a face image comparison request from a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set;

[0143] Using a preset image feature extraction network, extracting a feature vector of the first contrast image to obtain a first vector, extracting a feature vector of the second contrast image to obtain a second vector, extracting feature vectors of the first occlusion picture set to obtain a first occlusion vector set, and extracting feature vectors of the second occlusion picture set to obtain a second occlusion vector set;

[0144] Calculating the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set;

[0145] Calculating a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set based on the second cosine value set and the pixel value set, calculating a third correlation coefficient set based on the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set;

[0146] A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

[0147] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0148] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0149] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. It should be noted that if software tools or components other than those of the company appear in the embodiments of this application, they are only used for example and do not represent actual use.

Claims

1. A visual analysis method for facial image comparison, characterized in that: The method comprises: After receiving a face image comparison request from a user, obtaining a first comparison image and a second comparison image, fusing the first comparison image with a preset mask image set to generate a first occlusion image set, and fusing the second comparison image with the mask image set to generate a second occlusion image set; Using a preset image feature extraction network, extracting a feature vector of the first contrast image to obtain a first vector, extracting a feature vector of the second contrast image to obtain a second vector, extracting feature vectors of the first occlusion picture set to obtain a first occlusion vector set, and extracting feature vectors of the second occlusion picture set to obtain a second occlusion vector set; Calculating the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculating the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculating the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set; Calculating a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculating a second correlation coefficient set based on the second cosine value set and the pixel value set, calculating a third correlation coefficient set based on the third cosine value set and the pixel value set, obtaining a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtaining a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set; A first positive correlation heat map and a first negative correlation heat map are generated according to the fourth correlation coefficient set, a second positive correlation heat map and a second negative correlation heat map are generated according to the fifth correlation coefficient set, and the generated heat maps are returned to the user.

2. The visual analysis method for facial image comparison according to claim 1, characterized in that: The step of fusing the first contrast image with a preset mask picture set to generate a first occlusion picture set, and fusing the second contrast image with the mask picture set to generate a second occlusion picture set, includes: Extracting mask images from the mask image set one by one; Invoking a preset image fusion algorithm to fuse the mask image and the first contrast image to obtain a first occlusion image, and combining all the first occlusion images into a first occlusion image set; The mask image and the second contrast image are fused using the image fusion algorithm to obtain a second occlusion image, and all the second occlusion images are combined into a second occlusion image set.

3. The visual analysis method for facial image comparison according to claim 1, wherein: The generation of the mask image set includes: Randomly generate a preset number of grayscale images of the first size; Using a preset difference algorithm to convert the first size of the grayscale image into a second size to obtain a mask image; The preset number of mask images are combined into the mask image set.

4. The visual analysis method for facial image comparison according to claim 1, wherein: The generation of the mask image set includes: Creating a new folder containing a preset number of blank images, extracting the blank images from the folder using a preset script, obtaining a third size of the blank images, and determining a value range based on the third size; Using a preset random number generation function to generate a set of random numbers within the value range to obtain a random array, and using a preset occlusion area drawing function to draw an occlusion area in the blank image according to the random array to obtain an initial image; Using a preset difference algorithm to change the third size of the initial image to a fourth size to obtain a mask image; Repeat the above steps until a preset number of mask images are generated, and the preset number of mask images are combined into the mask image set.

5. The visual analysis method for facial image comparison according to claim 1, wherein: The obtaining of the fourth correlation coefficient set according to the first correlation coefficient set and the second correlation coefficient set, and the obtaining of the fifth correlation coefficient set according to the third correlation coefficient set and the second correlation coefficient set, comprises: Sequentially extracting the first correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the first correlation coefficient set and the second correlation coefficient set; Calculating an average of the first correlation coefficient and the second correlation coefficient to obtain a fourth correlation coefficient, and forming the fourth correlation coefficient corresponding to each coordinate into the fourth correlation coefficient set; Sequentially extract the third correlation coefficient and the second correlation coefficient corresponding to the same coordinate in the third correlation coefficient set and the second correlation coefficient set; The average value of the third correlation coefficient and the second correlation coefficient is calculated to obtain a fifth correlation coefficient, and the fifth correlation coefficient corresponding to each coordinate is formed into the fifth correlation coefficient set.

6. The visual analysis method for facial image comparison according to claim 1, wherein: The generating of a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, and generating a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, comprises: Extracting all positive correlation coefficients from the fourth correlation coefficient set, and overlaying the positive correlation coefficients onto the second comparison image to obtain a first positive correlation heat map; Extracting all negative correlation coefficients from the fourth correlation coefficient set, and overlaying the negative correlation coefficients onto the second comparison image to obtain a first negative correlation heat map; extracting all positive correlation coefficients from the fifth correlation coefficient set, and overlaying the positive correlation coefficients onto the first comparison image to obtain a second positive correlation heat map; All negative correlation coefficients in the fifth correlation coefficient set are extracted, and the negative correlation coefficients are overlaid onto the first comparison image to obtain a second negative correlation heat map.

7. The visual analysis method for facial image comparison according to claim 1, wherein: The step of returning the generated heat map to the user includes: Merging the first positive correlation heat map and the first negative correlation heat map into a first heat map; Merging the second positive correlation heat map and the second negative correlation heat map into a second heat map; The first heat map and the second heat map are returned to the user.

8. A visual analysis device for facial image comparison, characterized in that: The device comprises: The occlusion image generation module is configured to, upon receiving a face image comparison request from a user, obtain a first comparison image and a second comparison image, fuse the first comparison image with a preset mask image set to generate a first occlusion image set, and fuse the second comparison image with the mask image set to generate a second occlusion image set; An image feature extraction module: configured to use a preset image feature extraction network to extract a feature vector of the first contrast image to obtain a first vector, extract a feature vector of the second contrast image to obtain a second vector, extract a feature vector of the first occlusion picture set to obtain a first occlusion vector set, and extract a feature vector of the second occlusion picture set to obtain a second occlusion vector set; a cosine value calculation module configured to calculate the cosine value between the first vector and each occlusion vector in the first occlusion vector set to obtain a first cosine value set, calculate the cosine value between the second vector and each occlusion vector in the second occlusion vector set to obtain a second cosine value set, and calculate the cosine value between each occlusion vector in the first occlusion vector set and the corresponding occlusion vector in the second occlusion vector set to obtain a third cosine value set; a correlation coefficient calculation module configured to calculate a first correlation coefficient set based on the first cosine value set and pixel value sets of corresponding coordinates of all mask images, calculate a second correlation coefficient set based on the second cosine value set and the pixel value set, calculate a third correlation coefficient set based on the third cosine value set and the pixel value set, obtain a fourth correlation coefficient set based on the first correlation coefficient set and the second correlation coefficient set, and obtain a fifth correlation coefficient set based on the third correlation coefficient set and the second correlation coefficient set; Heat map return module: used to generate a first positive correlation heat map and a first negative correlation heat map according to the fourth correlation coefficient set, generate a second positive correlation heat map and a second negative correlation heat map according to the fifth correlation coefficient set, and return the generated heat maps to the user.

9. A device, characterized in that The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the visual analysis method for facial image comparison as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the visual analysis method for facial image comparison according to any one of claims 1 to 7 is implemented.

Images