Interest region generation method and device, electronic equipment and storage medium

Abstract

The application provides a method and device for generating an area of interest, electronic equipment and a storage medium. The method comprises inputting an obtained image to be processed into a first convolutional neural network trained in advance to obtain a feature vector, wherein the feature vector represents position information of a rectangular area of interest in the image to be processed. Then, the feature vector is binarized by using a set threshold to obtain a binary feature vector, and a vertical feature vector and a horizontal feature vector of the rectangular area of interest are determined based on the binary feature vector. Since the vertical feature vector and the horizontal feature vector can represent the vertical position and the horizontal position of the rectangular area of interest in the image to be processed, respectively, the vertical feature vector and the horizontal feature vector can be further matrixed to obtain a mask image of the rectangular area of interest. In this way, solving a two-dimensional mask is converted into solving a one-dimensional feature vector, the dimension is reduced, the calculation amount is also reduced, and thus the mask image of the area of interest in the image can be accurately and quickly obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, in particular to a method and device for generating an area of interest, an electronic device and a storage medium. BACKGROUND

[0002] In the application scenarios of face attribute prediction and face disguise, there is usually a problem that which part of the image is the most important, for example, in the prediction of whether a portrait is wearing glasses, for a face image, only the image of the eyes is helpful for the judgment of "whether wearing glasses". Then, where is the image of the eyes in the face image, which belongs to the setting problem of the image area of interest. For another example, in the application scenario of face disguise, in order to make the person in the face image be considered as another person by the face recognition model, the face image needs to be processed by coding first, and where to code is the best choice, which also belongs to the setting problem of the image area of interest.

[0003] However, how to accurately and quickly obtain the area of interest in the image is a technical problem to be solved. SUMMARY

[0004] The purpose of the present application is to provide a method and device for generating an area of interest, an electronic device and a storage medium to improve the problems existing in the prior art.

[0005] Embodiments of the present application can be implemented as follows:

[0006] In a first aspect, the present application provides a method for generating a mask of an area of interest, comprising:

[0007] obtaining a to-be-processed image and inputting the to-be-processed image into a pre-trained first convolutional neural network to obtain a feature vector; the feature vector represents position information of a rectangular area of interest in the to-be-processed image;

[0008] performing binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector;

[0009] determining a longitudinal feature vector and a transverse feature vector of the rectangular area of interest based on the binary feature vector; wherein the longitudinal feature vector and the transverse feature vector respectively represent the longitudinal position and the transverse position of the rectangular area of interest in the to-be-processed image;

[0010] performing matrixization on the longitudinal feature vector and the transverse feature vector to obtain a mask image of the rectangular area of interest.

[0011] In an optional embodiment, the feature vector includes a plurality of feature values, and the step of performing binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector includes:

[0012] For any of the feature values, when the feature value is greater than the set threshold, setting the binary feature value corresponding to the feature value to 1;

[0013] When the feature value is less than or equal to the set threshold, the binary feature value corresponding to the feature value is set to 0;

[0014] Iterate through each of the feature values to obtain the binary feature vector, the binary feature vector including the binary feature value corresponding to each of the feature values.

[0015] In an optional implementation, the step of determining the longitudinal feature vector and the transverse feature vector of the rectangular region of interest based on the binary feature vector comprises:

[0016] Splitting the binary feature vector based on the height and width of the image to be processed to obtain a first feature vector and a second feature vector;

[0017] Converting each feature value in the first feature vector into a longitudinal feature value using a first expression to obtain the longitudinal feature vector, the longitudinal feature vector including the longitudinal feature value corresponding to each feature value in the first feature vector;

[0018] Wherein the first expression is:

[0019]

[0020] In the formula, represents the i-th feature value in the first feature vector, represents the height of the image to be processed, represents the i-th longitudinal feature value in the longitudinal feature vector;

[0021] Converting each feature value in the second feature vector into a transverse feature value using a second expression to obtain the transverse feature vector, the transverse feature vector including the transverse feature value corresponding to each feature value in the second feature vector;

[0022] Wherein the second expression is:

[0023]

[0024] In the formula, represents the i-th feature value in the second feature vector, represents the width of the image to be processed, represents the i-th transverse feature value in the transverse feature vector; ​​​a transverse feature value.

[0025] In an optional embodiment, the step of matrixing the longitudinal feature vector and the transverse feature vector to obtain the mask image of the rectangular region of interest comprises:

[0026] respectively extending columns and rows of the longitudinal feature vector and the transverse feature vector to obtain a longitudinal matrix and a transverse matrix;

[0027] point-multiplying the longitudinal matrix and the transverse matrix to obtain the mask image.

[0028] In an optional embodiment, the step of respectively extending columns and rows of the longitudinal feature vector and the transverse feature vector to obtain a longitudinal matrix and a transverse matrix comprises:

[0029] for the longitudinal feature vector, taking the width of the image to be processed as the number of column vectors in the matrix, and taking the longitudinal feature vector as each column vector to obtain the longitudinal matrix;

[0030] for the transverse feature vector, taking the height of the image to be processed as the number of row vectors in the matrix, and taking the transverse feature vector as each row vector to obtain the transverse matrix.

[0031] In an optional embodiment, the mask image comprises a white region of the rectangular region of interest and a black region corresponding to a non-interest region, and the method further comprises:

[0032] point-multiplying the mask image and the image to be processed to obtain a processed image, the processed image comprising the rectangular region of interest and the black region in the image to be processed;

[0033] inputting the processed image into a second pre-trained convolutional neural network to obtain an attribute prediction feature vector; the attribute prediction feature vector representing probabilities of a predicted object in the image to be processed for each label attribute;

[0034] determining a target label attribute to which the predicted object belongs based on the attribute prediction feature vector.

[0035] In an optional embodiment, the mask image comprises a white region of the rectangular region of interest and a black region corresponding to a non-interest region of the image to be processed, and the image to be processed comprises a to-be-printed image and a reference image of the same size, and the method further comprises:

[0036] input the channel spliced image to a pre-trained generative adversarial network to obtain a coded image; the coded image has the same size as the image to be coded and the reference image;

[0037] performing point multiplication between the coded image and the mask image to obtain a target coded image, the target coded image including a black region corresponding to the non-interest region and a coded region corresponding to the rectangular interest region;

[0038] performing point multiplication between the inverted mask image and the image to be coded to obtain a target image to be coded; wherein the colors of the rectangular interest region and the non-interest region in the inverted mask image are opposite to those in the mask image; the target image to be coded includes a black region corresponding to the non-interest region and the rectangular interest region of the image to be processed;

[0039] superimposing the target coded image and the target image to be coded to obtain a coded image, so that a face recognition model recognizes a face in the coded image as a face in the reference image.

[0040] In a second aspect, the present application provides a mask generation device for an interest region, comprising:

[0041] an acquisition module configured to acquire an image to be processed and input the image to be processed to a pre-trained first convolutional neural network to obtain a feature vector; the feature vector represents position information of a rectangular interest region in the image to be processed;

[0042] a conversion module configured to:

[0043] performing binaryzation processing on the feature vector using a set threshold to obtain a binary feature vector;

[0044] determining a longitudinal feature vector and a transverse feature vector of the rectangular interest region based on the binary feature vector; wherein the longitudinal feature vector and the transverse feature vector represent the longitudinal position and the transverse position of the rectangular interest region in the image to be processed, respectively;

[0045] performing matrixization on the longitudinal feature vector and the transverse feature vector to obtain a mask image of the rectangular interest region.

[0046] In a third aspect, the present application provides an electronic device, comprising a memory and a processor, the memory storing machine readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine readable instructions to implement the mask generation method for an interest region as described in any one of the preceding embodiments.

[0047] In a fourth aspect, the present application provides a computer readable storage medium storing a computer program, which, when executed by a processor, implements the mask generation method of the interest region according to any one of the preceding embodiments.

[0048] Compared with the prior art, the embodiment of the present application provides a mask generation method, device, electronic equipment and storage medium of an interest region, which inputs the obtained to-be-processed image into a pre-trained first convolutional neural network to obtain a feature vector; the feature vector represents position information of a rectangular interest region in the to-be-processed image. Then, the feature vector is binarized by using a set threshold to obtain a binary feature vector, and the vertical feature vector and the horizontal feature vector of the rectangular interest region are determined based on the binary feature vector. Since the vertical feature vector and the horizontal feature vector can represent the vertical position and the horizontal position of the rectangular interest region in the to-be-processed image respectively, the vertical feature vector and the horizontal feature vector can be further matrixed to obtain a mask image of the rectangular interest region. In this way, solving a two-dimensional mask is converted into solving a one-dimensional feature vector, which reduces the dimension and the calculation amount, so that the mask image of the interest region in the image can be accurately and quickly obtained. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0050] Figure 1 One of the flowcharts of the mask generation method of the interest region provided by the embodiment of the present application.

[0051] Figure 2 An analysis diagram for the to-be-processed image.

[0052] Figure 3 The second flowchart of the mask generation method of the interest region provided by the embodiment of the present application.

[0053] Figure 4 The image change diagram of the face attribute prediction provided by the embodiment of the present application.

[0054] Figure 5 The mask image diagram when the face is disguised provided by the embodiment of the present application.

[0055] Figure 6 The third flowchart of the mask generation method of the interest region provided by the embodiment of the present application.

[0056] Figure 7 A code printing image schematic diagram when a face is disguised is provided for an embodiment of the present application.

[0057] Figure 8 A schematic diagram for obtaining a target code printing image by using a mask image and a code printing image is provided for an embodiment of the present application.

[0058] Figure 9 A schematic diagram for obtaining a target code printing image by using a mask image and a code printing image is provided for an embodiment of the present application.

[0059] Figure 10 A schematic diagram for obtaining a target code printing image by using a mask image and a code printing image is provided for an embodiment of the present application.

[0060] Figure 11 A structure schematic diagram of a mask generation device of an interest region is provided for an embodiment of the present application.

[0061] Figure 12 A structure schematic diagram of an electronic device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings can be arranged and designed in various different configurations.

[0063] Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0064] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0065] In addition, if the terms "first", "second" and the like are used, they are only used to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0066] It should be noted that, in the case of no conflict, the features in the embodiments of the present application can be combined with each other.

[0067] Please refer to Figure 1 ,Figure 1 A flowchart of a method for generating a mask of a region of interest is provided for an embodiment of the present application. The method comprises the following steps S110-S140:

[0068] S110, an image to be processed is obtained and input into a first convolutional neural network trained in advance to obtain a feature vector.

[0069] In this embodiment, the feature vector can include a plurality of feature values, and the number of feature values can be the height plus the width of the image to be processed. Here, the height and the width can refer to the pixels of the image to be processed. The feature vector can represent the position information of the rectangular region of interest in the image to be processed in terms of the size and position of the feature values.

[0070] S120, the feature vector is binarized using a set threshold to obtain a binary feature vector.

[0071] It can be understood that based on the set threshold, each feature value of the feature vector can be binarized using the set threshold to obtain each binary feature value in the binary feature vector.

[0072] S130, a longitudinal feature vector and a transverse feature vector of the rectangular region of interest are determined based on the binary feature vector.

[0073] In this embodiment, the longitudinal feature vector and the transverse feature vector represent the longitudinal position and the transverse position of the rectangular region of interest in the image to be processed, respectively.

[0074] S140, the longitudinal feature vector and the transverse feature vector are matrixed to obtain a mask image of the rectangular region of interest.

[0075] It can be understood that through the above steps, the mask image originally in two dimensions can be converted into a feature vector in one dimension, and then the longitudinal feature vector and the transverse feature vector representing the longitudinal position and the transverse position of the rectangular region of interest in the image to be processed can be obtained based on the feature vector in one dimension. After matrixing, a two-dimensional mask image can be obtained.

[0076] The method for generating regions of interest (ROIs) provided in this invention involves inputting the acquired image to be processed into a pre-trained first convolutional neural network to obtain feature vectors. These feature vectors represent the positional information of rectangular ROIs in the image. The feature vectors are then binarized using a set threshold to obtain binary feature vectors. Based on these binary feature vectors, the vertical and horizontal feature vectors of the rectangular ROI are determined. Since the vertical and horizontal feature vectors can respectively represent the vertical and horizontal positions of the rectangular ROI in the image, they can be further matrixed to obtain a mask image of the rectangular ROI. Thus, solving for a two-dimensional mask is transformed into solving for a one-dimensional feature vector, reducing both dimensionality and computational complexity, thereby enabling accurate and rapid generation of mask images of ROIs in the image.

[0077] To facilitate the explanation of the process of obtaining the mask image, the following description is provided in conjunction with the accompanying drawings.

[0078] Please see Figure 2 , Figure 2 This is a schematic diagram illustrating the analysis of the image to be processed. Assume the image to be processed has a height of H and a width of... If we place the image to be processed in a Cartesian coordinate system (horizontal axis corresponds to width, vertical axis corresponds to height), the width of the rectangular region of interest in the image to be processed is: Height is ,in, , ,and and .

[0079] In an optional implementation, the sub-steps of step S120 above may include S121 to S123:

[0080] S121. For any feature value, when the feature value is greater than the set threshold, set the corresponding binarized feature value to 1.

[0081] S122. When the feature value is less than or equal to the set threshold, the binarized feature value corresponding to the feature value is set to 0.

[0082] S123. Traverse each feature value to obtain a binary feature vector, which includes the binary feature value corresponding to each feature value.

[0083] In the optional example, assume the height of the image to be processed is... Width is The feature vector is The following formula is used to analyze the feature vector. Binarize the eigenvalues ​​in the vector to obtain binary eigenvectors. :

[0084]

[0085] wherein, represents a set threshold value, represents a feature vector is the first feature value in the feature vector, represents a binary feature vector the first binary feature value in the binary feature vector, .

[0086] In combination Figure 2 , in the first feature values in the binary feature vector , the first feature value with a value of 1 and the last feature value with a value of 1 are both in the position of the feature value, i.e. , .

[0087] In the last feature values in the binary feature vector , the first feature value with a value of 1 and the last feature value with a value of 1 are both in the position of the feature value, i.e. , .

[0088] In an optional embodiment, the sub-steps of step S130 can include S134-S136.

[0089] S134, based on the height and width of the image to be processed, the binary feature vector is split to obtain a first feature vector and a second feature vector.

[0090] In combination with the above examples, for a binary feature vector , it can be split to obtain a first feature vector and a second feature vector .

[0091]

[0092]

[0093] S135, each feature value in the first feature vector is converted into a longitudinal feature value by using a first expression to obtain a longitudinal feature vector.

[0094] It can be understood that the longitudinal feature vector includes a longitudinal feature value corresponding to each feature value in the first feature vector. The first expression can be:

[0095]

[0096] In the formula, represents the i-th feature value in the first feature vector, represents the height of the image to be processed, represents the i-th longitudinal feature value in the longitudinal feature vector.

[0097] Using the first expression, all feature values between the first feature value of 1 and the last feature value of 1 in the first feature vector can be set to 1, and the remaining feature values can be set to 0, to obtain the longitudinal feature vector .

[0098] S136, each feature value in the second feature vector is converted into a transverse feature value using a second expression to obtain a transverse feature vector.

[0099] It can be understood that the transverse feature vector can include a transverse feature value corresponding to each feature value in the second feature vector. The second expression can be:

[0100]

[0101] In the formula, represents the i-th feature value in the second feature vector, represents the width of the image to be processed, represents the i-th transverse feature value in the transverse feature vector.

[0102] Using the second expression, all feature values between the first feature value of 1 and the last feature value of 1 in the second feature vector can be set to 1, and the remaining feature values can be set to 0, to obtain the transverse feature vector .

[0103] In an optional embodiment, matrixing the longitudinal feature vector and the transverse feature vector can be performed by column expansion and row expansion, respectively. Correspondingly, the sub-steps of the step S140 can include S141-S142.

[0104] S141, respectively, column expansion and row expansion are performed on the longitudinal feature vector and the transverse feature vector to obtain corresponding longitudinal matrix and transverse matrix.

[0105] ​​​​​First, for the vertical eigenvectors, take the width of the image to be processed as the number of column vectors in the matrix, and take the vertical eigenvectors as each column vector, to obtain a vertical matrix.

[0106] Therefore, for the vertical eigenvectors , the corresponding vertical matrix is as follows:

[0107]

[0108] where the vertical matrix is a matrix of order .

[0109] Second, for the horizontal eigenvectors, take the height of the image to be processed as the number of row vectors in the matrix, and take the horizontal eigenvectors as each row vector, to obtain a horizontal matrix.

[0110] Therefore, for the horizontal eigenvectors , the corresponding horizontal matrix is as follows:

[0111]

[0112] where the horizontal matrix is a matrix of order .

[0113] S142, point-multiply the vertical matrix and the horizontal matrix to obtain a mask image.

[0114] Therefore, the mask image is:

[0115]

[0116] where the mask image is a matrix of order , corresponding to the height and the width of the image to be processed.

[0117] A simple example of converting eigenvectors to a mask image is given below.

[0118] Assume that the height of the image to be processed is , the width is , and the threshold is set to 0.5.

[0119] When the eigenvectors are:

[0120]

[0121]

[0122] Correspondingly, after binarization using a set threshold (0.5), the binary feature vector is:

[0123]

[0124]

[0125] Correspondingly, for binary eigenvectors The first feature vector is obtained by splitting the vector. Second eigenvector :

[0126]

[0127]

[0128] Then, using the first expression, the first feature vector is... Convert to vertical feature vector Using the second expression, the second feature vector Convert to lateral feature vectors :

[0129] , ~ All are 1

[0130] , ~ All are 1

[0131] Then, using the vertical feature vector Horizontal eigenvectors The vertical matrix is ​​obtained respectively. and horizontal matrix :

[0132] , It is a 96×96 matrix

[0133] , It is a 96×96 matrix

[0134] in this way, , Performing a dot product, the resulting mask image M is:

[0135]

[0136] It should be noted that this example is for illustrative purposes only. The height and width of the image to be processed, as well as the size of the feature vector and the set threshold, should be determined according to the actual application and are not limited here.

[0137] In the process of obtaining the mask image corresponding to the rectangular region of interest in the to-be-processed image, a one-dimensional feature vector is obtained by using machine learning, and the mask image is obtained by processing the one-dimensional feature vector.

[0138] In optional embodiments, the above-mentioned method of obtaining the mask image can be applied in different scenarios, and two applications will be briefly introduced below.

[0139] The first application is as follows:

[0140] In the scenario of face attribute prediction, for a frame of face image, it may be necessary to determine whether the face is wearing glasses, wearing a mask, having a beard, wearing a hat, etc. Correspondingly, the parts that need to be concerned in the image should be the eye part, the face, the mouth nearby, the head, etc. for recognition and judgment. The following will be illustrated by taking the example of determining whether the face is wearing glasses.

[0141] In order to determine whether the face in the face image is wearing glasses, a first convolutional neural network and a second convolutional neural network can be used to form a learnable end-to-end prediction model. The first convolutional neural network is used to obtain the mask image of the rectangular region of interest (i.e. the eye region), and then the second convolutional neural network is responsible for determining whether the face is wearing glasses.

[0142] Therefore, after the first neural network obtains the mask image of the rectangular region of interest (i.e. the eye region) in the to-be-processed image (i.e. the face image), the mask image can include the white region corresponding to the rectangular region of interest and the black region corresponding to the non-interest region, and then the second convolutional neural network is used to determine whether the face is wearing glasses. Correspondingly, please refer to Figure 3 After the above step S140, steps S210-S230 can also be included.

[0143] S210, point-multiply the mask image and the to-be-processed image to obtain a processed image.

[0144] In combination with Figure 4 , Figure 4 An image change schematic diagram of face attribute prediction provided by an embodiment of the present application. For the to-be-processed image A, the mask image A1 corresponding thereto includes the white region corresponding to the rectangular region of interest (eye region) and the black region corresponding to the non-interest region, and the processed image A2 corresponding thereto can include the eye region and the black region in the to-be-processed image.

[0145] S220, input the processed image into the pre-trained second convolutional neural network to obtain an attribute prediction feature vector.

[0146] S230, determining a target label attribute to which the prediction object belongs based on the attribute prediction feature vector.

[0147] It can be understood that the attribute prediction feature vector can represent the probability of the prediction object in the to-be-processed image for each label attribute. In the embodiment, the prediction object in the to-be-processed image is a face, and the label attributes include "wearing glasses" and "not wearing glasses".

[0148] Therefore, the attribute prediction feature vector can include two prediction feature values, and the attribute prediction feature vector , wherein, , correspond to "not wearing glasses" and "wearing glasses", respectively.

[0149] Then, if , it can be determined that the target label attribute to which the face belongs is "not wearing glasses"; if , it can be determined that the target label attribute to which the face belongs is "wearing glasses".

[0150] Here, the training of the learnable end-to-end prediction model composed of the first convolutional neural network and the second convolutional neural network is briefly introduced.

[0151] First, the training data set 1 is obtained. The training data set includes a plurality of standardized face image samples, The number of images in the training data set

[0152] may be greater than 100,000, and the sample number ratio of the glasses image to the no glasses image is 1:3.

[0153] The training data set 1 can be used to train the prediction model based on the Resnet50 network and the Resnet18 network. During the entire model training process, the initial learning rate is 0.0005, the learning rate is halved every 5 epochs (representing a training cycle: using all samples in the training data set to train once), and the training is completed after 25 epochs to obtain the trained prediction model.

[0154] In each iteration in the training process, the gradient is calculated by forward calculation and backward propagation, and the gradient is used to update the weight coefficient of the Resnet50 network and the Resnet18 network.

[0155] It can be understood that the training data set Taking a face image sample G as an example, during the training process:

[0156] (1) Input the face image sample G into the ResNet50 network to obtain the feature vector. (Corresponding to step S110 above);

[0157] (2) For the eigenvector Binarization is performed to obtain binary feature vectors. (Corresponding to step S120 above);

[0158] (3) Based on the binary eigenvectors Obtain the first eigenvector Second eigenvector (corresponding to step S130 above), however, get It is obtained by comparing the feature value with a set threshold and then assigning a value. In order to ensure that the gradient can be obtained during the backpropagation process of the sample, it is also necessary to first... Perform the following calculations:

[0159] S131, The feature vector Make a copy, get ;

[0160] S132, Calculation ;

[0161] S133, Calculation ;

[0162] Then execute S134. Split into , That's understandable. and The same principle applies; this is done to ensure that partial derivatives can be calculated during backpropagation to obtain gradients, thus enabling updates to the weight coefficients of the ResNet50 network. Similarly, during the application of the prediction model, steps S131 to S133 must be performed before executing step S134.

[0163] (4) Based on the first feature vector Second eigenvector Obtain the mask image ;

[0164] (5) Mask image Multiply the face image sample G by the dot product to obtain the processed image G1;

[0165] (6) G1 input The network yields attribute prediction feature vectors. ;

[0166] (7) Calculate the loss value based on the label of the face image sample G:

[0167] When the label corresponding to the face image sample G is , which represents "no glasses", then the loss value is:

[0168]

[0169] When the label corresponding to the face image sample G is , which represents "wearing glasses", then the loss value is:

[0170]

[0171] (8) Perform backpropagation training based on the loss value.

[0172] In this way, after the above (1) ~ (8) process, for the face image sample G, the training process is completed once.

[0173] The second application is as follows:

[0174] In the face disguise scene, it is necessary to code an image so that the face recognition model recognizes the face in the coded face image as a face, thereby achieving the purpose of face disguise.

[0175] Therefore, the first convolutional neural network and the generative adversarial network described above can be used to form a learnable end-to-end face disguise model. The first convolutional neural network is used to obtain a mask image of a rectangular region of interest (i.e. a coding region), and then a generative adversarial network is used to obtain a coding image, which is used to code the coding region.

[0176] In an optional example, the first convolutional neural network can use a Resnet50 network; the generative adversarial network can use the generator part in the mainstream adversarial generative network GAN.

[0177] In this scenario, the above to-be-processed image can include a to-be-coded image and a reference image with the same size, and the above step S110 can be: Figure 5

[0178] S111, obtain the to-be-coded image G1 and the reference image G2, and input the to-be-coded image G1 and the reference image G2 after channel splicing to the Resnet50 network in the pre-trained face disguise model to obtain a feature vector.

[0179] Then, the mask image M of the rectangular region of interest (i.e. the coding region) is obtained through S120~S140, and from Figure 5 ​It can be seen that the mask image M can include a white region corresponding to the rectangular interest region (i.e. the coding region) and a black region corresponding to the non-interest region.

[0180] Further, in order to code the coding region, please refer to Figure 6 After the above step S140, steps S310-S340 can also be included.

[0181] S310, input the to-be-coded image and the reference image to the pre-trained generative adversarial network after channel splicing to obtain a coded image.

[0182] In combination with Figure 7 It can be seen that the size of the coded image P1 is consistent with the to-be-coded image G1 and the reference image G2.

[0183] S320, point-multiply the coded image and the mask image to obtain a target coded image.

[0184] In combination with Figure 8 It can be seen that the target coded image can include a black region corresponding to the non-interest region and a coding region corresponding to the rectangular interest region.

[0185] S330, point-multiply the inverted mask image and the to-be-coded image to obtain a target to-be-coded image.

[0186] In combination with Figure 9 The colors of the rectangular interest region and the non-interest region in the inverted mask image (1-M) are opposite to those in the mask image, and the target to-be-coded image includes a black region corresponding to the non-interest region and the rectangular interest region of the to-be-processed image.

[0187] S340, superimpose the target coded image and the target to-be-coded image to obtain a coded image, so that the face recognition model recognizes the face in the coded image as the face in the reference image.

[0188] In combination with Figure 10 Superimposing (i.e. adding) the target coded image and the target to-be-coded image can obtain the final coded image, realizing face camouflage of the to-be-coded image G1. That is, the coded image For the face recognition model , the coded image has a high similarity with the reference image , achieving the purpose of face camouflage.

[0189] Here, the training of the face camouflage model composed of the first convolutional neural network and the generative adversarial network is briefly introduced.

[0190] First, the training data set is obtained, and the training data set The plurality of standardized face image pairs include a to-be-coded image sample and a reference image sample. The number of images in the plurality of standardized face image pairs can be greater than 100,000.

[0191] The training data set is used to train a face disguise model pre-constructed based on a Resnet50 network and a GAN network. During the entire model training process, the initial learning rate is 0.0005, the learning rate is halved every 5 epochs, and the training is completed after 25 epochs to obtain a trained prediction model.

[0192] It can be understood that, taking one face image pair (to-be-coded image sample G1 and reference image sample G2) in the training data set as an example, in the training process: (1) The to-be-coded image sample G1 and the reference image sample G2 are spliced in the channel and input into the Resnet50 network to obtain a feature vector

[0193] (corresponding to step S110 described above); (2) The feature vector

[0194] is binarized to obtain a binary feature vector (corresponding to step S120 described above); (3) The binary feature vector

[0195] is used to obtain a first feature vector and a second feature vector (corresponding to step S130 described above), but is obtained by comparing the feature value with a set threshold and then assigning a value, in order to ensure that the gradient can be obtained in the sample back propagation process, before step S134 described above, the following calculation is needed: is obtained by comparing the feature value with a set threshold and then assigning a value, in order to ensure that the gradient can be obtained in the sample back propagation process, before step S134 described above, the following calculation is needed: S131, a copy of the feature vector

[0196] is obtained;

[0197] S132, is calculated ;

[0198] S133, is calculated ;

[0199] Then, S134 is executed to split to obtain , . It can be understood that and ​​​are the same, so as to ensure that the partial derivative can be calculated in the process of back propagation to obtain the gradient to ensure that the weight coefficients of the Resnet50 network can be updated. Similarly, in the process of applying the face disguise model, the steps S131-S133 also need to be performed before the step S134 is performed.

[0200] (4) Based on the first feature vector and the second feature vector , a mask image is obtained .

[0201] (5) The to-be-printed image sample G1 and the reference image sample G2 are input into the GAN network after being spliced in the channel, and a printed image P1 is obtained.

[0202] (6) After the , a printed image sample P2 is obtained.

[0203] (7) The printed image sample P2 and the reference image sample G2 are respectively input into the pre-prepared face recognition model (the parameters in the face recognition model are fixed during training), and the face feature vectors and are respectively obtained.

[0204] (8) Based on the face feature vectors and , a loss value is calculated: the and are input into a loss function, and the loss value is calculated as:

[0205]

[0206] wherein, is the cosine distance of and ; is the loss value.

[0207] (9) Based on the loss value, back propagation training is performed.

[0208] In this way, after the above (1)-(9), for the face image pair (to-be-printed image sample G1 and reference image sample G2), one training in the training process is completed.

[0209] It should be noted that in the above two application scenarios, the size and position of the rectangular region of interest are determined according to the actual requirements in the scene and the convergence conditions during training, and are not limited by the above examples. The execution order of each step in the above method embodiment is not limited by the figure, and the execution order of each step is subject to the actual application situation.

[0210] To perform the above-mentioned method embodiments and respective steps in various possible implementations, an implementation of a mask generation device of a region of interest is given below.

[0211] Please refer to Figure 11 , Figure 11 A structure diagram of a mask generation device of a region of interest provided by an embodiment of the present application is shown. The mask generation device of the region of interest 200 comprises an acquisition module 210 and a conversion module 220.

[0212] The acquisition module 210 is configured to acquire a to-be-processed image and input the to-be-processed image into a pre-trained first convolutional neural network to obtain a feature vector; the feature vector represents position information of a rectangular region of interest in the to-be-processed image.

[0213] The conversion module 220 is configured to perform binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector; determine a longitudinal feature vector and a transverse feature vector of the rectangular region of interest based on the binary feature vector; wherein the longitudinal feature vector and the transverse feature vector respectively represent a longitudinal position and a transverse position of the rectangular region of interest in the to-be-processed image; and perform matrixization on the longitudinal feature vector and the transverse feature vector to obtain a mask image of the rectangular region of interest.

[0214] In an optional implementation, the conversion module 220 can be specifically configured to: for any feature value, when the feature value is greater than the set threshold, set a binaryzation feature value corresponding to the feature value to 1; when the feature value is less than or equal to the set threshold, set the binaryzation feature value corresponding to the feature value to 0; and traverse each feature value to obtain a binary feature vector, the binary feature vector comprising the binaryzation feature value corresponding to each feature value.

[0215] In an optional implementation, the conversion module 220 can be specifically configured to: split the binary feature vector based on a height and a width of the to-be-processed image to obtain a first feature vector and a second feature vector; and convert each feature value in the first feature vector into a longitudinal feature value by using a first expression to obtain a longitudinal feature vector, the longitudinal feature vector comprising the longitudinal feature value corresponding to each feature value in the first feature vector; wherein the first expression is:

[0216]

[0217] In the formula, represents the i-th feature value in the first feature vector, represents the height of the to-be-processed image, represents the i-th longitudinal feature value in the longitudinal feature vector;

[0218] ​​The second expression is used to convert each feature value in the second feature vector into a transverse feature value, to obtain a transverse feature vector, the transverse feature vector including a transverse feature value corresponding to each feature value in the second feature vector; wherein the second expression is:

[0219]

[0220] In the formula, represents the i-th feature value in the second feature vector, represents the width of the image to be processed, represents the i-th transverse feature value in the transverse feature vector.

[0221] In an optional implementation, the conversion module 220 can be specifically configured to: perform column expansion and row expansion on the longitudinal feature vector and the transverse feature vector respectively, to obtain corresponding longitudinal matrix and transverse matrix; and perform dot multiplication on the longitudinal matrix and the transverse matrix, to obtain the mask image.

[0222] In an optional implementation, the conversion module 220 can be specifically configured to: for the longitudinal feature vector, taking the width of the image to be processed as the number of column vectors in the matrix, and taking the longitudinal feature vector as each column vector, to obtain the longitudinal matrix; and for the transverse feature vector, taking the height of the image to be processed as the number of row vectors in the matrix, and taking the transverse feature vector as each row vector, to obtain the transverse matrix.

[0223] In an optional implementation, the mask generation apparatus 200 for the region of interest can further include a prediction module 230, and the mask image includes a white region of the rectangular region of interest and a black region corresponding to the non-region of interest. The prediction module 230 can be configured to: perform dot multiplication on the mask image and the image to be processed, to obtain a processed image, the processed image including the rectangular region of interest and the black region in the image to be processed; input the processed image into a second convolutional neural network trained in advance, to obtain an attribute prediction feature vector; the attribute prediction feature vector represents probabilities of a prediction object in the image to be processed for each label attribute; and determine a target label attribute to which the prediction object belongs based on the attribute prediction feature vector.

[0224] In an optional implementation, the mask generation apparatus 200 for the region of interest can further include a code printing module 240, the mask image includes a white region of the rectangular region of interest and a black region corresponding to the non-region of interest of the image to be processed, and the image to be processed includes a to-be-printed image and a reference image of the same size. The code printing module 240 can be configured to:

[0225] ​​The image to be coded and the reference image are spliced in channels and input to a pre-trained generative adversarial network to obtain a coded image; the size of the coded image is consistent with that of the image to be coded and the reference image; the coded image is multiplied by the mask image to obtain a target coded image, the target coded image including a black region corresponding to a non-interest region and a coded region corresponding to a rectangular interest region; the inverted mask image is multiplied by the image to be coded to obtain a target image to be coded; the colors of the rectangular interest region and the non-interest region in the inverted mask image are opposite to those in the mask image; the target image to be coded includes a black region corresponding to the rectangular interest region of the image to be processed; the target coded image is superimposed on the target image to be coded to obtain a coded image, so that the face recognition model recognizes the face in the coded image as the face in the reference image.

[0226] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the mask generation device 200 of the interest region described above can refer to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0227] Please refer to Figure 12 , Figure 12 A structural schematic diagram of an electronic device is provided for an embodiment of the present application. The electronic device 300 includes a processor 310, a memory 320, and a bus 330, the processor 310 being connected with the memory 320 through the bus 330.

[0228] The electronic device 300 is used to execute the above-mentioned mask generation method of the interest region, and the electronic device 300 can be, but is not limited to, a smart phone, a personal computer, a server, a notebook computer, etc.

[0229] The memory 320 can be used to store software programs, for example, Figure 11 The mask generation device 200 of the interest region is shown. The memory 320 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a flash memory, a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The processor 310 can be an integrated circuit chip with signal processing capability.

[0230] The processor 310 can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0231] The memory 320 stores machine readable instructions executable by the processor 310. When the processor 310 executes the machine readable instructions, the method for generating an area of interest disclosed in the above embodiments is implemented.

[0232] It can be understood that, Figure 12 The structure shown is only schematic, and the electronic device 300 can further include more or less components than those shown in the figures, or have a different configuration from that shown in the figures. Figure 12 The components shown in the figures can be implemented in hardware, software or a combination thereof. Figure 12 The components shown in the figures can be implemented in hardware, software or a combination thereof. Figure 12 The components shown in the figures can be implemented in hardware, software or a combination thereof.

[0233] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the method for generating an area of interest disclosed in the above embodiments is implemented. The computer readable storage medium can be, but is not limited to, a U disk, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a FLASH disk or an optical disk and various media that can store program codes.

[0234] To sum up, the embodiment of the present application provides a method and device for generating an interest region, electronic equipment and storage medium, by inputting the obtained image to be processed into the first convolutional neural network trained in advance to obtain a feature vector; the feature vector represents the position information of the rectangular interest region in the image to be processed. Then, the feature vector is binarized by using a set threshold to obtain a binary feature vector, and the longitudinal feature vector and the transverse feature vector of the rectangular interest region are determined based on the binary feature vector. Since the longitudinal feature vector and the transverse feature vector can represent the longitudinal position and the transverse position of the rectangular interest region in the image to be processed, respectively, the longitudinal feature vector and the transverse feature vector can be further matrixed to obtain a mask image of the rectangular interest region. In this way, solving a two-dimensional mask is converted into solving a one-dimensional feature vector, which reduces the dimension and the calculation amount, so that the mask image of the interest region in the image can be accurately and quickly obtained.

[0235] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of generating a mask of a region of interest, characterized by, The method comprises: acquiring a to-be-processed image and inputting the to-be-processed image into a pre-trained first convolutional neural network to obtain a feature vector; the feature vector represents position information of a rectangular region of interest in the to-be-processed image; performing binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector; determining a longitudinal feature vector and a transverse feature vector of the rectangular region of interest based on the binary feature vector; wherein the longitudinal feature vector and the transverse feature vector respectively represent longitudinal and transverse positions of the rectangular region of interest in the to-be-processed image; performing matrixization on the longitudinal feature vector and the transverse feature vector to obtain a mask image of the rectangular region of interest, the mask image comprising a white region of the rectangular region of interest and a black region corresponding to a non-interest region; the method further comprises: point-multiplying the mask image and the to-be-processed image to obtain a processed image, the processed image comprising the rectangular region of interest and the black region in the to-be-processed image; inputting the processed image into a pre-trained second convolutional neural network to obtain an attribute prediction feature vector; the attribute prediction feature vector represents probabilities of a predicted object in the to-be-processed image for each label attribute; and determining a target label attribute to which the predicted object belongs based on the attribute prediction feature vector; when the to-be-processed image comprises a to-be-printed image and a reference image of the same size, the method further comprises: inputting the to-be-printed image and the reference image after channel splicing into a pre-trained generative adversarial network to obtain a printed image; the printed image has the same size as the to-be-printed image and the reference image; point-multiplying the printed image and the mask image to obtain a target printed image, the target printed image comprising a black region corresponding to the non-interest region and a printed region corresponding to the rectangular region of interest; point-multiplying an inverted mask image and the to-be-printed image to obtain a target to-be-printed image; wherein the colors of the rectangular region of interest and the non-interest region in the inverted mask image are opposite to those in the mask image; and the target to-be-printed image comprises a black region corresponding to the non-interest region and the rectangular region of interest of the to-be-processed image; superimposing the target printed image and the target to-be-printed image to obtain a printed image, so that a face recognition model recognizes a face in the printed image as a face in the reference image.

2. The method of claim 1, wherein, The feature vector comprises a plurality of feature values, and the binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector comprises: for any feature value, when the feature value is greater than the set threshold, the binaryzation feature value corresponding to the feature value is set to 1; when the feature value is less than or equal to the set threshold, the binaryzation feature value corresponding to the feature value is set to 0; traversing each feature value to obtain the binary feature vector, the binary feature vector comprising a binaryzation feature value corresponding to each feature value.

3. The method of claim 1, wherein, The step of determining the longitudinal feature vector and the transverse feature vector of the rectangular region of interest based on the binary feature vector comprises: splitting the binary feature vector based on the height and the width of the to-be-processed image to obtain a first feature vector and a second feature vector; converting each feature value in the first feature vector into a longitudinal feature value by using a first expression to obtain the longitudinal feature vector, the longitudinal feature vector comprising a longitudinal feature value corresponding to each feature value in the first feature vector; wherein the first expression is: In the formula, Represents the first feature vector. 1 eigenvalue, This represents the height of the image to be processed. Represents the first in the vertical feature vector One vertical feature value; converting each feature value in the second feature vector into a transverse feature value by using a second expression to obtain the transverse feature vector, the transverse feature vector comprising a transverse feature value corresponding to each feature value in the second feature vector; wherein the second expression is: wherein represents the i-th feature value in the second feature vector, represents the width of the image to be processed, represents the i-th feature value in the transverse feature vector, represents the width of the image to be processed,​ 4. The method of claim 1, wherein, The step of matrixing the longitudinal feature vector and the transverse feature vector to obtain the mask image of the rectangular region of interest comprises: respectively performing column extension and row extension on the longitudinal feature vector and the transverse feature vector to obtain a corresponding longitudinal matrix and a transverse matrix; point-multiplying the longitudinal matrix and the transverse matrix to obtain the mask image.

5. The method of claim 4, wherein, The step of respectively performing column extension and row extension on the longitudinal feature vector and the transverse feature vector to obtain a corresponding longitudinal matrix and a transverse matrix comprises: for the longitudinal feature vector, taking the width of the to-be-processed image as the number of column vectors in the matrix and taking the longitudinal feature vector as each column vector to obtain the longitudinal matrix; for the transverse feature vector, taking the height of the to-be-processed image as the number of row vectors in the matrix and taking the transverse feature vector as each row vector to obtain the transverse matrix.

6. An interest region mask generation apparatus characterized by comprising: comprises: an acquisition module configured to acquire a to-be-processed image and input the to-be-processed image into a pre-trained first convolutional neural network to obtain a feature vector; The feature vector represents position information of a rectangular region of interest in the to-be-processed image. a conversion module configured to: perform binaryzation processing on the feature vector by using a set threshold to obtain a binary feature vector; determine a longitudinal feature vector and a transverse feature vector of the rectangular region of interest based on the binary feature vector; wherein the longitudinal feature vector and the transverse feature vector respectively represent a longitudinal position and a transverse position of the rectangular region of interest in the to-be-processed image; matrix the longitudinal feature vector and the transverse feature vector to obtain a mask image of the rectangular region of interest, the mask image comprising a white region of the rectangular region of interest and a black region corresponding to a non-interest region; The prediction module is configured to: multiply the mask image and the to-be-processed image to obtain a processed image, the processed image comprising a rectangular region of interest and the black region in the to-be-processed image; input the processed image into a second convolutional neural network trained in advance to obtain an attribute prediction feature vector; the attribute prediction feature vector representing probabilities of a predicted object in the to-be-processed image for each label attribute; and determine a target label attribute to which the predicted object belongs based on the attribute prediction feature vector. When the to-be-processed image comprises a to-be-printed image and a reference image of the same size, the device further comprises a printing module configured to: input the to-be-printed image and the reference image after channel splicing into a generative adversarial network trained in advance to obtain a printed image; the printed image having the same size as the to-be-printed image and the reference image; multiply the printed image and the mask image to obtain a target printed image, the target printed image comprising a black region corresponding to the non-region of interest and a printed region corresponding to the rectangular region of interest; multiply an inverted mask image and the to-be-printed image to obtain a target to-be-printed image; wherein the colors of the rectangular region of interest and the non-region of interest in the inverted mask image are opposite to those in the mask image; the target to-be-printed image comprising a non-region of interest and a black region corresponding to the rectangular region of interest of the to-be-processed image; superimpose the target printed image and the target to-be-printed image to obtain a printed image, so that a face recognition model recognizes a face in the printed image as a face in the reference image.

7. An electronic device, comprising: comprise: a memory and a processor, the memory storing machine-readable instructions executable by the processor, and the processor executing the machine-readable instructions to implement the mask generation method of the region of interest according to any one of claims 1-5 when the electronic device is running.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and the computer program is executed by the processor to implement the mask generation method of the region of interest according to any one of claims 1-5.

Images