Human body and face matching method, device, equipment and storage medium
By calculating the embedding features and target point distances between the face and the human body in the target image, and selecting the face with the largest or closest similarity as the pairing face, the problem of low accuracy of face and human body matching in the prior art is solved. Achieves high accuracy pairing effect.
Patent Information
- Application Number
- CN202110924095.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-08-12
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-08-12
AI Technical Summary
The prior art When multiple people and high-repetitive human bodies are used in scene images, the accuracy of face and body pairing is low.
By obtaining the target point positions of the face frame and the human body frame in the target image, using the detection model to calculate the embedded features and target point distances between the face and the human body, selecting the face with the largest or closest similarity as the paired face to achieve high Accuracy pairing.
It improves the accuracy of matching between the human body and face of the same person and reduces pairing errors.
Smart Images

Figure CN113591783B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for matching a human body and a human face. Background Art
[0002] Human body recognition and face recognition are two common methods of pedestrian recognition. In a practical scenario of pedestrian recognition, there is also a need to pair faces and bodies that belong to the same person in the scene. Generally, face detection and body detection are completed separately first, and then the Hungarian algorithm is used to pair faces and bodies that belong to the same person from the detected faces and bodies.
[0003] Specifically, the basic method of using the Hungarian algorithm to match faces and bodies belonging to the same person from the detected faces and bodies is that when it is found that the coverage of a face frame in the body frame exceeds a certain degree, the face frame and the body frame are considered to belong to the same person. However, when there are multiple people in the scene image and the bodies of multiple people are highly repeated, it will lead to wrong matching of faces and bodies. It can be seen from this that the current scheme has a low matching accuracy for matching faces and bodies belonging to the same person. Summary of the invention
[0004] In view of this, the embodiments of the present invention provide a method, apparatus, device and storage medium for matching a human body and a human face, so as to achieve pairing of human bodies belonging to the same person with a high accuracy.
[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:
[0006] In a first aspect, the present application provides a method for matching a human body and a human face, comprising:
[0007] Obtaining the face frame and the corresponding target point position in the target image, as well as the body frame and the corresponding target point position in the target image;
[0008] Processing the target point position corresponding to the target human frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame;
[0009] For the target person in the target image, a face corresponding to the target point distance that meets the conditions is selected as the paired face of the target person.
[0010] Optionally, for a target person in a target image, a human face that meets the conditions is selected as a paired human face of the target person, including:
[0011] Calculate the similarity value corresponding to the distance between the target human body in the target image and the target point of each face;
[0012] For the target person in the target image, the face with the largest similarity value is used as the paired face of the target person.
[0013] Optionally, obtaining a face frame in a target image and a corresponding target point position, as well as a human body frame in a target image and a corresponding target point position, includes:
[0014] The detection model is called to process the target image to obtain the face frame and its corresponding target point position in the target image, as well as the human body frame and its corresponding target point position in the target image.
[0015] Optionally, the detection model includes: a basic module and a prediction module;
[0016] The basic module is used for the target image to obtain the face frame and the body frame in the target image;
[0017] The prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame in the target image.
[0018] Optionally, the detection model is a one-stage detector, and the prediction module is arranged after the dilated convolution layer of the one-stage detector; or, the detection model is a two-stage detector, and the prediction module is arranged after the second fully connected layer of the two-stage detector.
[0019] Optionally, during the training process, the detection model uses an angle loss function to train the detection model's functions of obtaining the target point positions corresponding to the face frame and the target point positions corresponding to the body frame. The angle loss function is used to: prompt the predicted value of the target point position corresponding to the body frame in the target image to be closer to the actual value, and prompt the predicted value of the target point position corresponding to the face frame to be closer to the actual value.
[0020] Optionally, the human body frame includes: a human body full body frame, a human body upper body frame and a human body upper torso frame.
[0021] Optionally, the target point positions include: the center point position of the human head, the center point position of the neck, and one or two connection point positions of the shoulder and the upper arm.
[0022] In a second aspect, the present application provides a human body and face matching device, comprising:
[0023] A calling unit, used to obtain a face frame in a target image and a corresponding target point position, as well as a human body frame in a target image and a corresponding target point position;
[0024] a processing unit, configured to process a target point position corresponding to a target human body frame in a target image and a target point position corresponding to each face frame, to obtain a target point distance between a target human body in the target image and each face frame; the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame;
[0025] The screening unit is used to select a face corresponding to the target point distance closest to the target human body and the human face for representing the target human body in the target image as a paired face of the target human body.
[0026] Optionally, when the calling unit executes the acquisition of the face frame and the corresponding target point position in the target image, and the human body frame and the corresponding target point position in the target image, it is used to: call the detection model to process the target image, and obtain the face frame and the corresponding target point position in the target image, and the human body frame and the corresponding target point position in the target image;
[0027] The detection model includes: basic module and prediction module;
[0028] The basic module is used to process the target image and obtain the face frame and the body frame in the target image; the prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame in the target image;
[0029] The detection model is a one-stage detector, and the prediction module is set after the dilated convolutional layer of the one-stage detector; the detection model is a two-stage detector, and the prediction module is set after the second fully connected layer of the two-stage detector.
[0030] Optionally, a model training unit is also included. During the process of training the detection model, the model training unit uses an angle loss function to train the detection model to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.
[0031] In a third aspect, the present application provides an electronic device, including:
[0032] one or more processors;
[0033] a memory having a program stored therein;
[0034] When the program is executed by one or more processors, the one or more processors implement any method for matching a human body and a face as described in the first aspect.
[0035] In a fourth aspect, the present application provides an electronic device, including:
[0036] The first processing logic is configured to: receive image data of a target image and process the image data using a neural network model to output position information of a human body part and a target point position in the target image;
[0037] Among them, the neural network model includes:
[0038] The first sub-network receives feature expression data of the image data and outputs position data of the detected face and body;
[0039] The second sub-network receives the position data of the face and the body, and outputs the target point position corresponding to the face and the target point position corresponding to the body.
[0040] Optionally, it also includes:
[0041] The second processing logic is configured to: calculate the target point distance between the face and the body according to the target point position corresponding to the face and the target point position corresponding to the body output by the second sub-network to obtain the matching relationship between the face and the body.
[0042] Optionally, the neural network model is a two-stage detection network, and the second sub-network is arranged after the second fully connected layer of the two-stage detector;
[0043] The neural network model is a one-stage detection network, and the second subnetwork is set after the dilated convolutional layer of the one-stage detector.
[0044] In a fifth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for matching a human body and a face of any one of the first aspects.
[0045] It can be seen from the above technical scheme that in a method for matching a human body and a face provided in an embodiment of the present application, a face frame in a target image and its corresponding target point position, a human body frame in a target image and its corresponding target point position are obtained; the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each human face frame are processed to obtain the target point distance between the target human body and each human face in the target image; for the target human body in the target image, a face corresponding to the distance between the center points of the human head closest to the target human body and the face is selected as the matching face of the target human body, thereby achieving a high accuracy rate in matching human bodies belonging to the same person. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0047] Figure 1 A display diagram of the detection model provided for the application embodiment;
[0048] Figure 2 A diagram showing the generation process of the angle loss function provided in the embodiment of the present application;
[0049] Figure 3 A flowchart of a method for matching a human body and a human face provided in an embodiment of the present application;
[0050] Figure 4a A flowchart of a method for matching a human body and a human face provided in an embodiment of the present application;
[0051] Figure 4b A schematic diagram of the structure of a device for matching a human body and a human face provided in an embodiment of the present application;
[0052] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0054] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0055] The embodiments of the present invention provide a method, device, equipment and storage medium for matching a human body and a human face, so as to achieve high accuracy in pairing human bodies belonging to the same person.
[0056] In the method for matching a human body and a human face provided in an embodiment of the present application, a detection model needs to be called to process a target image to obtain a face frame in the target image and its corresponding embedded features, a human body frame in the target image and its corresponding embedded features. Optionally, the human body frame includes: a human body full body frame, a human body upper body frame, and a human body upper torso frame.
[0057] The detection model used in the present application is improved on the basis of the basic detection model of face and body, and can be improved on the common one-stage detector (One Stage) or two-stage detector (TwoStages) architecture.
[0058] For details, see Figure 1 , an embedding module 11 is added to the one-stage detector or the two-stage detector, and the embedding module 11 runs in parallel with the original module of the detector (referring to the one-stage detector and the two-stage detector). For the one-stage detector, the embedding module includes four ordinary convolutional layers and a dilated convolutional layer at the end of the four ordinary convolutional layers; for the two-stage detector, the embedding module includes two fully connected layers f c1 and f c2 The main function of the embedding module is to process the image and obtain the embedding features corresponding to the face frame and the body frame in the image.
[0059] In addition, a prediction module is added to the end of the original module of the one-stage detector or the two-stage detector, see also Figure 1 , the prediction module is added after the dilated convolutional layer in the original module of the one-stage detector, and the fully connected layer f in the original module of the two-stage detector c2 A prediction module is added later. The main function of the prediction module is to process the image and obtain the position of the center point of the head corresponding to the face frame and the position of the center point of the head corresponding to the body frame in the image.
[0060] After building the architecture of the detection model, it is necessary to use sample data to train the detection model, train the original module, embedding module and prediction module of the detection model, so that the original module (also called the basic module) can process the image and obtain the face frame and body frame in the image, so that the embedding module can process the image and obtain the embedded features corresponding to the face frame and the embedded features corresponding to the body frame in the image, so that the prediction module can process the image and obtain the head center point position corresponding to the face frame and the head center point position corresponding to the body frame in the image.
[0061] It should also be noted that, by processing the image using the prediction module, the center point position of the neck corresponding to the face frame in the image, as well as the positions of one or two connection points of the shoulder and upper arm, and the center point position of the neck corresponding to the human body frame, as well as the positions of one or two connection points of the shoulder and upper arm can also be obtained. Based on this, the prediction module processes the image, and the target point positions obtained include: the center point position of the head, the center point position of the neck, and the positions of one or two connection points of the shoulder and upper arm.
[0062] Specifically, in order to train the embedding module of the detection model, the loss function is constructed according to the following purpose: if two targets belong to the same pedestrian (such as the body and face of the same pedestrian), their embedded features are brought closer to each other; otherwise, they are moved away from each other.
[0063] For the human body frame and face frame belonging to the same person k in the sample data, the constructed loss function is:
[0064]
[0065] in, They represent the embedded features of the i-th body frame and the j-th face frame of the pedestrian, respectively. k and N k Represents the number of valid boxes around the human body and face. By optimizing the loss function, and The Euclidean distance between them is as small as possible (close to each other).
[0066] For the human body frame and face frame that do not belong to the same pedestrian in the sample data, the constructed loss function is:
[0067]
[0068] Among them, δ is a small positive number (set to 2); M k and N l Represent the number of human frames of pedestrian k and the number of face frames of pedestrian l respectively. By optimizing the loss function, and The Euclidean distance between them is as large as possible (stay away from each other).
[0069] Based on the above content, it can be seen that: in the training process of the detection model, for sample data in which the face and body belong to the same person, a loss function is used to encourage the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other, and the embedding module of the detection model is trained; for sample data in which the face and body do not belong to the same person, a loss function is used to encourage the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be far away from each other, and the embedding module of the detection model is trained.
[0070] After training with batches of sample data, the embedding module of the detection model learns the semantic association between the human body and the face belonging to the same person, including the positional relationship, morphological relationship, etc. Based on this, after processing the image with the trained detection model, the face frame and its corresponding embedded features, the human body frame and its corresponding embedded features are obtained. The embedded features corresponding to the face frame can represent the semantic information of the human body belonging to the same person as the face frame, and specifically, it can also be the semantic information explained from the positional relationship and morphological relationship between the face and the body; similarly, the embedded features corresponding to the human body frame can represent the semantic information of the face belonging to the same person as the human body frame, and of course, it can also be the semantic information explained from the positional relationship and morphological relationship between the face and the body.
[0071] The prediction module can be trained using an angle loss function, specifically, the prediction module is trained to obtain the aforementioned target point position corresponding to the face frame and the aforementioned target point position corresponding to the body frame. The purpose of the angle loss function is to provide a constraint based on relative position for the target point.
[0072] Figure 2 The method of constructing the angle loss function based on the human body 21, the human face 22 and the human head 23 is respectively shown. Now, the case based on the human body and the target point is the center point of the human head is used as an example for explanation.
[0073] b * Represents the actual position of the center point of the human body, h * and h represent the actual position and predicted position of the center point of the head (indicated by 23 in the head diagram), respectively. Construct vector v * and v, so that the angle θ between them is as small as possible. The loss function expression is as follows:
[0074]
[0075] Similarly, in Figure 2 In, f * represents the actual position of the center point of the face, h * and h represent the actual position and predicted position of the center point of the head respectively. Similarly, according to the above loss function expression, the vector v can be constructed * and v, so that the angle θ between the two is as small as possible.
[0076] It should be noted that the detection model, embedded features and the center point position of the head proposed above are solutions for pairing the human body and the face from two dimensions respectively, which can be integrated or separated. In other words, there can be two detection models, one with the original module of the detection model and the newly added embedding module; the other with the original module of the detection model and the newly added prediction module.
[0077] Of course, the method of matching human body and face can also be performed by respectively using the original module with the detection model and the newly added embedding module, and the original module with the detection model and the newly added prediction module.
[0078] The following uses the center point of the head as an example to introduce the matching method between the human body and the face.
[0079] The present application embodiment provides a method for matching a human body and a human face, such as Figure 3 As shown, including:
[0080] S301, calling the detection model to process the target image, obtaining the face frame in the target image and its corresponding embedded features and the position of the center point of the head, as well as the body frame in the target image and its corresponding embedded features and the position of the center point of the head.
[0081] The target image refers to an image that needs to be matched with a human face and a human body, and the image includes at least one human face and one human body. The target image is input into the trained detection model. The original module in the detection model processes the target image to obtain a human body frame and a face frame; the embedding module processes the target image to obtain the embedding features corresponding to the face frame and the embedding features corresponding to the human body frame; the prediction module processes the target image to obtain the center point position of the human head corresponding to the face frame and the center point position of the human head corresponding to the human body frame.
[0082] The embedded features corresponding to the face frame in the target image are used to represent the semantic information of the human body that belongs to the same person as the face, and the embedded features corresponding to the human body frame in the target image are used to represent the semantic information of the face that belongs to the same person as the human body. The center point of the head corresponding to the face frame refers to the position of the center of the head in the face frame; the center point position of the head corresponding to the human body frame refers to the position of the center of the head in the human body frame.
[0083] It should be noted that the face frame and the body frame refer to the coordinate data of the face frame and the coordinate data of the body frame respectively.
[0084] It should also be noted that this step is an implementation method of obtaining the face frame and its corresponding embedded features and the position of the center point of the head in the target image, as well as the human body frame and its corresponding embedded features and the position of the center point of the head in the target image. It is known that other methods that can achieve the acquisition of the face frame and its corresponding embedded features and the position of the center point of the head in the target image, as well as the human body frame and its corresponding embedded features and the position of the center point of the head in the target image, also fall within the scope of this application.
[0085] S302: Process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body and each face in the target image.
[0086] The target human body frame refers to each human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.
[0087] It should be pointed out that for each human frame in the target image, the embedded feature distance between it and the embedded features corresponding to each face frame (also called the embedded feature distance between the human body and the face) needs to be calculated. It can be seen that the embedded features corresponding to the face frame can represent the semantic information of the human body belonging to the same person as the face, and the embedded features corresponding to the human body frame are used to represent the semantic information of the face belonging to the same person as the human body. Therefore, the embedded feature distance calculated using the embedded features corresponding to a face frame and the embedded features corresponding to the human body frame can represent the probability that the face frame and the human body frame belong to the same person.
[0088] S303, processing the head center position corresponding to the target human body frame in the target image and the head center position corresponding to each face frame to obtain the head center distance between the target human body in the target image and each face.
[0089] The target human body frame refers to each human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.
[0090] The position of the center point of the head corresponding to the face frame refers to the position of the center point of the head in the face frame, which can also be understood as the coordinate data of the center point of the head. Similarly, the position of the center point of the head corresponding to the body frame also refers to the position of the center point of the head in the body frame, which is of course also the coordinate data of the center point of the head.
[0091] Similarly, it is also necessary to calculate the distance between the center point of the head corresponding to each human frame in the icon image and the center point of the head corresponding to each face frame (also called the head center point distance between the human body and the face). For a face frame and a body frame belonging to the same person, the center point of the head in the face frame should be the same as the center point of the head in the body frame, so the distance between the two should be the shortest. In this way, the distance between the center point of the head corresponding to the face frame and the center point of the head corresponding to the body frame can reflect whether the face frame and the body frame belong to the same person.
[0092] S304: For the target person in the target image, select a face corresponding to the closest embedding feature distance between the target person and the face and the closest head center distance between the target person and the face as the paired face of the target person.
[0093] According to the above content, we can know that the embedding feature distance calculated by the embedding feature corresponding to a face frame and the embedding feature corresponding to a body frame can represent the probability that the face frame and the body frame belong to the same person. In addition, the distance between the center point position of the head corresponding to a face frame and the center point position of the head corresponding to a body frame can also represent the probability that the face frame and the body frame belong to the same person. Therefore, the embedding feature distance and the head center point distance can be used to find a matching face for the body.
[0094] It can be understood that the embedding feature distance representing that the target human body and the human face are closest means that the embedding feature distance representing that the target human body and the human face have the smallest embedding feature distance. Similarly, the head center point distance representing that the target human body and the human face are closest can also mean that the head center point distance between the target human body and the human face is the smallest.
[0095] Optionally, in another embodiment of the present application, an implementation of step S304 includes:
[0096] A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: the similarity value corresponding to the distance between the target human body in the target image and the embedded features of each face, and the second similarity value refers to: the similarity value corresponding to the distance between the target human body in the target image and the center point of the head of each face.
[0097] The first similarity value and the second similarity value are combined to obtain a fused similarity value.
[0098] For the target person in the target image, the face with the largest fusion similarity value is used as the paired face of the target person.
[0099] The following is an example to illustrate. In this example, there are m human bodies and n human faces in a frame. According to the embedded feature distance between each human body and each human face, and the distance between the center points of each human body and each human face, two matrices S with a size of m×n can be obtained. e and S h .
[0100]
[0101] element is the similarity value obtained by the radial basis function between the embedding feature distance of the i-th human body and the j-th face; the element It is the similarity value obtained by the radial basis function between the center point of the head of the i-th human body and the j-th face.
[0102] It can be known that the calculation formula of the similarity value obtained by the radial basis function is:
[0103] Using the above formula to calculate the element and elements When ij They refer to the embedded feature distance between the i-th human body and the j-th face and the distance between the center points of the human head.
[0104] Use the following formula to fuse the two similarity matrices:
[0105]
[0106] In the above formula, J is an all-one matrix of size m×n. The size of is also m×n, and its elements are obtained by averaging the confidence scores of the human and face prediction boxes corresponding to the corresponding rows and columns; [.]° is an element-by-element exponential operation, and the exponent λ is set to 2; ⊙ represents the multiplication of elements at corresponding positions.
[0107] After obtaining the fused similarity matrix S, for each human body (each row), the face (column) with the greatest similarity is taken as its paired face, thereby obtaining the final human body-face pair.
[0108] The above embodiment shows that the embedded features and the center point position of the head are used to complete the matching of the face and the body. The center point position of the head can also be used alone (this embodiment is explained by taking the center point position of the head as an example. It is known that the center point position of the head can also be replaced by the other two target point positions mentioned above) to achieve the matching of the face and the body. For details, see Figure 4a , the matching method of human body and face includes:
[0109] S401, calling the detection model to process the target image, obtaining the face frame in the target image and the corresponding head center point position, and the body frame in the target image and the corresponding head center point position.
[0110] It should also be noted that this step is an implementation method of obtaining the face frame in the target image and the corresponding head center position, the human body frame in the target image and the corresponding head center position. It is known that other methods that can achieve obtaining the face frame in the target image and the corresponding head center position, the human body frame in the target image and the corresponding head center position also fall within the scope of this application.
[0111] S402, processing the head center position corresponding to the target human body frame in the target image and the head center position corresponding to each face frame to obtain the head center distance between the target human body and each face in the target image.
[0112] The target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.
[0113] The target image may acquire multiple human frames, and for each human frame, the distance between the human body it refers to and the center point of the head of each face may be calculated using step S402. Of course, for each of the partial human frames, the distance between the human body it refers to and the center point of the head of each face may also be calculated using step S402.
[0114] For the specific implementation process of step S401 and step S402, please refer to the corresponding Figure 3 The content of the embodiment.
[0115] S403 . For the target person in the target image, select a face corresponding to the distance between the center points of the human head closest to the target person and the human face as a paired face of the target person.
[0116] In a possible implementation, this step may be: calculating the similarity value corresponding to the distance between the target human body in the target image and the center point of the head of each human face; for the target human body in the target image, taking the face with the largest similarity value as the matching face of the target human body.
[0117] In the above example, we get the matrix S of size m×n h After that, we can directly use the matrix S h Select the human face pair, specifically, for the matrix S h For each human body (each row), the face (column) with the greatest similarity is taken as its paired face, thereby obtaining the final human body-face pair.
[0118] It should also be noted that, for the target human body in the target image, the distance between the center points of the human head that are closest to the target human body and the human face is configured as a set condition in the execution device of the present method. When the device executes step S403, this condition is used to screen out the matching face of the target human body in the target image.
[0119] The present application embodiment provides a human body and face matching device, see Figure 4b ,include:
[0120] The calling unit 401 is used to obtain a face frame and a corresponding target point position in a target image, and a body frame and a corresponding target point position in the target image.
[0121] The processing unit 402 processes the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.
[0122] The screening unit 403 is used to select, for the target person in the target image, a face corresponding to the target point distance that is closest to the target person and the face, as a paired face of the target person.
[0123] In a possible implementation, the screening unit 403 selects, for the target human body in the target image, a human face corresponding to the distance between the center points of the human head closest to the target human body and the human face as the paired human face of the target human body, and is used to:
[0124] The similarity value corresponding to the distance between the center point of the head of the target person and each face in the target image is calculated; for the target person in the target image, the face with the largest similarity value is used as the paired face of the target person.
[0125] In one possible implementation, when the calling unit executes the acquisition of a face frame and its corresponding target point position in a target image, as well as a human body frame and its corresponding target point position in a target image, it is used to: call a detection model to process the target image, obtain a face frame and its corresponding target point position, as well as a human body frame and its corresponding target point position.
[0126] In one possible implementation, the detection model includes: a basic module and a prediction module;
[0127] The basic module is used to process the target image and obtain the face frame and the body frame in the target image; the prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame in the target image;
[0128] The detection model is a one-stage detector, and the prediction module is set after the dilated convolutional layer of the one-stage detector; the detection model is a two-stage detector, and the prediction module is set after the second fully connected layer of the two-stage detector.
[0129] In a possible implementation, it also includes a model training unit. During the process of training the detection model, the model training unit uses an angle loss function to train the detection model to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.
[0130] In a possible implementation, the human body frame includes: a full body frame, an upper body frame and an upper torso frame.
[0131] In a possible implementation, the target point positions include: the center point position of the human head, the center point position of the neck, and one or two connection point positions of the shoulder and the upper arm.
[0132] In the above-mentioned embodiments, the specific working process of the calling unit 401, the processing unit 402 and the screening unit 403 can be found in the corresponding method embodiment content, which will not be repeated here. Of course, the working process of the model training unit can also be found in the corresponding method embodiment content.
[0133] Another embodiment of the present application further provides an electronic device, such as Figure 5 As shown, including:
[0134] One or more processors 501;
[0135] A memory 502 on which a program is stored;
[0136] When the program is executed by one or more processors 501, the one or more processors 501 implement the method for matching a human body and a human face provided in any one of the above embodiments.
[0137] In a fourth aspect, the present application provides an electronic device, including:
[0138] The first processing logic is configured to: receive image data of a target image and process the image data using a neural network model to output position information of a human body part and a target point position in the target image;
[0139] Among them, the neural network model includes:
[0140] The first sub-network receives feature expression data of the image data and outputs position data of the detected face and body;
[0141] The second sub-network receives the position data of the face and the body, and outputs the position of the target point corresponding to the body and the position of the target point corresponding to the body.
[0142] In this embodiment, the first processing logic belongs to an execution component in the electronic device, which can be understood as a processor or an execution unit in the processor.
[0143] The neural network model is the detection model mentioned above. The first sub-network belongs to an execution module in the original module of the detector, which has the function of obtaining the position information of the face and the position information of the body. The second sub-network belongs to the prediction module in the detector.
[0144] In one possible implementation, the neural network model is a two-stage detection network, and the second subnetwork is arranged after the second fully connected layer of the two-stage detector;
[0145] The neural network model is a one-stage detection network, and the second subnetwork is set after the dilated convolutional layer of the one-stage detector.
[0146] It should also be noted that the network before the first sub-network and the second sub-network in the neural network model is a well-known feature map related network, which will not be described in detail in this article.
[0147] In a possible implementation, the electronic device further includes:
[0148] The second processing logic is configured to: calculate the target point distance between the face and the body according to the target point position corresponding to the face and the target point position corresponding to the body output by the second sub-network to obtain the matching relationship between the face and the body.
[0149] In this implementation, the second processing logic is equivalent to the first processing logic, and belongs to an execution component in the electronic device, which can be understood as a processor or an execution unit in the processor.
[0150] The second processing logic is to calculate the target point distance between the face and the body according to the target point position corresponding to the face and the target point position corresponding to the body to obtain the matching relationship between the face and the body, which can be understood as:
[0151] Using the above correspondence Figure 4a In the embodiment of step S402 and step S403, the calculated target point distance between the face and the body can further be the similarity value corresponding to the target point distance between the face and the body.
[0152] Another embodiment of the present application further provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for matching a human body and a face provided in any one of the above embodiments is implemented.
[0153] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.
[0154] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0155] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for matching a human body and a human face, It is characterized in that include: Acquire a face frame and a corresponding target point position in a target image, as well as a human body frame and a corresponding target point position in the target image, wherein the target point position includes a center point position of a human head; Processing the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body and each face in the target image; Processing the center point position of the head corresponding to the target human body frame in the target image and the center point position of the head corresponding to each face frame to obtain the distance between the center point of the head of the target human body and each face in the target image; The target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each human face is the human face referred to by each human face frame; For the target human body in the target image, select a human face corresponding to the embedding feature distance and the head center distance that meets the conditions as the paired human face of the target human body; The step of selecting a human face that meets the conditions as a paired human face of the target human body in the target image includes: There are m human bodies and n human faces in the target image. According to the embedding feature distance between each human body and each human face, and the distance between the center points of each human body and each human face, two matrices S with a size of m×n are obtained: e and the matrix S h , the matrix S e and the matrix S h The specific formula is as follows: element is the similarity value obtained by the radial basis function between the embedding feature distance of the i-th human body and the j-th face; the element It is the similarity value obtained by radial basis function between the center point of the head of the i-th human body and the j-th face; The calculation formula of the similarity value obtained by the radial basis function is: e is a natural constant; Using the above formula to calculate the element and elements When ij They refer to the embedding feature distance between the i-th human body and the j-th face and the distance between the center points of the human head; The matrix S is expressed as follows: e and the matrix S h The fusion is performed to obtain the fused similarity matrix S. The specific formula of the similarity matrix S is as follows: Where J is an all-1 matrix of size m×n, The size of is also m×n, The elements in are obtained by averaging the confidence scores of the human and face prediction boxes corresponding to the corresponding rows and columns; [.]° is an element-by-element exponential operation, and the exponent λ is set to 2; ⊙ represents the multiplication of the elements at the corresponding positions; For the target human body in the target image, the face with the largest fusion similarity value is selected from the similarity matrix as the paired face of the target human body.
2. The method for matching a human body and a human face according to claim 1, It is characterized in that The step of obtaining a face frame in a target image and a corresponding target point position thereof, and a body frame in the target image and a corresponding target point position thereof, comprises: The detection model is called to process the target image to obtain a face frame and a corresponding target point position in the target image, and a human body frame and a corresponding target point position in the target image.
3. The method for matching a human body and a human face according to claim 2, It is characterized in that The detection model includes: a basic module and a prediction module; The basic module is used to process the target image to obtain the face frame and the body frame in the target image; The prediction module is used to obtain the target point position corresponding to the face frame in the target image and the target point position corresponding to the body frame.
4. The method for matching a human body and a human face according to claim 3, It is characterized in that The detection model is a one-stage detector, and the prediction module is arranged after the dilated convolutional layer of the one-stage detector; or, the detection model is a two-stage detector, and the prediction module is arranged after the second fully connected layer of the two-stage detector.
5. The method for matching a human body and a human face according to any one of claims 2 to 4, It is characterized in that During the training process of the detection model, an angle loss function is used to train the detection model's function of obtaining the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.
6. The method for matching a human body and a human face according to any one of claims 1 to 4, It is characterized in that The human body frame includes: a human body full body frame, a human body upper body frame and a human body upper torso frame.
7. The method for matching a human body and a human face according to any one of claims 1 to 4, It is characterized in that The target point positions include: the center point position of the human head, the center point position of the neck, and one or two connection point positions of the shoulder and the upper arm.
8. A device for matching a human body and a human face, It is characterized in that include: A calling unit, used to obtain a face frame in a target image and a corresponding target point position, as well as a human body frame in the target image and a corresponding target point position, wherein the target point position includes a center point position of a human head; a processing unit, configured to process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame, to obtain the embedded feature distance between the target human body and each face in the target image; to process the head center point position corresponding to the target human body frame in the target image and the head center point position corresponding to each face frame, to obtain the head center point distance between the target human body and each face in the target image; the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame; a screening unit, configured to select, for the target human body in the target image, a face corresponding to the embedded feature distance and head center distance closest to the target human body and the face, as a paired face of the target human body; The step of selecting a human face that meets the conditions as a paired human face of the target human body in the target image includes: There are m human bodies and n human faces in the target image. According to the embedding feature distance between each human body and each human face, and the distance between the center points of each human body and each human face, two matrices S with a size of m×n are obtained: e and the matrix S h , the matrix S e and the matrix S h The specific formula is as follows: element is the similarity value obtained by the radial basis function between the embedding feature distance of the i-th human body and the j-th face; the element It is the similarity value obtained by radial basis function between the center point of the head of the i-th human body and the j-th face; The calculation formula of the similarity value obtained by the radial basis function is: e is a natural constant; Using the above formula to calculate the element and elements When ij They refer to the embedding feature distance between the i-th human body and the j-th face and the distance between the center points of the human head; The matrix S is expressed as follows: e and the matrix S h The fusion is performed to obtain the fused similarity matrix S. The specific formula of the similarity matrix S is as follows: Where J is an all-1 matrix of size m×n, The size of is also m×n, The elements in are obtained by averaging the confidence scores of the human and face prediction boxes corresponding to the corresponding rows and columns; [·]° is an element-by-element exponential operation, and the exponent λ is set to 2; ⊙ represents the multiplication of the elements at the corresponding positions; For the target human body in the target image, the face with the largest fusion similarity value is selected from the similarity matrix as the paired face of the target human body.
9. The human body and face matching device according to claim 8, It is characterized in that When the calling unit executes the acquisition of the face frame and the corresponding target point position in the target image, and the human body frame and the corresponding target point position in the target image, the calling unit is used to: call the detection model to process the target image, and obtain the face frame and the corresponding target point position in the target image, and the human body frame and the corresponding target point position in the target image; The detection model includes: a basic module and a prediction module; The basic module is used to be in the target image, and obtain the face frame and the body frame in the target image; the prediction module is used to process the target image, and obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame in the target image; The detection model is a one-stage detector, and the prediction module is arranged after the dilated convolution layer of the one-stage detector; the detection model is a two-stage detector, and the prediction module is arranged after the second fully connected layer of the two-stage detector.
10. The human body and face matching device according to claim 9, It is characterized in that It also includes a model training unit. During the process of training the detection model, the model training unit uses an angle loss function to train the detection model's function of obtaining the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.
11. An electronic device, It is characterized in that include: one or more processors; a memory having a program stored therein; When the program is executed by the one or more processors, the one or more processors implement the method for matching a human body and a human face as claimed in any one of claims 1 to 7.
12. A computer-readable storage medium, It is characterized in that A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for matching a human body and a human face as described in any one of claims 1 to 7 is implemented.
