Human body and face matching method, device, equipment and storage medium

By acquiring and processing the embedded features and target point positions of the face and human body in the target image, using the detection model to pair the face and human body, the problem of low accuracy of face and human body matching in the prior art is solved, and the problem of low accuracy of face and human body matching is realized Higher matching accuracy.

CN113591786BActive Publication Date: 2024-11-26BEIJING AIBI TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202110925798.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-12
Publication Date
2024-11-26
Estimated Expiration
2041-08-12

AI Technical Summary

Technical Problem

In the prior art, in multiple human scenarios, the accuracy of face and body pairing is low, especially in high repetition scenarios, pairing errors are prone to occur.

Method used

By obtaining the embedded features and target point positions of the face frame and body frame in the target image, using the detection model to train and process the embedded features and target point positions, selecting the embedded feature distance and target point distance that meets specific conditions for face and For human body pairing, similarity value fusion is used to improve pairing accuracy.

Benefits of technology

It achieves improving the accuracy of face and human pairing in multiple human scenes, reducing pairing errors, and improving the matching accuracy of human and human faces that belong to the same person.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113591786B_ABST
    Figure CN113591786B_ABST
Patent Text Reader

Abstract

The present invention provides a method, device, equipment and storage medium for matching a human body and a human face. The method comprises obtaining a human face frame and its corresponding embedded features and target point positions, a human body frame and its corresponding embedded features and target point positions, processing the embedded features corresponding to the human body frame and the embedded features corresponding to each human face frame to obtain the embedded feature distance between the human body and each human face, processing the target point position corresponding to the human body frame and the target point position corresponding to each human face frame to obtain the target point distance between the human body and each human face, and selecting, for a human body in a target image, a human face corresponding to the closest embedded feature distance between the human body and the human face and the closest target point distance between the human body and the human face as a paired human face of the human body, so as to achieve a high accuracy in pairing human bodies belonging to the same person.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, device, equipment and storage medium for matching a human body and a human face. Background Art

[0002] Human body recognition and face recognition are two common methods of pedestrian recognition. In a practical scenario of pedestrian recognition, there is also a need to pair faces and bodies that belong to the same person in the scene. Generally, face detection and body detection are completed separately first, and then the Hungarian algorithm is used to pair faces and bodies that belong to the same person from the detected faces and bodies.

[0003] Specifically, the basic method of using the Hungarian algorithm to match faces and bodies belonging to the same person from the detected faces and bodies is that when it is found that the coverage of a face frame in the body frame exceeds a certain degree, the face frame and the body frame are considered to belong to the same person. However, when there are multiple people in the scene image and the bodies of multiple people are highly repeated, it will lead to wrong matching of faces and bodies. It can be seen from this that the current scheme has a low matching accuracy for matching faces and bodies belonging to the same person. Summary of the invention

[0004] In view of this, the embodiments of the present invention provide a method, apparatus, device and storage medium for matching a human body and a human face, so as to achieve pairing of human bodies belonging to the same person with a high accuracy.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, the present application provides a method for matching a human body and a human face, comprising:

[0007] Obtain a face frame in the target image and its corresponding embedded features and target point positions, as well as a human frame in the target image and its corresponding embedded features and target point positions; wherein the embedded features corresponding to the face frame in the target image represent semantic information of a human body belonging to the same person as the face frame, and the embedded features corresponding to the human frame in the target image represent semantic information of a face belonging to the same person as the human frame;

[0008] Process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body in the target image and each face; and process the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; wherein the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame;

[0009] For the target person in the target image, a face corresponding to the embedding feature distance that meets the first condition and the target point distance that meets the second condition is selected as the paired face of the target person.

[0010] Optionally, for a target person in a target image, selecting a face corresponding to an embedded feature distance that meets the first condition and a target point distance that meets the second condition as a paired face of the target person includes:

[0011] A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: a similarity value corresponding to the distance between the target human body in the target image and the embedded features of each face, and the second similarity value refers to: a similarity value corresponding to the distance between the target human body in the target image and the target point of each face;

[0012] Combining the first similarity value and the second similarity value to obtain a fused similarity value;

[0013] For the target person in the target image, the face with the largest fusion similarity value is used as the paired face of the target person.

[0014] Optionally, obtaining a face frame in a target image and its corresponding embedded features and target point positions, as well as a body frame in a target image and its corresponding embedded features and target point positions, includes:

[0015] The detection model is called to process the target image to obtain the face frame in the target image and its corresponding embedded features and target point positions, as well as the body frame in the target image and its corresponding embedded features and target point positions.

[0016] Optionally, the detection model includes: a basic module, an embedding module and a prediction module;

[0017] The basic module is used to process the target image and obtain the face frame and body frame in the target image;

[0018] The embedding module is used to process the target image and obtain the embedding features corresponding to the face frame and the embedding features corresponding to the body frame;

[0019] The prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the body frame in the target image.

[0020] Optionally, during the training process, for sample data in which the face and body belong to the same person, the detection model is trained using a loss function that encourages the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other; for sample data in which the face and body do not belong to the same person, the detection model is trained using a loss function that encourages the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be away from each other.

[0021] Optionally, during the training process, the detection model uses an angle loss function to train the detection model's functions of obtaining the target point positions corresponding to the face frame and the target point positions corresponding to the body frame. The angle loss function is used to: prompt the predicted value of the target point position corresponding to the body frame in the target image to be closer to the actual value, and prompt the predicted value of the target point position corresponding to the face frame to be closer to the actual value.

[0022] Optionally, the detection model is a one-stage detector, and the embedding module of the detection model includes: four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the prediction module is arranged after the dilated convolutional layer of the one-stage detector;

[0023] The detection model is a two-stage detector. The embedding module of the detection model includes: two fully connected layers; the prediction module is set after the second fully connected layer of the two-stage detector.

[0024] Optionally, the target point positions include: the center point position of the head, the center point position of the neck, and one or two connection point positions of the shoulder and the upper arm; the human body frame includes: a full body frame, an upper body frame and an upper torso frame.

[0025] In a second aspect, the present application provides a human body and face matching device, comprising:

[0026] The calling unit is used to obtain a face frame in a target image and its corresponding embedded features and target point positions, as well as a human body frame in a target image and its corresponding embedded features and target point positions; wherein the embedded features corresponding to the face frame in the target image represent semantic information of a human body belonging to the same person as the face frame, and the embedded features corresponding to the human body frame in the target image represent semantic information of a face belonging to the same person as the human body frame;

[0027] A processing unit is used to process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body and each face in the target image; and process the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body and each face in the target image; wherein the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame;

[0028] The screening unit is used to select, for a target human body in a target image, a face corresponding to an embedded feature distance that meets a first condition and a target point distance that meets a second condition as a paired face of the target human body.

[0029] Optionally, the screening unit selects, for a target human body in a target image, a face corresponding to an embedded feature distance that meets the first condition and a target point distance that meets the second condition as a paired face of the target human body, for:

[0030] A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: a similarity value corresponding to an embedded feature distance between a target human body in a target image and each human face, and the second similarity value refers to: a similarity value corresponding to a target point distance between a target human body in a target image and each human face; the first similarity value and the second similarity value are combined to obtain a fused similarity value; for a target human body in a target image, a face having the largest fused similarity value is used as a paired face of the target human body.

[0031] Optionally, when the calling unit executes obtaining a face frame in a target image and its corresponding embedded features and target point positions, and a human body frame in a target image and its corresponding embedded features and target point positions, it is used to:

[0032] The detection model is called to process the target image to obtain the face frame in the target image and its corresponding embedded features and target point positions, as well as the body frame in the target image and its corresponding embedded features and target point positions.

[0033] Optionally, the detection model includes: a basic module, an embedding module and a prediction module;

[0034] The basic module is used to process the target image and obtain the face frame and body frame in the target image;

[0035] The embedding module is used to process the target image and obtain the embedding features corresponding to the face frame and the embedding features corresponding to the body frame;

[0036] The prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the human body frame in the target image;

[0037] The detection model is a one-stage detector. The embedding module of the detection model includes: four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the prediction module is set after the dilated convolutional layer of the one-stage detector;

[0038] The detection model is a two-stage detector. The embedding module of the detection model includes: two fully connected layers; the prediction module is set after the second fully connected layer of the two-stage detector.

[0039] Optionally, the method further comprises a model training unit. During the process of training the detection model, for sample data in which the face and the body belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other; for sample data in which the face and the body do not belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be far away from each other;

[0040] The model training unit uses an angle loss function to train the detection model to obtain the function of the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.

[0041] In a third aspect, the present application provides an electronic device, including:

[0042] one or more processors;

[0043] a memory having a program stored therein;

[0044] When the program is executed by one or more processors, the one or more processors implement the method for matching a human body and a human face of any one of the first aspects described above.

[0045] In a fourth aspect, the present application provides an electronic device, including:

[0046] A first processing logic is configured to: receive image data of a target image and process the image data using a neural network model to output location information of a human body part in the target image, an embedded feature, and a target point location;

[0047] Among them, the neural network model includes:

[0048] The first sub-network receives feature expression data of the image data and outputs position data of the detected face and body;

[0049] A second sub-network receives feature expression data in parallel with the first sub-network, and outputs an embedded feature corresponding to the face and an embedded feature corresponding to the face;

[0050] The third sub-network receives the position data of the face and the body, and outputs the target point position corresponding to the face and the target point position corresponding to the body.

[0051] Optionally, it also includes:

[0052] The second processing logic is configured to calculate the embedded feature distance and target point distance between the face and the body based on the embedded features corresponding to the face and the embedded features corresponding to the body output by the second sub-network, and the target point positions corresponding to the face and the target point positions corresponding to the body output by the third sub-network, so as to obtain the matching relationship between the face and the body.

[0053] Optionally, the neural network model is a one-stage detection network, the feature expression data received by the second sub-network is a feature map of the target image, the second sub-network includes four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the third sub-network is arranged after the dilated convolutional layer of the one-stage detector;

[0054] The neural network model is a two-stage detection network. The feature expression data received by the second sub-network is the feature map of the candidate area of ​​the target image. The second sub-network includes two fully connected layers. The third sub-network is arranged after the second fully connected layer of the two-stage detector.

[0055] In a fifth aspect, the present application provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for matching a human body and a face as described in any one of the first aspects above is implemented.

[0056] It can be seen from the above technical scheme that in the method for matching a human body and a face provided in an embodiment of the present application, a face frame in a target image and its corresponding embedded features and target point positions, a human body frame and its corresponding embedded features and target point positions are obtained; the embedded features corresponding to the human body frame in the target image and the embedded features corresponding to each human face frame are processed to obtain the embedded feature distance between the human body in the target image and each human face; the target point position corresponding to the human body frame in the target image and the target point position corresponding to each human face frame are processed to obtain the target point distance between the human body in the target image and each human face; for the human body in the target image, a face corresponding to the embedded feature distance that meets the first condition and the target point distance that meets the second condition is selected as the paired face of the human body, so as to achieve a high accuracy rate in pairing human bodies belonging to the same person. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0058] Figure 1 A display diagram of the detection model provided for the application embodiment;

[0059] Figure 2 A diagram showing the generation process of the angle loss function provided in the embodiment of the present application;

[0060] Figure 3 A flowchart of a method for matching a human body and a human face provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the structure of a device for matching a human body and a human face provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0065] The embodiments of the present invention provide a method, device, equipment and storage medium for matching a human body and a human face, so as to achieve high accuracy in pairing human bodies belonging to the same person.

[0066] In the method for matching a human body and a human face provided in an embodiment of the present application, a detection model needs to be called to process a target image to obtain a face frame in the target image and its corresponding embedded features, a human body frame in the target image and its corresponding embedded features. Optionally, the human body frame includes: a human body full body frame, a human body upper body frame, and a human body upper torso frame.

[0067] It can be seen that the detection model used in the present application is improved on the basis of the basic detection model of face and body. For example, it can be improved on the common one-stage detector (One Stage) or two-stage detector (Two Stages) architecture.

[0068] For details, see Figure 1 , an embedding module 11 is added to the one-stage detector or the two-stage detector, and the embedding module 11 runs in parallel with the original module (also called the basic module) of the detector (referring to the one-stage detector and the two-stage detector). For the one-stage detector, the embedding module includes four ordinary convolutional layers and a dilated convolutional layer at the end of the four ordinary convolutional layers; for the two-stage detector, the embedding module includes two fully connected layers f c1 and f c2 The main function of the embedding module is to process the image and obtain the embedding features corresponding to the face frame and the body frame in the image.

[0069] In addition, a prediction module is added to the end of the original module of the one-stage detector or the two-stage detector, see also Figure 1 , the prediction module is added after the dilated convolutional layer in the original module of the one-stage detector, and the fully connected layer f in the original module of the two-stage detector c2 A prediction module is added later. The main function of the prediction module is to process the image and obtain the position of the center point of the head corresponding to the face frame and the position of the center point of the head corresponding to the body frame in the image.

[0070] After building the architecture of the detection model, you need to use sample data to train the detection model, train the original module, embedding module and prediction module of the detection model, so that the original module can process the image and obtain the face frame and body frame in the image, so that the embedding module can process the image and obtain the embedded features corresponding to the face frame and the embedded features corresponding to the body frame in the image, so that the prediction module can process the image and obtain the head center point position corresponding to the face frame and the head center point position corresponding to the body frame in the image.

[0071] It should also be noted that, by processing the image using the prediction module, the center point position of the neck corresponding to the face frame in the image, as well as the positions of one or two connection points of the shoulder and upper arm, and the center point position of the neck corresponding to the human body frame, as well as the positions of one or two connection points of the shoulder and upper arm can also be obtained. Based on this, the prediction module processes the image, and the target point positions obtained include: the center point position of the head, the center point position of the neck, and the positions of one or two connection points of the shoulder and upper arm.

[0072] Specifically, in order to train the embedding module of the detection model, the loss function is constructed according to the following purpose: if two targets belong to the same pedestrian (such as the body and face of the same pedestrian), their embedded features are brought closer to each other; otherwise, they are moved away from each other.

[0073] For the human body frame and face frame belonging to the same person k in the sample data, the constructed loss function is:

[0074]

[0075] in, They represent the embedded features of the i-th body frame and the j-th face frame of the pedestrian, respectively. k and N k Represents the number of valid boxes around the human body and face. By optimizing the loss function, and The Euclidean distance between them is as small as possible (close to each other).

[0076] For the human body frame and face frame that do not belong to the same pedestrian in the sample data, the constructed loss function is:

[0077]

[0078] Among them, δ is a small positive number (set to 2); M k and N l Represent the number of human frames of pedestrian k and the number of face frames of pedestrian l respectively. By optimizing the loss function, and The Euclidean distance between them is as large as possible (stay away from each other).

[0079] Based on the above content, it can be seen that: in the training process of the detection model, for sample data in which the face and body belong to the same person, a loss function is used to encourage the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other, and the embedding module of the detection model is trained; for sample data in which the face and body do not belong to the same person, a loss function is used to encourage the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be far away from each other, and the embedding module of the detection model is trained.

[0080] After training with batches of sample data, the embedding module of the detection model learns the semantic association between the human body and the face belonging to the same person, including the positional relationship, morphological relationship, etc. Based on this, after processing the image with the trained detection model, the face frame and its corresponding embedded features, the human body frame and its corresponding embedded features are obtained. The embedded features corresponding to the face frame can represent the semantic information of the human body belonging to the same person as the face frame, and specifically, it can also be the semantic information explained from the positional relationship and morphological relationship between the face and the body; similarly, the embedded features corresponding to the human body frame can represent the semantic information of the face belonging to the same person as the human body frame, and of course, it can also be the semantic information explained from the positional relationship and morphological relationship between the face and the body.

[0081] The prediction module can be trained using an angle loss function, specifically, the prediction module is trained to obtain the aforementioned target point position corresponding to the face frame and the aforementioned target point position corresponding to the body frame. The purpose of the angle loss function is to provide a constraint based on relative position for the target point.

[0082] Figure 2 The method of constructing the angle loss function based on the human body 21, the human face 22 and the human head 23 is respectively shown. Now, the case based on the human body and the target point is the center point of the human head is used as an example for explanation.

[0083] b * Represents the actual position of the center point of the human body, h * and h represent the actual position and predicted position of the center point of the head (indicated by 23 in the head diagram), respectively. Construct vector v * and v, so that the angle θ between them is as small as possible. The loss function expression is as follows:

[0084]

[0085] Similarly, in Figure 2 In, f * represents the actual position of the center point of the face, h * and h represent the actual position and predicted position of the center point of the head respectively. Similarly, according to the above loss function expression, the vector v can be constructed * and v, so that the angle θ between the two is as small as possible.

[0086] The following uses the center point of the head as an example to introduce the matching method between the human body and the face.

[0087] Based on the detection model trained in the above embodiment, the present application embodiment provides a method for matching a human body and a face, such as Figure 3 As shown, including:

[0088] S301, calling the detection model to process the target image, obtaining the face frame in the target image and its corresponding embedded features and the position of the center point of the head, as well as the body frame in the target image and its corresponding embedded features and the position of the center point of the head.

[0089] The target image refers to an image that needs to be matched with a human face and a human body, and the image includes at least one human face and one human body. The target image is input into the trained detection model. The original module in the detection model processes the target image to obtain a human body frame and a face frame; the embedding module processes the target image to obtain the embedding features corresponding to the face frame and the embedding features corresponding to the human body frame; the prediction module processes the target image to obtain the center point position of the human head corresponding to the face frame and the center point position of the human head corresponding to the human body frame.

[0090] The embedded features corresponding to the face frame in the target image are used to represent the semantic information of the human body that belongs to the same person as the face, and the embedded features corresponding to the human body frame in the target image are used to represent the semantic information of the face that belongs to the same person as the human body. The center point of the head corresponding to the face frame refers to the position of the center of the head in the face frame; the center point position of the head corresponding to the human body frame refers to the position of the center of the head in the human body frame.

[0091] It should be noted that the face frame and the body frame refer to the coordinate data of the face frame and the coordinate data of the body frame respectively.

[0092] It should also be noted that this step is an implementation method of obtaining the face frame and its corresponding embedded features and the position of the center point of the head in the target image, as well as the human body frame and its corresponding embedded features and the position of the center point of the head in the target image. It is known that other methods that can achieve the acquisition of the face frame and its corresponding embedded features and the position of the center point of the head in the target image, as well as the human body frame and its corresponding embedded features and the position of the center point of the head in the target image, also fall within the scope of this application.

[0093] S302: Process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body and each face in the target image.

[0094] The target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.

[0095] The target image may acquire multiple human frames, and for each human frame, the distance between the human body it refers to and the center point of the head of each face may be calculated using step S302. Of course, for each of the partial human frames, the distance between the human body it refers to and the center point of the head of each face may also be calculated using step S302.

[0096] It should be pointed out that for each human frame in the target image, the embedded feature distance between it and the embedded features corresponding to each face frame (also called the embedded feature distance between the human body and the face) needs to be calculated. It can be seen that the embedded features corresponding to the face frame can represent the semantic information of the human body belonging to the same person as the face, and the embedded features corresponding to the human body frame are used to represent the semantic information of the face belonging to the same person as the human body. Therefore, the embedded feature distance calculated using the embedded features corresponding to a face frame and the embedded features corresponding to the human body frame can represent the probability that the face frame and the human body frame belong to the same person.

[0097] S303, processing the head center position corresponding to the target human body frame in the target image and the head center position corresponding to each face frame to obtain the head center distance between the target human body in the target image and each face.

[0098] The target human body frame refers to each human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.

[0099] The position of the center point of the head corresponding to the face frame refers to the position of the center point of the head in the face frame, which can also be understood as the coordinate data of the center point of the head. Similarly, the position of the center point of the head corresponding to the body frame also refers to the position of the center point of the head in the body frame, which is of course also the coordinate data of the center point of the head.

[0100] Similarly, it is also necessary to calculate the distance between the center point of the head corresponding to each human frame in the icon image and the center point of the head corresponding to each face frame (also called the head center point distance between the human body and the face). For a face frame and a body frame belonging to the same person, the center point of the head in the face frame should be the same as the center point of the head in the body frame, so the distance between the two should be the shortest. In this way, the distance between the center point of the head corresponding to the face frame and the center point of the head corresponding to the body frame can reflect whether the face frame and the body frame belong to the same person.

[0101] S304: For the target person in the target image, select a face corresponding to the closest embedding feature distance between the target person and the face and the closest head center distance between the target person and the face as the paired face of the target person.

[0102] According to the above content, we can know that the embedding feature distance calculated by the embedding feature corresponding to a face frame and the embedding feature corresponding to a body frame can represent the probability that the face frame and the body frame belong to the same person. In addition, the distance between the center point position of the head corresponding to a face frame and the center point position of the head corresponding to a body frame can also represent the probability that the face frame and the body frame belong to the same person. Therefore, the embedding feature distance and the head center point distance can be used to find a matching face for the body.

[0103] It can be understood that the embedding feature distance representing that the target human body and the human face are closest means that the embedding feature distance representing that the target human body and the human face have the smallest embedding feature distance. Similarly, the head center point distance representing that the target human body and the human face are closest can also mean that the head center point distance between the target human body and the human face is the smallest.

[0104] It should also be noted that, for the target human body in the target image, the embedded feature distance between the target human body and the face is closest, and is configured as a setting condition in the execution device of this method. The distance between the center points of the head that are closest to the target human body and the face is also configured as a setting condition in the execution device of this method. When the device executes step S304, these two conditions are used to screen out the matching face of the target human body in the target image.

[0105] Optionally, in another embodiment of the present application, an implementation of step S304 includes:

[0106] A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: the similarity value corresponding to the distance between the target human body in the target image and the embedded features of each face, and the second similarity value refers to: the similarity value corresponding to the distance between the target human body in the target image and the center point of the head of each face.

[0107] The first similarity value and the second similarity value are combined to obtain a fused similarity value.

[0108] For the target person in the target image, the face with the largest fusion similarity value is used as the paired face of the target person.

[0109] The following is an example to illustrate. In this example, there are m human bodies and n human faces in a frame. According to the embedded feature distance between each human body and each human face, and the distance between the center points of each human body and each human face, two matrices S with a size of m×n can be obtained. e and S h .

[0110]

[0111] element is the similarity value obtained by the radial basis function between the embedding feature distance of the i-th human body and the j-th face; the element It is the similarity value obtained by the radial basis function between the center point of the head of the i-th human body and the j-th face.

[0112] It can be known that the calculation formula of the similarity value obtained by the radial basis function is:

[0113] Using the above formula to calculate the element and elements When ij They refer to the embedded feature distance between the i-th human body and the j-th face and the distance between the center points of the human head.

[0114] Use the following formula to fuse the two similarity matrices:

[0115]

[0116] In the above formula, J is an all-one matrix of size m×n. The size of is also m×n, and its elements are obtained by averaging the confidence scores of the human and face prediction boxes corresponding to the corresponding rows and columns; [.]° is an element-by-element exponential operation, and the exponent λ is set to 2; ⊙ represents the multiplication of elements at corresponding positions.

[0117] After obtaining the fused similarity matrix S, for each human body (each row), the face (column) with the greatest similarity is taken as its paired face, thereby obtaining the final human body-face pair.

[0118] The present application embodiment provides a device for matching a human body and a human face, see Figure 4 ,include:

[0119] The calling unit 401 is used to obtain a face frame in a target image and its corresponding embedded features and target point positions, as well as a human body frame in a target image and its corresponding embedded features and target point positions; wherein the embedded features corresponding to the face frame in the target image represent semantic information of a human body belonging to the same person as the face frame, and the embedded features corresponding to the human body frame in the target image represent semantic information of a face belonging to the same person as the human body frame.

[0120] The processing unit 402 is used to process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body in the target image and each face; and process the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; wherein the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame.

[0121] The screening unit 403 is used to select, for the target human body in the target image, a face corresponding to the embedded feature distance that meets the first condition and the target point distance that meets the second condition as a paired face of the target human body.

[0122] In a possible implementation, the screening unit 403 selects, for the target human body in the target image, a face corresponding to the embedded feature distance that meets the first condition and the target point distance that meets the second condition as the paired face of the target human body, for:

[0123] A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: a similarity value corresponding to the distance between the embedded features of the target human body in the target image and each human face, and the second similarity value refers to: a similarity value corresponding to the distance between the center point of the head of the target human body in the target image and each human face; the first similarity value and the second similarity value are combined to obtain a fused similarity value; for the target human body in the target image, a face with the largest fused similarity value is used as a paired face of the target human body.

[0124] In a possible implementation, when the calling unit executes the steps of acquiring a face frame in a target image and its corresponding embedded features and target point positions, and a body frame in a target image and its corresponding embedded features and target point positions, the calling unit is used to:

[0125] The detection model is called to process the target image to obtain the face frame in the target image and its corresponding embedded features and target point positions, as well as the body frame in the target image and its corresponding embedded features and target point positions.

[0126] In one possible implementation, the detection model includes: a basic module, an embedding module, and a prediction module;

[0127] The basic module is used to process the target image and obtain the face frame and body frame in the target image;

[0128] The embedding module is used to process the target image and obtain the embedding features corresponding to the face frame and the embedding features corresponding to the body frame;

[0129] The prediction module is used to obtain the target point position corresponding to the face frame and the target point position corresponding to the human body frame in the target image;

[0130] The detection model is a one-stage detector. The embedding module of the detection model includes: four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the prediction module is set after the dilated convolutional layer of the one-stage detector;

[0131] The detection model is a two-stage detector. The embedding module of the detection model includes: two fully connected layers; the prediction module is set after the second fully connected layer of the two-stage detector.

[0132] In a possible implementation, the method further includes a model training unit. During the process of training the detection model, for sample data in which the face and the body belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other; for sample data in which the face and the body do not belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be far away from each other;

[0133] The model training unit uses an angle loss function to train the detection model to obtain the function of the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.

[0134] In the above-mentioned embodiments of the present application, the specific working process of the calling unit 401, the processing unit 402 and the screening unit 403 can be referred to the corresponding method embodiment content, which will not be repeated here. Of course, the working process of the model training unit can also be referred to the corresponding method embodiment content.

[0135] Another embodiment of the present application further provides an electronic device, such as Figure 5 As shown, including:

[0136] One or more processors 501;

[0137] A memory 502 on which a program is stored;

[0138] When the program is executed by one or more processors 501, the one or more processors 501 implement the method for matching a human body and a human face provided in any one of the above embodiments.

[0139] Another embodiment of the present application further provides a readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for matching a human body and a face provided in any one of the above embodiments is implemented.

[0140] Another embodiment of the present application further provides an electronic device, including:

[0141] A first processing logic is configured to: receive image data of a target image and process the image data using a neural network model to output location information of a human body part in the target image, an embedded feature, and a target point location;

[0142] Among them, the neural network model includes:

[0143] The first sub-network receives feature expression data of the image data and outputs position data of the detected face and body;

[0144] A second sub-network receives feature expression data in parallel with the first sub-network, and outputs an embedded feature corresponding to the face and an embedded feature corresponding to the face;

[0145] The third sub-network receives the position data of the face and the body, and outputs the target point position corresponding to the face and the target point position corresponding to the body.

[0146] In this embodiment, the first processing logic belongs to an execution component in the electronic device, which can be understood as a processor or an execution unit in the processor.

[0147] The neural network model is the detection model mentioned above. The first sub-network belongs to an execution module in the original module of the detector, which has the function of obtaining the position information of the face and the body. The second sub-network belongs to the embedding module in the detector, and the third sub-network belongs to the prediction module in the detector.

[0148] In one possible implementation, the neural network model is a one-stage detection network, the feature expression data received by the second sub-network is a feature map of the target image, the second sub-network includes four convolutional layers and an expanded convolutional layer at the end of the four convolutional layers; the third sub-network is arranged after the expanded convolutional layer of the one-stage detector;

[0149] The neural network model is a two-stage detection network. The feature expression data received by the second sub-network is the feature map of the candidate area of ​​the target image. The second sub-network includes two fully connected layers. The third sub-network is arranged after the second fully connected layer of the two-stage detector.

[0150] In a possible implementation, the electronic device further includes:

[0151] The second processing logic is configured to calculate the embedded feature distance and target point distance between the face and the body based on the embedded features corresponding to the face and the embedded features corresponding to the body output by the second sub-network, and the target point positions corresponding to the face and the target point positions corresponding to the body output by the third sub-network, so as to obtain the matching relationship between the face and the body.

[0152] In this implementation, the second processing logic is equivalent to the first processing logic, and belongs to an execution component in the electronic device, which can be understood as a processor or an execution unit in the processor.

[0153] Using the above correspondence Figure 3 In the embodiment of step S302 and step S303, the calculated embedded feature distance and target point distance between the face and the body, further, can also be the similarity value corresponding to the embedded feature and target point distance between the face and the body.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0155] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0156] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for matching a human body and a human face, It is characterized in that include: Acquire a face frame in a target image and its corresponding embedded features and target point positions, as well as a human body frame in the target image and its corresponding embedded features and target point positions; wherein the embedded features corresponding to the face frame in the target image represent semantic information of a human body belonging to the same person as the face frame, and the embedded features corresponding to the human body frame in the target image represent semantic information of a face belonging to the same person as the human body frame; Processing the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body in the target image and each face; and processing the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; wherein the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame; For the target human body in the target image, select a face corresponding to an embedding feature distance that meets the first condition and a target point distance that meets the second condition as a paired face of the target human body; The acquiring of the face frame in the target image and its corresponding embedded features and target point positions, as well as the body frame in the target image and its corresponding embedded features and target point positions, includes: Calling the detection model to process the target image, obtaining a face frame in the target image and its corresponding embedded features and target point positions, as well as a human body frame in the target image and its corresponding embedded features and target point positions; The detection model includes: a basic module, an embedding module and a prediction module; The basic module is used to process the target image to obtain the face frame and the body frame in the target image; The embedding module is used to process the target image to obtain the embedding features corresponding to the face frame and the embedding features corresponding to the body frame; The prediction module is used to obtain the target point position corresponding to the face frame in the target image and the target point position corresponding to the body frame; the embedding module is trained in the following manner: Get sample data, and for the human body frame and face frame belonging to the same person k in the sample data, construct the loss function as follows: in, represent the embedding features of the i-th body frame and the j-th face frame of the same person k, respectively, M k and N k Represents the number of human frames and face frames of the same pedestrian k; For the human body frame and face frame that do not belong to the same pedestrian in the sample data, the constructed loss function is: Among them, δ is a small positive number; M k and N l They represent the number of human frames of the same pedestrian k and the number of face frames of pedestrian l respectively; During the training process of the detection model, for sample data in which the face and the body belong to the same person, a loss function is used to force the embedding features corresponding to the body frame and the embedding features corresponding to the face frame to be close to each other, and the detection model is trained; for sample data in which the face and the body do not belong to the same person, a loss function is used to force the embedding features corresponding to the body frame and the embedding features corresponding to the face frame to be far away from each other, and the detection model is trained; The prediction module is trained using an angle loss function, and the expression of the angle loss function is as follows: b * represents the actual position of the center point of the human body, h * and h represent the actual position and predicted position of the center point of the head, respectively, constructing the vector v * and a vector v, the vector v * From point b * and point h * The vector v is composed of point b * and point h, vector v * The angle between the vector and the vector is θ, so that the vector v * The angle θ between and the vector v is as small as possible.

2. The method for matching a human body and a human face according to claim 1, It is characterized in that The step of selecting, for the target human body in the target image, a face corresponding to an embedded feature distance that meets a first condition and a target point distance that meets a second condition as a paired face of the target human body includes: A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: a similarity value corresponding to the distance between the target human body in the target image and the embedded features of each face, and the second similarity value refers to: a similarity value corresponding to the distance between the target point of the target human body in the target image and each face; combining the first similarity value and the second similarity value to obtain a fused similarity value; For the target person in the target image, the face with the largest fusion similarity value is used as the paired face of the target person.

3. The method for matching a human body and a human face according to claim 1, It is characterized in that During the training process of the detection model, an angle loss function is used to train the detection model's function of obtaining the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.

4. The method for matching a human body and a human face according to claim 1, It is characterized in that When the detection model is a one-stage detector, the embedding module of the detection model includes: four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the prediction module is arranged after the dilated convolutional layer of the one-stage detector; When the detection model is a two-stage detector, the embedding module of the detection model includes: two fully connected layers; the prediction module is arranged after the second fully connected layer of the two-stage detector.

5. The method for matching a human body and a human face according to any one of claims 1 to 2, It is characterized in that The target point positions include: the center point position of the head, the center point position of the neck, and one or two connection point positions of the shoulder and the upper arm; the human body frame includes: a full body frame, an upper body frame and an upper torso frame.

6. A device for matching a human body and a human face, It is characterized in that include: The calling unit is used to obtain a face frame in a target image and its corresponding embedded features and target point positions, as well as a human body frame in the target image and its corresponding embedded features and target point positions; wherein the embedded features corresponding to the face frame in the target image represent semantic information of a human body belonging to the same person as the face frame, and the embedded features corresponding to the human body frame in the target image represent semantic information of a face belonging to the same person as the human body frame; A processing unit is used to process the embedded features corresponding to the target human body frame in the target image and the embedded features corresponding to each face frame to obtain the embedded feature distance between the target human body in the target image and each face; and process the target point position corresponding to the target human body frame in the target image and the target point position corresponding to each face frame to obtain the target point distance between the target human body in the target image and each face; wherein the target human body frame refers to at least one human body frame in the target image, the target human body is the human body referred to by the target human body frame, and each face is the face referred to by each face frame; A screening unit, for selecting, for the target human body in the target image, a face corresponding to an embedded feature distance that meets a first condition and a target point distance that meets a second condition as a paired face of the target human body; When the calling unit executes the steps of acquiring a face frame in a target image and its corresponding embedded features and target point positions, and a human body frame in the target image and its corresponding embedded features and target point positions, the calling unit is used to: Calling the detection model to process the target image, obtaining a face frame in the target image and its corresponding embedded features and target point positions, as well as a human body frame in the target image and its corresponding embedded features and target point positions; The detection model includes: a basic module, an embedding module and a prediction module; The basic module is used to process the target image to obtain the face frame and the body frame in the target image; The embedding module is used to process the target image to obtain the embedding features corresponding to the face frame and the embedding features corresponding to the body frame; The prediction module is used to obtain the target point position corresponding to the face frame in the target image and the target point position corresponding to the human body frame; When the detection model is a one-stage detector, the embedding module of the detection model includes: four convolutional layers and a dilated convolutional layer at the end of the four convolutional layers; the prediction module is arranged after the dilated convolutional layer of the one-stage detector; When the detection model is a two-stage detector, the embedding module of the detection model includes: two fully connected layers; the prediction module is arranged after the second fully connected layer of the two-stage detector; The embedding module is trained in the following way: Get sample data, and for the human body frame and face frame belonging to the same person k in the sample data, construct the loss function as follows: in, represent the embedding features of the i-th body frame and the j-th face frame of the same person k, respectively, M k and N k Represents the number of human frames and face frames of the same pedestrian k; For the human body frame and face frame that do not belong to the same pedestrian in the sample data, the constructed loss function is: Among them, δ is a small positive number; M k and N l They represent the number of human frames of the same pedestrian k and the number of face frames of pedestrian l respectively; A model training unit, wherein during the process of training the detection model, for sample data in which the face and the body belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be close to each other; and for sample data in which the face and the body do not belong to the same person, the detection model is trained by using a loss function that causes the embedded features corresponding to the body frame and the embedded features corresponding to the face frame to be far away from each other; The prediction module is trained using an angle loss function, and the expression of the angle loss function is as follows: b * represents the actual position of the center point of the human body, h * and h represent the actual position and predicted position of the center point of the head, respectively, constructing the vector v * and a vector v, the vector v * From point b * and point h * The vector v is composed of point b * and point h, vector v * The angle between the vector and the vector is θ, so that the vector v * The angle θ between and the vector v is as small as possible.

7. The human body and face matching device according to claim 6, It is characterized in that The screening unit selects, for the target human body in the target image, a face corresponding to an embedded feature distance that meets a first condition and a target point distance that meets a second condition as a paired face of the target human body, for: A first similarity value and a second similarity value are calculated; wherein the first similarity value refers to: a similarity value corresponding to an embedded feature distance between a target human body in the target image and each human face, and the second similarity value refers to: a similarity value corresponding to a target point distance between the target human body in the target image and each human face; the first similarity value and the second similarity value are combined to obtain a fused similarity value; for the target human body in the target image, a face having the largest fused similarity value is used as a paired face of the target human body.

8. The human body and face matching device according to claim 6, It is characterized in that It also includes a model training unit, The model training unit adopts an angle loss function to train the detection model to obtain the function of the target point position corresponding to the face frame and the target point position corresponding to the body frame. The angle loss function is used to: promote the predicted value of the target point position corresponding to the body frame in the target image to be close to the actual value, and promote the predicted value of the target point position corresponding to the face frame to be close to the actual value.

9. An electronic device, It is characterized in that include: one or more processors; a memory having a program stored therein; When the program is executed by the one or more processors, the one or more processors implement the method for matching a human body and a human face as described in any one of claims 1 to 5.

10. A readable storage medium, It is characterized in that A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for matching a human body and a human face as described in any one of claims 1 to 5 is implemented.

Citation Information

Patent Citations

  • User identification method and device, electronic equipment and storage medium

    CN109740516A

  • Pedestrian detection method and device based on deep learning and medium

    CN111178208A

  • Target object detection method and device, computer equipment and storage medium

    CN111738231A

  • Human face key point detection method and device, electronic equipment and storage medium

    CN112464809A

  • Human face and human body matching method and device and storage medium

    CN113205138A