Human body part alignment method and device, electronic equipment and computer program product

Abstract

The invention is suitable for the technical field of artificial intelligence, and provides a human body part alignment method and device, electronic equipment and a computer program product. According to the embodiment of the invention, by introducing the first human body key point heat map for guiding the human body feature extraction network to perform human body feature alignment, on one hand, the attention of the human body feature extraction network on human body key point parts can be enhanced, and the loss guidance of the human body feature extraction network in the aspect of human body key points is increased; therefore, the extraction capability of effective information of the human body is enhanced, and the extraction of visible human body features is enhanced. And on the other hand, the improvement cost of the network model is low, the accuracy of human body feature alignment can be quickly and effectively improved, and the effectiveness of human body recognition is improved. According to the embodiment of the invention, the extracted human body feature vector is stronger in robustness and higher in matching accuracy, and the conditions of misrecognition, missing detection and the like caused by shielding and different human body posture angles are effectively reduced.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular relates to a method, device, electronic device, and computer program product for aligning human body parts. Background Art

[0002] Person re-identification (ReID) technology is a computer vision-based image retrieval technique designed to identify and track the movements of specific pedestrians across multiple camera scenarios. Its core approach is to extract pedestrian features and perform similarity matching through deep learning. It is widely used in fields such as intelligent security and video surveillance. ReID technology can be used for intelligent person retrieval in high-traffic locations such as supermarkets, train stations, and exhibition halls. In surveillance camera networks installed in public areas such as streets, squares, and parks, ReID technology enables cross-view retrieval, improving surveillance efficiency.

[0003] Feature alignment refers to adjusting the extracted objects to the same pose and angle during the feature extraction process, or by enabling the model to determine the corresponding positions based on the provided information. For example, in face recognition, face alignment involves applying an affine transformation to the facial key points to adjust the pose and position of the facial features to the frontal view. Unlike faces, the human body exhibits greater spatial variation in pose, making affine transformations unsuitable. There are two common methods for body alignment. First, manual segmentation or segmentation is used to align body parts. For example, pedestrian images and features are manually segmented into blocks and then the block features are aligned. Another example is using instance segmentation methods for the human body to assist in feature alignment using segmented body parts. However, manual segmentation or segmentation is based on planar segmentation and cannot accurately align flexible body parts. Second, alignment is achieved by extracting key point positional features through AI model architecture design. For example, using an attention mechanism, the network can focus on certain body parts to obtain alignment features. Another example is using key points to construct graph relationships and using graph matching algorithms to learn robust alignment capabilities. However, extracting key human point position features through model structure design for alignment requires high standards for model design and training, and the results are unstable. In summary, existing human feature alignment methods are costly to implement and lack accuracy and effectiveness. Summary of the Invention

[0004] The embodiments of the present application provide a method, device, electronic device, and computer program product for aligning human body parts, which can solve the technical problems in the prior art of high implementation cost of human feature alignment methods and insufficient accuracy and effectiveness of human body alignment.

[0005] In a first aspect, an embodiment of the present application provides a method for aligning human body parts, comprising:

[0006] Heat map of human body image and first human body key point Figure 1 And input it into the human body feature extraction network to obtain the human body feature vector output by the human body feature extraction network, the first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts and guide the human body feature extraction network to align human body parts, and the human body feature vector is used for human body similarity calculation.

[0007] In a possible implementation of the first aspect, the human feature extraction network includes a feature extraction backbone network and a human feature vector generator, wherein the feature extraction backbone network is used to extract image feature information, and the human feature vector generator is used to generate a human feature vector.

[0008] In a possible implementation of the first aspect, the human feature extraction network also includes a key point heat map generator, which is used to generate a second human key point heat map about the key point parts of the human body, and the second human key point heat map is used to back-propagate and optimize the parameters of the human feature extraction network.

[0009] In a possible implementation of the first aspect, the human feature extraction network includes a human feature loss function, a key point loss function and a key point loss hyperparameter, and the key point loss hyperparameter is used to balance the proportion of human feature loss and key point loss, and is the largest in the initial stage of human feature extraction and gradually decreases during the training process.

[0010] In a possible implementation of the first aspect, the human body key point parts in the first human body key point heat map include at least any one of the following: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left arm, right arm, left knee, right knee, left ankle, and right ankle.

[0011] In a possible implementation of the first aspect, the first human key point heat map includes one or more Gaussian circles centered on the human key point, where each Gaussian circle corresponds to a human key point part, different human key point parts have different encoding values ​​k, and the image feature values ​​of the pixels in the Gaussian circle are calculated based on the following formula:

[0012]

[0013] Among them, heatmap(D i,j ) represents the image feature value of the pixel point D in the Gaussian circle centered on the key point i, j of the human body, i, j represent the key point coordinates of the key point part of the human body, d is the distance between the pixel point D in the Gaussian circle and the key point i, j, and k is the encoding value of the key point part of the human body.

[0014] In a possible implementation of the first aspect, the method further includes:

[0015] According to the human body key point positions in the second human body key point heat map, the position angle of the human body in the human body picture is adjusted during the training process, and the adjustment processing includes at least any one of the following: random rotation, affine transformation, and simulated occlusion.

[0016] In a second aspect, an embodiment of the present application provides a device for aligning human body parts, comprising:

[0017] The first extraction module is used to thermally analyze the human body image and the first human body key point. Figure 1 And input it into the human body feature extraction network to obtain the human body feature vector output by the human body feature extraction network, the first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts and guide the human body feature extraction network to align human body parts, and the human body feature vector is used for human body similarity calculation.

[0018] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device implements a method as described in any one of the above-mentioned first aspects.

[0019] In a fourth aspect, an embodiment of the present application provides a computer program product, comprising a computer program, wherein when the computer program is run, the method described in any one of the above-mentioned first aspects is executed.

[0020] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described in any one of the above-mentioned first aspects is implemented.

[0021] Compared with the prior art, the embodiments of the first aspect of the present application have the following beneficial effects:

[0022] The embodiment of the present application introduces a first human key point heat map for guiding the human feature extraction network to perform human feature alignment. On the one hand, it can enhance the attention of the human feature extraction network to the key points of the human body, increase the loss guidance of the human feature extraction network in terms of human key points, and thus enhance the ability to extract effective information of the human body, and enhance the extraction of visible human features. On the other hand, the cost of modifying the network model of the present application is low, and it can quickly and effectively improve the accuracy of human feature alignment and improve the effectiveness of human recognition. The human feature vector extracted according to the embodiment of the present application is more robust and has a higher matching accuracy, which effectively reduces the misidentification and missed detection caused by occlusion and different human posture angles.

[0023] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Figure 1 1 is a flow chart of a method for aligning human body parts provided in one embodiment of the present application;

[0026] Figure 2 This is a flow chart of a method for aligning human body parts provided in an application embodiment of the present application;

[0027] Figure 3 1 is a schematic structural diagram of a device for aligning human body parts provided in one embodiment of the present application;

[0028] Figure 4 It is a structural diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0029] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0030] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0031] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0032] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0033] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0034] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0035] Figure 1 1 is a flow chart of a method for aligning human body parts provided in one embodiment of the present application.

[0036] S11, heatmap the human body image with the first human body key point Figure 1 And input it into the human body feature extraction network to obtain the human body feature vector output by the human body feature extraction network, the first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts to guide the human body feature extraction network to align human body parts, and the human body feature vector is used for human body similarity calculation.

[0037] Human body images refer to images that include human bodies. Human body images generally refer to images containing human body content that are captured by image acquisition devices such as cameras.

[0038] The first human key point heatmap refers to a heatmap that uses different image feature values ​​to label different human key point locations to distinguish different human body parts. Image feature values ​​include, but are not limited to, brightness, grayscale, and color values. Different image feature values ​​can distinguish different human key point locations. For example, using grayscale as an image feature value, for example, the grayscale value used near the head is 15, the grayscale value used near the left shoulder is 25, and the grayscale value used near the right shoulder is 45. Furthermore, different human key point locations can also be distinguished based on the range of image feature values. For example, using the grayscale value range as an image feature value, for example, the grayscale value used near the head is in the range of [0 to 20], the grayscale value used near the left shoulder is in the range of [21 to 40], and the grayscale value used near the right shoulder is in the range of [41 to 60]. Preferably, the size of the first human key point heatmap can be consistent with the size of the human image. First, the human body key point heat map can guide the human body feature extraction network to align human body parts, enhance the attention of the human body feature extraction network to the key points of the human body, increase the loss guidance of the human body feature extraction network in terms of human body key points, and thus enhance the ability to extract effective human body information, so as to guide the human body feature extraction network to output a human feature vector with higher robustness.

[0039] The human feature extraction network is used to extract image feature information (such as feature maps, FeatureMaps) from human body pictures, and extract human feature vectors based on the extracted image feature information for human body similarity calculation, human body recognition and comparison. The human feature extraction network can be constructed based on models including but not limited to convolutional neural networks, recurrent neural networks, long short-term memory networks, generative adversarial networks, etc. The feature map can, for example, include 255 pictures, and the size of each picture can be, for example, 18*18. Human feature vectors can be used for human body similarity calculation. Human feature vectors can be understood as feature matrices. For example, the similarity can be calculated by calculating the angular distance between two human feature vectors, thereby determining whether the corresponding human body pictures and human bodies are similar. Those skilled in the art should understand that various methods of calculating human body similarity based on human feature vectors fall within the scope of protection of this application.

[0040] The embodiment of the present application introduces a first human key point heat map for guiding the human feature extraction network to perform human feature alignment. On the one hand, it can enhance the attention of the human feature extraction network to the key points of the human body, increase the loss guidance of the human feature extraction network in terms of human key points, and thus enhance the ability to extract effective information of the human body, and enhance the extraction of visible human features. On the other hand, the cost of modifying the network model of the present application is low, and it can quickly and effectively improve the accuracy of human feature alignment and improve the effectiveness of human recognition. The human feature vector extracted according to the embodiment of the present application is more robust and has a higher matching accuracy, which effectively reduces the misidentification and missed detection caused by occlusion and different human posture angles.

[0041] In one embodiment, the human feature extraction network includes a feature extraction backbone network and a human feature vector generator. The feature extraction backbone network is used to extract image feature information, and the human feature vector generator is used to generate a human feature vector.

[0042] The feature extraction backbone network in the human feature extraction network can be used to extract image feature information (such as feature maps) from human images. The human feature vector generator in the human feature extraction network can extract human feature vectors based on the extracted image feature information for human body similarity calculation, human body recognition, and comparison.

[0043] In one embodiment, the human feature extraction network further includes a key point heat map generator, which is used to generate a second human key point heat map about key point parts of the human body, and the second human key point heat map is used for back propagation optimization of the parameters of the human feature extraction network.

[0044] The keypoint heatmap generator can output a second human keypoint heatmap similar to the first human keypoint heatmap. Different human keypoints in the second human keypoint heatmap are labeled with different image feature values ​​to distinguish different human body parts. The second human keypoint heatmap can be used for backpropagation to optimize the parameters of the human feature extraction network.

[0045] The feature extraction backbone network includes but is not limited to a network constructed by models such as convolutional neural networks, recurrent neural networks, long short-term memory networks, and generative adversarial networks for extracting human feature vectors. The feature extraction backbone network can perform feature extraction on human body images to obtain image feature information (for example, a feature map), and can output human feature vectors of specified dimensions through a fully connected layer. Based on the feature extraction backbone network, the human body feature extraction network can also include a key point heat map extractor (also known as a key point heat map generation head), which is used to generate a second human body key point heat map of key point parts of the human body.

[0046] Figure 2 This is a flow chart of a method for aligning human body parts provided in an application embodiment of the present application.

[0047] Figure 2 In the human body feature extraction network, the human body feature vector extractor and the key point heat map extractor are combined to form the human body feature extraction network. Figure 1 And input into the human feature extraction network, the feature extraction backbone network in the human feature extraction network (for example, a convolutional neural network) can generate a feature map. Then, the human feature vector extractor in the human feature extraction network outputs the human feature vector. The key point heat map extractor in the human feature extraction network outputs the second human key point heat map. Figure 2 As shown, the first human body key point heat map and the second human body key point heat map use different image feature values ​​to mark different human body key point parts to distinguish different human body parts. For example, in the first human body key point heat map, the grayscale range near the left foot of the human body at the bottom is larger than the grayscale range near the left wrist of the human body at the middle, and the grayscale range near the left wrist of the human body at the middle is larger than the grayscale range near the left shoulder of the human body at the top.

[0048] Figure 2 The body part alignment method shown belongs to the network structure of the training phase. When deploying the network structure of the inference phase, the key point heat map generator can be removed, and only the feature extraction backbone network and the body feature vector extractor are retained to extract the body feature vector to reduce the amount of computation.

[0049] In one embodiment, the human feature extraction network includes a human feature loss function, a key point loss function and a key point loss hyperparameter, wherein the key point loss hyperparameter is used to balance the proportion of human feature loss and key point loss, is the largest in the initial stage of human feature extraction and gradually decreases during training.

[0050] The human feature loss function is used to control the loss of human features. On the basis of the human feature loss function, a key point loss function can be added to control the loss of the second human key point heat map and the first human key point heat map. In addition, a key point loss hyperparameter can be added. The key point loss hyperparameter is equivalent to a parameter used to control the weight of the human feature loss function and the key point loss function. When the key point loss hyperparameter decreases, the weight of the human feature loss function relative to the key point loss function increases. The range of the key point loss hyperparameter can be, for example, between [1 and 0.1]. During the training process, the key point hyperparameter can be used to balance the weight of the human feature loss and the key point loss and dynamically adjust in a linear decreasing trend. The value is maximized at the initial stage of human feature extraction to guide the model to focus on the key structural parts of the human body. The value is gradually reduced during the training process to reduce the loss guidance of the key points and reduce the impact on the human feature extraction of the model.

[0051] In one embodiment, the human body key point parts in the first human body key point heat map include at least any one of the following: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left arm, right arm, left knee, right knee, left ankle, and right ankle.

[0052] Those skilled in the art should understand that the key points of the human body in the first key point heat map can be arbitrarily set according to needs, for example, including any one, any several, or all of the key points of the human body described above.

[0053] The above key points of the human body can be marked by setting different numbers. For example, the key points of the human body can be marked by setting the following numbers: {0: "head", 1: "left shoulder", 2: "right shoulder", 3: "left elbow", 4: "right elbow", 5: "left wrist", 6: "right wrist", 7: "left hip", 8: "right hip", 9: "left knee", 10: "right knee", 11: "left ankle", 12: "right ankle"}, where, for example, 0 represents the head, 1 represents the left shoulder, 2 represents the right shoulder, and so on.

[0054] If a key point of the human body is blocked or missing, the corresponding key point of the human body may not be marked.

[0055] In one embodiment, the first human key point heat map includes one or more Gaussian circles centered on the human key point, wherein each Gaussian circle corresponds to a human key point part, different human key point parts have different encoding values ​​k, and the image feature values ​​of the pixels in the Gaussian circle are calculated based on the following formula:

[0056]

[0057] Among them, heatmap(D i,j) represents the image feature value of the pixel point D in the Gaussian circle centered on the key point i, j of the human body, i, j represent the key point coordinates of the key point part of the human body, d is the distance between the pixel point D in the Gaussian circle and the key point i, j, and k is the encoding value of the key point part of the human body.

[0058] Where N and σ are constants, for example, N = 15, σ = 15. l can be a natural constant (e). d is usually less than 15.

[0059] First, different encoding values ​​k can be set for different key points of the human body. For example, {"head" k value = 19, "left shoulder" k value = 39, "right shoulder" k value = 58, "left elbow" k value = 78, "right elbow" k value = 98, "left wrist" k value = 117, "right wrist" k value = 137, "left hip" k value = 156, "right hip" k value = 176, "left knee" k value = 196, "right knee" k value = 215, "left ankle" k value = 235, "right ankle" k value = 255}. By setting different encoding values ​​for different key points of the human body, the human feature extraction network and the human recognition model can learn different parts of the human body.

[0060] The size of the first human key point heat map can be consistent with the size of the human body image. Each key point (i, j) of the human body is mapped to a point in the heat map according to its coordinate position. A Gaussian circle is generated with each key point as the center. The image feature value in the Gaussian circle is calculated based on the above formula.

[0061] Those skilled in the art should understand that, in addition to the above method of generating the first key point heat map, other methods may also be used to generate the first key point heat map.

[0062] In one embodiment, different human body key point parts are marked with different image feature value ranges in the first human body key point heat map to distinguish different human body parts.

[0063] For example, different image feature values ​​may be set for different key points of the human body in different ways such as equal-proportional reduction or non-equal-proportional reduction to generate a first human key point heat map. For example, {the image feature values ​​near the "head" range from 0 to 19, the image feature values ​​near the "left shoulder" range from 20 to 39, the image feature values ​​near the "right shoulder" range from 40 to 58, the image feature values ​​near the "left elbow" range from 59 to 78, the image feature values ​​near the "right elbow" range from 79 to 98, the image feature values ​​near the "left wrist" range from 99 to 117, the image feature values ​​near the "right wrist" range from 118 to 137, the image feature values ​​near the "left hip" range from 138 to 156, the image feature values ​​near the "right hip" range from 157 to 176, the image feature values ​​near the "left knee" range from 177 to 196, the image feature values ​​near the "right knee" range from 197 to 215, the image feature values ​​near the "left ankle" range from 216 to 235, and the image feature values ​​near the "right ankle" range from 236 to 255}.

[0064] In one embodiment, the method for aligning human body parts further includes step S12.

[0065] S12, adjusting the position angle of the human body in the human body picture during the training process according to the human body key point positions in the second human body key point heat map, wherein the adjustment processing includes at least any one of the following: random rotation, affine transformation, and simulated occlusion.

[0066] Based on the guidance information about the key points and body positions in the second key point heat map, the position and angle of the human body in the input human body image can be adjusted during training, such as through random rotation, affine transformation and other data augmentation methods, to improve the model's ability to extract features of the same body position. In addition, key point positions can be randomly cut out (CutOut) to simulate the situation where key points on the human body are blocked.

[0067] Figure 3 Schematic diagram of the structure of a device for aligning human body parts provided in one embodiment of the present application.

[0068] like Figure 3 As shown, the device 3 for aligning human body parts includes:

[0069] The first extraction module 31 is used to extract the human body picture and the first human body key point thermal Figure 1And input it into the human body feature extraction network to obtain the human body feature vector output by the human body feature extraction network, the first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts and guide the human body feature extraction network to align human body parts, and the human body feature vector is used for human body similarity calculation.

[0070] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 On the basis of the corresponding embodiment, the human body feature extraction network includes a feature extraction backbone network and a human body feature vector generator. The feature extraction backbone network is used to extract image feature information, and the human body feature vector generator is used to generate a human body feature vector.

[0071] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 On the basis of the corresponding embodiment, the human feature extraction network also includes a key point heat map generator, which is used to generate a second human key point heat map about the key point parts of the human body, and the second human key point heat map is used to back-propagate and optimize the parameters of the human feature extraction network.

[0072] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 Based on the corresponding embodiment, the human feature extraction network includes a human feature loss function, a key point loss function and a key point loss hyperparameter. The key point loss hyperparameter is used to balance the proportion of human feature loss and key point loss, and is the largest in the initial stage of human feature extraction and gradually decreases during the training process.

[0073] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 Based on the corresponding embodiment, the human body key point parts in the first human body key point heat map include at least any one of the following: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left arm, right arm, left knee, right knee, left ankle, and right ankle.

[0074] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 On the basis of the corresponding embodiment, the first human key point heat map includes one or more Gaussian circles centered on the human key point, wherein each Gaussian circle corresponds to a human key point part, different human key point parts have different encoding values ​​k, and the image feature values ​​of the pixels in the Gaussian circle are calculated based on the following formula:

[0075]

[0076] Among them, heatmap(D i,j ) represents the image feature value of the pixel point D in the Gaussian circle centered on the key point i, j of the human body, i, j represent the key point coordinates of the key point part of the human body, d is the distance between the pixel point D in the Gaussian circle and the key point i, j, and k is the encoding value of the key point part of the human body.

[0077] Another embodiment of the present invention discloses a device 3 for aligning parts of the human body. Figure 3 Based on the corresponding embodiment, the device further includes:

[0078] The second adjustment module is used to adjust the position angle of the human body in the human body picture during the training process according to the human body key point position in the second human body key point heat map, and the adjustment processing includes at least any one of the following: random rotation, affine transformation, and simulated occlusion.

[0079] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0080] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0081] The present application also provides an electronic device, such as Figure 4 As shown, the electronic device 4 includes: at least one processor 40, a memory 41, and a computer program 42 stored in the memory 41 and executable on the at least one processor 40. When the processor 40 executes the computer program 42, the electronic device 4 implements the steps of any of the above-mentioned method embodiments.

[0082] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0083] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed, the methods in the above-mentioned various method embodiments are executed.

[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, it can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0085] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0086] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0087] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0088] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0089] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for aligning human body parts, characterized in that: include: The human body picture and the first human body key point heat map are input into the human body feature extraction network together to obtain the human body feature vector output by the human body feature extraction network. The first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts and guide the human body feature extraction network to align human body parts. The human body feature vector is used for human body similarity calculation.

2. The method according to claim 1, wherein The human body feature extraction network includes a feature extraction backbone network and a human body feature vector generator. The feature extraction backbone network is used to extract image feature information, and the human body feature vector generator is used to generate a human body feature vector.

3. The method according to claim 2, wherein The human feature extraction network also includes a key point heat map generator, which is used to generate a second human key point heat map about the key point parts of the human body, and the second human key point heat map is used for back propagation optimization of the parameters of the human feature extraction network.

4. The method according to claim 1, wherein The human feature extraction network includes a human feature loss function, a key point loss function and a key point loss hyperparameter. The key point loss hyperparameter is used to balance the proportion of human feature loss and key point loss, is the largest in the initial stage of human feature extraction, and gradually decreases during training.

5. The method according to claim 1, wherein The human body key point parts in the first human body key point heat map include at least any one of the following: head, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left arm, right arm, left knee, right knee, left ankle, right ankle.

6. The method according to claim 1, wherein The first human key point heat map includes one or more Gaussian circles centered on the human key point, wherein each Gaussian circle corresponds to a human key point part, different human key point parts have different encoding values ​​k, and the image feature values ​​of the pixels in the Gaussian circle are calculated based on the following formula: Among them, heatmap(D i,j ) represents the image feature value of the pixel point D in the Gaussian circle centered on the key point i, j of the human body, i, j represent the key point coordinates of the key point part of the human body, d is the distance between the pixel point D in the Gaussian circle and the key point i, j, and k is the encoding value of the key point part of the human body.

7. The method according to claim 1, wherein The method further comprises: According to the human body key point positions in the second human body key point heat map, the position angle of the human body in the human body picture is adjusted during the training process, and the adjustment processing includes at least any one of the following: random rotation, affine transformation, and simulated occlusion.

8. A device for aligning body parts, characterized in that: include: The first extraction module is used to input the human body picture and the first human body key point heat map into the human body feature extraction network to obtain the human body feature vector output by the human body feature extraction network. The first human body key point heat map uses different image feature values ​​to mark different human body key point parts to distinguish different human body parts and guide the human body feature extraction network to align human body parts. The human body feature vector is used to calculate the similarity of the human body.

9. An electronic device, characterized in that: The electronic device comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when being executed, enables the method according to any one of claims 1 to 7 to be performed.

Images