Human body key point detection method and device, electronic equipment and storage medium
Through the weakly supervised learning module and multi-task loss function training convolutional neural network, combined with 2D or 3D skeleton signals, the efficiency and accuracy of multi-person key point detection in financial scenarios is solved, and efficient identification in complex environments is achieved and tracking have improved the quality of financial services.
Patent Information
- Application Number
- CN202411059767.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-11-08
AI Technical Summary
In financial scenarios, it is difficult for the existing technology to effectively improve the efficiency and accuracy of multi-person key point detection, especially in multi-person scenarios, it is difficult to distinguish different human bodies and deal with factors such as occlusion, lighting, and background interference, resulting in a degradation of detection performance.
The weakly supervised learning module is used to train the convolutional neural network and the key point detection network based on the multi-task loss function, and the 2D key point or 3D skeleton is used as the weakly supervised signal. Combined with reconstruction loss, structural loss and smooth loss function, the peak extraction algorithm is used to obtain thermal force from the peak force. The location of key points is extracted from the figure to realize the detection and tracking of key points of multiple people.
Improve the accuracy and stability of multi-person key point detection, reduce the cost and difficulty of data acquisition and labeling, and can efficiently identify the postures, movements, and expressions of multiple human bodies in complex environments, improving financial service efficiency and customer satisfaction Spend.
Smart Images

Figure CN118918611A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of financial technology (Fi ntech) technology, and in particular to a method and device for detecting key points of a human body, an electronic device, and a storage medium. Background Art
[0002] Human key point detection is an important research direction in the field of computer vision. It aims to locate the coordinates of human anatomical key points (such as body joints, eyes, nose, etc.) from images or videos for applications such as human posture estimation, action recognition, and behavior analysis. At present, human key point detection is mainly divided into two categories: deep learning-based methods and traditional feature-based methods. Deep learning-based methods have high accuracy and robustness, but they also require a large amount of labeled data to train the network, which is particularly difficult and time-consuming in multi-person scenarios. Traditional feature-based methods have lower computational complexity and data requirements, but are also easily affected by factors such as occlusion, lighting, and background interference, resulting in reduced detection performance.
[0003] In financial scenarios, multi-person key point detection can be applied to the intelligent monitoring system of financial institutions. By analyzing the postures and movements of multiple people in the monitoring screen, abnormal behaviors can be detected and alarmed, such as fighting, emergencies, etc. It can also be applied to intelligent customer service systems. By analyzing the postures, movements, and expressions of multiple people when customers are consulting or handling business, customer satisfaction can be evaluated and optimized, such as judging whether the customer is interested, has questions, needs help, etc. Based on this, in financial scenarios, how to improve the efficiency and accuracy of multi-person key point detection has become a technical problem that needs to be solved urgently. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a method and device for detecting key points of a human body, an electronic device and a storage medium, which can improve the efficiency and accuracy of detecting key points of multiple people, especially in financial scenarios, and can efficiently and accurately identify the postures, movements and expressions of multiple human bodies, which helps to improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for detecting key points of a human body, the method comprising:
[0006] Acquire an image to be detected containing multiple human bodies;
[0007] Performing human body detection on the image to be detected to obtain multiple human body images;
[0008] Input each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map;
[0009] Input the human key point feature map into a pre-trained key point detection network for key point detection, and predict a thermal map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points;
[0010] A peak extraction algorithm is used to extract the position of each key point of the human body from the thermal map.
[0011] In some embodiments, the method further comprises:
[0012] In the case where the image to be detected is a video frame image, a pre-trained key point tracking network is used to match and associate the position of each of the human key points in adjacent video frame images to obtain a position association result of each of the human key points, wherein the key point tracking network is trained by the weakly supervised learning module based on a multi-task loss function, and the multi-task loss function also includes a smoothing loss function, and the smoothing loss function is used to measure the continuity and stability between the human key points in adjacent video frame images;
[0013] A Kalman filter is used to smooth and predict the position association result of each key point of the human body, so as to obtain the position tracking result of each key point of the human body.
[0014] In some embodiments, performing human body detection on the image to be detected to obtain a plurality of human body images includes:
[0015] Performing human body detection on the image to be detected to obtain a bounding box of each human body;
[0016] The human body image within each of the boundary boxes is cropped and scaled to obtain multiple human body images of uniform size.
[0017] In some embodiments, after inputting each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map, the method further includes:
[0018] Determine a first area related to a human body key point in the human body key point feature map;
[0019] Determine a second area in the human body key point feature map that is unrelated to the human body key points;
[0020] The first region is enhanced and the second region is suppressed through an attention mechanism.
[0021] In some embodiments, the reconstruction loss function is the average of the Euclidean distances between the positions of the human key points and the weak supervisory signals.
[0022] In some embodiments, the structural loss function is the negative logarithm of the structural similarity between the human body key points and a preset reference model.
[0023] In some embodiments, the smoothness loss function is the sum of squares of position changes between the key points of the human body in adjacent frames.
[0024] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a human key point detection device, the device comprising:
[0025] An acquisition module, used for acquiring an image to be detected containing multiple human bodies;
[0026] A human body detection module is used to perform human body detection on the image to be detected to obtain multiple human body images; a feature extraction module is used to input each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map;
[0027] A key point detection module, used for inputting the human key point feature map into a pre-trained key point detection network for key point detection, and predicting a thermal map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, the multi-task loss function includes a reconstruction loss and a structural loss, the reconstruction loss function is used to measure the consistency between the positions of the human key points and the weak supervision signals, and the structural loss function is used to measure the geometric constraints between the positions of the human key points;
[0028] The key point extraction module is used to extract the position of each key point of the human body from the thermal map using a peak extraction algorithm.
[0029] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0030] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0031] The human key point detection method and device, electronic device and storage medium proposed in the present application obtain an image to be detected containing multiple human bodies; perform human body detection on the image to be detected to obtain multiple human body images; input each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human key point feature map; input the human key point feature map into a pre-trained key point detection network for key point detection, and predict the heat map of each human key point, wherein the convolutional neural network and the key point detection network are obtained by training based on a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points; the peak extraction algorithm is used to extract the position of each human key point from the heat map. The embodiment of the present application can realize the detection of multiple human key points from a monocular image or video, and does not need to use complete human posture annotation data, but only needs to use part of the 2D key points or 3D skeletons as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of multi-task loss functions can effectively utilize the information in weak supervision signals, improving the model generalization ability and robustness of convolutional neural networks and key point detection networks. In multi-person scenarios, the embodiments of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handle factors such as occlusion, lighting, and background interference in complex environments, thereby improving the accuracy and stability of multi-person key point detection. Based on this, the embodiments of the present application can improve the efficiency and accuracy of multi-person key point detection, especially in financial scenarios, and can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, which helps to improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Figure 1 is a flow chart of a human body key point detection method provided in an embodiment of the present application;
[0033] Figure 2 is a flowchart from step S201 to step S202;
[0034] Figure 3 yes Figure 1 Flow chart of step S102 in FIG.
[0035] Figure 4 is a flowchart of steps S401 to S403;
[0036] Figure 5is a structural schematic diagram of a human body key point detection device provided in an embodiment of the present application;
[0037] Figure 6 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0038] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0039] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0041] First, some nouns involved in this application are analyzed:
[0042] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0043] Multi-head Attention Mechanism: It is a technique used in deep learning to enhance the learning ability of models. It is usually applied to neural network models, especially in models in the field of natural language processing, such as the Transformer model. In the multi-head attention mechanism, the input signal is processed by multiple parallel attention heads, and each attention head learns different representations. Each attention head calculates an attention weight matrix to guide the model to pay attention to information at different positions in the input sequence. In this way, the model can pay attention to different parts of the input sequence at the same time, so as to better capture the relationship and importance between the inputs. Through the multi-head attention mechanism, the model can learn richer and more comprehensive representations, which helps to improve the performance of the model on complex tasks. In the Transformer model, the multi-head attention mechanism is widely used in the encoder and decoder to help the model understand the relationship between the input sequence and the generated output sequence. In general, the multi-head attention mechanism enables the model to learn multiple different representations in parallel by introducing multiple parallel attention heads, thereby improving the representation and generalization capabilities of the model. This mechanism plays an important role in improving model performance and processing long-distance dependencies in sequence data.
[0044] Human key point detection is an important research direction in the field of computer vision. It aims to locate the coordinates of human anatomical key points (such as body joints, eyes, nose, etc.) from images or videos for applications such as human posture estimation, action recognition, and behavior analysis. At present, human key point detection is mainly divided into two categories: deep learning-based methods and traditional feature-based methods. Deep learning-based methods have high accuracy and robustness, but they also require a large amount of labeled data to train the network, which is particularly difficult and time-consuming in multi-person scenarios. Traditional feature-based methods have lower computational complexity and data requirements, but are also easily affected by factors such as occlusion, lighting, and background interference, resulting in reduced detection performance.
[0045] Weakly supervised learning methods can use incomplete, inaccurate or imprecise labeled data to train models, and use a large amount of unlabeled or weakly labeled data to improve the generalization ability and robustness of the model. However, existing methods based on weakly supervised learning still have the following problems:
[0046] 1. In multi-person scenarios, how to effectively distinguish different human bodies and perform key point detection and tracking to avoid incorrect matching and association.
[0047] Second, in complex environments, how to effectively deal with factors such as occlusion, lighting, and background interference to improve the accuracy and stability of key point detection.
[0048] Third, how to design appropriate weak supervision signals and loss functions so that the model can learn effective features and constraints from weakly labeled data instead of overfitting or underfitting.
[0049] In financial scenarios, multi-person key point detection can be applied to the intelligent monitoring system of financial institutions. By analyzing the postures and movements of multiple people in the monitoring screen, abnormal behaviors can be detected and alarmed, such as fighting, emergencies, etc. It can also be applied to intelligent customer service systems. By analyzing the postures, movements, and expressions of multiple people when customers are consulting or handling business, customer satisfaction can be evaluated and optimized, such as judging whether the customer is interested, has questions, needs help, etc. Based on this, in financial scenarios, how to improve the efficiency and accuracy of multi-person key point detection has become a technical problem that needs to be solved urgently.
[0050] Based on this, the embodiment of the present application provides a method and device for detecting key points of a human body, an electronic device and a storage medium, obtaining an image to be detected containing multiple human bodies; performing human body detection on the image to be detected to obtain multiple human body images; inputting each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; inputting the human body key point feature map into a pre-trained key point detection network for key point detection, and predicting a heat map of each human body key point, wherein the convolutional neural network and the key point detection network are obtained by training a weakly supervised learning module based on a multi-task loss function, the weakly supervised learning module uses 2D key points or 3D skeletons as weak supervision signals, the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human body key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; the position of each human body key point is extracted from the heat map using a peak extraction algorithm. The embodiment of the present application can realize the detection of multiple human body key points from a monocular image or video, and does not need to use complete human body posture annotation data, but only needs to use part of the 2D key points or 3D skeletons as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of multi-task loss functions can effectively utilize the information in weak supervision signals, improving the model generalization ability and robustness of convolutional neural networks and key point detection networks. In multi-person scenarios, the embodiments of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handle factors such as occlusion, lighting, and background interference in complex environments, thereby improving the accuracy and stability of multi-person key point detection. Based on this, the embodiments of the present application can improve the efficiency and accuracy of multi-person key point detection, especially in financial scenarios, and can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, which helps to improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0051] The human body key point detection method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the human body key point detection method in the embodiments of the present application is described.
[0052] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0053] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0054] The human body key point detection method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The human body key point detection method provided in the embodiment of the present application can be applied to the terminal, can also be applied to the server side, and can also be software running in the terminal or the server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or it can be configured as a server cluster or distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the human body key point detection method, etc., but is not limited to the above forms.
[0055] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0056] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.
[0057] Figure 1 is an optional flow chart of the human body key point detection method provided in the embodiment of the present application, Figure 1 The method may include but is not limited to steps S101 to S105.
[0058] Step S101, obtaining an image to be detected containing multiple human bodies;
[0059] Step S102, performing human body detection on the image to be detected to obtain a plurality of human body images;
[0060] Step S103, inputting each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map;
[0061] Step S104, inputting the key point feature map of the human body into a pre-trained key point detection network for key point detection, and predicting a heat map of each key point of the human body, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function;
[0062] Step S105, using a peak extraction algorithm to extract the position of each key point of the human body from the heat map.
[0063] In step S101 of some embodiments, an image to be detected containing multiple human bodies is obtained. The image to be detected can be a monocular image or a video frame image. A monocular image is relative to a binocular image and a multi-eye image. It refers to a single image. A monocular image does not rely on multiple images to obtain depth information, but directly uses a single visual sensor to capture the scene. A video frame image is a single static image in a video and constitutes the basic unit of a video. In a financial scenario, customers can be photographed in real time through the intelligent monitoring system or intelligent customer service system of a financial institution to obtain an image to be detected containing multiple customers.
[0064] In step S102 of some embodiments, human body detection is performed on the image to be detected to obtain multiple human body images. Specifically, human body detection can be performed on the input monocular image to obtain the bounding box of each human body, and the monocular human body image in each bounding box is cropped and scaled to obtain a monocular human body image of uniform size. Human body detection can also be performed on the input video frame image to obtain the bounding box of each human body, and the video frame human body image in each bounding box is cropped and scaled to obtain a video frame human body image of uniform size.
[0065] In step S103 of some embodiments, each human image is input into a pre-trained convolutional neural network for feature extraction to obtain a human key point feature map. A pre-trained convolutional neural network CNN model suitable for human key point detection tasks, CNN models include but are not limited to ResNet, MobileNet, etc. The pre-trained convolutional neural network model is trained on a large-scale data set and can extract the key point features of the image. Each human image is input into the CNN model, and the output of a certain middle layer is obtained as a human key point feature map. The selected middle layer should be able to retain enough spatial information and semantic information to effectively represent the human key points, thereby obtaining a human key point feature map.
[0066] In step S104 of some embodiments, the human key point feature map is input into the pre-trained key point detection network for key point detection, and the heat map of each human key point is predicted, wherein the convolutional neural network and the key point detection network are trained by the weak supervision learning module based on the multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key points and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points. Among them, 2D key points usually refer to the coordinate positions of human key points on the image plane, such as shoulders, knees, etc. A 3D skeleton can be a structure that describes the human skeleton in 3D space, usually composed of joint connection lines, and each joint connection represents a part of the skeleton. The reconstruction loss function is used to ensure that the generated key point heat map can accurately reconstruct the 2D key points or 3D skeletons in the training data. Common reconstruction loss functions can be mean square error (MSE) or some more complex loss functions, such as adaptive weighted loss, etc., which are used to measure the difference between the generated heat map and the real key points. The structural loss function is used to capture the structural information between key points and ensure that there is a reasonable spatial relationship or joint connection constraint between the predicted key points, which can help improve the accuracy and consistency of key point detection. Therefore, the reconstruction loss function ensures that the generated heat map can accurately reconstruct the real key point position, and the structural loss function ensures the spatial relationship and skeleton constraint of the key points. Based on a dataset containing human images and corresponding 2D key points or 3D skeleton information. This dataset will be used as the input for training the weakly supervised learning module, and the key point heat map can be generated using the input human image and weak supervision signal (2D key points or 3D skeleton). The weakly supervised learning module can be a convolutional neural network-based structure, and may also include appropriate upsampling or downsampling layers to generate heat maps of appropriate resolution. Through the weakly supervised learning module and the multi-task loss function, 2D key points or 3D skeletons can be effectively used as supervision signals to train the key point detection network. This method combines the powerful characteristics of deep learning with the prior knowledge of human structure, and can achieve better key point detection results without a large amount of labeled key point data. Since there is no need to use complete human posture annotation data, only some 2D key points or 3D skeletons need to be used as weak supervision signals, thus reducing the cost and difficulty of data acquisition and annotation.
[0067] In step S105 of some embodiments, a peak extraction algorithm is used to extract the position of each key point of the human body from the thermal map. The peak extraction algorithm is used to extract local maxima or peak points from the thermal map, which usually correspond to the positions of key points. For each pixel point, a neighborhood is defined, such as a 3x3 or 5x5 neighborhood centered on the pixel point. For each pixel point, the pixel value in its neighborhood is checked. If the pixel value is greater than or equal to all other pixel values in the neighborhood, the pixel point is marked as a potential peak point. A threshold can be set to retain only those potential peak points whose pixel values are greater than the threshold. For adjacent peak points, non-maximum suppression can be performed to retain only points with maximum pixel values. The peak point finally obtained is the position of the key point of the human body.
[0068] By adopting the human key point detection method of the embodiment of the present application, the detection and tracking of multiple human key points can be achieved from a monocular image or video without using complete human posture annotation data. Only partial 2D key points or 3D skeletons need to be used as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation.
[0069] The human key point detection method of the embodiment of the present application is based on a multi-task loss function, including reconstruction loss and structural loss. It comprehensively considers the consistency between human key points and weak supervision signals, the geometric constraints between human key points, and the mutual exclusivity between human key points of different human bodies. It effectively utilizes the information in weakly labeled data and improves the generalization ability and robustness of the model.
[0070] By using the human key point detection method of the embodiment of the present application, different human bodies can be effectively distinguished and key point detection can be performed in multi-person scenarios, avoiding erroneous matching and association. In complex environments, factors such as occlusion, lighting, and background interference can be effectively handled, thereby improving the accuracy and stability of key point detection.
[0071] The human body key point detection method of the embodiment of the present application can improve the efficiency and accuracy of key point detection for multiple people. Especially in financial scenarios, it can efficiently and accurately identify the postures, movements, and expressions of multiple people, which helps to improve the efficiency and quality of financial services, increase customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0072] Steps S101 to S105 shown in the embodiment of the present application are to obtain an image to be detected containing multiple human bodies; perform human body detection on the image to be detected to obtain multiple human body images; input each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; input the human body key point feature map into a pre-trained key point detection network for key point detection, and predict the heat map of each human body key point, wherein the convolutional neural network and the key point detection network are obtained by training a weakly supervised learning module based on a multi-task loss function, the weakly supervised learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human body key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; and the peak extraction algorithm is used to extract the position of each human body key point from the heat map. The embodiment of the present application can realize the detection of multiple human body key points from a monocular image or video, and does not need to use complete human body posture annotation data, but only needs to use part of the 2D key points or 3D skeletons as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of multi-task loss functions can effectively utilize the information in weak supervision signals, improving the model generalization ability and robustness of convolutional neural networks and key point detection networks. In multi-person scenarios, the embodiments of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handle factors such as occlusion, lighting, and background interference in complex environments, thereby improving the accuracy and stability of multi-person key point detection. Based on this, the embodiments of the present application can improve the efficiency and accuracy of multi-person key point detection, especially in financial scenarios, and can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, which helps to improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0073] See also Figure 2 In some embodiments, the human body key point detection method of the embodiment of the present application further includes but is not limited to step S201 to step S202:
[0074] Step S201, when the image to be detected is a video frame image, a pre-trained key point tracking network is used to match and associate the position of each human key point in adjacent video frame images to obtain a position association result of each human key point, wherein the key point tracking network is trained by a weakly supervised learning module based on a multi-task loss function, and the multi-task loss function also includes a smoothing loss function, which is used to measure the continuity and stability between human key points in adjacent video frame images;
[0075] Step S202: Use a Kalman filter to smooth and predict the position association result of each key point of the human body, and obtain the position tracking result of each key point of the human body.
[0076] In the case where the image to be detected is a video frame image, the embodiment of the present application can also realize human key point tracking. The human key points in the video frame are detected using a pre-trained key point tracking network. Each video frame is processed to obtain the key point position of each human body. Between adjacent frames, a matching algorithm (such as nearest neighbor matching) can be used to associate the human key points in the previous and next frames. Matching can be performed based on the distance or other features between the key points to ensure correct association. For the position of each key point, a Kalman filter is used for smoothing and prediction to improve the stability and accuracy of tracking. The Kalman filter can estimate and predict the position of the key point based on the current observation value. For the key points of multiple human bodies, an independent Kalman filter can be assigned to each human body to achieve multi-human tracking. The accuracy of tracking can also be further improved based on the historical trajectory and motion pattern of each human body. Among them, the smooth loss function is designed in the multi-task loss function to take into account the continuity and stability between the human key points in adjacent frames to reduce noise and discontinuity, thereby improving the accuracy and stability of the position tracking results of each human key point. In financial scenarios, it can effectively distinguish different customers and perform key point detection and tracking to avoid erroneous matching and association, effectively handle factors such as occlusion, lighting, and background interference, and improve the accuracy and stability of key point detection and tracking. It can efficiently and accurately identify multiple key points of the human body for tracking, which helps to improve the efficiency and quality of financial services, increase customer satisfaction, and enhance the competitiveness of financial enterprises.
[0077] See also Figure 3 In some embodiments, step S102 may include but is not limited to steps S301 to S302:
[0078] Step S301, performing human body detection on the image to be detected to obtain a bounding box of each human body;
[0079] Step S302 , cropping and scaling the human body image within each boundary box to obtain multiple human body images of uniform size.
[0080] In some embodiments, a pre-trained human detection model (such as Faster R-CNN, YOLO, SSD, etc.) can be used to perform human detection on the image to be detected to obtain the bounding box of each human body. The human body image within the bounding box is cropped to obtain a local image of the human body area. The cropped human body image is scaled and adjusted to a uniform size. Human body images of uniform size can be more easily used for subsequent tasks, such as human key point detection.
[0081] See also Figure 4 In some embodiments, after step S103, the following steps may also include but are not limited to steps S401 to S403:
[0082] Step S401, determining a first area related to a human body key point in a human body key point feature map;
[0083] Step S402, determining a second area in the human body key point feature map that is unrelated to the human body key points;
[0084] Step S403: enhance the first region and suppress the second region through an attention mechanism.
[0085] In some embodiments, using an attention mechanism to enhance the areas related to human key points in the feature map and suppress irrelevant areas is an effective method to improve the accuracy of key point detection. Specifically, an attention mechanism module is designed to dynamically adjust the importance of different areas in the feature map to enhance the areas related to human key points. The feature map is input into the attention mechanism module, and the attention weight of each position is calculated according to the content of the feature map. Through the learned weights, the first area related to the human key points is enhanced, and the irrelevant second area is suppressed. The attention-weighted feature map is multiplied or added to the original feature map to obtain an enhanced feature map. The enhanced feature map will pay more attention to the areas related to human key points, which helps to improve the accuracy of key point detection. Based on this, by introducing an attention mechanism to enhance the areas related to human key points in the feature map, the accuracy of key point detection can be effectively improved, while suppressing the influence of irrelevant areas, improving the quality and efficiency of feature expression, and thus improving the key point detection performance.
[0086] In some embodiments, a weakly supervised learning module is used to jointly train and optimize the convolutional neural network, key point detection network and key point tracking network. The weakly supervised learning module uses part of the 2D key points or 3D skeleton as weak supervision signals and designs a multi-task loss function, including the following parts:
[0087] 1. Reconstruction loss function: It is used to measure the consistency between the predicted human key points and the weak supervision signal. It can be defined as the average value of the Euclidean distance between the predicted human key points and the weak supervision signal, that is:
[0088]
[0089] 2. Structural loss function: It is used to measure whether the predicted human key points satisfy some prior geometric constraints, such as length, angle, symmetry, etc. It can be defined as the negative logarithm of the structural similarity between the predicted human key points and a reference model, that is:
[0090] L str = -logS(p,r)
[0091] Among them, p = (p1, p2, ..., pN) is the predicted position of all human key points, r = (r1, r2, ..., rN) is the position of all corresponding human key points in a reference model, and S (p, r) is a structural similarity function that can be designed according to different geometric constraints. For example, the following formula can be used to calculate structural similarity:
[0092]
[0093] 3. Smooth loss function: used to measure whether the predicted human key points in adjacent frames have continuity and stability. It can be defined as the sum of the squares of the position changes between the predicted human key points in adjacent frames, while alleviating the effects of occlusion, etc.
[0094]
[0095] Where T is the number of video frames, is the position of the predicted i-th human key point in the t-th frame.
[0096] Based on this, the multi-task loss function designed in the embodiment of the present invention includes a reconstruction loss function, a structural loss function and a smoothing loss function, which comprehensively considers the consistency between human key points and weak supervision signals, the geometric constraints between human key points, the continuity and stability between human key points in adjacent frames, and the mutual exclusivity between human key points of different human bodies. It effectively utilizes the information in the weakly labeled data and improves the generalization ability and robustness of the model.
[0097] The following is a further description of the human body key point detection method of the present application in conjunction with a specific embodiment, and the specific steps are as follows:
[0098] 1. Perform human body detection on the input image or video frame to obtain the bounding box of each human body, and crop and scale the image within each bounding box to obtain a human body image of uniform size.
[0099] Second, a pre-trained convolutional neural network (CNN) is used to extract feature maps for each human image, and an attention mechanism is used to enhance the areas related to the key points of the human body in the feature map and suppress irrelevant areas.
[0100] 3. Use a pre-trained key point detection network (KeyPoint Detection Network) to predict the heat map of each human key point for each human image, and use a peak extraction algorithm (Peak Extraction Algorithm) to extract the position of each human key point from the heat map.
[0101] 4. For video input, a pre-trained key point tracking network is used to match and associate the human key points in adjacent frames, and a Kalman filter is used to smooth and predict the position of each human key point to achieve locked tracking of multiple human key points.
[0102] 5. Use a weakly supervised learning module to jointly train and optimize the above convolutional neural network, key point detection network and key point tracking network. The weakly supervised learning module uses some 2D key points or 3D skeletons as weak supervision signals and designs a multi-task loss function, which includes the following parts:
[0103] 1. Reconstruction loss function: It is used to measure the consistency between the predicted human key points and the weak supervision signal. It can be defined as the average value of the Euclidean distance between the predicted human key points and the weak supervision signal, that is:
[0104]
[0105] 2. Structural loss function: It is used to measure whether the predicted human key points satisfy some prior geometric constraints, such as length, angle, symmetry, etc. It can be defined as the negative logarithm of the structural similarity between the predicted human key points and a reference model, that is:
[0106] L str = -logS(p,r)
[0107] Among them, p = (p1, p2, ..., pN) is the predicted position of all human key points, r = (r1, r2, ..., rN) is the position of all corresponding human key points in a reference model, and S (p, r) is a structural similarity function that can be designed according to different geometric constraints. For example, the following formula can be used to calculate structural similarity:
[0108]
[0109] 3. Smooth loss function: used to measure whether the predicted human key points in adjacent frames have continuity and stability. It can be defined as the sum of the squares of the position changes between the predicted human key points in adjacent frames, while alleviating the effects of occlusion, etc.
[0110]
[0111] Where T is the number of video frames, is the position of the predicted i-th human key point in the t-th frame.
[0112] Based on this, the embodiments of the present application can realize the detection and tracking of multiple human key points from a monocular image or video without using complete human posture annotation data. Only partial 2D key points or 3D skeletons need to be used as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation.
[0113] The embodiment of the present application uses an attention mechanism to enhance the areas related to the key points of the human body in the feature map and suppress irrelevant areas, thereby improving the quality and efficiency of feature expression.
[0114] The embodiment of the present application designs a multi-task loss function, including reconstruction loss, structural loss, and smoothness loss. It comprehensively considers the consistency between human key points and weak supervision signals, the geometric constraints between human key points, the continuity and stability between human key points in adjacent frames, and the mutual exclusivity between human key points of different human bodies. It effectively utilizes the information in weakly labeled data and improves the generalization ability and robustness of the model.
[0115] The embodiments of the present application can effectively distinguish different human bodies and perform key point detection and tracking in multi-person scenarios, avoiding erroneous matching and association. In complex environments, it can effectively handle factors such as occlusion, lighting, and background interference, thereby improving the accuracy and stability of key point detection and tracking.
[0116] See also Figure 5 The embodiment of the present application also provides a human key point detection device, which can implement the above human key point detection method, and the device includes:
[0117] An acquisition module 510 is used to acquire an image to be detected containing multiple human bodies;
[0118] A human body detection module 520, used to perform human body detection on the image to be detected to obtain multiple human body images;
[0119] A feature extraction module 530 is used to input each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map;
[0120] A key point detection module 540 is used to input a human key point feature map into a pre-trained key point detection network for key point detection, and predict a heat map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss and a structural loss, the reconstruction loss function is used to measure the consistency between the position of the human key points and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points;
[0121] The key point extraction module 550 is used to extract the position of each key point of the human body from the heat map using a peak extraction algorithm.
[0122] In some embodiments of the present application, the acquisition module 510 acquires an image to be detected containing multiple human bodies; the human body detection module 520 performs human body detection on the image to be detected to obtain multiple human body images; the feature extraction module 530 inputs each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; the key point detection module 540 inputs the human body key point feature map into a pre-trained key point detection network for key point detection, and predicts a heat map of each human body key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes reconstruction loss and structural loss, the reconstruction loss function is used to measure the consistency between the position of the human body key points and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; the key point extraction module 550 uses a peak extraction algorithm to extract the position of each human body key point from the heat map.
[0123] In some embodiments of the present application, an image to be detected containing multiple human bodies is obtained. The image to be detected can be a monocular image or a video frame image. A monocular image is relative to a binocular image and a multi-eye image. It refers to a single image. A monocular image does not rely on multiple images to obtain depth information, but directly uses a single visual sensor to capture the scene. A video frame image is a single static image in a video and constitutes the basic unit of a video. In a financial scenario, customers can be photographed in real time through the intelligent monitoring system or intelligent customer service system of a financial institution to obtain an image to be detected containing multiple customers.
[0124] In some embodiments of the present application, human body detection is performed on the image to be detected to obtain multiple human body images. Specifically, human body detection can be performed on the input monocular image to obtain the bounding box of each human body, and the monocular human body image in each bounding box can be cropped and scaled to obtain a monocular human body image of uniform size. Human body detection can also be performed on the input video frame image to obtain the bounding box of each human body, and the video frame human body image in each bounding box can be cropped and scaled to obtain a video frame human body image of uniform size.
[0125] In some embodiments of the present application, each human image is input into a pre-trained convolutional neural network for feature extraction to obtain a human key point feature map. A pre-trained convolutional neural network CNN model suitable for human key point detection tasks, CNN models include but are not limited to ResNet, MobileNet, etc. The pre-trained convolutional neural network model is trained on a large-scale data set and can extract the key point features of the image. Each human image is input into the CNN model, and the output of a certain middle layer is obtained as a human key point feature map. The selected middle layer should be able to retain enough spatial information and semantic information to effectively represent the human key points, thereby obtaining a human key point feature map.
[0126] In some embodiments of the present application, the human key point feature map is input into the pre-trained key point detection network for key point detection, and the heat map of each human key point is predicted, wherein the convolutional neural network and the key point detection network are trained by the weak supervision learning module based on the multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key points and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points. Among them, 2D key points usually refer to the coordinate positions of the human key points on the image plane, such as shoulders, knees, etc. A 3D skeleton can be a structure that describes the human skeleton in 3D space, usually composed of joint connection lines, and each joint connection represents a part of the skeleton. The reconstruction loss function is used to ensure that the generated key point heat map can accurately reconstruct the 2D key points or 3D skeletons in the training data. Common reconstruction loss functions can be mean square error (MSE) or some more complex loss functions, such as adaptive weighted loss, etc., which are used to measure the difference between the generated heat map and the real key points. The structural loss function is used to capture the structural information between key points and ensure that there is a reasonable spatial relationship or joint connection constraint between the predicted key points, which can help improve the accuracy and consistency of key point detection. Therefore, the reconstruction loss function ensures that the generated heat map can accurately reconstruct the real key point position, and the structural loss function ensures the spatial relationship and skeleton constraint of the key points. Based on a dataset containing human images and corresponding 2D key points or 3D skeleton information. This dataset will be used as the input for training the weakly supervised learning module, and the key point heat map can be generated using the input human image and weak supervision signal (2D key points or 3D skeleton). The weakly supervised learning module can be a convolutional neural network-based structure, and may also include appropriate upsampling or downsampling layers to generate heat maps of appropriate resolution. Through the weakly supervised learning module and the multi-task loss function, 2D key points or 3D skeletons can be effectively used as supervision signals to train the key point detection network. This method combines the powerful characteristics of deep learning with the prior knowledge of human structure, and can achieve better key point detection results without a large amount of labeled key point data. Since there is no need to use complete human posture annotation data, only some 2D key points or 3D skeletons need to be used as weak supervision signals, thus reducing the cost and difficulty of data acquisition and annotation.
[0127] In some embodiments of the present application, a peak extraction algorithm is used to extract the position of each key point of the human body from the thermal map. The peak extraction algorithm is used to extract local maxima or peak points from the thermal map, which usually correspond to the positions of key points. For each pixel point, a neighborhood is defined, such as a 3x3 or 5x5 neighborhood centered on the pixel point. For each pixel point, the pixel value in its neighborhood is checked. If the pixel value is greater than or equal to all other pixel values in the neighborhood, the pixel point is marked as a potential peak point. A threshold can be set to retain only those potential peak points whose pixel values are greater than the threshold. For adjacent peak points, non-maximum suppression can be performed to retain only the points with the maximum pixel value. The peak point finally obtained is the position of the key point of the human body.
[0128] Based on this, in the human key point detection device of the embodiment of the present application, the acquisition module 510 acquires an image to be detected containing multiple human bodies; the human body detection module 520 performs human body detection on the image to be detected to obtain multiple human body images; the feature extraction module 530 inputs each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; the key point detection module 540 inputs the human body key point feature map into a pre-trained key point detection network for key point detection, and predicts a thermal map of each human body key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes reconstruction loss and structural loss, the reconstruction loss function is used to measure the consistency between the position of the human body key points and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; the key point extraction module 550 uses a peak extraction algorithm to extract the position of each human body key point from the thermal map. The present application obtains an image to be detected containing multiple human bodies; performs human body detection on the image to be detected to obtain multiple human body images; inputs each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; inputs the human body key point feature map into a pre-trained key point detection network for key point detection, and predicts a heat map of each human body key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human body key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; and the peak extraction algorithm is used to extract the position of each human body key point from the heat map. The embodiment of the present application can realize the detection of multiple human body key points from a monocular image or video, and does not need to use complete human body posture annotation data, but only needs to use part of the 2D key points or 3D skeletons as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of multi-task loss functions can effectively utilize the information in weak supervision signals, improving the model generalization ability and robustness of convolutional neural networks and key point detection networks. In multi-person scenarios, the embodiments of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handle factors such as occlusion, lighting, and background interference in complex environments, thereby improving the accuracy and stability of multi-person key point detection.Based on this, the embodiments of the present application can improve the efficiency and accuracy of detecting key points of multiple people. Especially in financial scenarios, it can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, which helps to improve the efficiency and quality of financial services, increase customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0129] The specific implementation of the human body key point detection device is basically the same as the specific embodiment of the above-mentioned human body key point detection method, and will not be repeated here.
[0130] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned human body key point detection method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0131] See also Figure 6 , Figure 6 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0132] The processor 601 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.
[0133] The memory 602 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 602, and the processor 601 is called to execute the human key point detection method of the embodiment of the present application, that is, by obtaining an image to be detected containing multiple human bodies; performing human body detection on the image to be detected to obtain multiple human body images; inputting each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; inputting the human body key point feature map into a pre-trained key point detection network for key point detection, and predicting a thermal map of each human body key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human body key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human body key points; the peak extraction algorithm is used to extract the position of each human body key point from the thermal map. The embodiment of the present application can realize the detection of multiple human key points from a monocular image or video, and there is no need to use complete human posture annotation data, only part of the 2D key points or 3D skeletons need to be used as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of a multi-task loss function can effectively utilize the information in the weak supervision signal, improving the model generalization ability and robustness of the convolutional neural network and the key point detection network. In a multi-person scenario, the embodiment of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handling factors such as occlusion, illumination, and background interference in a complex environment, thereby improving the accuracy and stability of multi-person key point detection. Based on this, the embodiment of the present application can improve the efficiency and accuracy of multi-person key point detection, especially in a financial scenario, can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, help improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0134] The input / output interface 603 is used to implement information input and output.
[0135] The communication interface 604 is used to realize communication interaction between the device and other devices. Communication can be realized through wired methods (such as USB, network cable, etc.) or wireless methods (such as mobile network, WI FI, Bluetooth, etc.).
[0136] The bus transmits information between various components of the device (eg, processor 601 , memory 602 , input / output interface 603 , and communication interface 604 ).
[0137] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus.
[0138] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned human body key point detection method is implemented.
[0139] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0140] The human key point detection method, human key point detection device, electronic device and storage medium provided in the embodiments of the present application are as follows: acquiring an image to be detected containing multiple human bodies; performing human body detection on the image to be detected to obtain multiple human body images; inputting each human body image into a pre-trained convolutional neural network for feature extraction to obtain a human key point feature map; inputting the human key point feature map into a pre-trained key point detection network for key point detection, and predicting a heat map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, and the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points; and a peak extraction algorithm is used to extract the position of each human key point from the heat map. The embodiment of the present application can realize the detection of multiple human key points from a monocular image or video, and there is no need to use complete human posture annotation data, only part of the 2D key points or 3D skeletons need to be used as weak supervision signals, thereby reducing the cost and difficulty of data acquisition and annotation. At the same time, the use of a multi-task loss function can effectively utilize the information in the weak supervision signal, improving the model generalization ability and robustness of the convolutional neural network and the key point detection network. In a multi-person scenario, the embodiment of the present application can effectively distinguish different human bodies and perform key point detection, avoiding incorrect matching and association, and effectively handling factors such as occlusion, illumination, and background interference in a complex environment, thereby improving the accuracy and stability of multi-person key point detection. Based on this, the embodiment of the present application can improve the efficiency and accuracy of multi-person key point detection, especially in a financial scenario, can efficiently and accurately identify the postures, movements, and expressions of multiple human bodies, help improve the efficiency and quality of financial services, improve customer satisfaction, and thus enhance the competitiveness of financial enterprises.
[0141] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as a computer-readable program, a data structure, a program module or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, disk storage or other magnetic storage device, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media generally contain computer-readable programs, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0142] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0143] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0144] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0145] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0146] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0147] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0148] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0149] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0152] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for detecting key points of a human body, characterized in that: The method comprises: Acquire an image to be detected containing multiple human bodies; Performing human body detection on the image to be detected to obtain multiple human body images; Input each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; Input the human key point feature map into a pre-trained key point detection network for key point detection, and predict a thermal map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, the multi-task loss function includes a reconstruction loss function and a structural loss function, the reconstruction loss function is used to measure the consistency between the position of the human key point and the weak supervision signal, and the structural loss function is used to measure the geometric constraints between the positions of the human key points; A peak extraction algorithm is used to extract the position of each key point of the human body from the thermal map.
2. The method according to claim 1, characterized in that: The method further comprises: In the case where the image to be detected is a video frame image, a pre-trained key point tracking network is used to match and associate the position of each of the human key points in adjacent video frame images to obtain a position association result of each of the human key points, wherein the key point tracking network is trained by the weakly supervised learning module based on a multi-task loss function, and the multi-task loss function also includes a smoothing loss function, and the smoothing loss function is used to measure the continuity and stability between the human key points in adjacent video frame images; A Kalman filter is used to smooth and predict the position association result of each key point of the human body, so as to obtain the position tracking result of each key point of the human body.
3. The method according to claim 1, characterized in that The performing human body detection on the image to be detected to obtain a plurality of human body images comprises: Performing human body detection on the image to be detected to obtain a bounding box of each human body; The human body image within each of the boundary boxes is cropped and scaled to obtain multiple human body images of uniform size.
4. The method according to claim 1, characterized in that: After inputting each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map, the method further includes: Determine a first area related to a human body key point in the human body key point feature map; Determine a second area in the human body key point feature map that is unrelated to the human body key points; The first region is enhanced and the second region is suppressed through an attention mechanism.
5. The method according to claim 1 or 2, characterized in that: The reconstruction loss function is the average value of the Euclidean distance between the position of the human key point and the weak supervision signal.
6. The method according to claim 1 or 2, characterized in that: The structural loss function is the negative logarithm of the structural similarity between the key points of the human body and the preset reference model.
7. The method according to claim 2, characterized in that The smooth loss function is the sum of squares of position changes between the key points of the human body in adjacent frames.
8. A human body key point detection device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be detected containing multiple human bodies; A human body detection module, used for performing human body detection on the image to be detected to obtain a plurality of human body images; A feature extraction module is used to input each of the human body images into a pre-trained convolutional neural network for feature extraction to obtain a human body key point feature map; A key point detection module, used for inputting the human key point feature map into a pre-trained key point detection network for key point detection, and predicting a thermal map of each human key point, wherein the convolutional neural network and the key point detection network are trained by a weak supervision learning module based on a multi-task loss function, the weak supervision learning module uses 2D key points or 3D skeletons as weak supervision signals, the multi-task loss function includes a reconstruction loss and a structural loss, the reconstruction loss function is used to measure the consistency between the positions of the human key points and the weak supervision signals, and the structural loss function is used to measure the geometric constraints between the positions of the human key points; The key point extraction module is used to extract the position of each key point of the human body from the thermal map using a peak extraction algorithm.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the human body key point detection method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the human body key point detection method described in any one of claims 1 to 7 is implemented.
