Human pose recognition method and device based on artificial intelligence, and related equipment

Abstract

The present application relates to the field of artificial intelligence and financial technology, and provides a human posture recognition method and device based on artificial intelligence and related equipment, the method comprising: updating each feature extraction network in a first posture recognition model to obtain a second posture recognition model; performing feature extraction processing on a to-be-recognized picture by using the second posture recognition model, and inputting feature maps of each scale into a first target detection network and a second target detection network respectively, preprocessing output first detection results and second detection results, and performing fusion processing on third detection results and fourth detection results obtained by preprocessing to obtain human posture recognition results. The present application updates each feature extraction network in the first posture recognition model to recognize three-dimensional key points, thereby improving the accuracy of human posture recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and financial technology, and particularly relates to a human body posture recognition method and device based on artificial intelligence and related equipment. BACKGROUND

[0002] With the development of the times, it is more and more common to use a camera to collect pictures or videos for character behavior analysis. For example, using a home camera to detect whether an old person has fallen, using a fitness APP to detect whether the action is completed in a standard manner, or using a monitor to monitor whether a bank has a staff member who violates the rules, and the like.

[0003] However, in these scenarios, the prior art uses a two-dimensional target character detection or posture estimation method to recognize the posture of a target character in a picture collected by a camera, but is greatly affected by factors such as camera position and orientation, and cannot accurately recognize the human body posture of the target character in the picture.

[0004] Therefore, it is necessary to propose a method that can quickly and accurately recognize human body posture. SUMMARY

[0005] In view of the above, it is necessary to propose a human body posture recognition method and device based on artificial intelligence and related equipment, which updates each feature extraction network in the first posture recognition model to recognize three-dimensional key points, thereby improving the accuracy of human body posture recognition.

[0006] A first aspect of the present application provides a human body posture recognition method based on artificial intelligence, the method comprising:

[0007] obtaining a pre-trained first posture recognition model, and updating each feature extraction network in the first posture recognition model to obtain a second posture recognition model;

[0008] in response to a received human body posture recognition request, obtaining a picture to be recognized;

[0009] performing feature extraction processing on the picture to be recognized using the second posture recognition model to obtain a plurality of scale feature maps;

[0010] inputting each scale feature map into a first target detection network to obtain a first detection result corresponding to the scale feature map, and inputting each scale feature map into a second target detection network to obtain a second detection result corresponding to the scale feature map;

[0011] performing first preprocessing on each first detection result to obtain a third detection result, and performing second preprocessing on each second detection result to obtain a fourth detection result;

[0012] Fusing the third detection result and the fourth detection result to obtain a human posture recognition result.

[0013] Optionally, the updating each feature extraction network in the first posture recognition model to obtain a second posture recognition model comprises:

[0014] adding a preset number of feature extraction channels in each feature extraction network of the first posture recognition model to obtain a corresponding target feature extraction network;

[0015] replacing the corresponding feature extraction network in the first posture recognition model with the target feature extraction network to obtain the second posture recognition model.

[0016] Optionally, the feature extraction processing of the to-be-recognized picture by using the second posture recognition model to obtain a plurality of scale feature maps comprises:

[0017] performing down-sampling on the to-be-recognized picture by using a convolution layer in the target feature extraction network of the second posture recognition model to obtain a feature map corresponding to a scale of the convolution layer.

[0018] Optionally, the inputting each scale feature map into a first target detection network to obtain a first detection result of the corresponding scale feature map comprises:

[0019] detecting each scale feature map by using the first target detection network;

[0020] outputting a first detection box, a two-dimensional first key point and a three-dimensional first key point of a target person in each scale feature map to obtain the first detection result.

[0021] Optionally, the first preprocessing of each first detection result to obtain a third detection result comprises:

[0022] obtaining a plurality of first detection boxes and a confidence of each first detection box from each first detection result;

[0023] sorting the confidence of the plurality of first detection boxes to obtain a first detection box list;

[0024] selecting a first detection box with the highest confidence from the first detection box list and adding the first detection box to a preset first output list;

[0025] calculating a first overlap degree between the first detection box with the highest confidence and each first detection box remaining in the first detection box list;

[0026] Retain each first detection box with a first overlap degree less than a preset overlap degree threshold, and add it to the preset first output list to obtain a third detection result.

[0027] Optionally, the fusing the third detection result and the fourth detection result to obtain a human posture recognition result comprises:

[0028] Traversing the two-dimensional second key points and the three-dimensional second key points in the fourth detection result;

[0029] Calculating a first Euclidean distance between each two-dimensional second key point and a corresponding two-dimensional first key point, and calculating a second Euclidean distance between each three-dimensional second key point and a corresponding three-dimensional first key point;

[0030] Respectively judging whether the calculated first Euclidean distance and second Euclidean distance satisfy a preset replacement condition;

[0031] If the calculated first Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the two-dimensional first key point in the third detection result with the coordinates of the corresponding two-dimensional second key point in the fourth detection result, and / or, if the calculated second Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the three-dimensional first key point in the third detection result with the coordinates of the corresponding three-dimensional second key point in the fourth detection result to obtain a fifth detection result;

[0032] Thirdly preprocessing the fifth detection result and the fourth detection result to obtain a human posture recognition result.

[0033] Optionally, the thirdly preprocessing the fifth detection result and the fourth detection result to obtain a human posture recognition result comprises:

[0034] Merging and deduplicating the fifth detection result and the fourth detection result to obtain a human posture recognition result.

[0035] A second aspect of the present application provides a human posture recognition device based on artificial intelligence, the device comprising:

[0036] A first acquisition module is configured to acquire a pre-trained first posture recognition model, and update each feature extraction network in the first posture recognition model to obtain a second posture recognition model;

[0037] A second acquisition module is configured to acquire a to-be-recognized picture in response to a received human posture recognition request;

[0038] A feature extraction module is configured to perform feature extraction processing on the to-be-recognized picture by using the second posture recognition model to obtain a plurality of scale feature maps;

[0039] inputting each scale of feature map into the first target detection network to obtain a first detection result of the corresponding scale of feature map, and inputting each scale of feature map into the second target detection network to obtain a second detection result of the corresponding scale of feature map;

[0040] the pre-processing module is configured to perform first pre-processing on each first detection result to obtain a third detection result, and perform second pre-processing on each second detection result to obtain a fourth detection result;

[0041] the fusion processing module is configured to perform fusion processing on the third detection result and the fourth detection result to obtain a human posture recognition result.

[0042] A third aspect of the present application provides an electronic device, which comprises a processor and a memory, wherein the processor is configured to implement the human posture recognition method based on artificial intelligence when executing a computer program stored in the memory.

[0043] A fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is configured to implement the human posture recognition method based on artificial intelligence when executed by a processor.

[0044] In summary, the human posture recognition method based on artificial intelligence, the device and the related equipment can promote the construction of smart cities, and can be applied to the fields of smart buildings, smart security, smart communities, smart life, Internet of Things, financial technology and the like. By obtaining a pre-trained first posture recognition model, updating each feature extraction network in the first posture recognition model to obtain a second posture recognition model, adding a preset number of feature extraction channels in each feature extraction network of the first posture recognition model to predict the coordinate values of three-dimensional key points of a target person, i.e. to predict the posture information of the three-dimensional key points of the target person, the problem of inaccurate posture recognition caused by the lack of depth information in two-dimensional posture is solved. The second posture recognition model is used for feature extraction processing on the to-be-recognized picture to obtain a plurality of scales of feature maps. Each scale of feature map is input into the first target detection network to obtain a first detection result of the corresponding scale of feature map, and each scale of feature map is input into the second target detection network to obtain a second detection result of the corresponding scale of feature map. Each first detection result is pre-processed to obtain a third detection result, and each second detection result is pre-processed to obtain a fourth detection result. The redundant detection boxes are eliminated, the accuracy of the obtained first detection box and second detection box is improved, and the accuracy of posture recognition is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1is a flow chart of the human body posture recognition method based on artificial intelligence provided by the embodiment one of the present application.

[0046] Figure 2 is a schematic diagram of a target detection network outputting a detection result.

[0047] Figure 3 is a structural diagram of the human body posture recognition device based on artificial intelligence provided by the embodiment two of the present application.

[0048] Figure 4 is a structural schematic diagram of the electronic device provided by the embodiment three of the present application. DETAILED DESCRIPTION

[0049] In order to enable the above-mentioned objects, features and advantages of the present application to be more clearly understood, the present application will be described in detail below with reference to the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terminology used in the description of the present application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0051] Embodiment one

[0052] Figure 1 is a flow chart of the human body posture recognition method based on artificial intelligence provided by the embodiment one of the present application.

[0053] In the present embodiment, the human body posture recognition method based on artificial intelligence can be applied to an electronic device. For an electronic device that needs to perform human body posture recognition based on artificial intelligence, the function of human body posture recognition based on artificial intelligence provided by the method of the present application can be directly integrated on the electronic device, or run in the form of a software development kit (Software Development Kit, SDK) in the electronic device.

[0054] The embodiment of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (Artificial Intelligence, AI) is the theory, method, technology and application system for using digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results.

[0055] The basic technology of artificial intelligence generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The software technology of artificial intelligence mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0056] As shown in Figure 1 The human body posture recognition method based on artificial intelligence specifically includes the following steps, and the order of the steps in the flowchart can be changed, and some steps can be omitted according to different needs.

[0057] In the prior art, it is more and more common to use a camera to collect pictures or videos for human body posture behavior analysis, for example, using a household camera to detect whether an old person has fallen, using a fitness APP to detect whether the action is completed in a standard manner, or using a monitor to monitor whether a bank has a staff member with irregular operation, etc. However, in these scenarios, due to factors such as camera position and orientation, if the human body posture of the target person is accurately identified using a two-dimensional target detection box or a posture estimation method, the depth information of the key points of the target person needs to be identified, the posture of the target person in the three-dimensional space is predicted according to the depth information of the identified key points of the target person, and then the human body posture of the target person in the picture is better predicted.

[0058] 101, obtain a pre-trained first posture recognition model, and update each feature extraction network in the first posture recognition model to obtain a second posture recognition model.

[0059] In this embodiment, the first posture recognition model can be pre-trained, and the first posture recognition model can be an existing KAPAO (Keypoints And Poses As Objects) model. Specifically, a single picture is input into the KAPAO model, a feature extraction network in the KAPAO model is used to extract features of the single picture, and different scale feature maps corresponding to the single picture are obtained, wherein the feature extraction network is used to extract features of the single picture and output different scale feature maps corresponding to the single picture.

[0060] In this embodiment, the first posture recognition model includes a first prediction branch and a second prediction branch. The first prediction branch can well perceive the overall posture of the target person in each picture, such as the positions of key points such as head, left hand, right hand, left foot, right foot, etc., but is not accurate enough for predicting the local positions of the target person in each picture, such as eyes and mouth. The second prediction branch focuses on the local information of the target person in each picture, such as strong perception of the detailed positions of the eyes and mouth of the target person.

[0061] In this embodiment, after obtaining the feature maps of different scales of each picture, the feature maps of each scale of the single picture are respectively input into a first prediction branch and a second prediction branch of the first pose recognition model. The first prediction branch is used to predict whether a detection box of a target person and a two-dimensional key point position of the target person exist in each picture. The second prediction branch is used to take the two-dimensional key point of the target person predicted by the first prediction branch as a target detection task, and predict the position of a rectangular box generated with the two-dimensional key point as the center. Finally, the pose prediction result of the target person is output according to the detection box of the target person and the two-dimensional key point position of the target person predicted by the first prediction branch, and the position of the rectangular box generated with the two-dimensional key point as the center predicted by the second prediction branch.

[0062] In this embodiment, since the existing KAPAO model inputs two-dimensional pose information of a target person, it cannot accurately identify three-dimensional pose information of the target person in a picture. In order to improve the accuracy of pose prediction of the target person in the picture, the depth information of the key points of the target person needs to be detected, and the pose of the target person in the three-dimensional space is inferred, so that the human body pose of the target person can be more accurately predicted. Therefore, in this embodiment, a three-dimensional pose estimation network is built on the basis of the existing KAPAO model to accurately predict the human body pose of the target person in the three-dimensional space.

[0063] In an optional embodiment, the updating of each feature extraction network in the first pose recognition model to obtain a second pose recognition model comprises:

[0064] A preset number of feature extraction channels are added to each feature extraction network of the first pose recognition model to obtain a corresponding target feature extraction network.

[0065] The target feature extraction network is used to replace the corresponding feature extraction network in the first pose recognition model to obtain a second pose recognition model.

[0066] In this embodiment, a preset number of feature extraction channels are added to each feature extraction network of the first pose recognition model, for example, the feature extraction channels of each feature extraction network of the first pose recognition model are increased from the original 6+3K to 6+6K to obtain a target feature extraction network. The increased 3K channels are used to output x, y, and z values of three-dimensional key points of a target person. K represents the number of three-dimensional key points of the target person.

[0067] Specifically, the adding of the preset number of feature extraction channels in each feature extraction network of the first posture recognition model to obtain the corresponding target feature extraction network includes: modifying parameters of a convolutional layer of each feature extraction network in the first posture recognition model to increase the number of feature extraction channels, wherein the extra feature extraction channels are used to predict coordinate values of three-dimensional key points of a target person in a feature map.

[0068] In the embodiment, since the second posture recognition model is obtained by building a three-dimensional posture estimation network on the basis of the existing KAPAO model, in the training process of the second posture recognition model, a loss function used by the second posture recognition model when predicting three-dimensional key points of a target person is as follows:

[0069]

[0070] wherein K represents the number of three-dimensional key points of the target person, v k represents whether the three-dimensional key point is visible, O p represents the target person, G s represents a feature map of each scale output by the target feature extraction network, s e {8, 16, 32, 64}, wherein 8, 16, 32 and 64 respectively represent scales of the feature map, n represents the number of positive samples corresponding to each scale of the feature map, and δ represents a preset function, wherein if v k > 0, δ = 1, otherwise δ = 0, represents a predicted value of the coordinate of the three-dimensional key point, u k represents an actual value of the coordinate of the three-dimensional key point.

[0071] In the training process of the second posture recognition model, a multi-person three-dimensional posture estimation dataset is selected as a training set, and three-dimensional key point labels of a target person in the three-dimensional posture estimation dataset need to be obtained by using a professional motion capture system, and the scene and the target person obtained are limited. In order to improve the generalization of the second posture recognition model, the COCO two-dimensional posture estimation dataset is added to the training set to jointly train the second posture recognition model. For samples without three-dimensional key point labels, the three-dimensional key point loss in the training may be ignored. Since the COCO two-dimensional posture estimation dataset is added to the training set, the ignored three-dimensional key point loss is effectively generalized, and the accuracy of the key points recognized by the second posture recognition model is ensured.

[0072] In this embodiment, by adding a preset number of feature extraction channels in each feature extraction network of the first pose recognition model, the coordinates of the three-dimensional key points of the target person in each picture are predicted, that is, the pose of the three-dimensional key points of the target person is predicted, thereby solving the problem of inaccurate pose recognition of the target person caused by the lack of depth information in the two-dimensional pose information of the target person recognized by the existing first pose recognition model.

[0073] 102, in response to the received human pose recognition request, obtaining a to-be-recognized picture.

[0074] In this embodiment, the human pose recognition request is used to request to recognize the human pose of the target person in the picture.

[0075] In this embodiment, when the electronic device receives the human pose recognition request sent by the user end, the message of the human pose recognition request is parsed, and the to-be-recognized picture is obtained from the message, wherein the to-be-recognized picture can contain one or more target persons.

[0076] In this embodiment, the to-be-recognized picture can be an online face signing video frame of a borrower and a bank face auditor, a picture for identity verification of a user, a picture in a business process, etc., and the business can include applying for a loan, a credit card, or purchasing insurance, financial products, etc.

[0077] 103, performing feature extraction processing on the to-be-recognized picture by using the second pose recognition model to obtain a plurality of scale feature maps.

[0078] In this embodiment, the second pose recognition model contains a plurality of scale target feature extraction networks, and each scale target feature extraction network outputs a feature map.

[0079] In an optional embodiment, the performing feature extraction processing on the to-be-recognized picture by using the second pose recognition model to obtain a plurality of scale feature maps comprises:

[0080] Using the convolution layer in the target feature extraction network of the second pose recognition model to down-sample the to-be-recognized picture to obtain a feature map corresponding to the scale of the convolution layer.

[0081] In this embodiment, the second pose recognition model can perform multi-scale feature prediction on the to-be-recognized picture to obtain a plurality of scale feature maps. Specifically, the second pose recognition model contains a plurality of scale convolution layers, and each scale convolution layer is used to perform a down-sampling operation on the to-be-recognized picture to obtain a feature map of the corresponding scale.

[0082] 104, inputting each scale feature map into the first target detection network to obtain a first detection result of the corresponding scale feature map, and inputting each scale feature map into the second target detection network to obtain a second detection result of the corresponding scale feature map.

[0083] In the embodiment, the second posture recognition model comprises two branches, the first branch is the first target detection network, and the second branch is the second target detection network. After obtaining the feature maps of multiple scales of each picture, each scale feature map is input into the first target detection network and the second target detection network of the second posture recognition model, wherein the first target detection network is used to detect the first detection box, the two-dimensional first key point and the three-dimensional first key point of the target person in each scale feature map; and the second target detection network is used to detect the second detection box, the two-dimensional second key point and the three-dimensional second key point of the target person in each scale feature map.

[0084] Referring to Figure 2 As shown in the figure, the detection box 11 comprises detection box information of the target person, and the detection box information represents the first detection box or the second detection box of the target person corresponding to the detection box 11. The key point 12 comprises two-dimensional information and three-dimensional information of the target person, the two-dimensional information represents the two-dimensional first key point or the two-dimensional second key point of the target person corresponding to the key point 12, and the three-dimensional information represents the three-dimensional first key point or the three-dimensional second key point of the target person corresponding to the key point 12.

[0085] In an optional embodiment, the step of inputting each scale feature map into the first target detection network to obtain a first detection result of the corresponding scale feature map comprises:

[0086] detecting each scale feature map by using the first target detection network;

[0087] outputting the first detection box, the two-dimensional first key point and the three-dimensional first key point of the target person in each scale feature map to obtain the first detection result.

[0088] In the embodiment, the first target detection network comprises a multi-task convolutional neural network, which is used to detect the target person, the detection box of the target person and the key point position information of the target person in each scale feature map, and determine the first detection box, the two-dimensional first key point and the three-dimensional first key point of the target person obtained by detection as the first detection result.

[0089] In an optional embodiment, the step of inputting each scale feature map into the second target detection network to obtain a second detection result of the corresponding scale feature map comprises:

[0090] detect each scale of feature map by using the second target detection network;

[0091] output the second detection frame, the two-dimensional second key point and the three-dimensional second key point of the target person in each scale of feature map, and obtain a second detection result.

[0092] In this embodiment, the process of obtaining the second detection result is the same as that of obtaining the first detection result, which is not described here in detail.

[0093] Specifically, referring to FIG. 1, Figure 2 As shown in FIG. 1, for the first target detection network or the second target detection network, the first target detection network or the second target detection network can predict 6+6K values for each scale of feature map, wherein p0 represents whether there is a detection frame of a target person in each scale of feature map, when p0=0, it means that the probability of the existence of a target person in any scale of feature map is 0, when p0=1, it means that the probability of the existence of a target person in any scale of feature map is 1, t x ,t y ,t w ,t h represents the position of the detection frame, c1…c k+1 represents the preset category of the target person, wherein c1 represents whether the target is a person, c2 to c k+1 is used to determine the probability that each key point in k key points belongs to a key point of a certain preset category, for example, the preset category includes hand key points, leg key points, head key points, etc., k represents the index of the key point, v x1 ,v y1 …v xk ,v yk represents the x, y coordinates of k two-dimensional key points, u x1 ,u y1 ,u z1 …u xk ,u yk ,u zk represents the x, y, z coordinates of k three-dimensional key points relative to the preset position. For example, the preset position can be the hip bone.

[0094] In this embodiment, the three-dimensional key points of the target person are detected by the first target detection network and the second target detection network in the second pose recognition model, and when subsequent pose recognition is performed, the three-dimensional first key points and the three-dimensional second key points detected by the first target detection network and the second target detection network are considered, thereby improving the accuracy of the pose recognition result.

[0095] In the embodiment, the first target detection network of the second posture recognition model outputs the first detection frame, the two-dimensional first key point, the three-dimensional first key point of the target person, and the second target detection network outputs the second detection frame, the two-dimensional second key point, and the three-dimensional second key point of the target person, so that the posture recognition of the target person can be realized.

[0096] 105. The first preprocessing is performed on each first detection result to obtain a third detection result, and the second preprocessing is performed on each second detection result to obtain a fourth detection result.

[0097] In the embodiment, a large number of detection frames are generated at the same position in the target detection process, and the detection frames may overlap with each other. In order to find the best target detection frame, the first preprocessing is used to eliminate the redundant first detection frame, that is, the overlapping first detection frame in each first detection result is deleted; and the second preprocessing is used to eliminate the redundant second detection frame, that is, the overlapping second detection frame in each second detection result is deleted.

[0098] In an optional embodiment, the first preprocessing performed on each first detection result to obtain a third detection result includes:

[0099] A plurality of first detection frames and a confidence of each first detection frame are obtained from the each first detection result.

[0100] The confidence of the plurality of first detection frames is sorted to obtain a first detection frame list.

[0101] The first detection frame with the highest confidence is selected from the first detection frame list and added to a preset first output list.

[0102] A first overlap degree between the first detection frame with the highest confidence and each remaining first detection frame in the first detection frame list is calculated.

[0103] Each first detection frame with a first overlap degree less than a preset overlap threshold is retained and added to the preset first output list to obtain a third detection result.

[0104] In the embodiment, the confidence is used to represent a probability value that the first target detection network predicts the first detection frame to be correct. For example, the confidence of any first detection frame is 0.9, which represents that the probability value of the predicted first detection frame being a correct detection frame is 90%, wherein the correct detection frame represents that the target person exists in the predicted detection frame.

[0105] In an optional embodiment, the second preprocessing performed on each second detection result to obtain a fourth detection result includes:

[0106] obtain a plurality of second detection boxes and a confidence of each second detection box from each second detection result;

[0107] sort the confidence of the plurality of second detection boxes to obtain a second detection box list;

[0108] select a second detection box with the highest confidence from the second detection box list and add it to a preset second output list;

[0109] calculate a second overlap degree between the second detection box with the highest confidence and each remaining second detection box in the second detection box list;

[0110] retain each second detection box with a second overlap degree less than the preset overlap degree threshold and add it to the preset second output list to obtain a fourth detection result.

[0111] In this embodiment, the confidence is used to represent a probability value that the second target detection network predicts the second detection box correctly.

[0112] In this embodiment, the detection box list can be a first detection box list or a second detection box list, the detection box can be a first detection box or a second detection box, and the overlap degree refers to Intersection Over Union (IOU). The IOU refers to the intersection of any one detection box in the detection box list and the remaining detection boxes in the detection box list divided by the union of the corresponding detection box in the detection box list and the remaining detection boxes in the detection box list. The IOU is used to measure the overlap degree of two sets, where IOU is 0, the two boxes do not overlap and have no intersection; IOU is 1, the two boxes completely overlap; and IOU takes a value between 0 and 1, which represents the overlap degree of the two boxes, and the higher the value, the higher the overlap degree.

[0113] In this embodiment, the overlap degree threshold can be preset, and based on the preset overlap degree threshold, redundant first detection boxes and second detection boxes can be eliminated, and the best first detection boxes and second detection boxes can be retained.

[0114] In this embodiment, when the first overlap degree or the second overlap degree is less than or equal to the preset overlap degree threshold, it is determined that there is no overlapping detection box at the same position, and no deletion is performed.

[0115] In this embodiment, each first detection result and each second detection result is preprocessed respectively, and when the first overlap degree or the second overlap degree is greater than the preset overlap degree threshold, redundant detection boxes are eliminated, the accuracy of the obtained first detection boxes and second detection boxes is improved, and the accuracy of the pose recognition is improved.

[0116] 106, fuse the third detection result and the fourth detection result to obtain a human posture recognition result.

[0117] In this embodiment, the third detection result includes the preprocessed first detection box, the two-dimensional first key point and the three-dimensional first key point, the fourth detection result includes the preprocessed second detection box, the two-dimensional second key point and the three-dimensional second key point, and the fusion processing refers to fusing the two-dimensional key points and the three-dimensional key points in the third detection result and the fourth detection result to the target person to obtain a human posture recognition result of the target person.

[0118] In an optional embodiment, the fusion processing of the third detection result and the fourth detection result to obtain a human posture recognition result includes:

[0119] traversing the two-dimensional second key points and the three-dimensional second key points in the fourth detection result;

[0120] calculating a first Euclidean distance between each two-dimensional second key point and a two-dimensional first key point at a corresponding position, and calculating a second Euclidean distance between each three-dimensional second key point and a three-dimensional first key point at a corresponding position;

[0121] respectively judging whether the calculated first Euclidean distance and second Euclidean distance satisfy a preset replacement condition;

[0122] if the calculated first Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the two-dimensional first key point in the third detection result with the coordinates of the corresponding two-dimensional second key point in the fourth detection result, and / or, if the calculated second Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the three-dimensional first key point in the third detection result with the coordinates of the corresponding three-dimensional second key point in the fourth detection result, to obtain a fifth detection result;

[0123] thirdly preprocessing the fifth detection result and the fourth detection result to obtain a human posture recognition result.

[0124] Further, the third preprocessing of the fifth detection result and the fourth detection result to obtain a human posture recognition result includes merging and deduplicating the fifth detection result and the fourth detection result to obtain a human posture recognition result.

[0125] In this embodiment, the fifth detection result represents the third detection result after fusion processing, the human posture recognition result includes a 6+6K channel vector predicted by the second posture recognition model, and the vector includes the first detection box, the two-dimensional first key point, the three-dimensional first key point in the fifth detection result, and the second detection box, the two-dimensional second key point and the three-dimensional second key point in the fourth detection result.

[0126] In this embodiment, the Euclidean distance is also called Euclidean distance, which is used to measure the absolute distance between two points in a multi-dimensional space.

[0127] In this embodiment, the replacement condition can be set in advance, and the replacement condition can be set as the first Euclidean distance or the second Euclidean distance being less than a preset distance threshold.

[0128] In this embodiment, the third preprocessing is used to de-duplicate the same two-dimensional first key points and three-dimensional first key points, and the same two-dimensional second key points and three-dimensional second key points in the fifth detection result and the fourth detection result, and to merge the different two-dimensional first key points and three-dimensional first key points, and the different two-dimensional second key points and three-dimensional second key points in the fifth detection result and the fourth detection result, thereby obtaining the human posture recognition result.

[0129] For example, the first target detection network of the second posture recognition model outputs M key points of an arbitrary target person, and the second target detection network outputs N key points of the arbitrary target person. The key points can be two-dimensional key points or three-dimensional key points. Each key point output by the second target detection network is traversed. If the Euclidean distance between any key point and the key point at the corresponding position output by the first target detection network is less than a preset distance threshold, for example, the distance between the nose coordinate value output by the second target detection network and the nose coordinate value output by the first target detection network is less than the preset distance threshold, it is considered that the replacement condition is met, the nose coordinate value output by the first target detection network is replaced with the nose coordinate value output by the second target detection network, and the nose coordinate value output by the second target detection network is taken as the final coordinate value of the nose.

[0130] In this embodiment, the first target detection network has a good overall perception of the target, but is not very accurate in predicting these relatively local key points such as eyes and nose. These key points need to be repaired, and this repair relies on the key points output by the second target detection network. The second target detection network has a poorer overall perception than the first target detection network, but has a better perception of local key points than the first target detection network. If the Euclidean distance between the key point output by the first target detection network and the key point output by the second target detection network meets the replacement condition, the key point output by the second target detection network is used to replace the key point output by the first target detection network. The final key point combines the advantages of the overall perception of the first target detection network and the local perception of the second target detection network, improves the accuracy of the final key point, and further improves the accuracy of human posture recognition of the target person.

[0131] In the embodiment, the second posture recognition model estimates the coordinate values of the three-dimensional key points of the target person by increasing a preset number of feature extraction channels, solves the problem that the human body posture of the target person cannot be accurately recognized using a two-dimensional target detection frame or a posture estimation method, and improves the accuracy of human body posture recognition of the target person.

[0132] In the embodiment, the second posture recognition model is used for human body posture recognition of the target person in the to-be-recognized picture, accurately recognizes the posture of the target person, for example, if the to-be-recognized picture is a picture for identity verification of a user, the human body posture result of the target person in the picture for identity verification is that a face image in a mobile phone is used in the face recognition process, it is determined according to the human body posture recognition result that the identity verification is not passed, and the accuracy of identity verification is improved.

[0133] To sum up, the human body posture recognition method based on artificial intelligence provided in the embodiment includes the following steps: obtaining a pre-trained first posture recognition model, updating each feature extraction network in the first posture recognition model to obtain a second posture recognition model, increasing a preset number of feature extraction channels in each feature extraction network of the first posture recognition model to predict the coordinate values of the three-dimensional key points of the target person, i.e., to predict the posture information of the three-dimensional key points of the target person, solving the problem of inaccurate posture recognition caused by the lack of depth information of a two-dimensional posture. The second posture recognition model is used for feature extraction processing of the to-be-recognized picture, a plurality of scale feature maps are obtained, each scale feature map is input into a first target detection network to obtain a first detection result of the corresponding scale feature map, each scale feature map is input into a second target detection network to obtain a second detection result of the corresponding scale feature map, each first detection result is first preprocessed to obtain a third detection result, and each second detection result is second preprocessed to obtain a fourth detection result, so as to eliminate redundant detection frames, improve the accuracy of the obtained first detection frame and second detection frame, and further improve the accuracy of posture recognition.

[0134] Embodiment Two

[0135] Figure 3 is a structural diagram of the human body posture recognition device based on artificial intelligence provided in the embodiment two of the present application.

[0136] In some embodiments, the human body posture recognition device based on artificial intelligence 20 can include a plurality of function modules composed of program code segments. The program codes of each program segment in the human body posture recognition device based on artificial intelligence 20 can be stored in the memory of the electronic device and executed by the at least one processor to perform the functions of the human body posture recognition based on artificial intelligence (see Figure 1 and Figure 2 described) based on artificial intelligence.

[0137] In this embodiment, the human posture recognition device 20 based on artificial intelligence can be divided into multiple functional modules according to the functions performed by the human posture recognition device 20. The functional modules can include a first acquisition module 201, a second acquisition module 202, a feature extraction module 203, an input module 204, a preprocessing module 205, and a fusion processing module 206. The module referred to in the present application refers to a series of computer-readable instruction segments that can be executed by at least one processor and can complete a fixed function, which is stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0138] The first acquisition module 201 is configured to acquire a pre-trained first posture recognition model, and update each feature extraction network in the first posture recognition model to obtain a second posture recognition model.

[0139] The second acquisition module 202 is configured to acquire a to-be-recognized picture in response to a received human posture recognition request.

[0140] The feature extraction module 203 is configured to perform feature extraction processing on the to-be-recognized picture by using the second posture recognition model to obtain feature maps of multiple scales.

[0141] The input module 204 is configured to input the feature maps of each scale into a first target detection network to obtain first detection results of the corresponding scale feature maps, and input the feature maps of each scale into a second target detection network to obtain second detection results of the corresponding scale feature maps.

[0142] The preprocessing module 205 is configured to perform first preprocessing on each first detection result to obtain third detection results, and perform second preprocessing on each second detection result to obtain fourth detection results.

[0143] The fusion processing module 206 is configured to perform fusion processing on the third detection results and the fourth detection results to obtain a human posture recognition result.

[0144] In an optional embodiment, the first acquisition module 201 is configured to: add a preset number of feature extraction channels in each feature extraction network of the first posture recognition model to obtain a corresponding target feature extraction network; and replace the corresponding feature extraction network in the first posture recognition model with the target feature extraction network to obtain the second posture recognition model.

[0145] In an optional embodiment, the feature extraction module 203 is configured to: perform down-sampling on the to-be-recognized picture by using a convolutional layer in a target feature extraction network of the second posture recognition model to obtain a feature map corresponding to a scale of the convolutional layer.

[0146] In an optional embodiment, the input module 204 is configured to: detect each scale of feature map by using the first target detection network; and output a first detection frame, a two-dimensional first key point, and a three-dimensional first key point of a target person in each scale of feature map, to obtain a first detection result.

[0147] In an optional embodiment, the preprocessing module 205 is configured to: obtain a plurality of first detection frames and a confidence of each first detection frame from each first detection result; sort the confidence of the plurality of first detection frames to obtain a first detection frame list; select a first detection frame with the highest confidence from the first detection frame list and add it to a preset first output list; calculate a first overlap degree between the first detection frame with the highest confidence and each remaining first detection frame in the first detection frame list; retain each first detection frame with a first overlap degree less than a preset overlap degree threshold and add it to the preset first output list, to obtain a third detection result.

[0148] In an optional embodiment, the preprocessing module 205 is further configured to: obtain a plurality of second detection frames and a confidence of each second detection frame from each second detection result; sort the confidence of the plurality of second detection frames to obtain a second detection frame list; select a second detection frame with the highest confidence from the second detection frame list and add it to a preset second output list; calculate a second overlap degree between the second detection frame with the highest confidence and each remaining second detection frame in the second detection frame list; retain each second detection frame with a second overlap degree less than the preset overlap degree threshold and add it to the preset second output list, to obtain a fourth detection result.

[0149] In an optional embodiment, the fusion processing module 206 is configured to: traverse the two-dimensional second key points and the three-dimensional second key points in the fourth detection result; calculate a first Euclidean distance between each two-dimensional second key point and a two-dimensional first key point at a corresponding position, and calculate a second Euclidean distance between each three-dimensional second key point and a three-dimensional first key point at a corresponding position; respectively determine whether the calculated first Euclidean distance and the second Euclidean distance satisfy a preset replacement condition; if the calculated first Euclidean distance satisfies the preset replacement condition, replace a coordinate of the two-dimensional first key point in the third detection result with a coordinate of the corresponding two-dimensional second key point in the fourth detection result, or if the calculated second Euclidean distance satisfies the preset replacement condition, replace a coordinate of the three-dimensional first key point in the third detection result with a coordinate of the corresponding three-dimensional second key point in the fourth detection result, to obtain a fifth detection result; and perform third preprocessing on the fifth detection result and the fourth detection result, to obtain a human pose recognition result.

[0150] Further, the third preprocessing of the fifth detection result and the fourth detection result to obtain the human posture recognition result comprises: performing a merging and deduplication processing on the fifth detection result and the fourth detection result to obtain the human posture recognition result.

[0151] To sum up, the human posture recognition device based on artificial intelligence provided by the embodiment can obtain a pre-trained first posture recognition model, update each feature extraction network in the first posture recognition model to obtain a second posture recognition model, increase a preset number of feature extraction channels in each feature extraction network of the first posture recognition model, and predict the coordinate values of three-dimensional key points of a target person, i.e., predict the posture information of the three-dimensional key points of the target person, so as to solve the problem of inaccurate posture recognition caused by the lack of depth information of two-dimensional posture. The second posture recognition model is used for feature extraction processing on the picture to be recognized to obtain a plurality of scale feature maps, each scale feature map is input into a first target detection network to obtain a first detection result of the corresponding scale feature map, each scale feature map is input into a second target detection network to obtain a second detection result of the corresponding scale feature map, each first detection result is subjected to first preprocessing to obtain a third detection result, and each second detection result is subjected to second preprocessing to obtain a fourth detection result, so as to eliminate redundant detection boxes, improve the accuracy of the obtained first detection box and second detection box, and further improve the accuracy of posture recognition.

[0152] Embodiment three

[0153] Referring to Figure 4 Fig. 3 shows a structural schematic diagram of an electronic device provided by the embodiment three of the present application. In the preferred embodiment of the present application, the electronic device 3 comprises a memory 31, at least one processor 32, at least one communication bus 33, and a transceiver 34.

[0154] Those skilled in the art should understand that Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation of the embodiment of the present application, and can be a bus structure or a star structure. The electronic device 3 can also comprise more or less other hardware or software, or different component arrangements.

[0155] In some embodiments, the electronic device 3 is an electronic device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions, and the hardware thereof includes, but is not limited to, a microprocessor, an application-specific integrated circuit, a programmable gate array, a digital processor, an embedded device, etc. The electronic device 3 can also include a client device, which includes, but is not limited to, any electronic product capable of human-computer interaction with a client through a keyboard, a mouse, a remote controller, a touchpad, a voice control device, etc., such as a personal computer, a tablet computer, a smart phone, a digital camera, etc.

[0156] It should be noted that the electronic device 3 is only an example, and other existing or future electronic products, such as those that can be adapted to the present application, should also be included in the protection scope of the present application and are hereby incorporated by reference.

[0157] In some embodiments, the memory 31 is used to store program codes and various data, such as the artificial intelligence-based human body posture recognition device 20 installed in the electronic device 3, and to realize high-speed and automatic access of programs or data during the operation of the electronic device 3. The memory 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc memories, a magnetic disk memory, a magnetic tape memory, or any other computer-readable medium capable of carrying or storing data.

[0158] In some embodiments, the at least one processor 32 can be composed of integrated circuits, for example, can be composed of a single packaged integrated circuit, or can be composed of multiple packaged integrated circuits of the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, combinations of various control chips, etc. The at least one processor 32 is the control core (Control Unit) of the electronic device 3, which connects various components of the entire electronic device 3 through various interfaces and lines, and executes programs or modules stored in the memory 31 and calls data stored in the memory 31 to perform various functions and process data of the electronic device 3.

[0159] In some embodiments, the at least one communication bus 33 is configured to realize the connection and communication between the memory 31, the at least one processor 32, etc.

[0160] Although not shown, the electronic device 3 can also include a power supply (such as a battery) for powering various components. Optionally, the power supply can be logically connected to the at least one processor 32 through a power management device, so as to realize the functions of managing charging, discharging, and power consumption management, etc. through the power management device. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, etc. The electronic device 3 can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described here.

[0161] It should be understood that the embodiments are only for illustration and are not limited in the scope of the patent application by the structure.

[0162] The integrated units in the form of software function modules described above can be stored in a computer readable storage medium. The software function modules described above are stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, an electronic device, or a network device, etc.) or a processor to execute part of the method described in each embodiment of the present application.

[0163] In further embodiments, in combination with Figure 3 , the at least one processor 32 can execute the operating device of the electronic device 3 and various application programs (such as the artificial intelligence-based human body posture recognition device 20 described above) installed, program codes, etc., for example, the various modules described above.

[0164] The memory 31 stores program codes, and the at least one processor 32 can invoke the program codes stored in the memory 31 to perform related functions. For example, Figure 3 Each module described in the above embodiments is a program code stored in the memory 31 and executed by the at least one processor 32, so as to realize the functions of the modules to achieve the purpose of human pose recognition based on artificial intelligence.

[0165] For example, the program codes can be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete the present application. The one or more modules / units can be a series of computer readable instruction segments capable of completing a specific function, which are used to describe the execution process of the program codes in the electronic device 3. For example, the program codes can be divided into a first acquisition module 201, a second acquisition module 202, a feature extraction module 203, an input module 204, a preprocessing module 205, and a fusion processing module 206.

[0166] In an embodiment of the present application, the memory 31 stores a plurality of computer readable instructions, which are executed by the at least one processor 32 to realize the functions of human pose recognition based on artificial intelligence.

[0167] Specifically, the specific implementation method of the at least one processor 32 to the above instructions can refer to the descriptions of the related steps in the corresponding embodiments, which will not be repeated here. Figure 1 and Figure 2 The descriptions of the related steps in the corresponding embodiments, which will not be repeated here.

[0168] In several embodiments provided by the present application, it should be understood that the disclosed apparatus and method can be implemented in other manners. For example, the described apparatus embodiments are merely schematic. For example, the division of the modules is merely a logical function division. There can be another division manner for the actual implementation.

[0169] The modules illustrated as separated components can or can not be physically separated, and the components illustrated as modules can or can not be physical units. They can be located in one position or distributed on a plurality of network units. Some or all of the modules can be selected according to the actual needs to achieve the purposes of the embodiments.

[0170] In addition, each functional module in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically independently, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware, or in the form of hardware plus software function module.

[0171] It will be obvious to a person skilled in the art that the application is not limited to the details of the foregoing exemplary embodiments and can be implemented in other embodiments without departing from the scope of the application. The embodiments are to be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalents of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims to the features to which the reference signs are attached. Furthermore, it should be noted that the word "comprising" does not exclude other elements or steps than those listed and the single units or relative devices recited in the application can also be implemented by software or hardware. The words "first", "second", etc. do not imply any particular order but are used for naming purposes only.

[0172] Finally, it should be noted that the above-mentioned embodiments are merely intended to illustrate the technical solutions of the present application, rather than limit the present application. Even though the present application has been described in detail with reference to the preferred embodiments, a person skilled in the art should understand that the technical solutions of the present application can be modified or equivalent replaced without departing from the spirit and scope of the present application.

Claims

1. A human pose recognition method based on artificial intelligence, characterized in that, The method includes: Obtain a pre-trained first pose recognition model, and update each feature extraction network in the first pose recognition model to obtain a second pose recognition model; In response to a received human pose recognition request, obtain the image to be recognized; The second pose recognition model is used to perform feature extraction processing on the image to be recognized to obtain feature maps at multiple scales; The feature map at each scale is input into the first target detection network to obtain the first detection result of the corresponding scale feature map, and the feature map at each scale is input into the second target detection network to obtain the second detection result of the corresponding scale feature map; Each first detection result undergoes a first preprocessing step to obtain a third detection result, and each second detection result undergoes a second preprocessing step to obtain a fourth detection result; The third and fourth detection results are fused to obtain a human pose recognition result, including: traversing the two-dimensional second keypoints and three-dimensional second keypoints in the fourth detection result; calculating the first Euclidean distance between each two-dimensional second keypoint and the corresponding two-dimensional first keypoint, and calculating the second Euclidean distance between each three-dimensional second keypoint and the corresponding three-dimensional first keypoint; determining whether the calculated first and second Euclidean distances satisfy a preset replacement condition; if the calculated first Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the two-dimensional first keypoints in the third detection result with the coordinates of the corresponding two-dimensional second keypoints in the fourth detection result, and / or, if the calculated second Euclidean distance satisfies the preset replacement condition, replacing the coordinates of the three-dimensional first keypoints in the third detection result with the coordinates of the corresponding three-dimensional second keypoints in the fourth detection result, to obtain a fifth detection result; and performing a third preprocessing on the fifth and fourth detection results to obtain a human pose recognition result.

2. The human pose recognition method based on artificial intelligence as described in claim 1, characterized in that, The step of updating each feature extraction network in the first pose recognition model to obtain the second pose recognition model includes: A preset number of feature extraction channels are added to each feature extraction network of the first pose recognition model to obtain the corresponding target feature extraction network. The second pose recognition model is obtained by replacing the corresponding feature extraction network in the first pose recognition model with a target feature extraction network.

3. The human pose recognition method based on artificial intelligence as described in claim 1, characterized in that, The step of using the second pose recognition model to extract features from the image to be recognized, resulting in feature maps at multiple scales, includes: The convolutional layer in the target feature extraction network of the second pose recognition model is used to downsample the image to be recognized, thereby obtaining a feature map corresponding to the scale of the convolutional layer.

4. The human pose recognition method based on artificial intelligence as described in claim 1, characterized in that, The step of inputting the feature map at each scale into the first object detection network to obtain the first detection result of the corresponding scale feature map includes: The first object detection network is used to detect the feature map at each scale; Output the first detection box, the first two-dimensional key point, and the first three-dimensional key point of the target person in the feature map of each scale to obtain the first detection result.

5. The human pose recognition method based on artificial intelligence as described in claim 1, characterized in that, The first preprocessing of each first detection result to obtain the third detection result includes: From each of the first detection results, obtain multiple first detection boxes and the confidence level of each first detection box; The confidence scores of the plurality of first detection boxes are sorted to obtain a list of first detection boxes; Select the first detection box with the highest confidence from the first detection box list and add it to the preset first output list; Calculate the first overlap between the first detection box with the highest confidence and each of the remaining first detection boxes in the first detection box list; Each first detection box with an overlap less than a preset overlap threshold is retained and added to the preset first output list to obtain a third detection result.

6. The human pose recognition method based on artificial intelligence as described in claim 1, characterized in that, The third preprocessing of the fifth and fourth detection results to obtain the human pose recognition result includes: The fifth and fourth detection results are merged and deduplicated to obtain the human posture recognition result.

7. A human posture recognition device based on artificial intelligence, characterized in that, The device includes: The first acquisition module is used to acquire a pre-trained first pose recognition model and update each feature extraction network in the first pose recognition model to obtain a second pose recognition model. The second acquisition module is used to acquire the image to be recognized in response to the received human posture recognition request; The feature extraction module is used to perform feature extraction processing on the image to be identified using the second pose recognition model to obtain feature maps at multiple scales; The input module is used to input the feature map of each scale into the first target detection network to obtain the first detection result of the feature map of the corresponding scale, and to input the feature map of each scale into the second target detection network to obtain the second detection result of the feature map of the corresponding scale. The preprocessing module is used to perform a first preprocessing on each first detection result to obtain a third detection result, and to perform a second preprocessing on each second detection result to obtain a fourth detection result; A fusion processing module is used to fuse the third detection result and the fourth detection result to obtain a human pose recognition result. This includes: traversing the two-dimensional second keypoints and three-dimensional second keypoints in the fourth detection result; calculating the first Euclidean distance between each two-dimensional second keypoint and the corresponding two-dimensional first keypoint, and calculating the second Euclidean distance between each three-dimensional second keypoint and the corresponding three-dimensional first keypoint; determining whether the calculated first and second Euclidean distances satisfy preset replacement conditions; if the calculated first Euclidean distance satisfies the preset replacement conditions, replacing the coordinates of the two-dimensional first keypoints in the third detection result with the coordinates of the corresponding two-dimensional second keypoints in the fourth detection result, and / or, if the calculated second Euclidean distance satisfies the preset replacement conditions, replacing the coordinates of the three-dimensional first keypoints in the third detection result with the coordinates of the corresponding three-dimensional second keypoints in the fourth detection result to obtain a fifth detection result; and performing a third preprocessing on the fifth and fourth detection results to obtain a human pose recognition result.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory, wherein the processor is used to execute a computer program stored in the memory to implement the artificial intelligence-based human posture recognition method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the artificial intelligence-based human posture recognition method as described in any one of claims 1 to 6.

Images