Methods, devices, electronic equipment and storage media for detecting the orientation of a target object

Abstract

The present invention provides a method, apparatus, electronic device, and storage medium for detecting the orientation of a target object. The method includes: determining a target region in a first image; capturing a second image of the target region to determine key human body points of the target object; determining the orientation of the target object in the first image based on a first human body angle score calculated from at least one set of specific key point pairs when at least one set of specific key point pairs exists; and determining the orientation of the target object in the first image based on a second human body angle score calculated from at least one set of backup key point pairs when no specific key point pairs exist. The present invention quantitatively calculates human body angle scores, which accurately determine the orientation of the human body, solving the technical problems of low accuracy and high cost in human body orientation detection in related technologies.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a method, apparatus, electronic device and storage medium for detecting the orientation of a target object. Background Technology

[0002] With the development of computer technology and the widespread application of computer vision principles, research on real-time target tracking using computer image processing technology is becoming increasingly popular. Dynamic real-time target tracking and positioning has broad application value in intelligent transportation systems, intelligent monitoring systems, military target detection, and surgical instrument positioning in medical navigation surgeries. For ordinary users, it's frequently used to obtain information such as the user's body orientation (facing forward / sideways / backwards from the camera) from their exercise videos, and then analyze body posture using this information for sports training analysis. Similarly, in film and television, detecting the body orientation of characters (facing forward / sideways / backwards from the camera) allows for the analysis of body posture and keyframe selection.

[0003] Currently, there are generally two methods for detecting the orientation of people in videos: one is to classify human posture using deep learning algorithms, and the other is to determine the orientation using 3D information of the human body. However, both existing methods have inherent problems that prevent them from meeting user needs and from being widely adopted. For example, the method of classifying human posture using deep learning algorithms suffers from low accuracy and unclear boundaries between labels. For instance, when a person is facing the camera at an angle between 30 and 60 degrees, it is difficult to determine whether to classify them as facing directly or sideways to the camera. Similarly, the method of determining the orientation using 3D information suffers from high computational costs and difficulty in data collection. 3D data requires specialized equipment for collection and annotation, resulting in significant data acquisition costs.

[0004] Currently, no effective solution has been proposed to address the issues of low accuracy and high cost in human orientation detection in related technologies. Summary of the Invention

[0005] Therefore, embodiments of the present invention provide a method, apparatus, electronic device, and storage medium for detecting the orientation of a target object, which can effectively improve the accuracy of human body orientation detection and reduce detection costs.

[0006] In a first aspect, embodiments of the present invention provide a method for detecting the orientation of a target object, wherein the method includes:

[0007] The target region where the target object is located is determined in the first image; wherein, the first image is a single frame image in the video sequence;

[0008] A second image of the target area is captured, and key human body points of the target object are determined based on the second image;

[0009] In the case where there is at least one set of specific key point pairs, a first human body angle score of the target object is calculated based on at least one set of specific key points, and the orientation of the target object in the first image is determined based on the first human body angle score; wherein, each set of specific key point pairs includes a first specific key point and a second specific key point, and the first human body part corresponding to the first specific key point and the second human body part corresponding to the second specific key point are symmetrical human body parts in the human body structure.

[0010] In the case where there is no specific key point pair on the human body, but there is at least one set of backup key point pairs, the second human body angle score of the target object is calculated based on the at least one set of backup key point pairs, and the orientation of the target object in the first image is determined based on the second human body angle score; wherein, each set of backup key point pairs is different from the specific key point pair, each set of backup key points includes a first backup key point and a second backup key point, and the third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

[0011] In one possible implementation, determining the target region where the target object is located in the first image includes:

[0012] The first image is input into the first neural network model to extract positional features from the first image using the first neural network model, thereby obtaining the first feature map of the first image;

[0013] For each point on the first feature map, generate multiple anchor boxes, where each anchor box has a different size;

[0014] For each anchor box, calculate the confidence that the target object exists within the anchor box;

[0015] The first target anchor frame with a confidence level greater than or equal to the first threshold is retained, and the area enclosed by the first target anchor frame is determined as the target area where the target object is located.

[0016] In one possible implementation, determining key human body points of the target object based on the second image includes:

[0017] The second image is input into the second neural network model to extract human features from the second image and obtain the second feature map of the second image.

[0018] The second feature map is converted into multiple heat maps, where each heat map corresponds to a preset human body key point;

[0019] Determine the thermal value of each point on each heatmap, and sort each point on the heatmap according to the thermal value;

[0020] The coordinates of the points before the target sorting position are weighted and averaged to obtain the coordinate information of the predicted human key point corresponding to each preset human key point. The human key point includes the coordinate information of the predicted human key point.

[0021] Based on the coordinate information, predict the key points of the human body in the second image.

[0022] In one possible implementation, calculating a first human angle score of the target object based on at least a set of specific key points includes:

[0023] For each specific key pair, the first lateral coordinate information of the first specific key point and the second lateral coordinate information of the second specific key point are obtained in the second image;

[0024] The first score corresponding to a specific key pair is calculated based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image.

[0025] The first human angle score of the target object is obtained by weighting at least one first score.

[0026] In one possible implementation, calculating a first score corresponding to a specific key pair based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image includes:

[0027] The angle normalization score of the first key point is determined based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image.

[0028] The front and back views are determined based on the first and second horizontal coordinate information;

[0029] The first score is calculated based on the normalized score of the first key point angle and the score for judging the front and back sides.

[0030] In one possible implementation, determining the front / back identification score based on the first and second lateral coordinate information includes:

[0031] If the first horizontal coordinate information is greater than the second horizontal coordinate information, the first preset value is determined as the front / back determination score; wherein, the first preset value is a positive value; or,

[0032] If the first horizontal coordinate information is not greater than the second horizontal coordinate information, the second preset value is determined as the front / back judgment score; wherein, the second preset value is a negative value.

[0033] In one possible implementation, determining the orientation of the target object in the first image based on a first human body angle score includes:

[0034] The first target score range corresponding to the first human body angle score is found in the direction lookup table; the direction lookup table stores multiple different directions and the score range corresponding to each direction.

[0035] The first target direction corresponding to the first target score range is determined as the direction of the target object in the first image.

[0036] In one possible implementation, calculating a second human angle score for the target object based on at least one set of alternative key points includes:

[0037] For each pair of backup key points, the third lateral coordinate information of the first backup key point and the fourth lateral coordinate information of the second backup key point are obtained in the second image.

[0038] The second score corresponding to the backup key point pair is calculated based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image.

[0039] The second human angle score of the target object is obtained by weighting at least one second score.

[0040] In one possible implementation, the second score corresponding to the backup keypoint pair is calculated based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image, including:

[0041] The angle normalization score of the second key point is determined based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image.

[0042] The second score is calculated based on the normalized score of the second key point angle and the first preset value; wherein, the first preset value is a positive value.

[0043] In one possible implementation, determining the orientation of the target object in the first image based on the second human body angle score includes:

[0044] The direction lookup table is used to find the range of the second target score corresponding to the second human body angle score. The direction lookup table stores multiple different directions and the score range corresponding to each direction.

[0045] The second target direction corresponding to the second target score range is determined as the direction of the target object in the first image.

[0046] In one possible implementation, the method further includes:

[0047] If no specific keypoint pair exists for the human body and no alternative keypoint pair exists, a prompt message indicating that the human body is incomplete is generated.

[0048] Secondly, embodiments of the present invention provide a target object orientation detection device, wherein the device includes:

[0049] The first determining module is used to determine the target region where the target object is located in the first image; wherein, the first image is a single frame image in a video sequence;

[0050] The second determining module is used to capture a second image of the target area and determine the key human body points of the target object based on the second image.

[0051] The first calculation and determination module is used to calculate a first human angle score of the target object based on at least one set of specific key point pairs when there are at least one set of specific key point pairs of human body key points, and to determine the orientation of the target object in the first image based on the first human angle score; wherein each set of specific key point pairs includes a first specific key point and a second specific key point, and the first human body part corresponding to the first specific key point and the second human body part corresponding to the second specific key point are symmetrical human body parts in the human body structure.

[0052] The second calculation and determination module is used to calculate the second human body angle score of the target object based on at least one set of backup key point pairs when there is no specific key point pair and there is at least one set of backup key point pairs; and to determine the orientation of the target object in the first image based on the second human body angle score; wherein each set of backup key point pairs is different from the specific key point pair, each set of backup key points includes a first backup key point and a second backup key point, and the third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

[0053] Thirdly, embodiments of the present invention provide an electronic device, comprising: a processor and a memory, wherein the processor is configured to execute a target object orientation detection program stored in the memory to implement the target object orientation detection method described above.

[0054] Fourthly, embodiments of the present invention provide a storage medium, wherein the storage medium stores one or more programs, which can be executed by one or more processors to implement the above-described target object orientation detection method.

[0055] The present invention provides a method, apparatus, electronic device, and storage medium for detecting the orientation of a target object, comprising: determining a target region in a first image where the target object is located; capturing a second image of the target region; determining human body key points of the target object based on the second image; calculating a first human body angle score of the target object based on at least one set of specific key point pairs when at least one set of specific key point pairs exists; determining the orientation of the target object in the first image based on the first human body angle score; calculating a second human body angle score of the target object based on at least one set of backup key point pairs when no specific key point pairs exist and at least one set of backup key point pairs exists; and determining the orientation of the target object in the first image based on the second human body angle score. Based on a 2D keypoint detection algorithm for specific or backup keypoint pairs, this invention can quantitatively calculate human body angle scores. Based on these scores, the direction of the human body can be accurately determined. This not only avoids the problem of unclear label boundaries in deep learning classification algorithms but also improves detection accuracy. Furthermore, compared to data acquisition and annotation of 3D model algorithms, the data acquisition cost of this 2D keypoint detection algorithm is much lower, solving the technical problems of low accuracy and high cost in human body direction detection in related technologies. Attached Figure Description

[0056] Figure 1 This is a schematic diagram of the hardware environment for a target object orientation detection method provided in an embodiment of the present invention;

[0057] Figure 2 A flowchart illustrating a method for detecting the orientation of a target object according to an embodiment of the present invention;

[0058] Figure 3 This is a schematic diagram of the distribution of preset human body key points provided in an embodiment of the present invention;

[0059] Figure 4 A schematic diagram of the orientation of a target object provided in an embodiment of the present invention;

[0060] Figure 5 A block diagram illustrating an embodiment of a target object orientation detection device provided by this invention;

[0061] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] To facilitate understanding of the embodiments of the present invention, further explanations and descriptions will be provided below with reference to the accompanying drawings and specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0064] In this embodiment, the above-described target object orientation detection method can be applied to, for example... Figure 1 The hardware environment shown consists of terminal 101 and server 103. Figure 1 As shown, server 103 is connected to terminal 101 via a network and can be used to provide services (such as direction detection services) to the terminal or clients installed on the terminal. Database 105 can be set up on the server or independently of the server to provide data storage services for server 103. The network mentioned above includes, but is not limited to, wide area network, metropolitan area network or local area network. Terminal 101 includes, but is not limited to, PC, mobile phone, tablet computer, etc.

[0065] The target object orientation detection method in this embodiment can be executed by server 103, or it can be jointly executed by server 103 and terminal 101, such as... Figure 2 As shown, the method may include the following steps:

[0066] Step 201: Determine the target region where the target object is located in the first image;

[0067] The first image is a single frame image in a video sequence; specifically, the first image is a single frame image in a video sequence, which includes, but is not limited to, videos in various existing business fields, such as surveillance videos in the security field, recording videos of sports and fitness, videos of cultural and film works, etc., and the single frame image is an image including the target object obtained by extracting frames from the video sequence.

[0068] Step 202: Capture a second image of the target area and determine the key human body points of the target object based on the second image;

[0069] Step 203: If there is at least one set of specific key point pairs, calculate the first human angle score of the target object based on at least one set of specific key point pairs, and determine the orientation of the target object in the first image based on the first human angle score.

[0070] Each pair of specific key points includes a first specific key point and a second specific key point. The first human body part corresponding to the first specific key point and the second human body part corresponding to the second specific key point are symmetrical human body parts in the human body structure.

[0071] The specific key point pair is preset. Usually, two symmetrical key points on the upper body of the human body are set as the specific key point pair. In actual application, one or more specific key point pairs can be set. The number of specific key point pairs can be set according to actual needs and is not limited here.

[0072] When there is at least one specific key point pair on the human body, it indicates that the human body of the target object is relatively complete. The angle score of the human body facing the camera, i.e., the first human body angle score, can be calculated based on the specific key point pair. Based on the first human body angle score, the direction of the target object facing the camera in the first image can be accurately detected, which effectively improves the accuracy of human body direction detection.

[0073] Step 204: When there is no specific key point pair for human body key points and there is at least one set of backup key point pairs, calculate the second human body angle score of the target object based on at least one set of backup key point pairs, and determine the orientation of the target object in the first image based on the second human body angle score.

[0074] Each set of backup key point pairs is different from the specific key point pairs. Each set of backup key points includes a first backup key point and a second backup key point. The third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

[0075] The backup key point pair is preset. Usually, two symmetrical key points of the upper body are also set as backup key point pairs. The backup key point pairs are not the specific key point pairs set above. In actual application, one or more backup key point pairs can be set. The number of backup key point pairs can be set according to actual needs and is not limited here.

[0076] In practical applications, there may be cases where specific key point pairs are not detected. If a specific key point pair is not detected, the operation in step 203 is not performed. The strategy is changed to calculate the angle score of the target object facing the camera, i.e., the second human body angle score, using the detected backup key point pairs. Based on the second human body angle score, the direction of the target object facing the camera in the first image can be accurately detected, which effectively improves the accuracy of human body direction detection.

[0077] When neither a specific keypoint pair nor a backup keypoint pair exists for the human body, it indicates that the target object's human body is incomplete. Therefore, in this embodiment, when neither a specific keypoint pair nor a backup keypoint pair exists, a prompt message indicating that the human body is incomplete is generated. The generated prompt message can be sent to the display end of the execution entity of the direction detection method for display.

[0078] The target object orientation detection method provided in this invention, based on the human keypoint detection algorithm of specific keypoint pairs or backup keypoint pairs in 2D, can quantitatively calculate the human body angle score. Based on the human body angle score, the human body orientation can be accurately determined. This not only avoids the problem of unclear label boundaries in deep learning classification algorithms, but also improves the detection accuracy. Moreover, compared with the data acquisition and annotation of 3D model algorithms, the data acquisition cost of the 2D keypoint detection algorithm in this application is much lower, solving the technical problems of low accuracy and high cost of human body orientation detection in related technologies.

[0079] In some embodiments, step 201 above can be implemented through the following steps:

[0080] Step A1: Input the first image into the first neural network model to extract the positional features of the first image using the first neural network model, and obtain the first feature map of the first image;

[0081] To meet the input requirements of the first neural network model, the first image can be scaled to a fixed size and then input into the first neural network model. The first neural network model can be trained based on deep learning networks such as convolutional neural networks and recurrent neural networks. The first neural network model is used to extract the positional features of the human body region in the first image to obtain the first feature map.

[0082] Step A2: Generate multiple anchor boxes for each point on the first feature map, wherein each anchor box has a different size;

[0083] Each generated anchor frame has a different size and can be used to match human targets of different sizes. The number of anchor frames can be set according to actual needs and is not limited here.

[0084] Step A3: For each anchor frame, calculate the confidence that the target object exists within the anchor frame;

[0085] Step A4: Retain the first target anchor frame with a confidence level greater than or equal to the first threshold, and determine the area enclosed by the first target anchor frame as the target area where the target object is located.

[0086] Specifically, the presence of a human body in each anchor frame of the first feature map can be determined by calculating a score (confidence level) for each frame. Anchor frames containing a human body are first target anchor frames with a confidence level greater than or equal to a first threshold. The area enclosed by the first target anchor frame is the target region in the first image where the target object is located. The first threshold can be set according to actual needs and is not limited here.

[0087] To avoid duplicate detections when multiple anchor boxes hit the same object, a non-maximum suppression algorithm can be used to select the best one from multiple anchor boxes. Specifically, when there are multiple first target anchor boxes, the intersection-union ratio (CIU) of the multiple first target anchor boxes is determined; when the CIU is greater than or equal to a second threshold, the second target anchor box with the highest confidence is retained, and the area enclosed by the second target anchor box is determined as the target area where the target object is located.

[0088] First, the intersection-union ratio (IUR) of multiple first target anchor boxes is calculated. The IUR is used to identify which first target anchor boxes hit the same human body. Among them, the first target anchor boxes with an IUR greater than or equal to a second threshold are anchor boxes that hit the same human body. The second threshold can be set according to actual needs. Then, from the multiple first target anchor boxes that hit the same human body, the second target anchor box with the highest confidence is selected as the anchor box of the final human body region detection result, and the area enclosed by the second target anchor box is determined as the target region where the target object is located.

[0089] In this embodiment, in addition to using the anchor-frame-based method in step 201 above to determine the target area of ​​the target object, other target detection methods can also be used, and no limitation is made here.

[0090] In some embodiments, step 202 above can be implemented through the following steps:

[0091] Step B1: Input the second image into the second neural network model to extract human features from the second image using the second neural network model, and obtain the second feature map of the second image;

[0092] To meet the input requirements of the second neural network model, the second image can be scaled to a fixed size before being input into the model. The second neural network model can be trained using deep learning networks such as convolutional neural networks and recurrent neural networks. This model is used to extract human features of the target object in the second image, resulting in a second feature map.

[0093] Step B2: Convert the second feature map into multiple heatmaps, where each heatmap corresponds to a preset human body key point;

[0094] Step B3: Determine the thermal value of each point on each heat map, and sort each point on the heat map according to the thermal value;

[0095] like Figure 3 As shown, the actual position of multiple preset human body key points can be predicted. These preset human body key points include key points of the human body parts such as the top of the head, left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, left ankle, right knee, right ankle, left chest, right chest, and navel.

[0096] The number of heatmaps is the same as the number of preset human body key points and they correspond one-to-one. The heat value of each point on each heatmap is the probability value that the point belongs to the corresponding preset human body key point. This probability value is calculated by the second neural network model.

[0097] Step B4: Take a weighted average of the coordinates of the points before the target sorting position to obtain the coordinate information of the predicted human key point corresponding to each preset human key point. The human key point includes the coordinate information of the predicted human key point.

[0098] The target's ranking position can be among the top 2, top 3, etc., and can be set according to actual needs. By taking a weighted average of the coordinates of the points preceding the target's ranking position, the actual position of the predicted human keypoint corresponding to the preset human keypoint can be obtained.

[0099] Step B5: Draw the predicted human body key points in the second image according to the coordinate information.

[0100] The predicted human body key points are the human body key points of the target object determined based on the second image. The determined human body key points include one or more of the following: top of head, left ear, right ear, left eye, right eye, nose, left shoulder, right shoulder, left elbow, left wrist, right elbow, right wrist, left hip, right hip, left knee, left ankle, right knee, right ankle, left chest, right chest, and navel.

[0101] In this embodiment, in addition to the heatmap-based human key point detection method in step 202 above, other human key point detection methods can also be used, and no limitation is made here.

[0102] In some embodiments, the calculation of the first human body angle score of the target object based on at least one set of specific key points in step 203 above can be specifically implemented through the following steps:

[0103] Step C1: For each specific key pair, obtain the first lateral coordinate information of the first specific key point and the second lateral coordinate information of the second specific key point in the second image;

[0104] Typically, the lower left corner of the image is taken as the origin of the image coordinates of the second image. The first horizontal coordinate information is the horizontal distance of the first specific key point relative to the origin of the image coordinates, and the second horizontal coordinate information is the horizontal distance of the second specific key point relative to the origin of the image coordinates.

[0105] Step C2: Calculate the first score corresponding to a specific key pair based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image;

[0106] The horizontal width information is the actual horizontal width of the second image. This horizontal width information can be obtained by calculating the coordinates of the lower right corner of the second image and the origin of the image coordinate system. For example, if the coordinates of the lower right corner of the second image are (4, 0), then the horizontal width information is 4.

[0107] The first score can be calculated through steps D1 to D3:

[0108] Step D1: Determine the angle normalization score of the first key point based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image.

[0109] The specific process for calculating the normalized score of the first key point angle is as follows: first, the horizontal distance between the first specific key point and the second specific key point is calculated based on the first horizontal coordinate information and the second horizontal coordinate information; then, the calculated horizontal distance is divided by the horizontal width information to obtain the normalized score of the first key point angle.

[0110] In this embodiment, the left shoulder is taken as the first specific key point and the right shoulder as the second specific key point for illustration. The first horizontal coordinate information of the left shoulder is x1, the second horizontal coordinate information of the right shoulder is x2, and the horizontal width information of the second image is w. The angle normalization score of the first key point can be calculated by the following formula: |x1-x2| / w.

[0111] Step D2: Determine the front / back identification score based on the first and second horizontal coordinate information;

[0112] In this embodiment, the front / back view score is used to indicate the orientation of the target object towards the camera. If the front / back view score is positive, it indicates that the target object is facing the camera from the front or side. If the front / back view score is negative, it indicates that the target object is facing the camera from the back.

[0113] The specific process for determining the front / back identification score is as follows: if the first horizontal coordinate information is greater than the second horizontal coordinate information, the first preset value is determined as the front / back identification score; wherein, the first preset value is a positive value; or, if the first horizontal coordinate information is not greater than the second horizontal coordinate information, the second preset value is determined as the front / back identification score; wherein, the second preset value is a negative value.

[0114] The first and second preset values ​​mentioned above can be set according to actual needs, and are not limited here.

[0115] Step D3: Calculate the first score based on the normalized score of the first key point angle and the front / back judgment score.

[0116] The first score is obtained by multiplying the normalized score of the first key point angle by the score for judging front and back views.

[0117] Step C3: Weight at least one first score to obtain the first human angle score of the target object.

[0118] In this embodiment, if the human body key points at the left and right shoulders are set as a specific key point pair, and the human body key points at the left and right chests are set as a specific key point pair, if there is only one specific key point pair among the human body key points, only one first score can be calculated, then the first score can be directly used as the first human body angle score; or the first human body angle score can be obtained according to the weighted value set in advance for the specific key point pair and the first score, and the first human body angle score is the first score multiplied by the weighted value.

[0119] If the above two sets of specific keypoint pairs exist among the key points of the human body, then according to steps D1 to D3, a first score 1 and a first score 2 can be calculated for each set of specific keypoint pairs. If the weighted value of the specific keypoint pairs of the left and right shoulders is set to weighted value 1, and the weighted value of the specific keypoint pairs of the left and right chests is set to weighted value 2, then the first human body angle score is: (first score 1 multiplied by weighted value 1) + (first score 2 multiplied by weighted value 2). The values ​​of the weighted values ​​and the specific keypoint pairs can be set according to actual needs and are not limited here.

[0120] Through the above steps C1 to C3, the first human body angle score of the target object facing the lens can be quantitatively calculated, thereby accurately determining the direction of the target object in the first image based on the first human body angle score, which can effectively improve the accuracy of human body direction detection.

[0121] In some embodiments, determining the orientation of the target object in the first image based on the first human body angle score in step 203 above can be achieved through the following steps:

[0122] Step E1: Find the range of the first target score corresponding to the first human body angle score in the direction lookup table;

[0123] The direction lookup table stores multiple different directions and the corresponding score range for each direction. In this embodiment, directions are mainly divided into front, side, and back. For ease of understanding, as follows: Figure 4 As shown, different directions are illustrated. Based on the above description, Table 1 below is an example of a direction lookup table:

[0124]

[0125]

[0126] It should be noted that Table 1 only shows an example of the correspondence between direction and score range. The specific correspondence between direction and score range can be set according to actual needs, and is not limited here.

[0127] Step E2: Determine the first target direction corresponding to the first target score range as the direction of the target object in the first image.

[0128] If the first human body angle score is 0.3, referring to Table 1, the first human body angle score is in the range of [0.45, -0.45], then the first target direction corresponding to the first human body angle score is the side, and the direction of the target object in the first image can be determined to be the side.

[0129] In some embodiments, the calculation of the first human angle score of the target object based on at least one set of backup key points in step 204 above can be specifically implemented through the following steps:

[0130] Step F1: For each pair of backup key points, obtain the third lateral coordinate information of the first backup key point and the fourth lateral coordinate information of the second backup key point in the second image.

[0131] Similarly, the third horizontal coordinate information is the horizontal distance of the first backup key point relative to the image coordinate origin, and the fourth horizontal coordinate information is the horizontal distance of the second backup key point relative to the image coordinate origin.

[0132] Step F2: Calculate the second score corresponding to the backup key point pair based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image;

[0133] The specific implementation process of step F2 above is as follows: determine the normalized score of the second key point angle based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image; calculate the second score according to the normalized score of the second key point angle and the first preset value; wherein, the first preset value is a positive value.

[0134] The specific process for calculating the normalized score of the second key point angle is as follows: first, the horizontal axis distance between the first and second backup key points is calculated based on the third and fourth horizontal coordinate information; then, the calculated horizontal axis distance is divided by the horizontal width information to obtain the normalized score of the second key point angle.

[0135] In this embodiment, the left eye is taken as the first backup key point and the right eye as the second backup key point for illustration. The third horizontal coordinate information of the left eye is x3, the fourth horizontal coordinate information of the right eye is x4, and the horizontal width information of the second image is w. The angle normalization score of the second key point can be calculated by the following formula: |x3-x4| / w.

[0136] The first preset value is the positive front / back judgment score mentioned in the above embodiment. In this embodiment, the above-mentioned backup key point is only detected when the front or side of the target object's human body is facing the camera. Therefore, the first preset value is positive. Thus, the second score is obtained by multiplying the normalized score of the second key point angle by the first preset value.

[0137] Step F3 involves weighting at least one second score to obtain the second human angle score of the target object.

[0138] In this embodiment, if the human body key points at the left and right eyes are set as a set of backup key point pairs, and the human body key points at the left and right ears are set as a set of backup key point pairs, if there is only one set of backup key point pairs among the human body key points, only one second score can be calculated, and the second score can be directly used as the second human body angle score; or the second human body angle score can be obtained according to the weighted value and the second score set in advance for the backup key point pairs, and the second human body angle score is the second score multiplied by the weighted value.

[0139] If the aforementioned two sets of backup keypoint pairs exist among the human body keypoints, then step F2 can calculate the second score 1 and the second score 2 for each backup keypoint pair. If the weighted value of the backup keypoint pair for the left and right eyes is set to weighted value 3, and the weighted value of the backup keypoint pair for the left and right ears is set to weighted value 4, then the second human body angle score is: (second score 1 multiplied by weighted value 3) + (second score 2 multiplied by weighted value 4). The weighted values ​​and backup keypoint pairs can be set according to actual needs and are not limited here.

[0140] When a specific key pair of human body key points is not available and the first human body angle score cannot be calculated to determine the direction of the target object in the first image, the second human body angle score of the target object facing the camera can be calculated based on the backup key point pair through the above steps F1 to F3. Thus, the direction of the target object in the first image can be accurately determined based on the second human body angle score, which can effectively improve the accuracy of human body direction detection.

[0141] In some embodiments, the determination of the direction of the target object in the first image based on the second human body angle score in step 204 can be achieved by the following steps: searching for the second target score range corresponding to the second human body angle score in the direction lookup table; wherein, the direction lookup table stores multiple different directions and the score range corresponding to each direction; and determining the second target direction corresponding to the second target score range as the direction of the target object in the first image.

[0142] The direction lookup table is the same as the direction lookup table described in the above embodiments. In this embodiment, the process of determining the direction of the target object in the first image based on the second human body angle score is the same as the process of determining the direction of the target object in the first image based on the first human body angle score, and will not be repeated here.

[0143] See Figure 5 This is a block diagram illustrating an embodiment of a target object orientation detection device provided by an embodiment of the present invention; as shown below. Figure 5 As shown, the device includes:

[0144] The first determining module 501 is used to determine the target region where the target object is located in the first image; wherein, the first image is a single frame image in a video sequence;

[0145] The second determining module 502 is used to capture a second image of the target area and determine the human body key points of the target object based on the second image.

[0146] The first calculation and determination module 503 is used to calculate a first human angle score of the target object based on at least one set of specific key point pairs when there are at least one set of specific key point pairs of human body key points, and to determine the orientation of the target object in the first image based on the first human angle score; wherein each set of specific key point pairs includes a first specific key point and a second specific key point, and the first human body part corresponding to the first specific key point and the second human body part corresponding to the second specific key point are symmetrical human body parts in the human body structure.

[0147] The second calculation determination 504 is used to calculate the second human body angle score of the target object based on at least one set of backup key point pairs when there is no specific key point pair and there is at least one set of backup key point pairs, and to determine the orientation of the target object in the first image based on the second human body angle score; wherein, each set of backup key point pairs is different from the specific key point pair, each set of backup key points includes a first backup key point and a second backup key point, and the third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

[0148] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 5 The illustrated electronic device 500 includes at least one processor 501, a memory 502, at least one network interface 504, and other user interfaces 503. The various components in the electronic device 500 are coupled together via a bus system 505. It is understood that the bus system 505 is used to implement communication between these components. In addition to a data bus, the bus system 505 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 6 The general designated all buses as Bus System 505.

[0149] The user interface 503 may include a display, keyboard, or clicking device (e.g., mouse, trackball, touchpad, or touchscreen).

[0150] It is understood that the memory 502 in the embodiments of the present invention can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0151] In some implementations, memory 502 stores elements, executable units or data structures, or subsets thereof, or extended sets thereof: operating system 5021 and application program 5022.

[0152] The operating system 5021 includes various system programs, such as the framework layer, core library layer, and driver layer, used to implement various basic business functions and handle hardware-based tasks. The application program 5022 includes various applications, such as a media player and a browser, used to implement various application functions. The program implementing the method of this embodiment can be included in the application program 5022.

[0153] In this embodiment of the invention, the processor 501 executes the method steps provided in each method embodiment by calling the program or instructions stored in the memory 502, specifically the program or instructions stored in the application program 5022.

[0154] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in the form of software. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of the present invention can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software units in the decoding processor. The software units may be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 502. Processor 501 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0155] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0156] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0157] The electronic device provided in this embodiment may be as follows: Figure 6 The electronic device shown can perform the following: Figure 2 All steps of the orientation detection method for the target object are implemented, thereby achieving... Figure 2 For details on the technical effectiveness of the orientation detection method for the target object shown, please refer to [link / reference needed]. Figure 2 The relevant descriptions are presented concisely and will not be elaborated upon here.

[0158] This invention also provides a storage medium (computer-readable storage medium). This storage medium stores one or more programs. The storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk, or solid-state drive; the memory may also include combinations of the above types of memory.

[0159] One or more programs in the storage medium can be executed by one or more processors to implement the above-mentioned orientation detection method for the target object.

[0160] The processor is used to execute the orientation detection program of the target object stored in the memory to implement the steps of the orientation detection method of the target object.

[0161] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0162] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0163] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting the orientation of a target object, characterized in that, The method includes: The target region where the target object is located is determined in the first image; wherein, the first image is a single frame image in a video sequence; A second image of the target region is captured, and key human body points of the target object are determined based on the second image; When at least one set of specific key point pairs exists, a first human body angle score of the target object is calculated based on the at least one set of specific key point pairs, and the orientation of the target object in the first image is determined according to the first human body angle score; wherein, each set of specific key point pairs includes a first specific key point and a second specific key point, and the first human body part corresponding to the first specific key point and the second human body part corresponding to the second specific key point are symmetrical human body parts in the human body structure; the first human body angle score is determined by the lateral coordinate information of the at least one set of specific key point pairs and the lateral width information of the second image; The step of calculating the first human angle score of the target object based on the at least one set of specific key point pairs includes: for each set of specific key point pairs, obtaining the first horizontal coordinate information of the first specific key point and the second horizontal coordinate information of the second specific key point in the second image; calculating the first score corresponding to the specific key point pair based on the first horizontal coordinate information, the second horizontal coordinate information and the horizontal width information of the second image; and weighting at least one of the first scores to obtain the first human angle score of the target object. The step of calculating the first score corresponding to the specific key pair based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image includes: determining the angle normalization score of the first key point based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image; determining the front / back identification score based on the first horizontal coordinate information and the second horizontal coordinate information; and calculating the first score based on the angle normalization score of the first key point and the front / back identification score. The step of determining the front / back identification score based on the first horizontal coordinate information and the second horizontal coordinate information includes: when the first horizontal coordinate information is greater than the second horizontal coordinate information, determining a first preset value as the front / back identification score; wherein the first preset value is a positive value; or, when the first horizontal coordinate information is not greater than the second horizontal coordinate information, determining a second preset value as the front / back identification score; wherein the second preset value is a negative value. The step of determining the direction of the target object in the first image based on the first human body angle score includes: searching for a first target score range corresponding to the first human body angle score in a direction lookup table; wherein, the direction lookup table stores multiple different directions and a score range corresponding to each direction; and determining the first target direction corresponding to the first target score range as the direction of the target object in the first image. When the specific key point pair does not exist and at least one set of backup key point pairs exists, a second human body angle score of the target object is calculated based on the at least one set of backup key point pairs, and the orientation of the target object in the first image is determined based on the second human body angle score; wherein, each set of backup key point pairs is different from the specific key point pair, and each set of backup key points includes a first backup key point and a second backup key point, and the third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

2. The method according to claim 1, characterized in that, Determining the target region where the target object is located in the first image includes: The first image is input into the first neural network model to extract positional features from the first image using the first neural network model, thereby obtaining the first feature map of the first image; For each point on the first feature map, generate multiple anchor boxes, where each anchor box has a different size; For each anchor frame, calculate the confidence level that the target object exists in the anchor frame; The first target anchor frame with a confidence level greater than or equal to the first threshold is retained, and the area enclosed by the first target anchor frame is determined as the target area where the target object is located.

3. The method according to claim 1, characterized in that, The step of determining the key human body points of the target object based on the second image includes: The second image is input into the second neural network model to extract human features from the second image using the second neural network model, thereby obtaining the second feature map of the second image; The second feature map is converted into multiple heat maps, where each heat map corresponds to a preset human body key point; In each of the heatmaps, a thermal value is determined for each point on the heatmap, and each point on the heatmap is sorted according to the thermal value. The coordinates of the points before the target sorting position are weighted and averaged to obtain the coordinate information of the predicted human key point corresponding to each preset human key point, wherein the human key point includes the coordinate information of the predicted human key point. The predicted human body key points are drawn in the second image according to the coordinate information.

4. The method according to claim 1, characterized in that, The calculation of the second human angle score of the target object based on the at least one set of backup key points includes: For each pair of backup key points, the third lateral coordinate information of the first backup key point and the fourth lateral coordinate information of the second backup key point are obtained in the second image. The second score corresponding to the backup key point pair is calculated based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image; The second human body angle score of the target object is obtained by weighting at least one of the second scores.

5. The method according to claim 4, characterized in that, The calculation of the second score corresponding to the backup keypoint pair based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image includes: The normalized score of the second key point angle is determined based on the third horizontal coordinate information, the fourth horizontal coordinate information, and the horizontal width information of the second image. The second score is calculated based on the normalized score of the second key point angle and the first preset value; wherein the first preset value is a positive value.

6. The method according to claim 1, characterized in that, Determining the orientation of the target object in the first image based on the second human body angle score includes: The second target score range corresponding to the second human body angle score is found in the direction lookup table; wherein, the direction lookup table stores multiple different directions and the score range corresponding to each direction; The second target direction corresponding to the second target score range is determined as the direction of the target object in the first image.

7. The method according to claim 1, characterized in that, The method further includes: If the specific key point pair does not exist at the key points of the human body, and the alternative key point pair does not exist, a prompt message indicating that the human body is incomplete is generated.

8. A target object orientation detection device, characterized in that, The device includes: The first determining module is used to determine the target region where the target object is located in the first image; wherein, the first image is a single frame image in a video sequence; The second determining module is used to capture a second image of the target area and determine the human body key points of the target object based on the second image. A first calculation and determination module is configured to, when at least one set of specific keypoint pairs exists, calculate a first human angle score of the target object based on the at least one set of specific keypoints, and determine the orientation of the target object in a first image based on the first human angle score; wherein each set of specific keypoint pairs includes a first specific keypoint and a second specific keypoint, and the first human body part corresponding to the first specific keypoint and the second human body part corresponding to the second specific keypoint are symmetrical human body parts in the human body structure; the first human angle score is determined by the lateral coordinate information of the at least one set of specific keypoint pairs and the lateral width information of the second image; The first calculation and determination module is further configured to: for each pair of specific key points, obtain the first horizontal coordinate information of the first specific key point and the second horizontal coordinate information of the second specific key point in the second image; calculate the first score corresponding to the specific key point based on the first horizontal coordinate information, the second horizontal coordinate information and the horizontal width information of the second image; and calculate the first human angle score of the target object by weighting at least one of the first scores. The first calculation and determination module is further configured to: determine a normalized score for the angle of a first key point based on the first horizontal coordinate information, the second horizontal coordinate information, and the horizontal width information of the second image; determine a front / back view judgment score based on the first horizontal coordinate information and the second horizontal coordinate information; and calculate a first score based on the normalized score for the angle of the first key point and the front / back view judgment score. The first calculation and determination module is further configured to: determine a first preset value as the front / back judgment score when the first horizontal coordinate information is greater than the second horizontal coordinate information; wherein the first preset value is a positive value; or, determine a second preset value as the front / back judgment score when the first horizontal coordinate information is not greater than the second horizontal coordinate information; wherein the second preset value is a negative value. The first calculation and determination module is further configured to: search for a first target score range corresponding to the first human body angle score in a direction lookup table; wherein, the direction lookup table stores multiple different directions and a score range corresponding to each direction; and determine the first target direction corresponding to the first target score range as the direction of the target object in the first image; The second calculation and determination module is used to calculate a second human body angle score of the target object based on the at least one set of backup key point pairs when the specific key point pair does not exist and at least one set of backup key point pairs exists, and to determine the orientation of the target object in the first image based on the second human body angle score; wherein each set of backup key point pairs is different from the specific key point pair, each set of backup key points includes a first backup key point and a second backup key point, and the third human body part corresponding to the first backup key point and the fourth human body part corresponding to the second backup key point are symmetrical human body parts in the human body structure.

9. An electronic device, characterized in that, include: A processor and a memory, the processor being configured to execute a target object orientation detection program stored in the memory to implement the target object orientation detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores one or more programs, which can be executed by one or more processors to implement the orientation detection method for the target object according to any one of claims 1 to 7.

Images