Pedestrian heading angle determination method and system, electronic device, and storage medium

Abstract

The application provides a heading angle determination method and system, electronic equipment and a storage medium, and relates to the field of image processing. The heading angle determination method comprises: extracting a pedestrian key point from an image containing a pedestrian; wherein the pedestrian key point comprises a first type of key point away from the ground and a second type of key point close to the ground; determining a target pedestrian position of a target pedestrian in a world coordinate system according to the second type of key point; and determining a heading angle of the target pedestrian according to the first type of key point and the target pedestrian position. The heading angle determination method provided by the embodiment of the application can combine the pedestrian feature and the pedestrian trajectory to accurately calculate the heading angle of the pedestrian, and greatly overcome the problem of inaccurate calculation and untimely updating of the heading angle caused by the use of a single motion trajectory to calculate the heading angle of the pedestrian.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to a method, system, electronic device, and storage medium for determining the heading angle of a pedestrian. Background Technology

[0002] Roadside perception utilizes sensors such as cameras, millimeter-wave radar, and lidar, combined with roadside edge computing, to achieve intelligent perception of traffic participants and road conditions within a specific area. Roadside perception can be applied to the digitalization of smart cities or to blind spot detection on vehicles. In practical applications, obtaining the accurate heading angle of detected targets is crucial. Accurate heading angles can make pedestrian visibility more realistic in smart city digitalization, or enable pedestrian trajectory prediction in vehicle-side blind spot detection.

[0003] Currently, roadside sensing localization typically involves detecting rectangular boxes in images and calculating the heading angle using only the motion trajectory. However, due to the variability of pedestrian motion, this method cannot accurately predict the heading angle of pedestrians during movement, making it difficult to accurately predict pedestrian trajectories. Summary of the Invention

[0004] The purpose of this application is to provide a method, system, electronic device and storage medium for determining pedestrian heading angle, which combines pedestrian features with pedestrian trajectory to accurately calculate pedestrian heading angle; it greatly overcomes the problem of inaccurate heading angle calculation caused by only using rectangular frame detection to calculate pedestrian heading angle.

[0005] In a first aspect, embodiments of this application provide a method for determining the heading angle of a pedestrian. The method includes: extracting pedestrian key points from an image containing a pedestrian; wherein the pedestrian key points include a first type of key points far from the ground and a second type of key points close to the ground; determining the position of the target pedestrian in a world coordinate system based on the second type of key points; determining the heading angle direction of the target pedestrian based on the first type of key points and the position of the target pedestrian; and calculating the angle value of the heading angle in the heading angle direction based on the heading angle direction and the position of the target pedestrian.

[0006] In the above implementation process, the heading angle determination method provided in this application extracts pedestrian key points from pedestrian images; further, the key points are divided into first-class key points and second-class key points, and the heading angle of the target pedestrian in the world coordinate system is obtained by processing the first-class key points and second-class key points. Therefore, it can be seen that the heading angle determination method provided in this application incorporates pedestrian features into the determination of the pedestrian heading angle, which can greatly overcome the problem of inaccurate heading angle calculation caused by only using bounding box detection to calculate the pedestrian heading angle.

[0007] Optionally, in this embodiment of the application, extracting pedestrian key points from an image containing pedestrians includes: labeling first-class and second-class key points of pedestrians in the target image to obtain a labeled image; training a key point extraction model based on the labeled image; and inputting the image containing pedestrians into the key point extraction model to extract pedestrian key points.

[0008] In the above implementation process, in order to extract the desired key points from pedestrian images, this application embodiment trains a key point extraction model. Based on the key point extraction model, key points in pedestrian images are extracted, which can accurately and quickly extract key points in pedestrian images, greatly improving the efficiency of determining heading angles.

[0009] Optionally, in this embodiment of the application, the second type of key points includes the left ankle point and the right ankle point of the target pedestrian; determining the target pedestrian's position in the world coordinate system based on the second type of key points includes: calculating the ankle center point of the left ankle point and the right ankle point of the target pedestrian in the image coordinate system; transforming the ankle center point from the image coordinate system to the world coordinate system according to the calibration relationship between the image coordinate system and the world coordinate system, so as to obtain the target pedestrian's position in the world coordinate system.

[0010] In the above implementation process, calculating the heading angle requires determining the position of the target pedestrian in the world coordinate system. By processing the second type of key points, such as the left ankle point and the right ankle point, the center point of the second type of key points in the image coordinate system is transformed into the world coordinate system, so as to accurately obtain the position of the target pedestrian in the world coordinate system, and thus accurately obtain the heading angle of the target pedestrian.

[0011] Optionally, in this embodiment, the first type of key points includes a left shoulder point and a right shoulder point; determining the heading angle direction of the target pedestrian based on the first type of key points and the target pedestrian's position includes: determining the Z coordinate of the target pedestrian's position as the reference Z coordinate in the world coordinate system; transforming the Z coordinates of the left shoulder point and the right shoulder point to the world coordinate system, and transforming the Z coordinates of the left shoulder point and the right shoulder point in the world coordinate system to the reference Z coordinates to obtain the transformed left shoulder point and the transformed right shoulder point; calculating the shoulder point distance between the transformed left shoulder point and the transformed right shoulder point; and determining the heading angle direction of the target pedestrian based on the shoulder point distance.

[0012] In the above implementation process, in order to determine the heading angle direction of the target pedestrian, the left and right shoulder points are transformed using the reference Z coordinates to obtain the transformed left and right shoulder points in the world coordinate system. Furthermore, the shoulder point distance is calculated in the world coordinate system, and the heading angle direction of the target pedestrian is determined based on the calculated shoulder point distance. Therefore, it can be seen that using the heading angle determination method provided in this application embodiment, the heading angle direction of the target pedestrian can be accurately determined based on the first and second types of key points, thereby improving the accuracy of the target pedestrian's heading angle.

[0013] Optionally, in this embodiment of the application, determining the heading angle direction of the target pedestrian based on the shoulder point distance includes: determining whether the shoulder point distance is greater than a preset distance; if the shoulder point distance is greater than the preset distance, determining whether the target pedestrian is moving towards or away from the calibration camera; obtaining the positional relationship between the left shoulder point and the right shoulder point in the image coordinate system; and determining the heading angle direction of the target pedestrian based on the positional relationship.

[0014] In the above implementation process, when the distance between the shoulder points is greater than a preset distance, the target pedestrian is moving towards or away from the calibration camera. To further distinguish whether the target pedestrian is moving towards or away from the camera, it is necessary to obtain the positional relationship between the left and right shoulder points; that is, the heading angle determination method provided in this application determines whether the target pedestrian is moving towards or away from the calibration camera based on the preset distance, and then determines the heading angle direction based on the shoulder point positional relationship; it can accurately distinguish between the two situations of the target pedestrian moving towards or away from the calibration camera, thereby accurately obtaining the heading angle of the target pedestrian.

[0015] Optionally, in this embodiment of the application, determining the heading angle direction of the target pedestrian based on the positional relationship includes: if it is determined that the left shoulder point is to the left of the right shoulder point, then the opposite direction of the calibration camera is taken as the heading angle direction of the target pedestrian; if it is determined that the left shoulder point is to the right of the right shoulder point, then the direction of the calibration camera is taken as the heading angle direction of the target pedestrian.

[0016] In the above implementation process, after determining that the target pedestrian is moving towards or away from the calibration camera, the specific direction of movement of the target pedestrian is determined by the positional relationship between the left and right shoulder points. If the left shoulder point is to the left of the right shoulder point, the direction opposite to the calibration camera is taken as the heading angle direction of the target pedestrian; if the left shoulder point is to the right of the right shoulder point, the direction opposite to the calibration camera is taken as the heading angle direction of the target pedestrian. Using the heading angle determination method provided in this application embodiment, the direction of movement of the target pedestrian can be accurately distinguished based on the markings in the key point extraction process, thereby improving the accuracy of the heading angle of the target pedestrian.

[0017] Optionally, in this embodiment of the application, determining the heading angle direction of the target pedestrian based on the shoulder point distance further includes: if it is determined that the shoulder point distance is less than or equal to a preset distance, then the target pedestrian is determined to be laterally calibrated by a camera, and the direction of the laterally calibrated camera is taken as the heading angle direction of the target pedestrian.

[0018] In the above implementation process, apart from the two cases where the target pedestrian moves towards or away from the calibration camera, there is also the case where the target pedestrian moves towards the calibration camera. That is, when the distance between the shoulder points is less than the preset distance, it can be determined that the target pedestrian is moving towards the calibration camera. Then, the direction of the lateral calibration camera is used as the heading angle direction of the target pedestrian, thereby accurately determining other movement cases besides the two cases of moving towards or away from the calibration camera.

[0019] Optionally, in this embodiment of the application, the angle value of the heading angle is calculated in the heading angle direction based on the heading angle direction and the target pedestrian position, including: obtaining the target pedestrian position of two adjacent frames of pedestrian images; calculating the slope of the straight line where the coordinates of the target pedestrian position of the two adjacent frames of pedestrian images are located in the heading angle direction in the world coordinate system, and taking the angle value corresponding to the slope as the angle value of the heading angle of the target pedestrian.

[0020] In the above implementation process, after determining the direction of the heading angle, it is necessary to further calculate the magnitude of the heading angle. In this embodiment, the heading angle of the target pedestrian is calculated by using the target pedestrian's position points in two adjacent frames of images. Therefore, it can be seen that the heading angle determination method provided in this embodiment can determine the heading angle of the target pedestrian by combining pedestrian characteristics and pedestrian trajectory, and can accurately reflect the movement direction of the target pedestrian.

[0021] Secondly, embodiments of this application provide a pedestrian heading angle determination system, the system comprising: a key point extraction module, a target pedestrian position acquisition module, and a heading angle determination module; the key point extraction module is used to extract pedestrian key points from a pedestrian image; wherein, the pedestrian key points include a first type of key points far from the ground and a second type of key points close to the ground; the target pedestrian position acquisition module is used to determine the target pedestrian position in the world coordinate system based on the second type of key points; the heading angle determination module is used to determine the heading angle direction of the target pedestrian based on the first type of key points and the target pedestrian position; the heading angle determination module is also used to calculate the angle value of the heading angle in the heading angle direction based on the heading angle direction and the target pedestrian position.

[0022] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores program instructions, and when the processor reads and runs the program instructions, it executes the steps in the implementation of the first aspect described above.

[0023] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in the implementation of the first aspect described above. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A flowchart for determining the pedestrian heading angle provided in this application embodiment;

[0026] Figure 2 A flowchart for pedestrian key point extraction provided in this application embodiment;

[0027] Figure 3 A flowchart for determining the location of a target pedestrian provided in this application embodiment;

[0028] Figure 4 A first flowchart for determining the heading angle direction of a target pedestrian provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram illustrating the extraction of key points from embodiments of this application;

[0030] Figure 6 A second flowchart for determining the heading angle direction of a target pedestrian provided in this application embodiment;

[0031] Figure 7 A third flowchart for determining the heading angle of a target pedestrian provided in an embodiment of this application;

[0032] Figure 8 A flowchart for calculating the heading angle value provided in the embodiments of this application;

[0033] Figure 9 The pedestrian heading angle determination system provided in the embodiments of this application

[0034] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. For example, the flowcharts and block diagrams in the drawings illustrate the architecture, functions, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0036] During the research process, the applicant discovered that current roadside sensing localization typically relies on bounding box detection in images. This involves using a pre-defined transformation relationship between the ground and the image in a world coordinate system to convert the center point of the bottom edge of the bounding box in the image to the corresponding ground point in the world coordinate system, and then using this point to mark the target's location. However, when pedestrian occlusion is severe, inaccurate detection of the bottom edge of the bounding box can lead to excessively large localization errors.

[0037] On the other hand, the current calculation of pedestrian heading angle is often based on pedestrian trajectory; however, since pedestrian activities are usually very flexible and unpredictable, the calculated heading angle may not be the pedestrian's current heading angle, and cannot be updated in a timely manner.

[0038] Based on this, embodiments of this application provide a heading angle determination method. This heading angle determination method combines the pedestrian characteristics and movement trajectory of the target pedestrian to calculate the movement trajectory of the target pedestrian, which can achieve accurate positioning of the target pedestrian and accurate calculation of the pedestrian's heading angle.

[0039] Please refer to Figure 1 , Figure 1 A flowchart for determining the pedestrian heading angle is provided for an embodiment of this application; the embodiment of this application includes the following steps:

[0040] Step S100: Extract pedestrian key points from the image containing pedestrians.

[0041] In step S100 above, in order to calculate the pedestrian heading angle of the target pedestrian, pedestrian key points are first extracted from the pedestrian image; wherein, pedestrian key points include two types of key points, namely, the first type of key points and the second type of key points; the first type of key points are key points far away from the ground, and the second type of key points are key points close to the ground; both the first type of key points and the second type of key points represent pedestrian features, and those skilled in the art will understand that key points can also be understood as feature points.

[0042] Step S101: Determine the position of the target pedestrian in the world coordinate system based on the second type of key points.

[0043] In step S101 above, the position of the target pedestrian in the world coordinate system is determined based on the second type of key points near the ground. This position is called the target pedestrian position. That is, the position of the target pedestrian in the world coordinate system is determined based on the second type of key points in the image coordinate system. The transformation relationship between the image coordinate system and the world coordinate system involved is something that can be performed by those skilled in the art, and will not be elaborated here.

[0044] Step S102: Determine the heading angle of the target pedestrian based on the first type of key points and the position of the target pedestrian.

[0045] In step S102 above, the direction of the target pedestrian's heading angle in the world coordinate system is determined based on the position of the target pedestrian in the world coordinate system obtained from the second type of key points in the image coordinate system and the first type of key points in the image coordinate system.

[0046] Step S103: Calculate the angle value of the heading angle in the heading angle direction based on the heading angle direction and the position of the target pedestrian.

[0047] In step S103 above, after determining the heading angle direction of the target pedestrian in the world coordinate system, the heading angle angle value is calculated in the heading angle direction based on the heading angle direction and the position of the target pedestrian; thus determining the heading angle of the target pedestrian.

[0048] pass Figure 1 As can be seen, the heading angle determination method provided in this application extracts pedestrian key points from pedestrian images; further, the key points are divided into first-class key points and second-class key points, and the heading angle of the target pedestrian in the world coordinate system is obtained by processing the first-class key points and second-class key points. Therefore, the heading angle determination method provided in this application incorporates pedestrian features into the determination of the pedestrian heading angle, which can largely overcome the problem of inaccurate heading angle calculation caused by using only bounding box detection to calculate the pedestrian heading angle.

[0049] Please refer to Figure 2 , Figure 2A flowchart for pedestrian key point extraction provided in this application embodiment; in an optional embodiment of this application, extracting pedestrian key points from a pedestrian image includes the following steps:

[0050] Step S200: Label the first and second type key points of the pedestrian in the target image to obtain the labeled image.

[0051] In step S200 above, one or more target images including pedestrians are selected, and the first key point and the second type of key point in the target image are labeled to obtain a labeled image.

[0052] Step S201: Train a key point extraction model based on the labeled images.

[0053] In step S201 above, a key point extraction model is trained based on the labeled image; for example, the labeled image is trained using a model based on a deep learning method; wherein, the deep learning method includes, but is not limited to, methods such as OpenPose.

[0054] Step S202: Input the image containing pedestrians into the key point extraction model to extract pedestrian key points.

[0055] In step S202 above, after the key point extraction model is trained, the key point extraction model can be used to extract key points from the image; that is, several pedestrian images of this application embodiment are input into the trained key point extraction model to extract key points from the pedestrian images.

[0056] pass Figure 2 As can be seen, in order to extract the desired key points from pedestrian images, this application embodiment trains a key point extraction model. Based on the key point extraction model, key points in pedestrian images are extracted, which can accurately and quickly achieve the extraction of key points in pedestrian images, and greatly improve the efficiency of determining the heading angle.

[0057] Please refer to Figure 3 , Figure 3 This is a flowchart illustrating the target pedestrian location determination process provided in this application embodiment. It should be noted that both the first and second type of key points in this application embodiment represent pedestrian features; the principle for extracting the second type of points is that they are close to the ground and can reflect the current position of the target pedestrian, such as the left ankle point and right ankle point, the left toe point and right toe point, the left knee point and right knee point, etc.

[0058] Based on the principle of relatively easy extraction, the processing procedure for the second type of key points is illustrated using the left ankle point and the right ankle point as examples. In an optional embodiment of this application, determining the target pedestrian's position in the world coordinate system based on the second type of key points can be achieved in the following way:

[0059] Step S300: In the image coordinate system, calculate the center point of the left and right ankles of the target pedestrian.

[0060] In step S300 above, the center points of the left and right ankles are calculated in the image coordinate system. Regardless of the state of the target pedestrian's left and right feet, the center points of the left and right ankles can accurately reflect the current position of the target pedestrian.

[0061] Step S301: Based on the calibration relationship between the image coordinate system and the world coordinate system, transform the ankle center point from the image coordinate system to the world coordinate system to obtain the target pedestrian's position in the world coordinate system.

[0062] In step S301 above, the center point of the ankle is transformed from the image coordinate system to the world coordinate system, thereby obtaining the position of the center point of the ankle in the world coordinate system. This position is the position of the pedestrian in the world coordinate system, that is, the position of the target pedestrian.

[0063] It should be noted that the reason for performing coordinate transformation in this application embodiment is that, in the image coordinate system, the distance between two points closer to the camera appears to be greater than the distance between two points farther from the camera, often referred to as "nearer is larger and farther is smaller," which cannot reflect the true size relationship; therefore, it is necessary to transform the coordinates in the image coordinate system to the world coordinate system, and obtain the true size relationship based on the relationship between the image coordinate system and the world coordinate system.

[0064] pass Figure 3 As can be seen, calculating the heading angle requires determining the position of the target pedestrian in the world coordinate system. By processing the second type of key points, such as the left ankle point and the right ankle point, the center point of the second type of key points in the image coordinate system is transformed into the world coordinate system, so as to accurately obtain the position of the target pedestrian in the world coordinate system, and thus accurately obtain the heading angle of the target pedestrian.

[0065] Please refer to Figure 4 , Figure 4 A first flowchart for determining the heading angle direction of a target pedestrian provided in this application embodiment; please refer to the following: Figure 5 , Figure 5This is a schematic diagram of key point extraction in an embodiment of this application. It should be noted that the first type of key points and the second type of key points in this embodiment both represent pedestrian features. Among them, the first type of key points are key points far from the ground, such as left eye point and right eye point, left shoulder point and right shoulder point, left ear point and right ear point, etc. For the first type of key points, the extraction principle is that they are easy to extract and their position on the human body will not change due to human body movements. For example, left hand point and right hand point do not meet the extraction requirements of the first type of key points. The reason is that when a person's hands are crossed, the left hand point and right hand point may overlap or exchange positions.

[0066] Based on the principle of relatively easy extraction, this application's embodiments use the left and right shoulder points as examples to illustrate the processing process for the first type of key points; and use the left and right ankle points as examples to illustrate the processing process for the second type of key points. Figure 5 As shown, points A and B are the left and right shoulder points, and points C and D are the left and right ankle points.

[0067] In an optional embodiment of this application, determining the heading angle of the target pedestrian based on the first type of key points and the target pedestrian's position includes the following steps:

[0068] Step S400: In the world coordinate system, determine the Z coordinate of the target pedestrian's position as the reference Z coordinate.

[0069] Step S401: Transform the Z coordinates of the left and right shoulder points into the reference Z coordinates and then transform them into the world coordinate system. Also, transform the Z coordinates of the left and right shoulder points in the world coordinate system into the reference Z coordinates to obtain the transformed left and right shoulder points.

[0070] In step S401 above, in the world coordinate system, based on the reference Z-coordinate of the target pedestrian's position, the Z-coordinates of the left and right shoulder points are transformed to the world coordinate system, and the Z-coordinates of the left and right shoulder points in the world coordinate system are transformed to the reference Z-coordinates. This process can be understood as first transforming the coordinates of the left and right shoulder points from the image coordinate system to the world coordinate system, and then making the Z-coordinates of the left and right shoulder points after transformation to the world coordinate system consistent with the reference Z-coordinates. It can be understood that at this point, the transformed left and right shoulder points are located on the xoy plane of the world coordinate system.

[0071] It should be noted that the most commonly used world coordinate system is the standard three-dimensional Cartesian coordinate system; the pedestrian heading angle determination method provided in this application uses the standard three-dimensional Cartesian coordinate system as the world coordinate system.

[0072] Step S402: Calculate the shoulder point distance between the transformed left shoulder point and the transformed right shoulder point.

[0073] In step S402 above, after transforming the left shoulder point and the right shoulder point, the transformed left shoulder point and the transformed right shoulder point are obtained; further, the distance between the transformed left shoulder point and the transformed right shoulder point is calculated in the world coordinate system.

[0074] Step S403: Determine the heading angle direction of the target pedestrian based on the shoulder point distance.

[0075] In step S403 above, after determining the shoulder point distance between the left and right shoulder points in the world coordinate system, the heading angle of the target pedestrian is determined based on the shoulder point distance.

[0076] pass Figure 4 As can be seen, in order to determine the heading angle direction of the target pedestrian, the left and right shoulder points are transformed using the reference Z coordinates to obtain the transformed left and right shoulder points in the world coordinate system. Furthermore, the shoulder point distance is calculated in the world coordinate system, and the heading angle direction of the target pedestrian is determined based on the calculated shoulder point distance. Therefore, it can be seen that using the heading angle determination method provided in this application embodiment, the heading angle direction of the target pedestrian can be accurately determined based on the first and second types of key points, thereby improving the accuracy of the target pedestrian's heading angle.

[0077] Please refer to Figure 6 , Figure 6 A second flowchart for determining the heading angle direction of a target pedestrian provided in this application embodiment; in an optional embodiment of this application, determining the heading angle direction of a target pedestrian based on shoulder point distance can be achieved through the following steps:

[0078] Step S500: Determine whether the distance to the shoulder point is greater than the preset distance.

[0079] In step S500 above, after calculating the shoulder point distance in the world coordinate system, it is determined whether the shoulder point distance is greater than a preset distance. It should be noted that the preset distance can be used to determine whether the target pedestrian is facing the camera directly or sideways; for example, the preset distance can be any value between 15cm and 25cm, and the specific value can be determined based on actual experience, such as setting the preset distance to 20cm, using 20cm as the benchmark for judging the target pedestrian's posture.

[0080] Step S501: If the distance to the shoulder point is determined to be greater than the preset distance, then the target pedestrian is determined to be moving towards or away from the calibration camera.

[0081] In step S501 above, if it is determined that the calculated shoulder point distance is greater than the preset distance, then the target pedestrian moves toward or away from the calibration camera.

[0082] Step S502: In the image coordinate system, obtain the positional relationship between the left shoulder point and the right shoulder point.

[0083] In step S502 above, if it is necessary to further distinguish whether the target pedestrian is moving towards the camera or away from the camera, it is necessary to obtain the positional relationship between the left shoulder point and the right shoulder point in the image coordinate system. It should be understood that when extracting key points, the shoulder points will be clearly marked. Therefore, there may be cases where the left shoulder point is to the left of the right shoulder point or to the right of the right shoulder point. The positional relationship between the left shoulder point and the right shoulder point can be obtained based on the markings during key point extraction.

[0084] Those skilled in the art will understand that in a coordinate system that includes pedestrians, the results of left and right shoulder recognition in an image may be the opposite of the pedestrian's own left and right sides. For example, when the target pedestrian moves to the side away from the calibration camera, the recognition result in the image will be that the left shoulder point is to the right of the right shoulder point. When the target pedestrian moves to the side towards the calibration camera, the key point recognition result in the image will also be that the left shoulder point is to the right of the right shoulder point. Therefore, the orientation of the target pedestrian cannot be determined solely by the positional relationship between the left and right shoulder points.

[0085] Step S503: Determine the heading angle direction of the target pedestrian based on the positional relationship.

[0086] In step S503 above, after obtaining the positional relationship between the left shoulder point and the right shoulder point, the heading angle of the target pedestrian is determined based on the positional relationship between the left shoulder point and the right shoulder point.

[0087] pass Figure 6 It is known that when the distance between the shoulder points is greater than a preset distance, the target pedestrian is moving towards or away from the calibration camera. To further distinguish whether the target pedestrian is moving towards or away from the camera, it is necessary to obtain the positional relationship between the left and right shoulder points; that is, the heading angle determination method provided in this application determines whether the target pedestrian is moving towards or away from the calibration camera by using a preset distance, and then determines the heading angle direction based on the positional relationship of the shoulder points; it can accurately distinguish between the two situations where the target pedestrian is moving towards or away from the calibration camera.

[0088] Please refer to Figure 7 , Figure 7 A third flowchart for determining the heading angle direction of a target pedestrian provided in this application embodiment; in optional embodiments of this application embodiment, in order to further determine whether the target pedestrian is moving towards or away from the calibration camera; based on the positional relationship, determining the heading angle direction of the target pedestrian may include the following steps:

[0089] Step S600: If it is determined that the left shoulder point is to the left of the right shoulder point, then the direction opposite to the calibration camera is taken as the heading angle direction of the target pedestrian.

[0090] Step S601: If it is determined that the left shoulder point is to the right of the right shoulder point, then the direction of the calibration camera is used as the heading angle direction of the target pedestrian.

[0091] In steps S600-S601 above, when the distance between the shoulder points is greater than a preset distance, there are two possible outcomes for determining the positional relationship between the left and right shoulder points: one is that the left shoulder point is to the left of the right shoulder point, and the other is that the left shoulder point is to the right of the right shoulder point. If the left shoulder point is to the left of the right shoulder point, it indicates that the target pedestrian is moving towards the calibration camera; if the left shoulder point is to the right of the right shoulder point, it indicates that the target pedestrian is moving away from the calibration camera. When the target pedestrian is moving towards the calibration camera, the opposite direction of the calibration camera is used as the heading angle direction of the target pedestrian; when the target pedestrian is moving away from the calibration camera, the direction of the calibration camera is used as the heading angle direction of the target pedestrian, thereby improving the accuracy of the target pedestrian's heading angle.

[0092] pass Figure 7 It can be seen that after determining that the target pedestrian is moving towards or away from the calibration camera, the specific direction of movement of the target pedestrian is determined by the positional relationship between the left and right shoulder points. If the left shoulder point is to the left of the right shoulder point, the direction opposite to the calibration camera is taken as the heading angle direction of the target pedestrian; if the left shoulder point is to the right of the right shoulder point, the direction opposite to the calibration camera is taken as the heading angle direction of the target pedestrian. Using the heading angle determination method provided in this application embodiment, the direction of movement of the target pedestrian can be accurately distinguished based on the markings in the key point extraction process.

[0093] In an optional implementation, determining the heading angle direction of the target pedestrian based on the shoulder point distance further includes: if the shoulder point distance is determined to be less than or equal to a preset distance, then determining the target pedestrian's lateral calibration camera and using the direction of the lateral calibration camera as the heading angle direction of the target pedestrian.

[0094] Therefore, it can be seen that, apart from the two cases where the target pedestrian moves towards or away from the calibration camera, there is also the case where the target pedestrian moves to the side of the calibration camera; that is, when the distance between the shoulder points is less than the preset distance, it can be determined that the target pedestrian is moving to the side of the calibration camera. Then, the direction of the side calibration camera is used as the heading angle direction of the target pedestrian, thereby accurately determining the other movement cases besides the two cases of moving towards or away from the calibration camera.

[0095] Please refer to Figure 8 , Figure 8 A flowchart for calculating the heading angle value provided in this application embodiment; in an optional embodiment of this application, the heading angle value is calculated in the heading angle direction based on the heading angle direction and the target pedestrian position, which can be achieved through the following steps:

[0096] Step S700: Obtain the location of the target pedestrian in two adjacent pedestrian images.

[0097] In step S700 above, the target pedestrian position corresponding to two adjacent frames of pedestrian images is obtained; it can be understood that, based on the above, the target pedestrian position is a point on the xoy plane in the world coordinate system; wherein, two adjacent frames refer to two consecutive adjacent frames of pedestrian images, or, two preset frames of pedestrian images, such as the current frame and the second frame after the current frame.

[0098] Step S701: In the heading angle direction in the world coordinate system, calculate the slope of the straight line containing the coordinates of the target pedestrian position in two adjacent pedestrian images, and use the angle value corresponding to the slope as the heading angle value of the target pedestrian.

[0099] In step S701 above, after obtaining the position coordinates of two target pedestrians in the world coordinate system, the slope of the straight line containing the position coordinates of the two target pedestrians is calculated, and the angle value corresponding to the slope is taken as the heading angle of the target pedestrian.

[0100] For example, if the coordinates of the two target pedestrians are (x1, y1) and (x2, y2) respectively, then the heading angle of the target pedestrians is Θ = arctan((y2-y1) / (x2-x1)).

[0101] pass Figure 8 As can be seen, after determining the direction of the heading angle, it is necessary to further calculate the magnitude of the heading angle. In this embodiment, the heading angle of the target pedestrian is calculated by using the target pedestrian's position points in two adjacent frames of images. Therefore, it can be seen that the heading angle determination method provided in this embodiment can determine the heading angle of the target pedestrian by combining pedestrian characteristics and pedestrian trajectory, and can accurately reflect the direction of movement of the target pedestrian.

[0102] Please refer to Figure 9 , Figure 9 The pedestrian heading angle determination system provided in this application embodiment includes a key point extraction module 110, a target pedestrian position acquisition module 120, and a heading angle determination module 130.

[0103] The key point extraction module 110 is used to extract pedestrian key points from pedestrian images; wherein, pedestrian key points include a first type of key points far from the ground and a second type of key points close to the ground.

[0104] The target pedestrian location acquisition module 120 is used to determine the target pedestrian's location in the world coordinate system based on the second type of key points.

[0105] The heading angle determination module 130 is used to determine the heading angle direction of the target pedestrian based on the first type of key point and the position of the target pedestrian; the heading angle determination module 130 is also used to calculate the angle value of the heading angle in the heading angle direction based on the heading angle direction and the position of the target pedestrian.

[0106] In an optional implementation, the key point extraction module 110 extracts pedestrian key points from the pedestrian image, including: the key point extraction module 110 annotates the first type of key points and the second type of key points of the pedestrian in the target image to obtain an annotated image; the key point extraction module 110 trains a key point extraction model based on the annotated image; and inputs the pedestrian image into the key point extraction model to extract pedestrian key points.

[0107] In an optional embodiment, the second type of key points includes the left ankle point and the right ankle point of the target pedestrian; based on the second type of key points, the target pedestrian position acquisition module 120 includes a center point calculation module 121; the target pedestrian position acquisition module 120 determines the target pedestrian position in the world coordinate system, including: the center point calculation module 121 calculates the ankle center point of the left ankle point and the right ankle point in the image coordinate system; the target pedestrian position acquisition module 120 transforms the ankle center point from the image coordinate system to the world coordinate system according to the calibration relationship between the image coordinate system and the world coordinate system, so as to obtain the target pedestrian position in the world coordinate system.

[0108] In an optional embodiment, the first type of key points includes a left shoulder point and a right shoulder point; the heading angle determination module 130 includes a reference coordinate determination module 131, a shoulder point distance calculation module 132, and a heading angle direction determination module 133; the heading angle determination module 130 determines the heading angle direction of the target pedestrian based on the first type of key points and the target pedestrian's position by: the reference coordinate determination module 131 obtaining the reference Z coordinate of the target pedestrian's position in the world coordinate system; the shoulder point distance calculation module 132 transforming the Z coordinates of the left and right shoulder points into the reference Z coordinates to obtain the transformed left and right shoulder points; the shoulder point distance calculation module 132 calculating the shoulder point distance between the transformed left and right shoulder points; and the heading angle direction determination module 133 determining the heading angle direction of the target pedestrian based on the shoulder point distance.

[0109] In an optional embodiment, the heading angle direction determination module 133 includes a distance judgment module 1331 and a shoulder point position acquisition module 1332; the heading angle direction determination module 133 determines the heading angle direction of the target pedestrian based on the shoulder point distance, including: the distance judgment module 1331 judging whether the shoulder point distance is greater than a preset distance; if the shoulder point distance is greater than the preset distance, then judging whether the target pedestrian is moving towards or away from the calibration camera; the shoulder point position acquisition module 1332 acquiring the positional relationship between the left shoulder point and the right shoulder point in the image coordinate system; the heading angle direction determination module 133 determines the heading angle direction of the target pedestrian based on the positional relationship.

[0110] In an optional implementation, the heading angle direction determination module 133 determines the heading angle direction of the target pedestrian based on the positional relationship, including: if the heading angle direction determination module 133 determines that the left shoulder point is to the left of the right shoulder point, then the opposite direction of the calibration camera is taken as the heading angle direction of the target pedestrian; if the heading angle direction determination module 133 determines that the left shoulder point is to the right of the right shoulder point, then the direction of the calibration camera is taken as the heading angle direction of the target pedestrian.

[0111] In an optional embodiment, the heading angle direction determination module 133 determines the heading angle direction of the target pedestrian based on the shoulder point distance, and further includes: if the distance judgment module 1331 determines that the shoulder point distance is less than or equal to a preset distance, then the target pedestrian is determined to be laterally calibrated by the camera, and the heading angle direction determination module 133 takes the direction of the laterally calibrated camera as the heading angle direction of the target pedestrian.

[0112] In an optional embodiment, the heading angle determination module 130 further includes a heading angle calculation module 134; the heading angle determination module 130 calculates the angle value of the heading angle in the heading angle direction according to the heading angle direction and the target pedestrian position, including: the heading angle calculation module 134 obtains the target pedestrian position of two adjacent frames of pedestrian images; the heading angle calculation module 134 calculates the slope of the straight line where the coordinates of the target pedestrian position of the two adjacent frames of pedestrian images are located in the heading angle direction in the world coordinate system, and uses the angle value corresponding to the slope as the angle value of the heading angle of the target pedestrian.

[0113] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. An electronic device 300 provided in this application includes: a processor 301 and a memory 302. The memory 302 stores machine-readable instructions executable by the processor 301. When the machine-readable instructions are executed by the processor 301, the method described above is performed.

[0114] Based on the same inventive concept, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the steps in any of the above implementations.

[0115] The computer-readable storage medium can be any medium capable of storing program code, such as Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM). The storage medium stores the program, and the processor executes the program after receiving an execution instruction. The method executed by the electronic terminal as defined in any embodiment of this invention can be applied to the processor or implemented by the processor.

[0116] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0117] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0118] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0119] It can be replaced and can be implemented, wholly or partially, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, wholly or partially, in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated.

[0120] The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0121] In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, without necessarily requiring or implying any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0122] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for determining a pedestrian's heading angle, characterized in that, The method includes: From an image containing pedestrians, pedestrian key points are extracted; wherein, the pedestrian key points include a first type of key points far from the ground and a second type of key points close to the ground; Based on the second type of key points, determine the position of the target pedestrian in the world coordinate system; Based on the first type of key points and the location of the target pedestrian, determine the heading angle of the target pedestrian; Calculate the angle value of the heading angle in the heading angle direction based on the heading angle direction and the position of the target pedestrian; The first type of key points includes a left shoulder point and a right shoulder point; determining the heading angle direction of the target pedestrian based on the first type of key points and the target pedestrian's position includes: In the world coordinate system, the Z coordinate of the target pedestrian's position is determined as the reference Z coordinate; Transform the left shoulder point and the right shoulder point to the world coordinate system, and transform the Z coordinates of the left shoulder point and the right shoulder point in the world coordinate system to the reference Z coordinates to obtain the transformed left shoulder point and the transformed right shoulder point; Calculate the shoulder point distance between the transformed left shoulder point and the transformed right shoulder point; Based on the shoulder point distance, determine the heading angle direction of the target pedestrian; Determining the heading angle of the target pedestrian based on the shoulder point distance includes: Determine whether the distance between the shoulder points is greater than a preset distance; If the distance between the shoulder points is determined to be greater than the preset distance, then the target pedestrian is determined to be moving towards or away from the calibration camera; in the image coordinate system, the positional relationship between the left and right shoulder points is obtained; based on the positional relationship, the heading angle of the target pedestrian is determined; If the distance to the shoulder point is determined to be less than or equal to the preset distance, then the target pedestrian is determined to be laterally facing the calibration camera, and the direction laterally facing the calibration camera is taken as the heading angle direction of the target pedestrian; Determining the heading angle of the target pedestrian based on the positional relationship includes: If it is determined that the left shoulder point is to the left of the right shoulder point, then the opposite direction of the calibration camera is taken as the heading angle direction of the target pedestrian; If it is determined that the left shoulder point is to the right of the right shoulder point, then the direction of the calibration camera is taken as the heading angle direction of the target pedestrian.

2. The method according to claim 1, characterized in that, The step of extracting pedestrian key points from an image containing pedestrians includes: Label the first type of key points and the second type of key points of pedestrians in the target image to obtain a labeled image; Based on the labeled images, a key point extraction model is trained; The image containing pedestrians is input into the key point extraction model to extract the pedestrian key points.

3. The method according to claim 1, characterized in that, in, The second type of key points includes the left and right ankle points of the target pedestrian; determining the target pedestrian's position in the world coordinate system based on the second type of key points includes: In the image coordinate system, calculate the center point of the left ankle and the right ankle of the target pedestrian; Based on the calibration relationship between the image coordinate system and the world coordinate system, the center point of the ankle is transformed from the image coordinate system to the world coordinate system to obtain the position of the target pedestrian in the world coordinate system.

4. The method according to claim 1, characterized in that, The step of calculating the heading angle value in the heading angle direction based on the heading angle direction and the target pedestrian position includes: Obtain the location of the target pedestrian in two adjacent frames of pedestrian images; In the heading angle direction in the world coordinate system, the slope of the straight line containing the coordinates of the target pedestrian's position in the two adjacent frames of the pedestrian image is calculated, and the angle value corresponding to the slope is taken as the heading angle value of the target pedestrian.

5. A pedestrian heading angle determination system, characterized in that, The system includes: a key point extraction module, a target pedestrian location acquisition module, and a heading angle determination module; The key point extraction module is used to extract pedestrian key points from an image containing pedestrians; wherein, the pedestrian key points include a first type of key points far from the ground and a second type of key points close to the ground; The target pedestrian location acquisition module is used to determine the target pedestrian's location in the world coordinate system based on the second type of key points; The heading angle determination module is used to determine the heading angle direction of the target pedestrian based on the first type of key points and the position of the target pedestrian; The heading angle determination module is also used to calculate the angle value of the heading angle in the heading angle direction based on the heading angle direction and the position of the target pedestrian; The first type of key points includes the left shoulder point and the right shoulder point; the heading angle determination module includes a reference coordinate determination module, a shoulder point distance calculation module, and a heading angle direction determination module. The reference coordinate determination module determines the Z coordinate of the target pedestrian's position as the reference Z coordinate in the world coordinate system; The shoulder point distance calculation module transforms the left shoulder point and the right shoulder point into the world coordinate system, and transforms the Z coordinates of the left shoulder point and the right shoulder point in the world coordinate system into the reference Z coordinates to obtain the transformed left shoulder point and the transformed right shoulder point; and calculates the shoulder point distance between the transformed left shoulder point and the transformed right shoulder point. The heading angle direction determination module determines the heading angle direction of the target pedestrian based on the shoulder point distance; The heading angle direction determination module includes a distance judgment module and a shoulder point position acquisition module; The distance determination module determines whether the distance to the shoulder point is greater than a preset distance; if the distance to the shoulder point is greater than the preset distance, it determines whether the target pedestrian is moving towards or away from the calibration camera. The shoulder point position acquisition module obtains the positional relationship between the left shoulder point and the right shoulder point in the image coordinate system; The heading angle direction determination module determines the heading angle direction of the target pedestrian based on the positional relationship; If the distance judgment module determines that the shoulder point distance is less than or equal to the preset distance, it determines that the target pedestrian is facing the calibration camera, and the heading angle direction determination module takes the direction facing the calibration camera as the heading angle direction of the target pedestrian. If the heading angle direction determination module determines that the left shoulder point is to the left of the right shoulder point, then the opposite direction of the calibration camera is taken as the heading angle direction of the target pedestrian; if it determines that the left shoulder point is to the right of the right shoulder point, then the direction of the calibration camera is taken as the heading angle direction of the target pedestrian.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, perform the steps of the method according to any one of claims 1-4.

Images