Long-distance sight line estimation method and device based on human skeleton key points

Abstract

The present invention discloses a method and device for estimating long-distance sight lines based on human skeleton key points, the method comprising: separating pedestrians in an image to be tested and cutting out a human bounding box image; inputting the human bounding box image into a pre-trained human key point detection network model to obtain the position coordinates of multiple human key points in the human bounding box image, the human key points at least including the left eye, right eye, left ear, right ear, nose, left shoulder and right shoulder; obtaining an initial human face orientation angle according to the position coordinates of the multiple human key points; and obtaining the coordinates of the sight line estimation point using the initial human face orientation angle. The present invention can satisfactorily identify and estimate the sight lines of pedestrians at a distance in real scenes and game scenes.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a long-distance sight line estimation method and device based on key points of human skeleton. Background Art

[0002] Humans can quickly and easily understand and interpret head orientation and movement, and use this important behavioral language to quickly infer the intentions of others nearby. As society continues to develop, there are more and more scenes with multiple people, and more and more data is recorded by machines. In order to solve the safety problems caused by a large number of images or videos, the ability to quickly infer the line of sight of pedestrians in images or videos is becoming more and more important.

[0003] At present, according to the characteristics of gaze estimation, gaze detection methods can be roughly divided into two types. One gaze detection method is based on facial key points. This type of method performs alignment by establishing a correspondence between key points and a 3D head model, and restores the 3D posture of the head. Its accuracy depends on whether enough facial key points can be detected and the quality of the corresponding 3D head model. The other is a gaze detection method based on head features. This type of method extracts relevant head texture features for analysis to achieve the purpose of gaze detection, such as extracting and detecting head features of various posture angles left by a CNN (Convolutional Neural Network) model. The accuracy of this type of method depends on the head features extracted by the network. Head pose estimation is intrinsically related to gaze estimation, that is, the ability to characterize the direction of a person's eye gaze focus. In itself, head pose estimation only provides a rough characterization when the person's eyes are not visible (e.g., low resolution, occlusion). In the context of computer vision, head pose estimation is most often interpreted as the ability to infer the direction of a person's head relative to the camera. It is generally believed that the human head can be modeled as a non-substantial rigid object. Under this assumption, the posture of the human head is limited to three degrees of freedom, which are generally pitch, roll, and yaw. Under the deep learning method, a multi-loss network is used to predict the head posture angle directly from the image features. The network has a loss for each angle, that is, three separate losses, each of which has two components: posture classification and regression. After training, the network can directly predict the head posture angle after inputting image features, which can be visualized as head posture estimation.

[0004] However, when environmental factors and video quality factors change, the accuracy of both facial key points and head features will decrease. For example, at a long distance and low resolution, the face appears blurred, and the first method cannot effectively detect the face. The number of key points is seriously insufficient, and it cannot be aligned with the 3D head model, and thus cannot detect the focus of the head's line of sight. On the other hand, the 3D head model is customized, and when the detected key points are seriously inconsistent with the 3D head model, the accuracy of the head line of sight estimation will also be reduced. Facial features are seriously insufficient, and it is impossible to estimate the head posture of pedestrians in long-distance and low-resolution scenes, and it is impossible to distinguish the features at each posture angle, because some features of the head will disappear in long-distance and low-resolution situations, resulting in a low overall accuracy. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method and device for long-distance line of sight estimation based on key points of human skeleton. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0006] One aspect of the present invention provides a long-distance sight line estimation method based on human skeleton key points, comprising:

[0007] S1: Separate the pedestrians in the image to be tested and crop the human body bounding box image;

[0008] S2: inputting the human body bounding box image into a pre-trained human body key point detection network model to obtain position coordinates of a plurality of human body key points in the human body bounding box image, wherein the human body key points at least include a left eye, a right eye, a left ear, a right ear, a nose, a left shoulder and a right shoulder;

[0009] S3: Obtaining an initial human face orientation angle according to the position coordinates of the plurality of human key points;

[0010] S4: using the initial human face orientation angle to obtain the estimated sight point coordinates.

[0011] In one embodiment of the present invention, the S1 includes:

[0012] A human detector is used to separate all pedestrians in the image to be tested, a human bounding box image of each pedestrian is cropped, and the upper left corner coordinates and the lower right corner coordinates of the human bounding box image are obtained.

[0013] In one embodiment of the present invention, S3 includes:

[0014] S31: extracting the position coordinates of key points of the human body, including: left eye (x1, y1), right eye (x2, y2), left ear (x3, y3), right ear (x4, y4), nose (x5, y5), left shoulder (x6, y6) and right shoulder (x7, y7);

[0015] S32: Selecting a coordinate origin (x0, y0) in the human body boundary frame image according to the position coordinates of the human body key points;

[0016] S33: Using the coordinate origin (x0, y0), obtain the initial human face orientation angle:

[0017] head angel1 =arctan(y0 / x0).

[0018] In one embodiment of the present invention, the S32 includes:

[0019] S321: Calculate the distance l between the left ear and the left eye dist , the distance from the right ear to the right eye r dist And the midpoint coordinates ms between the left shoulder and the right shoulder;

[0020] S322: Judgement l dist and r dist The size of the coordinate system is selected and the origin is (x0, y0). If l dist >r dist , then x0=ms x -x5,y0=y3-y5, if l dist <r dist , then x0=ms x -x5,y0=y4-y5, where ms x The horizontal coordinate representing the midpoint between the left shoulder and the right shoulder.

[0021] In one embodiment of the present invention, the S4 includes:

[0022] S41: dividing the turning range of the human face into multiple angle ranges;

[0023] S42: Determine the initial human face orientation angle head angel1 The angle range it falls into, and the median value of the current angle range is selected as the final human face heading angle head angel2 ;

[0024] S43: Obtaining the estimated coordinates of the human body's sight point according to the selected final human body face orientation angle.

[0025] In one embodiment of the present invention, the S43 includes:

[0026] According to the selected final human face angle head angle2 And the visual line length L set in the visualization, get the relative coordinates (x8, y8) of the estimated line of sight point with the nose as the origin, the calculation formula is:

[0027]

[0028] The coordinates (x, y) of the estimated point of sight are calculated based on the relative coordinates (x8, y8):

[0029] x=x5+x8,y=y5+y8.

[0030] Another aspect of the present invention provides a long-distance sight line estimation device based on human skeleton key points, comprising:

[0031] Human detector, used to separate pedestrians in the image to be tested and crop the human bounding box image;

[0032] A human key point detection module, used for inputting the human body bounding box image into a pre-trained human body key point detection network model to obtain position coordinates of a plurality of human body key points in the human body bounding box image, wherein the human body key points at least include a left eye, a right eye, a left ear, a right ear, a nose, a left shoulder and a right shoulder;

[0033] A facial orientation angle acquisition module, used to obtain an initial human facial orientation angle according to the position coordinates of the plurality of human key points;

[0034] The long-distance sight line estimation module is used to obtain the coordinates of the sight line estimated point using the initial human face orientation angle.

[0035] In one embodiment of the present invention, the human key point detection module includes a pre-trained human key point detection network model.

[0036] In one embodiment of the present invention, the facial orientation angle acquisition module is specifically used to:

[0037] Extracting the position coordinates of key points of the human body; selecting a coordinate origin in the human body boundary frame image according to the position coordinates of the key points of the human body; and obtaining an initial human face orientation angle using the coordinate origin.

[0038] In one embodiment of the present invention, the long-distance sight line estimation module is specifically used for:

[0039] The human face turning range is equally divided into a plurality of angle ranges; the angle range into which the initial human face orientation angle falls is determined, and the median of the current angle range is selected as the final human face orientation angle; and the estimated landing point coordinates of the human body's line of sight are obtained according to the selected final human face orientation angle.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The present invention is a long-distance sight line estimation device based on the key points of the human skeleton. Due to the addition of the operation of the key points of the skeleton, it is possible to analyze pedestrians at a distance. For example, through different head postures, not only can the movement trajectory of the pedestrians be inferred, but also because of the direction of their head, the attention of the pedestrians to different areas can be judged, and the sight lines of the long-distance pedestrians can be satisfactorily identified and estimated in real scenes and game scenes. Secondly, due to the use of the pre-trained human key point detection network model, the execution process of the method does not involve the training of the network model, has the advantages of low computing resource consumption and applicability to various computers, and the test video can achieve real-time effects.

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flow chart of a long-distance sight line estimation method based on human skeleton key points provided by an embodiment of the present invention;

[0044] Figure 2 is a schematic diagram of separating all pedestrians in an image to be detected by using a human body detector provided by an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a cropped human body bounding box image provided by an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of key points of a human skeleton provided by an embodiment of the present invention;

[0047] Figure 5 It is a schematic diagram of detecting key points of human skeleton on a human body boundary frame image provided by an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of a human face orientation angle calculation process provided by an embodiment of the present invention;

[0049] Figure 7 is a schematic diagram of a sight focus calculation process provided by an embodiment of the present invention;

[0050] Figure 8 It is a module schematic diagram of a long-distance sight line estimation device based on human skeleton key points provided by an embodiment of the present invention;

[0051] Fig. 9 It is a violin diagram of the three-part video results provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a long-distance line of sight estimation method and device based on human skeleton key points proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0053] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0054] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0055] Embodiment 1

[0056] See also Figure 1 , Figure 1 yes Figure 1 : is a flow chart of a long-distance sight line estimation method based on human skeleton key points provided by an embodiment of the present invention. As shown in the figure, the long-distance sight line estimation method of this embodiment includes:

[0057] S1: Separate the pedestrians in the image to be tested and crop the human body bounding box image.

[0058] Specifically, a human detector is used to separate all pedestrians in the image to be tested, and a human bounding box image of each pedestrian and the upper left corner coordinates and lower right corner coordinates of the human bounding box image are obtained, such as Figure 2 As shown in FIG. 1 , the upper left corner coordinates and the lower right corner coordinates can be used to define the uppermost boundary, the leftmost boundary, the lowermost boundary and the rightmost boundary of the human body bounding box image. Subsequently, the upper left corner coordinates and the lower right corner coordinates of the human body bounding box image are used to crop the human body bounding box image of each pedestrian from the human body bounding box image, as shown in FIG. Figure 3 It should be noted that the position coordinates in this embodiment all represent the pixel position of the point.

[0059] S2: Input the human body bounding box image into a pre-trained human body key point detection network model to obtain the position coordinates of multiple human body key points in the human body bounding box image, where the human body key points include at least a left eye, a right eye, a left ear, a right ear, a nose, a left shoulder and a right shoulder.

[0060] In this embodiment, the human key point detection network model uses the Alpha pose network. The Alpha pose network is an existing neural network used for multi-person posture recognition. It can predict the position of human key points. Each human key point will predict multiple groups. Each group has a score. The one with the highest score will be selected as the final output result of the human key point position coordinates. The specific process will not be repeated here. It should be noted that the human key point detection network model will output the following: Figure 4 The 17 human skeleton key points shown include: left and right eyes, nose, left and right ears, left and right shoulders, left and right elbows, left and right hands, left and right hips, left and right knees and left and right ankles. The human key points used in the embodiment of the present invention include left eye, right eye, left ear, right ear, nose, left shoulder and right shoulder. For the situation that some human key points are unclear or blocked, the human key point detection network model can also estimate the position coordinates of other required key points according to the position and coordinates of the known key points. In other embodiments, other detection networks that can obtain human key points can also be used for processing, which is not limited here.

[0061] S3: Obtaining an initial human face orientation angle according to the position coordinates of the plurality of human body key points.

[0062] In this embodiment, step S3 specifically includes:

[0063] S31: extracting the position coordinates of the human body key points in the current human body bounding box image from the human body key point detection network model, including: left eye (x1, y1), right eye (x2, y2), left ear (x3, y3), right ear (x4, y4), nose (x5, y5), left shoulder (x6, y6) and right shoulder (x7, y7);

[0064] S32: Selecting a coordinate origin (x0, y0) in the human body boundary frame image according to the position coordinates of the human body key points.

[0065] Specifically, the S32 includes:

[0066] S321: Calculate the distance l between the left ear and the left eye dist , the distance from the right ear to the right eye r dist And the midpoint coordinates ms of the left shoulder and the right shoulder:

[0067] ldist =(x1-x3) 2 +(y1-y3) 2

[0068] r dist =(x2-x4) 2 +(y2-y4) 2

[0069]

[0070] S322: Judgement l dist and r dist The size of the coordinate system is selected and the coordinate origin (x0, y0) is selected. The coordinates of the midpoints of the left and right shoulders and the coordinates of the nose are used to select x0. The distance between the ear coordinates and the eye coordinates is used to select y0. If l dist >r dist , then x0=ms x -x5,y0=y3-y5, if l dist <r dist , then x0=ms x -x5,y0=y4-y5, where ms x Represents the horizontal coordinate of the midpoint of the left shoulder and the right shoulder, thereby obtaining the coordinate origin O(x0,y0), such as Figure 6 shown.

[0071] S33: Using the coordinate origin (x0, y0), obtain the initial human face orientation angle:

[0072] head angel1 =arctan(y0 / x0).

[0073] S4: using the initial human face orientation angle to obtain the estimated sight point coordinates.

[0074] In this embodiment, S4 includes:

[0075] S41: Divide the turning range of the human face into multiple angle ranges.

[0076] Specifically, the human head turning range is equally divided into 12 intervals, that is, within 360°, it is divided every 30°: [-180, -150], [-150, -120], [-120, -90], [-90, -60], [-60, -30], [-30, 0], [0, 30], [30, 60], [60, 90], [90, 120], [120, 150] and [150, 180], and then the landing point position is determined according to whether the initial human face orientation angle falls within the range.

[0077] S42: Determine the initial human face orientation angle head angel1 The angle range it falls into, and the median value of the current angle range is selected as the final human face heading angle head angel2 ;

[0078] If the pedestrian in the image to be tested is at a long distance, there will be an influence due to the irregular rotation of the pedestrian's head. If the initial human face orientation angle head calculated above is directly used angel1 , which will cause the orientation angle between video frames to change too much and be detrimental to visual perception. In addition, very high accuracy is not required for pedestrians at a long distance. Therefore, this embodiment adopts the median method to adjust the orientation angle of the human face.

[0079] Specifically, after dividing the sight range, the two-dimensional spatial coordinates of the specific sight focus are obtained. If the initial human face is facing the angle head angel1 If it falls into a certain sight range, the middle angle of this range is selected as the final human face heading angle head angel2 For example, if the initial human face orientation angle head calculated in step S33 angel1 =12, it falls within the range of [0,30] degrees, and the middle angle within the range of [0,30] degrees, i.e. 15°, is selected as the head orientation angle.

[0080] S43: Obtaining the estimated coordinates of the human body's sight point according to the selected final human body face orientation angle.

[0081] According to the selected final human face angle head angle2 And the visual line length L in the setting, get the relative coordinates (x8, y8) of the estimated line of sight point with the nose as the origin, such as Figure 7 As shown, the calculation formula is:

[0082]

[0083] Preferably, L = 20. Then, the coordinates (x, y) of the estimated sight point are calculated according to the relative coordinates (x8, y8):

[0084] x=x5+x8,y=y5+y8.

[0085] The present embodiment is a long-distance sight line estimation device based on human skeleton key points. Due to the addition of skeleton key point operations, it is possible to analyze long-distance pedestrians. For example, different head postures can not only infer the movement trajectory of pedestrians, but also judge the pedestrian's attention to different areas because of their head orientation, and satisfactorily identify and estimate the sight lines of long-distance pedestrians in real scenes and game scenes. Secondly, since a pre-trained human key point detection network model is used, the execution process of the method does not involve the training of the network model, has the advantages of low computing resource consumption and applicability to various computers, and the test video can achieve real-time effects.

[0086] Embodiment 2

[0087] Based on the above embodiment, this embodiment provides a long-distance sight line estimation device based on human skeleton key points, such as Figure 8 As shown, the long-distance sight line estimation device of this embodiment includes a human detector 1, a human key point detection module 2, a facial orientation angle acquisition module 3 and a long-distance sight line estimation module 4. The human detector 1 is used to separate pedestrians in the image to be tested and crop out a human bounding box image. Specifically, the human detector 1 can separate all pedestrians in the image to be tested, obtain a human bounding box image of each pedestrian and the upper left corner coordinates and lower right corner coordinates of the human bounding box image, and then use the upper left corner coordinates and lower right corner coordinates of the human bounding box image to crop out the human bounding box image of each pedestrian from the human bounding box image.

[0088] The human key point detection module 2 is used to input the human body boundary frame image into a pre-trained human body key point detection network model to obtain the position coordinates of multiple human body key points in the human body boundary frame image, wherein the human body key points at least include the left eye, the right eye, the left ear, the right ear, the nose, the left shoulder and the right shoulder. The human body key point detection module includes a pre-trained human body key point detection network model. In this embodiment, the human body key point detection network model uses the Alpha pose network.

[0089] The facial orientation angle acquisition module 3 is used to obtain the initial human facial orientation angle according to the position coordinates of the multiple human key points.

[0090] Furthermore, the facial orientation angle acquisition module 3 is specifically used to: extract the position coordinates of the key points of the human body; select a coordinate origin in the human body boundary frame image according to the position coordinates of the key points of the human body; and obtain an initial human facial orientation angle using the coordinate origin.

[0091] The long-distance sight line estimation module 4 is used to obtain the coordinates of the estimated sight line point using the initial human face orientation angle. Further, the long-distance sight line estimation module 4 is specifically used to: divide the human face turning range into multiple angle ranges; determine the angle range in which the initial human face orientation angle falls, and select the median of the current angle range as the final human face orientation angle; obtain the coordinates of the estimated sight line point of the human body according to the selected final human face orientation angle.

[0092] The long-distance sight line estimation method based on human skeleton key points provided by the experimental example of the present invention is verified and explained through simulation experiments as follows.

[0093] (1) Simulation conditions

[0094] In order to verify the effect of the above method, this embodiment obtains data sets of multiple perspectives and multiple recording environments, where the multiple perspectives include: horizontal view and overhead view, the multiple recording environments include: handheld camera (i.e., the camera moves with the person), vehicle-mounted camera (i.e., the camera moves with the vehicle) and fixed-position camera, and the multiple scenes include: artificially collected data sets of real scenes and artificially collected data sets of game scenes. The data set includes M frames of video images, where M is a natural number greater than 0.

[0095] Pytorch1.7 was used for simulation, and the data set used open source data sets such as MOT17 in real scenes, some scene videos of MTA in in-game monitoring scenes, and surveillance perspective videos used in public news reports in TikTok.

[0096] The results of the simulation experiment of this embodiment are used in the form of a questionnaire to obtain the final performance results. Specifically, three parts of test video sources are used in the questionnaire, including: Douyin, MTA public data set, MOT public data set, each of which contains 5 short videos, and each video is about 10 seconds long. Evaluation index: 0-10 points are used to evaluate the performance of long-distance pedestrian line of sight estimation in each video. The higher the score, the better the performance of the long-distance pedestrian line of sight estimation of this method, and the higher the accuracy. The people who answered the questionnaire are all experts and researchers with scientific research backgrounds in this field.

[0097] Please see Table 1. The average scores of the three videos are 7.38, 7.12, and 7.51, with standard deviations of 1.27, 1.36, and 1.06, and variances of 1.59, 1.82, and 1.11.

[0098] Table 1 Scoring results of three parts of videos

[0099]

[0100] See also Fig. 9,This embodiment also visualizes these three parts of the video as a violin plot for analysis, and the results show that the median, confidence interval, and quartile range indicators all reach excellent indicators.

[0101] It can be seen from the experimental results that the long-distance line of sight estimation method based on human skeleton key points in this embodiment achieves satisfactory recognition performance for long-distance pedestrian line of sight estimation in both real scenes and game scenes.

[0102] In the several embodiments provided by the present invention, it should be understood that the apparatus and method disclosed by the present invention can be implemented in other ways. For example, the apparatus embodiments described above are only illustrative, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation, for example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0103] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of hardware plus software functional modules.

[0104] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A long-distance sight line estimation method based on human skeleton key points, characterized in that: include: S1: Separate the pedestrians in the image to be tested and crop the human body bounding box image; S2: inputting the human body bounding box image into a pre-trained human body key point detection network model to obtain position coordinates of a plurality of human body key points in the human body bounding box image, wherein the human body key points at least include a left eye, a right eye, a left ear, a right ear, a nose, a left shoulder and a right shoulder; S3: Obtaining an initial human face orientation angle according to the position coordinates of the plurality of human key points; S4: using the initial human face orientation angle to obtain the estimated sight point coordinates; The S4 includes: S41: dividing the turning range of the human face into multiple angle ranges; S42: Determine the initial human face orientation angle head angel1 The angle range it falls into, and the median value of the current angle range is selected as the final human face heading angle head angel2 ; S43: Obtaining the estimated landing point coordinates of the human body's sight line according to the selected final human body face orientation angle, The S43 includes: According to the selected final human face orientation angle head angle2 And the visual line length L set in the visualization, get the relative coordinates (x8, y8) of the estimated line of sight point with the nose as the origin, the calculation formula is: The coordinates (x, y) of the estimated point of sight are calculated based on the relative coordinates (x8, y8): x=x5+x8,y=y5+y8, Among them, (x5, y5) represents the position coordinates of the nose.

2. The long-distance sight line estimation method based on human skeleton key points according to claim 1, characterized in that: The S1 includes: A human detector is used to separate all pedestrians in the image to be tested, a human bounding box image of each pedestrian is cropped, and the upper left corner coordinates and the lower right corner coordinates of the human bounding box image are obtained.

3. The long-distance sight line estimation method based on human skeleton key points according to claim 1 is characterized in that: The S3 includes: S31: extracting the position coordinates of key points of the human body, including: left eye (x1, y1), right eye (x2, y2), left ear (x3, y3), right ear (x4, y4), nose (x5, y5), left shoulder (x6, y6) and right shoulder (x7, y7); S32: Selecting a coordinate origin (x0, y0) in the human body boundary frame image according to the position coordinates of the human body key points; S33: Using the coordinate origin (x0, y0), obtain the initial human face orientation angle: head angel1 =arctan(y0 / x0)。 4. The long-distance sight line estimation method based on human skeleton key points according to claim 3 is characterized in that: The S32 includes: S321: Calculate the distance l between the left ear and the left eye dist , the distance from the right ear to the right eye r dist And the midpoint coordinates ms between the left shoulder and the right shoulder; S322: Judgement l dist and r dist The size of the coordinate system is selected and the origin is (x0, y0). If l dist >r dist , then x0=ms x -x5,y0=y3-y5, if l dist <r dist , then x0=ms x -x5,y0=y4-y5, where ms x The horizontal coordinate representing the midpoint between the left shoulder and the right shoulder.

5. A long-distance sight line estimation device based on human skeleton key points, characterized in that: include: Human detector, used to separate pedestrians in the image to be tested and crop the human bounding box image; A human key point detection module, used for inputting the human body bounding box image into a pre-trained human body key point detection network model to obtain position coordinates of a plurality of human body key points in the human body bounding box image, wherein the human body key points at least include a left eye, a right eye, a left ear, a right ear, a nose, a left shoulder and a right shoulder; A facial orientation angle acquisition module, used to obtain an initial human facial orientation angle according to the position coordinates of the plurality of human key points; A long-distance sight line estimation module is used to obtain the coordinates of the sight line estimated point using the initial human face orientation angle, The long-distance sight line estimation module is specifically used for: Divide the turning range of the human face into multiple angle ranges; Determine the initial human face orientation angle head angel1 The angle range it falls into, and the median value of the current angle range is selected as the final human face heading angle head angel2 ; The estimated coordinates of the human body's sight point are obtained according to the selected final human body face orientation angle. Specifically, according to the selected final human body face orientation angle head angle2 And the visual line length L set in the visualization, get the relative coordinates (x8, y8) of the estimated line of sight point with the nose as the origin, the calculation formula is: The coordinates (x, y) of the estimated point of sight are calculated based on the relative coordinates (x8, y8): x=x5+x8,y=y5+y8, Among them, (x5, y5) represents the position coordinates of the nose.

6. The long-distance sight line estimation device based on human skeleton key points according to claim 5, characterized in that: The human key point detection module includes a pre-trained human key point detection network model.

7. The long-distance sight line estimation device based on human skeleton key points according to claim 6, characterized in that: The facial orientation angle acquisition module is specifically used for: Extracting the position coordinates of key points of the human body; selecting a coordinate origin in the human body boundary frame image according to the position coordinates of the key points of the human body; and obtaining an initial human face orientation angle using the coordinate origin.

Images