A Reversing Parking Method Based on Kinect Pose Recognition

Abstract

This invention discloses a reversing parking method based on Kinect pose recognition, comprising five steps: model building, data acquisition, preprocessing, pose calculation, and decision control. Model building involves pre-constructing a model of the driver to obtain their identity image and dominant hand. Data acquisition uses a Kinect camera to capture depth images of the driver. Preprocessing combines the pre-constructed model with real-time acquired data to identify the driver and determine their orientation. Pose calculation uses trigonometric functions to calculate the driver's skeletal coordinates, and the resulting angle information serves as the pose calculation result. Decision control analyzes the pose calculation result and drives the vehicle to complete the corresponding actions, ultimately completing the reversing parking maneuver. This invention allows drivers to control the vehicle from outside using different poses, solving the problem of drivers being unable to open car doors after reversing in narrow spaces, while also reducing the occurrence of scrapes and significantly improving the safety of reversing into parking spaces.

Description

Technical Field

[0001] This invention relates to the field of intelligent parking, and more particularly to a method for reversing into a parking space based on Kinect pose recognition. This method uses a binocular Kinect camera to perform pre-reconstruction of a model, then acquires images, determines the human pose based on the skeletal model, analyzes the determination results, and converts them into specific control commands to drive the vehicle to complete the reversing into the parking space. Background Technology

[0002] The rapid increase in the number of vehicles in cities has led to a shortage of parking spaces and their dense arrangement. Coupled with some improper parking practices, this has significantly increased the difficulty of parking. For many novice drivers, the primary challenge is how to quickly and safely reverse into a parking space. When operating the vehicle from inside, the driver's field of vision is limited, relying only on rearview mirrors and onboard radar to observe the surrounding environment. They also need to simultaneously control the accelerator, brakes, and steering wheel, making the operation difficult and time-consuming. Especially in confined parking spaces, scrapes and collisions are highly likely. After parking, the presence of adjacent vehicles and obstacles often prevents the car door from opening, requiring re-parking. This complex parking environment leads to economic losses, wasted energy, injuries, and arguments arising from collisions with adjacent vehicles.

[0003] The Kinect camera is a depth camera officially produced by Microsoft, capable of acquiring RGB and depth images. To address the aforementioned difficulty in reversing into a parking space, this invention proposes a reversing parking method based on Kinect pose recognition. Summary of the Invention

[0004] The purpose of this invention is to provide a reversing parking method based on Kinect pose recognition, addressing the shortcomings of existing technologies. This method comprises five modules: a model building module, a data acquisition module, a preprocessing module, a pose calculation module, and a decision control module. Specifically, the model building module pre-obtains the driver's model information; the data acquisition module, powered by the Kinect camera, acquires depth images within the observation range; the preprocessing module combines the model information to obtain processed skeletal coordinates; and the pose calculation and decision control modules, performed by the onboard system, calculate angle information based on the skeletal coordinates, determine the pose, and control the vehicle's movement to safely and quickly complete the reversing parking maneuver.

[0005] The specific technical solution for achieving the objective of this invention is as follows:

[0006] A method for reversing into a parking space based on Kinect pose recognition, characterized by the following steps:

[0007] Step 1: Model Construction: Initialize the Kinect camera. Obtain the driver's skeletal information using the depth image captured by the Kinect camera and the Kinect for Windows software development kit (SDK). Obtain the coordinates of the shoulder center joint, head joint, spine joint, left shoulder joint, and right shoulder joint. Each coordinate position is represented by (x, y, z). Calculate the driver's identity map for driver identification. Based on the coordinates of the shoulder center joint, left shoulder joint, and right shoulder joint, calculate the driver's dominant hand for determining the driver's orientation. To ensure accuracy, at least three calculations are required, and the dominant hand determination result must be consistent across all three calculations.

[0008] Step 2: Data Acquisition: Initialize the Kinect camera, set the camera elevation angle and placement distance, and the Kinect camera acquires a depth image including the driver;

[0009] Step 3: Preprocessing: Using the depth image captured by the Kinect camera and the Kinect SDK, obtain human skeletal data information, namely the coordinate values ​​of the joints of the human skeleton; based on the real-time collected skeletal data information, combined with the identity image from Step 1 and the dominant hand, identify the driver and match the orientation.

[0010] Step 4: Pose calculation: Select the left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint of the human body, and divide them into two groups of three points each. Since the driver is facing the camera, ignore the value in the X-axis direction, obtain the two-dimensional coordinates of the shoulder joint, elbow joint, and wrist joint, form a triangle, and combine trigonometric functions to calculate the angle.

[0011] Step 5: Decision Control: Based on the decision judgment rules, the position calculation results of the left and right hands are analyzed to obtain the position judgment results. Then, based on the decision control rules, decision control commands are issued to the vehicle to move forward, backward, turn left, and turn right, driving the vehicle to complete the reverse parking.

[0012] Preferably, the identity diagram is calculated as follows: taking the shoulder center joint as the center, calculate the distance from the shoulder center joint to the head joint, the distance from the shoulder center joint to the spine joint, the distance from the shoulder center joint to the left shoulder joint, and the distance from the shoulder center joint to the right shoulder joint, and obtain the proportional relationship of the four to serve as the driver's identity diagram; preferably, the dominant hand is calculated as follows: compare the distance from the shoulder center joint to the left shoulder joint with the distance from the shoulder center joint to the right shoulder joint, and determine whether the driver's dominant hand is left or right based on the relationship between the two. If the distance from the shoulder center joint to the left shoulder joint is larger, the dominant hand is considered to be the right hand, otherwise it is the left hand.

[0013] Preferably, the Kinect camera includes a color camera, an infrared projector, and a depth camera. The color camera captures color images within its field of view. The infrared projector actively projects near-infrared light, which is reflected off rough objects and captured by the depth camera. The depth image is obtained by calculating the time difference of light. The Kinect camera features skeletal tracking technology. The Kinect SDK provides a solid foundation for developers to create and deploy interactive programs that respond to human actions, gestures, and voice commands. Using the official Kinect SDK, the skeletal information of the human body can be correctly read, namely the coordinate positions of 20 preset joints, each represented by (x, y, z).

[0014] Preferably, the driver identification process is as follows: the coordinates of the real-time collected head joint, shoulder center joint, spine joint, left shoulder joint, and right shoulder joint are proportionally calculated and compared with the identity image in step 1. If they match, the driver's identity is considered verified. Preferably, the driver's orientation matching process is as follows: the orientation of the skeletal model is calibrated according to the dominant hand in step 1. If the dominant hand is right, the distance from the real-time collected shoulder center joint to the left shoulder joint must be greater than the distance from the shoulder center joint to the right shoulder joint. If the driver does not meet this judgment standard three times in a row, it is considered that all the received left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint need to be converted to left or right.

[0015] As a preferred method, pose calculation involves selecting six joints: the left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint. These are divided into two groups of three points each, and the angle at the elbow joint is calculated. Taking the angle of the left elbow joint as an example, the calculation method is as follows: ignoring the X-axis data, the two-dimensional coordinates of the left shoulder joint, left elbow joint, and left wrist joint are obtained. Then, the lengths of the sides between each pair are calculated, and the angle at the elbow joint is determined based on these lengths. To ensure calculation efficiency, the judgment frequency is set to twice per second.

[0016] As a preferred option, the decision-making rule is as follows: The calculated pose angle, i.e., the angle at the elbow joint, is analyzed. The analysis results correspond to "top," "upward," "downward," and "bottom." When the elbow joint angle is within the range of 0°-5°, the driver's pose is determined to be top; when the elbow joint angle is within the range of 20°-80°, the driver's pose is determined to be upward; when the elbow joint angle is within the range of 100°-160°, the driver's pose is determined to be downward; and when the elbow joint angle is within the range of 175°-180°, the driver's pose is determined to be bottom. (5°-20°...) 80°-100° and 160°-175° serve as invalid angle boundaries separating the effective angle range, ensuring the accuracy of the judgment results. As the preferred method, the decision control rules are as follows: both hands are at the top, corresponding to activating the posture recognition reversing parking function; both hands are upward, corresponding to the vehicle moving forward; both hands are downward, corresponding to the vehicle reversing; left hand upward and right hand downward, corresponding to the vehicle turning left; left hand downward and right hand upward, corresponding to the vehicle turning right; both hands are at the bottom, corresponding to successful reversing parking and deactivation of the posture recognition reversing parking function.

[0017] The data acquisition steps employed in this invention can acquire depth images in real time within a specified distance range at a low cost and with controllable expenses. The pose calculation steps, combined with the officially provided software development kit, obtain skeletal coordinate data of human joints in a simple manner. The model building steps, which pre-model driver information and combine with preprocessing steps, solve the authentication problem of the acquired data, ensuring data accuracy to a certain extent. The pose calculation steps are simple and have strong real-time performance. The decision control steps, which judge the pose calculation results to control vehicle movement, are simple and easy to understand. In summary, this invention provides an intuitive, simple, and low-cost method that enables drivers to safely and efficiently reverse their vehicles into parking spaces, solving the problem of narrow parking spaces and the inability to open car doors after reversing, while also preventing scratches to some extent. Attached Figure Description

[0018] Figure 1 A schematic diagram of a system for implementing the present invention;

[0019] Figure 2 A schematic diagram of the human skeletal nodes that Kinect can capture;

[0020] Figure 3 This is a schematic diagram of the pose calculation algorithm;

[0021] Figure 4 A system flowchart for implementing the method of the present invention. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the present invention are all common knowledge and general knowledge in the art, and the present invention does not have any particular limitations.

[0023] This invention provides a method for reversing into a parking space based on Kinect pose recognition. The invention pre-builds a model of the driver, and during the reversing into a parking space operation, uses a Microsoft Kinect camera to capture depth images of the driver. The Kinect SDK is then used to identify the driver's skeletal model, calculate the driver's pose, and analyze the calculation results to control the vehicle's movement.

[0024] See Figure 1 The hardware component involved in this invention is primarily a Kinect dual infrared camera, which connects directly to the vehicle system via a USB cable. The Kinect camera should face outwards from the vehicle with no obstructions in front, enabling it to capture the external scene and obtain good depth images, eliminating the influence of the environment on the acquired images. Simultaneously, it should be ensured that the Kinect camera does not interfere with the driver's normal activities inside the vehicle. The Kinect camera has a resolution of 1920*1080, a frame rate of 30fps, a detectable depth range of 0.5m to 4m, a recognition range of 70° horizontally and 60° vertically, and a depth error within 0.5% of the captured images.

[0025] After the device is installed, the driver stands in front of the Kinect camera, which then takes a picture of the driver. The Kinect SDK is used to obtain the driver's skeletal information. Kinect can generate a human skeleton model represented by 20 joints, with the position of each joint indicated by (x, y, z) coordinates. See also... Figure 2The system acquires the coordinates of the head joint, left shoulder joint, right shoulder joint, shoulder center joint, and spine joint. Using the shoulder center joint as the center, it calculates the distances from the shoulder center joint to the head joint, the shoulder center joint to the spine joint, the shoulder center joint to the left shoulder joint, and the shoulder center joint to the right shoulder joint, obtaining their proportional relationships. This information serves as the driver's identification map for driver identification. Additionally, the distances from the shoulder center joint to the left shoulder joint and to the right shoulder joint are compared. Based on their relative magnitudes, the driver's dominant hand (left or right) is determined, serving as a reference for body orientation. If the distance from the shoulder center joint to the left shoulder joint is larger, the dominant hand is considered right; otherwise, it is left. To ensure accuracy, measurements must be taken three or more times, and the dominant hand determination results must be consistent across three consecutive measurements. The final driver modeling information is generated. The specific information of the 3D model mainly includes the identity diagram, that is, the proportional relationship between the shoulder center joint point and the head, spine, left shoulder, and right shoulder joint points, as well as the dominant hand, that is, the size relationship between the shoulder center joint point and the left and right joint points.

[0026] In situations where parking spaces are narrow and obstructed, the driver maneuvers the vehicle to a position with ample room for movement. The driver then opens the car door, exits, and stands approximately 2 meters in front of the Kinect camera to perform pose manipulation. The Kinect camera captures a depth image of the driver. Using the Kinect SDK, a corresponding skeletal model is generated by comparing the image with a pre-stored human model, and the coordinates of the skeletal points are obtained. First, the coordinates of the head joint, shoulder center joint, spine joint, left shoulder joint, and right shoulder joint are proportionally calculated and compared with a pre-generated identity image. If they are roughly consistent, the driver's identity is confirmed. Next, the orientation of the skeletal model is calibrated using a pre-stored dominant hand. If the dominant hand is right-handed, the distance from the shoulder center joint to the left shoulder joint should be greater than the distance from the shoulder center joint to the right shoulder joint. If this criterion is not met three times consecutively, all received left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint are considered to need to be converted between left and right.

[0027] The pose calculation algorithm of this invention uses six joints: left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint. The poses to be identified are left hand up, left hand down, left hand at the top, left hand at the bottom, right hand up, right hand down, right hand at the top, and right hand at the bottom. The algorithms used for identification are conceptually consistent. Taking the left hand as an example, identifying whether the left hand is up or down requires the coordinate values ​​of three joints: left shoulder joint, left elbow joint, and left wrist joint. When the driver operates in front of the Kinect camera, the x-values ​​of the three joints are the same; therefore, the calculation only involves the y-values ​​and z-values ​​of the joint coordinates.

[0028] Let the three-dimensional coordinates of the shoulder joint be (x1, y1, z1), the elbow joint be (x2, y2, z2), and the wrist joint be (x3, y3, z3). Since the driver operates the Kinect camera, the x-coordinates of the three joints are approximately the same. Therefore, this triangle can be projected onto a two-dimensional coordinate system formed by the Y and Z axes. (See also...) Figure 3 After transformation, the coordinates are as follows: the shoulder joint coordinates are (y1, z1), the elbow joint coordinates are (y2, z2), and the wrist joint coordinates are (y3, z3). Line segment a connects the elbow and wrist joints, line segment b connects the shoulder and wrist joints, and line segment c connects the shoulder and elbow joints, forming a triangle. Next, the length of each side of the triangle is calculated. Based on the elbow joint coordinates (y2, z2) and the wrist joint coordinates (y3, z3), the length of side a can be calculated as shown in the formula:

[0029]

[0030] The length of side b can be calculated from the coordinates of the shoulder joint (y1, z1) and the wrist joint (y3, z3), as shown in the formula:

[0031]

[0032] The length of side c can be calculated from the coordinates of the shoulder joint (y1, z1) and the elbow joint (y2, z2), as shown in the formula:

[0033]

[0034] Let 'a' be the line connecting the elbow joint and the wrist joint, and 'c' be the line connecting the elbow joint and the shoulder joint. Let 'θ' be the angle between them. Using the side length of the triangle, we can calculate the size of the angle θ, as shown in the formula:

[0035] 2*a*c*cosθ=a 2 +c 2 -b2

[0036] See Figure 3 , Figure 3 The elbow joint angle calculated in Figure (A) is an obtuse angle of approximately 145°, while the elbow joint angle calculated in Figure (B) is an acute angle of approximately 45°. Furthermore, to improve program speed, the pose is calculated every 15 frames in this invention. Combined with Kinect's 30fps frame rate, this ensures two pose recognitions per second, meeting practical requirements.

[0037] The elbow joint angle is the pose calculation result. Referring to Table 1, which shows the correspondence between pose calculation results and pose determination results, the decision control module needs to map the pose calculation results to the pose determination results. When the elbow joint angle is between 0° and 5°, the driver's pose is determined to be at the top; when the elbow joint angle is between 5° and 20°, the range is invalid and no determination is made; when the elbow joint angle is between 20° and 80°, the driver's pose is determined to be upward; when the elbow joint angle is between 80° and 100°, the range is invalid and no determination is made; when the elbow joint angle is between 100° and 160°, the driver's pose is determined to be downward; when the elbow joint angle is between 160° and 175°, the range is invalid and no determination is made; when the elbow joint angle is between 175° and 180°, the driver's pose is determined to be at the bottom.

[0038] Table 1

[0039] Pose calculation results Pose determination result 0°-5° To the top 5°-20° invalid 20°-80° up 80°-100° invalid 100°-160° down 160°-175° invalid 175°-180° To the bottom

[0040] Refer to Table 2 for the correspondence between pose recognition results and decision control results. Both hands at the top correspond to enabling the Kinect pose recognition reverse parking function; both hands upward correspond to the vehicle moving forward; both hands downward correspond to the vehicle reversing; left hand upward and right hand downward correspond to the vehicle turning left; left hand downward and right hand upward correspond to the vehicle turning right; both hands at the bottom correspond to successful reverse parking and disabling the Kinect pose recognition reverse parking function.

[0041] Table 2

[0042] Left hand pose determination results Right hand pose determination results Decision control results To the top To the top Enable reverse parking function up up Vehicle moving forward down down Vehicle reverse up down Vehicles turning left down up Vehicles turn right To the bottom To the bottom Disable reverse parking function

[0043] See Figure 4In this invention, the Kinect camera is used for model building and data acquisition. In the model building section, model information, including an identity image and dominant hand, is transmitted to the pose calculation module. In the data acquisition section, the model information and depth image are first transmitted to the preprocessing module to determine if the driver has operating authority and to determine the driver's orientation, obtaining processed skeletal information. This skeletal information is then transmitted to the pose calculation module to calculate the elbow joint angle. The pose calculation result is then transmitted to the decision control module, which determines the pose calculation result and obtains the pose determination result. Finally, the pose determination results of both hands are combined to control the vehicle's movement. Through... Figure 4 The process shown can complete the reverse parking method based on Kinect pose recognition of this invention.

[0044] The scope of protection of this invention is not limited to the above embodiments. Any variations and advantages that can be conceived by those skilled in the art without departing from the spirit and scope of this invention are included in this invention and are protected by the appended claims.

Claims

1. A method for reversing into a parking space based on Kinect pose recognition, characterized in that, The method includes the following specific steps: Step 1: Model Construction: Initialize the Kinect camera. Obtain the driver's skeletal information using the depth image captured by the Kinect camera and the Kinect for Windows software development kit (SDK). Obtain the coordinate values ​​of the shoulder center joint, head joint, spine joint, left shoulder joint, and right shoulder joint. Each coordinate position is represented by (x, y, z). Calculate the driver's identity map for driver identification. Based on the coordinate values ​​of the shoulder center joint, left shoulder joint, and right shoulder joint, calculate the driver's dominant hand for determining the driver's orientation. Step 2: Data Acquisition: Initialize the Kinect camera, set the camera elevation angle and placement distance, and the Kinect camera acquires a depth image including the driver; Step 3: Preprocessing: Using the depth image captured by the Kinect camera and the Kinect SDK, human skeletal data information, namely the coordinate values ​​of the joints of the human skeleton, is obtained; based on the real-time collected skeletal data information, combined with the identity image from step 1 and the dominant hand, the driver's identity is identified and orientation is matched. Step 4: Pose calculation: Select the left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint, and right wrist joint of the human body, and divide them into two groups of three points each. Since the driver is facing the camera, ignore the value in the X-axis direction, obtain the two-dimensional coordinates of the shoulder joint, elbow joint, and wrist joint, form a triangle, and combine trigonometric functions to calculate the angle. Step 5: Decision Control: Based on the decision judgment rules, the position calculation results of the left and right hands are analyzed to obtain the position judgment results. Then, based on the decision control rules, decision control commands are issued to the vehicle to move forward, backward, turn left, and turn right, driving the vehicle to complete the reverse parking.

2. The reversing parking method based on Kinect pose recognition according to claim 1, characterized in that, Step 1 involves calculating the driver's identity diagram. The calculation process is as follows: using the shoulder center joint as the center, calculate the distances from the shoulder center joint to the head joint, the shoulder center joint to the spine joint, the shoulder center joint to the left shoulder joint, and the shoulder center joint to the right shoulder joint, obtaining the proportional relationship of these four distances, which serves as the driver's identity diagram. Next, the driver's dominant hand is calculated. The calculation process is as follows: compare the distance from the shoulder center joint to the left shoulder joint with the distance from the shoulder center joint to the right shoulder joint. Based on the relationship between the two distances, determine whether the driver's dominant hand is left or right. If the distance from the shoulder center joint to the left shoulder joint is larger, the dominant hand is considered to be the right hand; otherwise, it is the left hand.

3. The reversing parking method based on Kinect pose recognition according to claim 1, characterized in that, The Kinect camera includes a color camera, an infrared projector, and a depth camera. The color camera captures color images within its field of view. The infrared projector actively projects near-infrared light spectrum, which is reflected when it hits a rough object and is then captured by the depth camera. The depth is calculated by calculating the time difference of light, thus obtaining a depth image.

4. The reversing parking method based on Kinect pose recognition according to claim 1, characterized in that, Step 3 describes the identification and orientation matching of the driver. The identification process involves calculating the coordinates of the real-time collected head joint, shoulder center joint, spine joint, left shoulder joint, and right shoulder joint and comparing them with the identity map from Step 1. If they match, the driver's identity is considered verified. Orientation matching: Based on the dominant hand in step 1, the orientation of the skeletal model is calibrated. If the dominant hand is right, the distance from the shoulder center joint to the left shoulder joint should be greater than the distance from the shoulder center joint to the right shoulder joint. If the orientation does not meet this criterion three times in a row, it is considered that all the received left shoulder joint, left elbow joint, left wrist joint, right shoulder joint, right elbow joint and right wrist joint need to be converted to left and right.

5. The reversing parking method based on Kinect pose recognition according to claim 1, characterized in that, Step 4, which involves angle calculation, refers to the elbow joint angle calculation. Let the three-dimensional coordinates of the shoulder joint be (x1, y1, z1), the corresponding three-dimensional coordinates of the elbow joint be (x2, y2, z2), and the corresponding three-dimensional coordinates of the wrist joint be (x3, y3, z3). Since the driver is operating in front of the Kinect camera, the x-coordinates of the three joints are nearly identical. Therefore, the triangle formed by the shoulder, elbow, and wrist joints is projected onto a two-dimensional coordinate system formed by the Y-axis and Z-axis. For the film, the coordinates of the shoulder joint are (y1, z1), the elbow joint is (y2, z2), and the wrist joint is (y3, z3). Line segment a connects the elbow and wrist joints, line segment b connects the shoulder and wrist joints, and line segment c connects the shoulder and elbow joints, forming a triangle. Next, the length of each side of the triangle is calculated. Based on the coordinates of the elbow joint (y2, z2) and the wrist joint (y3, z3), the length of side a is calculated as shown in the formula: Similarly, calculate the lengths of side lengths b and c; let the angle at the elbow joint be θ, and the size of the angle is calculated using the following formula: 2*a*c*cosθ=a 2 +c 2 -b 2 To ensure computational efficiency, the computation frequency is set to twice per second.

6. The reversing parking method based on Kinect pose recognition according to claim 1, characterized in that, The decision-making rule in step 5 is as follows: The result of the pose calculation, i.e., the angle at the elbow joint, is analyzed. The analysis result corresponds to "top," "upward," "downward," and "bottom." When the elbow joint angle is within the range of 0°-5°, the driver's pose is determined to be "top"; when the elbow joint angle is within the range of 20°-80°, the driver's pose is determined to be "upward"; when the elbow joint angle is within the range of 100°-160°, the driver's pose is determined to be "downward"; and when the elbow joint angle is within the range of 175°-180°, the driver's pose is determined to be "bottom." 5°-20°, 80°-100°, and 160°-175° are used as invalid angle boundaries to separate the effective angle range, ensuring the accuracy of the judgment results. The decision control rules are as follows: both hands are at the top, corresponding to activating the reverse parking function; both hands are up, corresponding to the vehicle moving forward; both hands are down, corresponding to the vehicle reversing; left hand up, right hand down, corresponding to the vehicle turning left; left hand down, right hand up, corresponding to the vehicle turning right; both hands are at the bottom, corresponding to successful reverse parking and deactivation of the reverse parking function.

Images