Method and system for roadside parking management based on key point moving posture

Abstract

The application discloses a roadside parking management method and system based on key point moving posture. The method comprises the following steps: inputting a to-be-tested vehicle image into a trained key point detection model, outputting a plurality of first posture key points, a plurality of offsets and a plurality of key point heat maps of each to-be-tested vehicle; calculating a plurality of second posture key points of each to-be-tested vehicle according to a posture center point and the plurality of offsets of each to-be-tested vehicle, and obtaining a plurality of third posture key points of each to-be-tested vehicle according to the plurality of key point heat maps of each to-be-tested vehicle; performing average value calculation according to the first posture key points, the second posture key points and the third posture key points to obtain a plurality of mean value posture key points of each to-be-tested vehicle; inputting the plurality of mean value posture key points of each to-be-tested vehicle into a trained key point position smoothing network model to perform position smoothing, and obtaining a plurality of smoothed posture key points of each to-be-tested vehicle for roadside parking management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a roadside parking management method and system based on key point moving posture. BACKGROUND

[0002] In recent years, high-position video technology has developed rapidly. By installing high-position video cameras at the roadside, the parking of vehicles can be judged and managed. Through vehicle detection and vehicle body posture detection, data analysis is performed with known parking position to judge the state of the vehicle, such as whether the vehicle is located in the parking space, whether the vehicle is parked in the line, whether the vehicle is parked in the forbidden parking area, etc. According to the vehicle body posture, it is judged whether the vehicle is illegally parked, thereby positively promoting various aspects of city traffic management, driving safety, etc.

[0003] The traditional method realizes roadside parking management by two-dimensional vehicle rectangular detection frame for vehicle posture estimation. However, due to the influence of the surrounding traffic scene, there may be a situation that the vehicle is blocked. Relying only on the two-dimensional vehicle rectangular detection frame to judge the posture of the vehicle cannot accurately determine the position of the vehicle, resulting in low efficiency of roadside parking management. SUMMARY

[0004] The purpose of the present application is to solve the technical problem that the traditional method cannot accurately determine the position of the vehicle. In order to achieve the above purpose, the present application provides a roadside parking management method and system based on key point moving posture.

[0005] The present application provides a roadside parking management method based on key point moving posture, comprising:

[0006] Obtaining multiple frames of vehicle images of each vehicle within a continuous time, each frame of the vehicle image being labeled with multiple real posture key points corresponding to a three-dimensional bounding box of the vehicle and multiple real offset amounts of the multiple real posture key points to a real posture center point respectively;

[0007] Inputting each frame of the vehicle image into a backbone network and a feature aggregation network of a key point detection model for feature extraction and fusion to obtain multiple aggregated feature maps;

[0008] Parallelly inputting each of the aggregated feature maps into a key point regression network, a key point offset regression network and a key point heat map prediction network of the key point detection model for prediction to obtain multiple first predicted posture key points, multiple predicted offset amounts and multiple predicted key point heat maps of each frame of the vehicle image of each vehicle;

[0009] According to the plurality of real attitude key points of each vehicle image of each vehicle in continuous time, a plurality of real key point position moving speeds of each vehicle in continuous time are calculated, and the plurality of real key point position moving speeds in continuous time and the plurality of real attitude key points are input into a key point position smoothing network model for smoothing processing, to obtain a plurality of predicted key point position moving speeds and a plurality of second predicted attitude key points.

[0010] According to the plurality of first predicted attitude key points, the plurality of predicted offsets, the plurality of predicted key point heat maps, the plurality of real attitude key points, the plurality of real offsets, and a heat map target value, a loss function of the key point detection model is constructed for model training, to obtain a trained key point detection model.

[0011] According to the plurality of predicted key point position moving speeds, the plurality of second predicted attitude key points, the plurality of real key point position moving speeds, and the plurality of real attitude key points, a loss function of the key point position smoothing network model is constructed for model training, to obtain a trained key point position smoothing network model.

[0012] Obtain a to-be-tested vehicle image and an attitude center point of each to-be-tested vehicle, and input the to-be-tested vehicle image into the trained key point detection model, to output a plurality of first attitude key points, a plurality of offsets, and a plurality of key point heat maps of each to-be-tested vehicle.

[0013] According to the attitude center point and the plurality of offsets of each to-be-tested vehicle, a plurality of second attitude key points of each to-be-tested vehicle are calculated, and according to the plurality of key point heat maps of each to-be-tested vehicle, a plurality of third attitude key points of each to-be-tested vehicle are obtained.

[0014] According to the first attitude key points, the second attitude key points, and the third attitude key points, average value calculation is performed, to obtain a plurality of mean attitude key points of each to-be-tested vehicle.

[0015] Input the plurality of mean attitude key points of each to-be-tested vehicle into the trained key point position smoothing network model for position smoothing, to obtain a plurality of smoothed attitude key points of each to-be-tested vehicle, and perform roadside parking management on the to-be-tested vehicle according to the plurality of smoothed attitude key points.

[0016] In one embodiment, the aggregated feature map of each vehicle is input into a key point detection model including a key point regression network, a key point offset regression network, and a key point heat map prediction network to obtain a plurality of first predicted posture key points, a plurality of predicted offsets, and a plurality of predicted key point heat maps of each vehicle image of each frame of the vehicle, including:

[0017] The aggregated feature map of each vehicle is input into a Gaussian convolution kernel to generate a two-dimensional normal distribution to generate the plurality of predicted key point heat maps, each of which is represented as:

[0018]

[0019] wherein x and y represent the coordinates of the predicted key points in the predicted key point heat map, x' and y' represent the quantized coordinates of the predicted key points, and σ represents the standard deviation of the Gaussian distribution.

[0020] In one embodiment, the loss function of the key point detection model is constructed based on the plurality of first predicted posture key points, the plurality of predicted offsets, the plurality of predicted key point heat maps, the plurality of real posture key points, the plurality of real offsets, and the heat map target value to train the model and obtain a trained key point detection model, including:

[0021] Based on each of the predicted key point heat maps, a predicted heat map value is calculated according to a focal loss function;

[0022] The heat map target value is set, an average absolute error loss function is constructed based on the predicted heat map value and the heat map target value, and a heat map regression loss function is formed according to the average absolute error loss function;

[0023] wherein the focal loss function is:

[0024] FL(p t )=-α t (1-p t ) γ log(p t );

[0025]

[0026] g=1 represents the predicted key point coordinate value in the predicted key point heat map being 1, otherwise represents the coordinate value of other positions other than the key point, P represents the probability of the predicted key point coordinate value being 1, and α t represents a first coefficient for controlling positive and negative samples, and γ represents a second coefficient for controlling positive and negative samples;

[0027] The average absolute error loss function is:

[0028]

[0029] y represents the heat map target value, and m represents the number of samples.

[0030] In one embodiment, the second attitude key points of each vehicle are calculated according to the attitude center point of each vehicle and the plurality of offsets, and the third attitude key points of each vehicle are obtained according to the plurality of key point heat maps of each vehicle.

[0031] The third attitude key points are obtained by finding the maximum value of each key point heat map according to the argument maximum evaluation function.

[0032] In one embodiment, the real key point position moving speed of each vehicle in continuous time is calculated according to the plurality of real attitude key points of each vehicle in each frame of vehicle image in continuous time.

[0033] The coordinate positions of the real attitude key points of each category corresponding to the first frame of vehicle image and the coordinate positions of the real attitude key points of each category corresponding to the second frame of vehicle image are obtained.

[0034] The x-direction moving speed of the real attitude key points of each category in the x-coordinate direction and the y-direction moving speed in the y-coordinate direction are respectively calculated according to the coordinate positions of the real attitude key points of each category corresponding to the first frame of vehicle image and the coordinate positions of the real attitude key points of each category corresponding to the second frame of vehicle image.

[0035] The x-direction moving speed and the y-direction moving speed of the real attitude key points of each category are respectively:

[0036]

[0037] x represents the x-coordinate position, y represents the y-coordinate position, i represents the category of the real attitude key point, t represents the first frame of vehicle image, t+1 represents the second frame of vehicle image, represents the coordinate position of the real attitude key point of each category corresponding to the first frame of vehicle image, represents the coordinate position of the real attitude key point of each category corresponding to the second frame of vehicle image, represents the x-direction moving speed, representing the y-direction movement speed.

[0038] In one embodiment, the application provides a roadside parking management system based on key point moving posture, comprising:

[0039] a data acquisition module configured to acquire multiple frames of vehicle images of each vehicle within continuous time, each frame of the vehicle images being labeled with multiple real posture key points corresponding to a three-dimensional bounding box of the vehicle and multiple real offset amounts of the multiple real posture key points to a real posture center point;

[0040] an aggregated feature map acquisition module configured to input each frame of the vehicle images into a backbone network and a feature aggregation network of a key point detection model for feature extraction and fusion to obtain multiple aggregated feature maps;

[0041] a parallel network module configured to input each of the aggregated feature maps into a key point regression network, a key point offset amount regression network and a key point heat map prediction network of the key point detection model in parallel for prediction to obtain multiple first predicted posture key points, multiple predicted offset amounts and multiple predicted key point heat maps of each frame of the vehicle images of each vehicle;

[0042] a smoothing network module configured to calculate multiple real key point position movement speeds of each vehicle within continuous time according to the multiple real posture key points of each frame of the vehicle images of each vehicle within continuous time, and input the multiple real key point position movement speeds within continuous time and the multiple real posture key points into a key point position smoothing network model for smoothing processing to obtain multiple predicted key point position movement speeds and multiple second predicted posture key points;

[0043] a first model training module configured to construct a loss function of the key point detection model according to the multiple first predicted posture key points, the multiple predicted offset amounts, the multiple predicted key point heat maps, the multiple real posture key points, the multiple real offset amounts and a heat map target value for model training to obtain a trained key point detection model;

[0044] a second model training module configured to construct a loss function of the key point position smoothing network model according to the multiple predicted key point position movement speeds, the multiple second predicted posture key points, the multiple real key point position movement speeds and the multiple real posture key points for model training to obtain a trained key point position smoothing network model;

[0045] The detection module is configured to acquire a to-be-tested vehicle image and a pose center point of each to-be-tested vehicle, input the to-be-tested vehicle image into the trained key point detection model, and output a plurality of first pose key points, a plurality of offsets, and a plurality of key point heat maps of each to-be-tested vehicle.

[0046] The pose key point acquisition module is configured to calculate a plurality of second pose key points of each to-be-tested vehicle according to the pose center point and the plurality of offsets of each to-be-tested vehicle, and obtain a plurality of third pose key points of each to-be-tested vehicle according to the plurality of key point heat maps of each to-be-tested vehicle.

[0047] The mean pose key point acquisition module is configured to perform mean value calculation according to the first pose key points, the second pose key points, and the third pose key points, and obtain a plurality of mean pose key points of each to-be-tested vehicle.

[0048] The key point position smoothing processing module is configured to input the plurality of mean pose key points of each to-be-tested vehicle into the trained key point position smoothing network model to perform position smoothing, obtain a plurality of smoothed pose key points of each to-be-tested vehicle, and perform roadside parking management on the to-be-tested vehicle according to the plurality of smoothed pose key points.

[0049] In an embodiment, the parallel network module comprises:

[0050] The heat map acquisition module is configured to input each aggregated feature map into a Gaussian convolution kernel to perform two-dimensional normal distribution, and generate the plurality of predicted key point heat maps, each of which is represented as:

[0051]

[0052] Wherein, x and y represent the coordinates of the key points in the predicted key point heat map, x' and y' represent the integer-quantized key point coordinates in the predicted key point heat map, and σ represents the standard deviation of the Gaussian distribution.

[0053] In an embodiment, the first model training module comprises:

[0054] The predicted heat map value acquisition module is configured to calculate a predicted heat map value according to a focus loss function based on each predicted key point heat map.

[0055] The heat map regression loss function construction module is configured to set the heat map target value, construct a mean absolute error loss function according to the predicted heat map value and the heat map target value, and form a heat map regression loss function according to the mean absolute error loss function.

[0056] Wherein, the focus loss function is:

[0057] FL(p t )=-α t (1-p t ) γ log(p t );

[0058]

[0059] g=1 represents that the predicted key point coordinate value in the predicted key point heat map is 1, otherwise represents the coordinate value of other positions except the key point, P represents the probability that the predicted key point coordinate value is 1, and alpha t represents the first coefficient for controlling positive and negative samples, and gamma represents the second coefficient for controlling positive and negative samples.

[0060] The mean absolute error loss function is:

[0061]

[0062] represents the predicted heat map value, y represents the heat map target value, and m represents the sample number.

[0063] In one embodiment, the pose key point acquisition module comprises:

[0064] The third pose key point acquisition module is configured to find the maximum value of each key point heat map according to the argument maximum evaluation function, and obtain the plurality of third pose key points.

[0065] In one embodiment, the smoothing network module comprises:

[0066] The adjacent frame key point position acquisition module is configured to acquire the coordinate positions of the real pose key points of each category corresponding to the first frame vehicle image and the second frame vehicle image.

[0067] The moving speed calculation module is configured to calculate the x-direction moving speed of the real pose key points of each category in the x coordinate direction and the y-direction moving speed in the y coordinate direction, respectively, according to the coordinate positions of the real pose key points of each category corresponding to the first frame vehicle image and the second frame vehicle image.

[0068] Wherein, the x-direction moving speed and the y-direction moving speed of the real pose key points of each category are respectively:

[0069]

[0070] x represents the x-coordinate position, y represents the y-coordinate position, i represents the category of the real pose key point, t represents the first frame of the vehicle image, t+1 represents the second frame of the vehicle image, representing the coordinate position of the real pose key point of each category corresponding to the first frame of the vehicle image, representing the coordinate position of the real pose key point of each category corresponding to the second frame of the vehicle image, representing the x-direction moving speed, representing the y-direction moving speed.

[0071] In the above key point moving pose-based roadside parking management method and system, the backbone network and the feature aggregation network are gradually used for feature extraction and fusion on multiple frames of vehicle images labeled with label data in continuous time, multiple aggregated feature maps are obtained, and multiple aggregated feature maps are respectively and in parallel used for key point prediction, offset prediction and heat map prediction, the key points are detected from multiple different angles, the first pose key point, the second pose key point and the third pose key point are calculated to obtain the mean value pose key point, and the mean value pose key point is input into the key point position smoothing network model to perform position smoothing, so as to obtain more stable and smooth vehicle pose changes, thereby being more conducive to vehicle parking management. BRIEF DESCRIPTION OF DRAWINGS

[0072] Figure 1 is a step flowchart of the key point moving pose-based roadside parking management method provided by the present application.

[0073] Figure 2 is a schematic diagram of eight key points in an embodiment provided by the present application.

[0074] Figure 3 is a structure schematic diagram of the key point moving pose-based roadside parking management system provided by the present application. DETAILED DESCRIPTION

[0075] The technical solutions of the present application will be further described in detail below with the aid of drawings and embodiments.

[0076] Please refer to Figure 1 , the present application provides a key point moving pose-based roadside parking management method, comprising:

[0077] S10, obtaining multiple frames of vehicle images of each vehicle in continuous time, each frame of vehicle image being labeled with multiple real pose key points corresponding to a three-dimensional bounding box of the vehicle and multiple real offsets of the multiple real pose key points to a real pose center point;

[0078] S20, input each frame of vehicle image into the backbone network and feature aggregation network of the key point detection model for feature extraction and fusion to obtain a plurality of aggregated feature maps;

[0079] S30, input each aggregated feature map into the key point regression network, key point offset regression network and key point heat map prediction network of the key point detection model respectively and in parallel for prediction to obtain a plurality of first predicted posture key points, a plurality of predicted offsets and a plurality of predicted key point heat maps of each frame of vehicle image of each vehicle;

[0080] S40, according to a plurality of real posture key points of each frame of vehicle image of each vehicle in continuous time, calculate a plurality of real key point position moving speeds of each vehicle in continuous time, and input the plurality of real key point position moving speeds and the plurality of real posture key points in continuous time into the key point position smoothing network model for smoothing processing to obtain a plurality of predicted key point position moving speeds and a plurality of second predicted posture key points;

[0081] S50, according to the plurality of first predicted posture key points, the plurality of predicted offsets, the plurality of predicted key point heat maps, the plurality of real posture key points, the plurality of real offsets and the heat map target value, construct a loss function of the key point detection model for model training to obtain a trained key point detection model;

[0082] S60, according to the plurality of predicted key point position moving speeds, the plurality of second predicted posture key points, the plurality of real key point position moving speeds and the plurality of real posture key points, construct a loss function of the key point position smoothing network model for model training to obtain a trained key point position smoothing network model;

[0083] S70, obtain a to-be-tested vehicle image and a posture center point of each to-be-tested vehicle, and input the to-be-tested vehicle image into the trained key point detection model to output a plurality of first posture key points, a plurality of offsets and a plurality of key point heat maps of each to-be-tested vehicle;

[0084] S80, according to the posture center point of each to-be-tested vehicle and the plurality of offsets, calculate a plurality of second posture key points of each to-be-tested vehicle, and according to the plurality of key point heat maps of each to-be-tested vehicle, obtain a plurality of third posture key points of each to-be-tested vehicle;

[0085] S90, according to the first posture key points, the second posture key points and the third posture key points, calculate the average value to obtain a plurality of mean posture key points of each to-be-tested vehicle;

[0086] S910, input the plurality of mean pose key points of each to-be-tested vehicle into the trained key point position smoothing network model for position smoothing, obtain a plurality of smoothed pose key points of each to-be-tested vehicle, and perform roadside parking management on the to-be-tested vehicle according to the plurality of smoothed pose key points.

[0087] In this embodiment, in S10, the continuous time can be understood as a time period in which 60 continuous vehicle images are acquired. The three-dimensional bounding box is a three-dimensional box surrounding the vehicle. The plurality of real pose key points represent a plurality of vertices of the three-dimensional bounding box, and can represent the position of the three-dimensional bounding box, and further represent the position of the vehicle. Please refer to Figure 2 In one embodiment, each vehicle corresponds to eight key points, which represent the eight vertices of the three-dimensional box, respectively, that is, four key points in contact with the ground and four key points on the roof of the vehicle, which can also be understood as, according to the orientation of the vehicle head, taking the key point on the left side of the vehicle head in contact with the ground as the reference, which is marked as No. 1 point bottom-front-1, then rotating clockwise, the other three contact points are marked as bottom-front-2, bottom-back-3, and bottom-back-4, respectively, and the key point on the left side of the vehicle head on the roof is marked as No. 5 point top-front-5, and the other three key points on the roof are marked as top-front-6, top-back-7, and top-back-8, respectively. Through the eight vertices of the three-dimensional bounding box surrounding the vehicle, eight types of key points of the vehicle body pose are described. The coordinates of the real pose key points are the x and y coordinates in the image coordinate system. The attributes of the real pose key points include visible and invisible, and the attribute of the visible point is marked as 1, and the attribute of the invisible point is marked as 0. The invisible point can be understood as the vehicle body blocking the vehicle image in which the pose key point is captured, for example Figure 2 The midpoint 1 and the point 4 are blocked, and have the invisible point attribute. Conversely, the visible point can be understood as the vehicle body not blocking the vehicle image in which the pose key point is captured, for example Figure 2 The midpoint 2, the point 3, the point 5, the point 6, the point 7, and the point 8 are not blocked, and have the visible point attribute.

[0088] According to the two-dimensional detection box real information of the vehicle, the coordinate position of the corresponding real attitude center point can be obtained, and then the plurality of real offsets of the plurality of real attitude key points to the real attitude center point can be obtained. The two-dimensional detection box real information of each vehicle includes the x coordinate and y coordinate information of the upper left corner and the lower right corner of the vehicle rectangular box, which are (x1, y1) and (x2, y2) respectively. The real attitude center point represents the center point coordinate of the two-dimensional detection box, which can be obtained through the coordinate information of the two-dimensional detection box. The plurality of real offsets of the real attitude key points to the real attitude center point relatively compares the position information of the two-dimensional detection box with the position information of the three-dimensional bounding box, so that the specific position positioning of the vehicle is more accurate and can be more accurately positioned to the specific position of the vehicle. The plurality of real attitude key points and the plurality of real offsets can be used as label data of each frame of vehicle image. In an embodiment, the label data of each frame of vehicle image can further include a plurality of two-dimensional detection box real information and a plurality of real attitude center points. Each frame of vehicle image containing label data is used as the input of the key point detection model, and further model training optimization is performed.

[0089] In S20, the backbone network, the feature aggregation network, the key point regression network, the key point offset regression network and the key point heat map prediction network constitute the key point detection model. The backbone network is used for feature extraction of the image, and includes a plurality of convolution combination layers obtained by stacking and operating a convolution combination mode of a convolution layer, a normalization layer and an activation function layer. The backbone network includes but is not limited to ResNet, VGG, MobileNet, etc. The feature aggregation network is used to aggregate high and low layer features extracted by different layers in the backbone network, to provide more feature representation for subsequent key point prediction. Since the semantic features of the high layer are close to the output end of the network but have low resolution, and the high-resolution features are close to the input end but have less semantic features, the features between different layers in the backbone network are aggregated, which can realize the fusion of high and low layer features, thereby improving the detection accuracy of the subsequent key point detection task.

[0090] In S30, the inputs of the key point regression network, the key point offset regression network and the key point heat map prediction network are all the aggregated feature maps after feature aggregation, i.e. the aggregated feature maps, which are processed in parallel to realize feature classification and position positioning of the key points. In this step, the key point regression network and the key point offset regression network can adopt an algorithm capable of realizing regression prediction. For the key point regression network, according to the information of the plurality of real attitude key points, the plurality of real attitude key points are marked as C i , i∈(1...8), C i which is composed of x coordinate and y coordinate. The coordinate positions of the plurality of real attitude key points are used as the learning target of the key point regression network.

[0091] For the key point offset regression network, a plurality of real pose key points of a three-dimensional bounding box of a vehicle are respectively mapped to a plurality of real pose center points of a two-dimensional detection box as a learning target of the key point offset regression network. For the key point heat map prediction network, a plurality of real pose key points are distributed on a Gaussian heat map to form a plurality of predicted key point heat maps of each vehicle by using a Gaussian convolution kernel to generate a two-dimensional normal distribution. One real pose key point corresponds to one predicted key point heat map. By increasing the key point heat map prediction network, more spatial feature information can be introduced, avoiding the overfitting problem caused by only relying on the key point regression network and the key point offset regression network, and more spatial information can be added to further improve the accuracy of key point prediction.

[0092] In S40, the key point position smoothing network model is used to smooth the key point positions output by the key point detection model, so that the changes of the vehicle key point positions in consecutive video frames are smoother and more fluent. The key point position smoothing network model is formed by a combination of fully connected layers and activation function layers, and performs superposition operation of a plurality of fully connected combination layers. The nonlinear activation layer includes but is not limited to ReLU, Leaky ReLU, and other nonlinear activation functions. The input of the key point position smoothing network model is the moving speed of a plurality of real key point positions and the positions of a plurality of real pose key points of a vehicle in a plurality of frames of vehicle images in consecutive time. The moving speed of the plurality of real key point positions can be calculated by the coordinate positions of the plurality of real pose key points of the current frame and the previous frame or the next frame.

[0093] In S50, the key point detection model outputs a plurality of first predicted pose key points, a plurality of predicted offsets, and a plurality of predicted key point heat maps in the learning iteration process, which are compared with a plurality of real pose key points, a plurality of real offsets, and heat map target values, respectively. In the learning iteration process, the error is gradually reduced, so that the predicted values output by the key point detection model gradually approach the plurality of real pose key points, the plurality of real offsets, and the heat map target values, so that the error reaches a minimum, and a trained key point detection model is obtained.

[0094] In S60, the key point position smoothing network model outputs a plurality of predicted key point position moving speeds and a plurality of second predicted pose key points in the learning iteration process, which are compared with a plurality of real key point position moving speeds and a plurality of real pose key points, respectively. In the learning iteration process, the error is gradually reduced, so that the predicted values output by the key point position smoothing network model gradually approach the plurality of real key point position moving speeds and the plurality of real pose key points, so that the error reaches a minimum, and a trained key point position smoothing network model is obtained.

[0095] In S70, the pose center point of the to-be-tested vehicle can be obtained through the two-dimensional detection frame of the to-be-tested vehicle. The trained key point detection model includes a trained key point regression network, a trained key point offset regression network, and a trained key point heat map prediction network. According to the trained key point regression network, the trained key point offset regression network, and the trained key point heat map prediction network, the to-be-tested vehicle image is detected to obtain a plurality of first pose key points, a plurality of offsets, and a plurality of key point heat maps of the to-be-tested vehicle.

[0096] In S80, each offset is added to the pose center point to calculate a corresponding second pose key point. The second pose key point l k = (x_center, y_center) + J j The position C car of the pose center point of the to-be-tested vehicle is (x_center, y_center), J j represents the offset, and j e 1, 2…k. The offset can be calculated by using the L1 distance calculation method. One key point heat map corresponds to one third pose key point. According to the key point heat map, the third pose key point can be obtained. The position C car of the pose center point of the to-be-tested vehicle can be calculated through the x coordinate and y coordinate information of the upper left corner and the lower right corner of the two-dimensional detection frame of the to-be-tested vehicle, or can be understood as the center point of the two-dimensional detection frame.

[0097] In S90, the first pose key point, the second pose key point, and the third pose key point can be understood as three different methods obtained by the same category of pose key point, or can be understood as a key point of the three-dimensional bounding box of the to-be-tested vehicle obtained by three different methods. The key point can be detected from multiple different angles, and the error caused by a single method calculation can be avoided. For example: the key point of the bottom-front-1 category corresponds to one first pose key point, one second pose key point, and one third pose key point, and the key point of the bottom-front-2 category corresponds to one first pose key point, one second pose key point, and one third pose key point. The average value of each first pose key point, each second pose key point, and each third pose key point is calculated to obtain the average value of each pose key point of the to-be-tested vehicle after calculation.

[0098] In S910, the plurality of mean pose key points are obtained by performing position smoothing on the plurality of mean pose key points through the trained key point position smoothing network model. The position smoothing of the key points in consecutive frames can make the key point position coordinates more stable. The plurality of smoothed pose key points represent the pose of the vehicle to be measured. The plurality of smoothed pose key points in the image coordinate system are converted into the world coordinate system, and are compared with the parking space position in the same world coordinate system, so as to determine whether the vehicle to be measured is parked in the parking space, or whether there is a line-pressing parking, or whether there is a forbidden parking area parking, and the like, thereby realizing the parking management of the roadside parking.

[0099] The roadside parking management method based on the key point moving pose provided in the present application gradually performs feature extraction and fusion of the backbone network and the feature aggregation network on the plurality of vehicle images with labeled data in consecutive time, obtains a plurality of aggregated feature maps, and performs key point prediction, offset prediction and heat map prediction on the plurality of aggregated feature maps in parallel, detects the key points from different angles, calculates the mean of the first pose key point, the second pose key point and the third pose key point to obtain the mean pose key point, and inputs the mean pose key point into the key point position smoothing network model to perform position smoothing, so as to obtain a more stable and smooth vehicle pose change, thereby being more conducive to the parking management of the vehicle.

[0100] In one embodiment, in S10, the roadside parking scene is modeled by using Carla, Arisim and the like simulation software based on Unity, Unreal and the like game engines, and labeled data is obtained, such as two-dimensional bounding box coordinate information of the vehicle, three-dimensional bounding box information of the vehicle, and the like, to form labeled data as training sample data for subsequent model training.

[0101] In one embodiment, in S10, the plurality of vehicle images in consecutive time can be cut out from the roadside parking scene image.

[0102] In one embodiment, the roadside parking management method based on the key point moving pose further includes size transformation, label data reconstruction and data enhancement processing of each vehicle image.

[0103] In one embodiment, in S10, the real pose center point C real-car (x real-car , y real-car ) is calculated as follows: x real-car =(x1+x2) / 2, y real-car =(y1+y2) / 2. The real offset of the real pose key point to the real pose center point is calculated by using the L1 distance calculation method. The real offset is represented as follows:

[0104] wherein, I1, I2 respectively represent two vectors of real attitude key points and real attitude center points, and p represents the dimension of the two vectors.

[0105] In one embodiment, S30, each aggregated feature map is respectively and in parallel input into a key point regression network, a key point offset regression network and a key point heat map prediction network of the key point detection model for prediction, to obtain multiple first predicted attitude key points, multiple predicted offsets and multiple predicted key point heat maps of each vehicle, including:

[0106] S310, each aggregated feature map is input into a Gaussian convolution kernel for two-dimensional normal distribution to generate multiple predicted key point heat maps, each predicted key point heat map being represented as:

[0107]

[0108] wherein, x and y represent the coordinates of the key points in the predicted key point heat map, x' and y' represent the integer-quantized key point coordinates in the predicted key point heat map, and σ represents the standard deviation of the Gaussian distribution.

[0109] In this embodiment, the size of the aggregated feature map is (W / R)*(H / R)*C, wherein R represents a down-sampling coefficient, which can be set as 4, the aggregated feature map is input into a Gaussian convolution kernel for two-dimensional normal distribution to distribute the attitude key points onto a Gaussian heat map, so that the attitude key point position value is 1 and the rest of the point positions are less than 1, presenting Gaussian distribution, and outputting the heat map of the key points in the aggregated feature map, H∈[0, 1] (W / R)*(H / R)*C , and C represents the number of key point categories, which can be set as 8. The standard deviation of the Gaussian distribution can be adaptively calculated from the width and height of the two-dimensional detection frame of the vehicle. By introducing the Gaussian heat map for model learning, more spatial information can be added to improve the accuracy of key point prediction.

[0110] In one embodiment, S40, according to the multiple real attitude key points of each frame of vehicle image within the continuous time of each vehicle, the multiple real key point position moving speeds within the continuous time of each vehicle are calculated, including:

[0111] S410, the coordinate positions of the real attitude key points of each category corresponding to the first frame of vehicle image of each vehicle and the coordinate positions of the real attitude key points of each category corresponding to the second frame of vehicle image are obtained.

[0112] S420, respectively calculate the x direction moving speed of the real posture key point of each category in the x coordinate direction and the y direction moving speed of the real posture key point of each category in the y coordinate direction according to the coordinate position of the real posture key point of each category corresponding to the first frame vehicle image and the coordinate position of the real posture key point of each category corresponding to the second frame vehicle image.

[0113] The x direction moving speed and the y direction moving speed of the real posture key point of each category are respectively:

[0114]

[0115] x represents the x coordinate position, y represents the y coordinate position, i represents the category of the real posture key point, t represents the first frame vehicle image, t+1 represents the second frame vehicle image, represents the coordinate position of the real posture key point of each category corresponding to the first frame vehicle image, represents the coordinate position of the real posture key point of each category corresponding to the second frame vehicle image, represents the x direction moving speed, represents the y direction moving speed.

[0116] In the embodiment, the first frame vehicle image and the second frame vehicle image are two consecutive vehicle images, and are not limited to the first frame and the second frame image, but can be any two consecutive vehicle images. Through the coordinate positions of the real posture key points in the adjacent two vehicle images of the multiple vehicle images in the continuous time, the x direction moving speed of the real posture key point in the x coordinate direction and the y direction moving speed of the real posture key point in the y coordinate direction can be calculated as the input of the key point position smoothing network model for model training.

[0117] represents the coordinate position of the real posture key point of each category corresponding to the first frame vehicle image. It can also be understood that the coordinates of the eight key points corresponding to the first frame vehicle image can be obtained through represents. i represents the category of the real posture key point, which can be bottom-front-1, bottom-front-2, bottom-back-3, bottom-back-4, top-front-5, and top-front-6, top-back-7, top-back-8 eight key points. t represents the frame number. The moving speed of the position of the key point can represent the position change of the key point, and can reflect the change of the key point in the continuous video frame.

[0118] In one embodiment, S50, model training is performed according to the loss function of the key point detection model constructed based on the plurality of first predicted posture key points, the plurality of predicted offset amounts, the plurality of predicted key point heat maps, the plurality of real posture key points, and the plurality of real offset amounts, to obtain a trained key point detection model, including:

[0119] S510, based on each predicted key point heat map, a predicted heat map value is calculated according to a focal loss function;

[0120] S520, a heat map target value is set, an average absolute error loss function is constructed according to the predicted heat map value and the heat map target value, and a heat map regression loss function is formed according to the average absolute error loss function;

[0121] The focal loss function is:

[0122] FL(p t )=-α t (1-p t ) γ log(p t );

[0123]

[0124] g=1 represents that the predicted key point coordinate value in the predicted key point heat map is 1, otherwise represents the coordinate value of other positions except the key point, P represents the probability that the predicted key point coordinate value is 1, and α t represents the first coefficient for controlling positive and negative samples, and γ represents the second coefficient for controlling positive and negative samples;

[0125] The average absolute error loss function is:

[0126]

[0127] represents the predicted heat map value, y represents the heat map target value, and m represents the number of samples.

[0128] In this embodiment, the loss function of the key point detection model includes a key point regression loss function, a key point offset regression loss function, and a heat map regression loss function. The key point regression loss function and the key point offset regression loss function can be L1 Loss average absolute error loss function. Based on the L1 Loss average absolute error loss function, the key point regression loss function is formed according to the plurality of first predicted posture key points and the plurality of real posture key points. Based on the L1 Loss average absolute error loss function, the key point offset regression loss function is formed according to the plurality of predicted offset amounts and the plurality of real offset amounts.

[0129] The key point heat map prediction network of the key point detection model outputs a plurality of predicted key point heat maps, each of which corresponds to a predicted key point coordinate value of 1 and represents a predicted pose key point. According to a focal loss function (also referred to as a FocalLoss loss function), a predicted heat map value FL(p t ) is calculated. In an embodiment, a in the formula of the focal loss function can be a numerical value 2, and g can be a numerical value 4, which can also be limited according to actual conditions. t

[0130] The heat map target value is a target value that the predicted heat map value approaches in the training process of the key point heat map prediction network. The heat map target value represents the result value of FL(p t ) corresponding to P = 1 when g = 1. Then, the predicted heat map value is compared with the heat map target value, and the key point heat map prediction network is continuously trained, so that the learning of the model reaches a more optimal state. By constructing the heat map regression loss function, the weights of positive and negative samples and difficult and easy samples can be controlled, so as to alleviate the imbalance of positive and negative samples and difficult and easy samples, and make the learning of the key point heat map prediction network reach a more optimal state.

[0131] In an embodiment, S60, according to the plurality of predicted key point position moving speeds, the plurality of second predicted pose key points, the plurality of real key point position moving speeds, and the plurality of real pose key points, a loss function of the key point position smoothing network model is constructed for model training, and a trained key point position smoothing network model is obtained, including:

[0132] S610, according to the plurality of predicted key point position moving speeds and the plurality of real key point position moving speeds, a speed error loss function is constructed;

[0133] S620, according to the plurality of second predicted pose key points and the plurality of real pose key points, a position error loss function is constructed;

[0134] S630, according to the speed error loss function and the position error loss function, a loss function of the key point position smoothing network model is constructed.

[0135] In this embodiment, the speed error loss function and the position error loss function can be L1 Loss average absolute error loss function. The L1 Loss average absolute error loss function can be expressed as:

[0136]

[0137] wherein, ​y is the position or velocity of the key point predicted by the key point position smoothing network model, or the real position or velocity of the key point, and T represents the total number of video frames in the continuous video. The final goal of the velocity error loss function is to minimize the error of the key point moving velocity. The final goal of the position error loss function is to minimize the error of the key point position.

[0138] In one embodiment, in S70, the to-be-tested vehicle image of each to-be-tested vehicle can be cut out from the to-be-tested roadside parking scene image. The to-be-tested vehicle is detected by a vehicle target detection algorithm to obtain a two-dimensional detection box of the to-be-tested vehicle.

[0139] In one embodiment, in S80, a plurality of third pose key points of each to-be-tested vehicle are obtained according to the plurality of key point heat maps of each to-be-tested vehicle, including:

[0140] S810, the maximum value of each key point heat map is found according to the argument maximum evaluation function to obtain a plurality of third pose key points.

[0141] In this embodiment, the maximum value of a key point heat map is found by using the argument maximum evaluation function (also referred to as the argmax function), and a third pose key point can be obtained. One key point heat map corresponds to one third pose key point. Thus, the first pose key point output by the trained key point regression network, the second pose key point obtained by calculating the offset, and the third pose key point obtained by the key point heat map are averaged to obtain the mean value pose key point after different angle averaging, which can make the finally obtained key point position more stable and more accurate.

[0142] Please refer to Figure 3 In one embodiment, the present application provides a roadside parking management system 100 based on key point moving pose. The roadside parking management system 100 based on key point moving pose includes a data acquisition module 10, an aggregated feature map acquisition module 20, a parallel network module 30, a smoothing network module 40, a first model training module 50, a second model training module 60, a detection module 70, a pose key point acquisition module 80, a mean value pose key point acquisition module 90, and a key point position smoothing processing module 910.

[0143] The data acquisition module 10 is configured to acquire a plurality of vehicle images of each vehicle in continuous time, each vehicle image being labeled with a plurality of real pose key points corresponding to a three-dimensional bounding box of the vehicle and a plurality of real offset amounts from the real pose key points to a real pose center point. The aggregated feature map acquisition module 20 is configured to input each vehicle image into a backbone network and a feature aggregation network of the key point detection model for feature extraction and fusion to obtain a plurality of aggregated feature maps. The parallel network module 30 is configured to input each aggregated feature map into a key point regression network, a key point offset regression network, and a key point heat map prediction network of the key point detection model in parallel to obtain a plurality of first predicted pose key points, a plurality of predicted offset amounts, and a plurality of predicted key point heat maps for each vehicle image of each vehicle.

[0144] The smoothing network module 40 is configured to calculate a plurality of real key point position moving speeds of each vehicle in continuous time according to the plurality of real pose key points of each vehicle image of each vehicle in continuous time, and input the plurality of real key point position moving speeds and the plurality of real pose key points into a key point position smoothing network model for smoothing processing to obtain a plurality of predicted key point position moving speeds and a plurality of second predicted pose key points. The first model training module 50 is configured to construct a loss function of the key point detection model according to the plurality of first predicted pose key points, the plurality of predicted offset amounts, the plurality of predicted key point heat maps, the plurality of real pose key points, the plurality of real offset amounts, and a heat map target value to train the model to obtain a trained key point detection model.

[0145] The second model training module 60 is configured to construct a loss function of the key point position smoothing network model according to the plurality of predicted key point position moving speeds, the plurality of second predicted pose key points, the plurality of real key point position moving speeds, and the plurality of real pose key points to train the model to obtain a trained key point position smoothing network model. The detection module 70 is configured to acquire a to-be-detected vehicle image and a pose center point of each to-be-detected vehicle, and input the to-be-detected vehicle image into the trained key point detection model to output a plurality of first pose key points, a plurality of offset amounts, and a plurality of key point heat maps for each to-be-detected vehicle.

[0146] The posture key point acquisition module 80 is configured to calculate a plurality of second posture key points of each vehicle to be detected according to the posture center point of each vehicle to be detected and a plurality of offsets, and obtain a plurality of third posture key points of each vehicle to be detected according to a plurality of key point heat maps of each vehicle to be detected. The mean posture key point acquisition module 90 is configured to calculate a mean value according to the first posture key points, the second posture key points and the third posture key points, and obtain a plurality of mean posture key points of each vehicle to be detected. The key point position smoothing processing module 910 is configured to input the plurality of mean posture key points of each vehicle to be detected into a trained key point position smoothing network model to perform position smoothing, obtain a plurality of smoothed posture key points of each vehicle to be detected, and perform roadside parking management on the vehicle to be detected according to the plurality of smoothed posture key points.

[0147] In the embodiment, the related description of the data acquisition module 10 can refer to the related description of S10 in the above embodiment. The related description of the aggregated feature map acquisition module 20 can refer to the related description of S20 in the above embodiment. The related description of the parallel network module 30 can refer to the related description of S30 in the above embodiment. The related description of the smoothing network module 40 can refer to the related description of S40 in the above embodiment. The related description of the first model training module 50 can refer to the related description of S50 in the above embodiment. The related description of the second model training module 60 can refer to the related description of S60 in the above embodiment. The related description of the detection module 70 can refer to the related description of S70 in the above embodiment. The related description of the posture key point acquisition module 80 can refer to the related description of S80 in the above embodiment. The related description of the mean posture key point acquisition module 90 can refer to the related description of S90 in the above embodiment. The related description of the key point position smoothing processing module 910 can refer to the related description of S910 in the above embodiment.

[0148] In one embodiment, the parallel network module 30 includes a heat map acquisition module. The heat map acquisition module is configured to input each aggregated feature map into a Gaussian convolution kernel to perform two-dimensional normal distribution, and generate a plurality of predicted key point heat maps, each of which is represented as:

[0149]

[0150] wherein x and y represent the coordinates of the key points in the predicted key point heat map, x' and y' represent the integer-quantized key point coordinates in the predicted key point heat map, and σ represents the standard deviation of the Gaussian distribution.

[0151] In the embodiment, the related description of the heat map acquisition module can refer to the related description of S310 in the above embodiment.

[0152] In an embodiment, the first model training module 50 comprises a predicted heat map value obtaining module and a heat map regression loss function constructing module. The predicted heat map value obtaining module is configured to calculate a predicted heat map value according to the focal loss function based on each predicted key point heat map. The heat map regression loss function constructing module is configured to set a heat map target value, construct a mean absolute error loss function according to the predicted heat map value and the heat map target value, and form a heat map regression loss function according to the mean absolute error loss function.

[0153] wherein the focal loss function is:

[0154] FL(p t )=-α t (1-p t ) γ log(p t );

[0155]

[0156] g=1 represents a coordinate value of a predicted key point in the predicted key point heat map, otherwise represents a coordinate value of other positions except the key point, P represents a probability of the predicted key point coordinate value being 1, and a t represents a first coefficient for controlling positive and negative samples, and γ represents a second coefficient for controlling positive and negative samples.

[0157] The mean absolute error loss function is:

[0158]

[0159] represents a predicted heat map value, y represents a heat map target value, and m represents a sample number.

[0160] In the embodiment, the related description of the predicted heat map value obtaining module can refer to the related description of S510 in the above embodiment. The related description of the heat map regression loss function constructing module can refer to the related description of S520 in the above embodiment.

[0161] In an embodiment, the pose key point obtaining module 80 comprises a third pose key point obtaining module. The third pose key point obtaining module is configured to perform maximum value finding on each key point heat map according to the argument maximum evaluation function to obtain a plurality of third pose key points.

[0162] In the embodiment, the related description of the third pose key point obtaining module can refer to the related description of S810 in the above embodiment.

[0163] In one embodiment, the smoothing network module 40 comprises a neighboring frame key point position obtaining module and a moving speed calculating module. The neighboring frame key point position obtaining module is configured to obtain the coordinate positions of the real attitude key points of each category corresponding to the first frame vehicle image and the coordinate positions of the real attitude key points of each category corresponding to the second frame vehicle image. The moving speed calculating module is configured to calculate the x-direction moving speed of the real attitude key points of each category in the x coordinate direction and the y-direction moving speed of the real attitude key points of each category in the y coordinate direction, respectively, according to the coordinate positions of the real attitude key points of each category corresponding to the first frame vehicle image and the coordinate positions of the real attitude key points of each category corresponding to the second frame vehicle image.

[0164] wherein the x-direction moving speed and the y-direction moving speed of the real attitude key points of each category are respectively:

[0165]

[0166] x represents the x coordinate position, y represents the y coordinate position, i represents the category of the real attitude key point, t represents the first frame vehicle image, t+1 represents the second frame vehicle image, represents the coordinate positions of the real attitude key points of each category corresponding to the first frame vehicle image, represents the coordinate positions of the real attitude key points of each category corresponding to the second frame vehicle image, represents the x-direction moving speed, represents the y-direction moving speed.

[0167] In this embodiment, the related description of the neighboring frame key point position obtaining module can refer to the related description of S410 in the above embodiment. The related description of the moving speed calculating module can refer to the related description of S420 in the above embodiment.

[0168] The above detailed description further explains the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above detailed description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method for managing roadside parking based on key point moving postures, characterized by, The method comprises the following steps: acquiring multiple frames of vehicle images of each vehicle in continuous time, each frame of the vehicle image being labeled with multiple real attitude key points corresponding to a three-dimensional bounding box of the vehicle and multiple real offsets from the real attitude key points to a real attitude center point; inputting each frame of the vehicle image into a backbone network and a feature aggregation network of a key point detection model for feature extraction and fusion to obtain multiple aggregated feature maps; parallel inputting each of the aggregated feature maps into a key point regression network, a key point offset regression network, and a key point heat map prediction network of the key point detection model for prediction to obtain multiple first predicted attitude key points, multiple predicted offsets, and multiple predicted key point heat maps of each frame of the vehicle image of each vehicle; calculating multiple real key point position moving speeds of each vehicle in continuous time according to the multiple real attitude key points of each frame of the vehicle image of each vehicle in continuous time, and inputting the multiple real key point position moving speeds and the multiple real attitude key points in continuous time into a key point position smoothing network model for smoothing processing to obtain multiple predicted key point position moving speeds and multiple second predicted attitude key points; constructing a loss function of the key point detection model according to the multiple first predicted attitude key points, the multiple predicted offsets, the multiple predicted key point heat maps, the multiple real attitude key points, the multiple real offsets, and a heat map target value to perform model training to obtain a trained key point detection model; constructing a loss function of the key point position smoothing network model according to the multiple predicted key point position moving speeds, the multiple second predicted attitude key points, the multiple real key point position moving speeds, and the multiple real attitude key points to perform model training to obtain a trained key point position smoothing network model; acquiring a to-be-tested vehicle image and an attitude center point of each to-be-tested vehicle, and inputting the to-be-tested vehicle image into the trained key point detection model to output multiple first attitude key points, multiple offsets, and multiple key point heat maps of each to-be-tested vehicle; calculating multiple second attitude key points of each to-be-tested vehicle according to the attitude center point and the multiple offsets of each to-be-tested vehicle, and obtaining multiple third attitude key points of each to-be-tested vehicle according to the multiple key point heat maps of each to-be-tested vehicle; performing average value calculation according to the first attitude key points, the second attitude key points, and the third attitude key points to obtain multiple mean attitude key points of each to-be-tested vehicle; inputting the multiple mean attitude key points of each to-be-tested vehicle into the trained key point position smoothing network model for position smoothing to obtain multiple smoothed attitude key points of each to-be-tested vehicle, and performing road side parking management on the to-be-tested vehicle according to the multiple smoothed attitude key points. 2.The keypoint movement based gesture based roadside parking management method of claim 1, wherein, The multiple first predicted posture key points, the multiple predicted offsets, and the multiple predicted key point heat maps of each of the vehicle images of each of the vehicles are obtained by respectively and in parallel inputting each of the aggregated feature maps into a key point regression network, a key point offset regression network, and a key point heat map prediction network of the key point detection model for prediction. Each of the aggregated feature maps is input into a Gaussian convolution kernel for two-dimensional normal distribution to generate a plurality of predicted key point heat maps, each of which is represented as: ; wherein x and y represent the coordinates of a predicted key point in the predicted key point heat map, x' and y' represent the coordinates of the predicted key point quantized to an integer in the predicted key point heat map, and σ represents the standard deviation of the Gaussian distribution. 3.The keypoint movement based gesture based roadside parking management method of claim 2, wherein, The loss function of the key point detection model is constructed according to the multiple first predicted posture key points, the multiple predicted offsets, the multiple predicted key point heat maps, the multiple real posture key points, the multiple real offsets, and a heat map target value, and model training is performed, so as to obtain a trained key point detection model. Based on each of the predicted key point heat maps, a predicted heat map value is calculated according to a focal loss function; The heat map target value is set, an average absolute error loss function is constructed according to the predicted heat map value and the heat map target value, and a heat map regression loss function is formed according to the average absolute error loss function; Wherein, the focus loss function is: g = 1 represents that the predicted key point coordinate value in the predicted key point heat map is 1, otherwise represents the coordinate value of other positions except the predicted key point, P represents the probability that the predicted key point coordinate value is 1, The first coefficient for controlling positive and negative samples is represented by α, and the second coefficient for controlling positive and negative samples is represented by γ; the mean absolute error loss function is: , The predicted heat map value is represented by z, the heat map target value is represented by z, and the total number of samples is m+1. 4.The keypoint movement based gesture based roadside parking management method of claim 1, wherein, The multiple second posture key points of each of the to-be-tested vehicles are calculated according to the posture center point and the multiple offsets of each of the to-be-tested vehicles, and the multiple third posture key points of each of the to-be-tested vehicles are obtained according to the multiple key point heat maps of each of the to-be-tested vehicles, including: The maximum value of each of the key point heat maps is found according to an independent variable maximum evaluation function, and the multiple third posture key points are obtained. 5.The keypoint movement based gesture based roadside parking management method of claim 1, wherein, The multiple real key point position moving speeds of each of the vehicles in continuous time are calculated according to the multiple real posture key points of each of the vehicle images of each of the vehicles in continuous time, including: The coordinate positions of the real posture key points of each category corresponding to the first frame of the vehicle image and the coordinate positions of the real posture key points of each category corresponding to the second frame of the vehicle image of each of the vehicles are obtained; The x-direction moving speed of the real posture key points of each category in the x coordinate direction and the y-direction moving speed of the real posture key points of each category in the y coordinate direction are respectively calculated according to the coordinate positions of the real posture key points of each category corresponding to the first frame of the vehicle image and the coordinate positions of the real posture key points of each category corresponding to the second frame of the vehicle image. Wherein, the x direction moving speed and the y direction moving speed of each category of real posture key points are respectively: x represents the x coordinate position, y represents the y coordinate position, i represents the category of the real posture key point, t represents the first frame of the vehicle image, t+1 represents the second frame of the vehicle image, represents the coordinate position of each category of real posture key points corresponding to the first frame of the vehicle image, represents the coordinate position of each category of real posture key points corresponding to the second frame of the vehicle image, represents the x direction moving speed, represents the y direction moving speed.

6. A keypoint movement based gesture based road side parking management system characterized by, The data acquisition module is configured to obtain multiple frames of vehicle images of each vehicle in continuous time, and each frame of the vehicle image is labeled with multiple real posture key points corresponding to a three-dimensional bounding box of a vehicle and multiple real offsets from the multiple real posture key points to a real posture center point. The aggregated feature map acquisition module is configured to input each frame of the vehicle image into a backbone network and a feature aggregation network of a key point detection model for feature extraction and fusion, to obtain multiple aggregated feature maps. ​ The parallel network module is configured to input each of the aggregated feature maps into a key point regression network, a key point offset regression network, and a key point heat map prediction network of the key point detection model in parallel to obtain a plurality of first predicted attitude key points, a plurality of predicted offsets, and a plurality of predicted key point heat maps of each of the vehicle images of each of the vehicles. The smoothing network module is configured to calculate a plurality of real key point position moving speeds of each of the vehicles in continuous time based on the plurality of real attitude key points of each of the vehicle images in continuous time, and input the plurality of real key point position moving speeds and the plurality of real attitude key points in continuous time into a key point position smoothing network model to perform smoothing processing, to obtain a plurality of predicted key point position moving speeds and a plurality of second predicted attitude key points. The first model training module is configured to train the key point detection model based on the plurality of first predicted attitude key points, the plurality of predicted offsets, the plurality of predicted key point heat maps, the plurality of real attitude key points, the plurality of real offsets, and a heat map target value to obtain a trained key point detection model. The second model training module is configured to train the key point position smoothing network model based on the plurality of predicted key point position moving speeds, the plurality of second predicted attitude key points, the plurality of real key point position moving speeds, and the plurality of real attitude key points to obtain a trained key point position smoothing network model. The detection module is configured to obtain a to-be-detected vehicle image and an attitude center point of each to-be-detected vehicle, and input the to-be-detected vehicle image into the trained key point detection model to output a plurality of first attitude key points, a plurality of offsets, and a plurality of key point heat maps of each to-be-detected vehicle. The attitude key point acquisition module is configured to calculate a plurality of second attitude key points of each to-be-detected vehicle based on the attitude center point and the plurality of offsets of each to-be-detected vehicle, and obtain a plurality of third attitude key points of each to-be-detected vehicle based on the plurality of key point heat maps of each to-be-detected vehicle. The mean attitude key point acquisition module is configured to calculate mean values based on the first attitude key points, the second attitude key points, and the third attitude key points to obtain a plurality of mean attitude key points of each to-be-detected vehicle. The key point position smoothing processing module is configured to input the plurality of mean attitude key points of each to-be-detected vehicle into the trained key point position smoothing network model to perform position smoothing to obtain a plurality of smoothed attitude key points of each to-be-detected vehicle, and perform roadside parking management on the to-be-detected vehicle based on the plurality of smoothed attitude key points.

7. The keypoint movement pose-based roadside parking management system of claim 6, wherein, The parallel network module includes: The heat map acquisition module is configured to input each of the aggregated feature maps into a Gaussian convolution kernel to generate a two-dimensional normal distribution, and generate a plurality of predicted key point heat maps, wherein each of the predicted key point heat maps is represented as: ; wherein x and y represent the coordinates of the predicted key points in the predicted key point heat map, x' and y' represent the coordinates of the quantized predicted key points in the predicted key point heat map, and σ represents the standard deviation of the Gaussian distribution.

8. The keypoint movement pose-based roadside parking management system of claim 7, wherein, The first model training module includes: The predicted heat map value acquisition module is configured to calculate a predicted heat map value based on each of the predicted key point heat maps according to a focus loss function. The heat map regression loss function construction module is configured to set the heat map target value, construct a mean absolute error loss function according to the predicted heat map value and the heat map target value, and form a heat map regression loss function according to the mean absolute error loss function. wherein the focus loss function is: g = 1 represents a predicted key point coordinate value in the predicted key point heat map, otherwise represents a coordinate value of other positions except the predicted key point, P represents a probability of the predicted key point coordinate value being 1, represents a first coefficient for controlling positive and negative samples, and γ represents a second coefficient for controlling positive and negative samples. The average absolute error loss function is: , represents the predicted heat map value, z represents the heat map target value, and the total number of samples is m+1.

9. The keypoint movement pose-based roadside parking management system of claim 6, wherein, The posture key point acquisition module comprises: The third posture key point acquisition module is configured to perform maximum value searching on each of the key point heat maps according to an independent variable maximum evaluation function, and obtain the plurality of third posture key points.

10. The keypoint movement pose-based roadside parking management system of claim 6, wherein, The smoothing network module comprises: The adjacent frame key point position acquisition module is configured to acquire coordinate positions of each category of real posture key points corresponding to the vehicle image of the first frame and coordinate positions of each category of real posture key points corresponding to the vehicle image of the second frame of each of the vehicles. The moving speed calculation module is configured to calculate an x-direction moving speed of each category of real posture key points in an x coordinate direction and a y-direction moving speed of each category of real posture key points in a y coordinate direction according to the coordinate positions of each category of real posture key points corresponding to the vehicle image of the first frame and the coordinate positions of each category of real posture key points corresponding to the vehicle image of the second frame, respectively. Wherein, the x direction moving speed and the y direction moving speed of each category of real posture key points are respectively: x represents the x coordinate position, y represents the y coordinate position, i represents the category of the real posture key point, t represents the first frame of the vehicle image, t+1 represents the second frame of the vehicle image, represents the coordinate position of each category of real posture key points corresponding to the first frame of the vehicle image, represents the coordinate position of each category of real posture key points corresponding to the second frame of the vehicle image, represents the x direction moving speed, represents the y direction moving speed.

Images