Method for recognizing the appearance of a driver and automatic adjustment of the seat and airbag

Abstract

The application discloses a kind of driver's appearance feature recognition and seat and airbag automatic adjustment method, utilize 2DCNN+LSTM action recognition algorithm, capture the image in the process of driver getting on, select key frame by adaptive screening strategy, for accurate prediction of the height and body shape of driver.By labeling skeleton point, generate two-dimensional skeleton connection diagram based on VitPose algorithm, and complete three-dimensional reconstruction in combination with JOTR algorithm, significantly improve the prediction accuracy of body shape parameters.The application adjusts the seat height, backrest angle and seat belt position according to the body shape of the driver, and optimizes the deployment position, trigger strategy and deployment range of the airbag in combination with the three-dimensional skeleton points and body shape parameters, to accurately protect the driver's head, chest and shoulders in the event of a collision, and reduce the risk of secondary injury.The application supports local data storage, protects privacy, improves driving safety and comfort, and provides an intelligent driving experience for users.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of intelligent driving technology, covering multiple disciplines such as computer vision, machine learning, human pose recognition, and car-human interaction, and specifically relates to a method for recognizing the physical features of a driver or passenger and automatically adjusting the seat and airbag. BACKGROUND

[0002] In today's rapidly developing intelligent driving technology, personalized adaptation of drivers has become an important research direction for improving driving experience and road safety. Traditional vehicle facility adjustment methods, such as seat position and airbag configuration, mostly rely on manual adjustment by the driver. This method is not only time-consuming and tedious, but also prone to discomfort or safety hazards due to improper adjustment.

[0003] To simplify this process, some existing technologies have proposed intelligent seat adjustment solutions. For example, CN110316027 stores the information of the first manual setting of the seat and rearview mirror position in the vehicle system, so as to automatically restore it the next time it is used. However, this method is still limited to fixed memory adjustment after initial setting and fails to adaptively adjust according to changes in the driver's body size or posture. In addition, CN107232822 proposes an intelligent seat adjustment system based on pressure sensors and posture sensors, which adjusts the seat position by detecting the pressure and posture changes on the seat. Although this technology provides posture recognition function, the accuracy of the sensors it relies on is limited, and the function is relatively simple, which cannot accurately predict the driver's height and body size.

[0004] Based on the above analysis, the shortcomings of existing technologies can be summarized as follows: (1) Lack of dynamic body size adaptability: most existing systems are based on fixed initial settings or single sensor information, and cannot adjust in real time according to changes in the driver's body size and posture; (2) Limited adjustment accuracy: although some systems use sensors to detect seat pressure or posture, they cannot accurately predict the driver's body size, resulting in insufficient accuracy of seat adjustment; (3) Single function: existing technologies mainly focus on basic position adjustment of the seat, without considering the impact of other key factors (such as skeletal point posture) in the overall driving environment on the driver's comfort and safety. In this context, there is an urgent need for a more intelligent vehicle seat adjustment system that can accurately predict the driver's height and body size through advanced pose recognition algorithms and machine learning models, and automatically adjust the seat position to meet the individual needs of different drivers, improving driving experience and road safety; (4) Lack of convenience and safety: without deploying locally, it cannot identify the driver instantly and unobtrusively, and the information is not stored locally, which cannot guarantee the driver's personal privacy information. SUMMARY

[0005] In the context of the continuous development of intelligent driving and vehicle systems, intelligent adjustment technology based on physical characteristics has become an important innovative direction for improving driver driving experience and road safety. Traditional adjustment methods mostly rely on manual adjustment of seats, rearview mirrors and other facilities by drivers to adapt to individual height, body shape and comfort needs. This method is not only tedious and time-consuming, but also prone to poor driving environment due to improper adjustment, affecting the accuracy of vision and operation, and thus reducing driving safety. At the same time, existing intelligent adjustment technologies mostly focus on single functions, such as seat position memory or pressure-sensitive adjustment, failing to achieve comprehensive adaptation to the overall driving environment and lacking dynamic real-time adjustment capability.

[0006] The present application captures key skeleton points in multiple frames of images based on a VitPose-based pose recognition algorithm, and accurately predicts the height and body shape of the driver using a deep learning algorithm. This not only solves the problem of lack of consideration of driver body shape differences in existing technologies, but also enables automatic adjustment of seats and airbags, allowing each driver to enjoy personalized, accurate comfort and safety experience. This intelligent adjustment method avoids the tediousness of traditional manual operation, significantly improving user driving experience and safety.

[0007] To achieve the above-mentioned application purposes, the present application proposes a method for recognizing the physical characteristics of drivers and passengers and automatically adjusting seats and airbags, comprising the following steps:

[0008] S1 Image capture and dataset construction Captures continuous image sequences of the entire process of the driver getting into the car through the in-vehicle camera, analyzes the continuous image sequences using a 2D CNN+LSTM-based action recognition algorithm, obtains the action features of the driver to form a key frame dataset, and filters the key frame dataset and divides it into a training set and a test set in proportion;

[0009] S2 Key skeleton point labeling Labeling of each key frame image I i is performed using an automatic labeling tool to generate a labeled skeleton point coordinate set P label ={(x label ,y label )},x label represents the horizontal coordinate of a certain skeleton point on the image plane, and y label represents the vertical coordinate of a certain key skeleton point on the image plane. This labeling data is used for subsequent two-dimensional skeleton point recognition system training;

[0010] S3 Two-dimensional skeleton point recognition system construction Feature extraction is performed on the key frame image I i to generate a preliminary skeleton point coordinate set P i and a corresponding skeleton point line graph S i; combined with the physical layout inside the vehicle, an optimized result set is generated through multi-modal fusion and self-attention mechanism and The dynamic changes of multiple image skeleton points are analyzed, and a dynamic optimization result is generated using spatio-temporal convolution and Through human body proportion correction, a two-dimensional skeleton point coordinate set is output and the corresponding skeleton point connection diagram S final ;

[0011] S4 obtains the in-vehicle space data based on the three-dimensional reconstruction of the two-dimensional skeleton points, models the in-vehicle three-dimensional space using the modeling tool Blender; uses the two-dimensional skeleton point coordinate set combined with the JOTR algorithm to calculate the three-dimensional skeleton point coordinates uses the self-supervised learning framework of the generative adversarial network to correct the three-dimensional coordinates after correction Based on the correction result, the Euclidean distance d between the three-dimensional skeleton points is calculated ij , and through the multi-layer perception MLP, the height and body shape parameters ShapeParams of the driver are mapped and output the accurate height of the driver and body shape parameters ShapeParams

[0012] S5 seat automatic adjustment according to the height body shape parameters ShapeParams of the driver, analyzes the position and posture angle of the driver's head, shoulders and waist, and dynamically adjusts the height, backrest angle and seat cushion angle of the seat to match the body shape requirements of the driver; adopt dynamic adjustment strategy, real-time monitoring of the driver's posture changes in the long driving process, and intelligent response to optimize the seat settings, reduce fatigue.

[0013] S6 automatic adjustment of airbag through analysis of the three-dimensional skeleton point coordinates body shape parameters ShapeParams and real-time posture information of the driver, dynamically optimize the deployment position, expansion range and trigger strategy of the airbag; adopt real-time monitoring mechanism to ensure that the airbag can accurately protect the key parts of the driver such as the head, chest and shoulders when it is deployed, and reduce the risk of secondary injury in the collision.

[0014] Further, the image capture and data set construction in the above step S1 specifically include:

[0015] S1.1 2D convolutional neural network extracts spatial features such as the contour of the human body, key limb parts and their motion trajectories from each frame of the continuous image sequence;

[0016] S1.2 The LSTM model is used to process the time sequence information in the continuous image sequence, capture the continuous action features of the driver during the boarding process, analyze the action transition features in a group of adjacent frames, and identify the turning points of the action;

[0017] S1.3 Analyzing the time curve of the amplitude change of the action, marking the key frame at the moment of sharp change in amplitude, and capturing the starting frame and the stopping frame of each action;

[0018] S1.4 Formulate an adaptive screening strategy to select key frame images I i , and form a key frame dataset {I i}. Divide the screened key frame dataset into training set and test set according to the proportion, and ensure the generalization ability and accuracy of the model.

[0019] Further, the adaptive screening strategy includes the following contents:

[0020] S1.4.1) Action fluency quantification: the action fluency is quantified by calculating the optical flow between two adjacent frames of images;

[0021] Let image frames I t and I t+1 be two consecutive frames of images, and the optical flow F t is calculated as follows:

[0022]

[0023] Where T is the time interval, F t is the displacement vector between the two frames, representing the movement of the object between the frames; if the displacement exceeds the threshold, it is considered that the action has changed greatly and does not meet the fluency requirement;

[0024] Calculate the mean square error MSE of the optical flow between adjacent frames, let F t and F t+1 be the optical flow of adjacent frames, and calculate the mean square error as follows:

[0025]

[0026] Select the index, if MSE(F t , F t+1 ) is higher than the threshold value 0.15, it is considered that the action is not smooth, and the frame is removed;

[0027] S1.4.2) Quantification of limb part visibility: use OpenPose to quickly detect the skeleton of the image and obtain the preliminary visibility score of the key parts; the visibility V(p i) According to the detection confidence score and the image quality, the score range is [0, 1]; using the spatial information of the fixed structure in the vehicle, such as the seat and the steering wheel, to determine whether a certain part is blocked; the occlusion area O of the fixed structure in the vehicle and the total visible area V(p i ) of the part are quantified using the following occlusion index:

[0028]

[0029] Where |V(p i )∩O| is the area of the part that is blocked, and |V(p i )| is the total visible area of the part. If the occlusion ratio is greater than the set threshold of 40%, the part is considered to be invisible.

[0030] S1.4.3) Occlusion evaluation: set the occlusion area percentage as an evaluation index to determine whether the fixed structure in the vehicle (such as the seat and the steering wheel) causes a large occlusion to the driver's key parts (such as the upper limbs and legs); process the image through the image segmentation technology Mask R-CNN to extract the vehicle structure area R car and the driver area R driver , calculate the intersection and overlapping area of the two areas; let |R car ∩R driver | be the intersection area of the vehicle structure and the driver area, and |R driver | be the total area of the driver area, then the occlusion degree is:

[0031]

[0032] If the occlusion ratio exceeds 40%, it is considered that the frame has a large occlusion problem.

[0033] Further, the specific method of the above steps S2 data labeling and skeleton point selection is:

[0034] S2.1 Use the CVAT automatic labeling tool to label the skeleton points of each key frame image I i selected, and design a dynamic labeling strategy according to the special scene in the vehicle and the continuous action of the driver from opening the door to holding the steering wheel, and the labeling scheme is as follows:

[0035] Opening the door stage: focus on labeling the shoulder, elbow, wrist, knee, ankle and other skeleton points to capture the driver's action trajectory of stretching the hand to pull the door, and ensure that the core posture features of the action are recorded. This stage is the only one to show the complete knee to ankle stage of the driver;

[0036] Sitting stage: focus on labeling key skeleton points such as the spine and knees to obtain the sitting angle and body rotation state of the driver.

[0037] Safety belt stage: focus on capturing subtle changes in the positions of upper body skeletal points such as shoulders, elbows, and waist, accurately recording the driver's side movement, and providing data support for subsequent analysis;

[0038] Steering wheel stage: focus on labeling skeletal points such as shoulders, elbows, wrists, and neck to ensure accurate recording of the final driving posture;

[0039] S2.2 The key skeletal points labeled include the following parts: (1) nose; (2) neck; (3) left shoulder; (4) right shoulder; (5) chest; (6) left elbow; (7) right elbow; (8) left hip; (9) right hip; (10) left knee; (11) right knee; (12) left ankle; (13) right ankle.

[0040] Further, the above step S3 includes:

[0041] S3.1 The feature extraction stage uses the VitPose network to perform feature extraction on each key frame image I i , generating a preliminary feature representation related to skeletal points;

[0042] Input the key frame image I i , assuming it is an image of size HxWxC, where H and W are the height and width of the image, respectively, and C is the number of channels; divide the image into image blocks of size PXP, with P 2 ·C pixels in each image block, and assuming there are a total of N image blocks, the formula for calculating N is:

[0043]

[0044] Embed each image block into a d-dimensional feature vector space through linear transformation, obtaining the embedding vector e ij ∈R d for each image block, the specific formula is as follows:

[0045] e ij =W patch P ij +b patch

[0046] where W patch is the weight of linear transformation, and b ij is the bias term, stack all embedding vectors e N×d into a matrix E ∈ R , and input it into the Transformer encoder, which captures global context information through self-attention mechanism, and its core calculation formula is:

[0047]

[0048] where Q, K, and V are matrices of queries, keys, and values, respectively, d k is a scaling factor of feature dimension, after being processed by multiple Transformer layers, the output feature matrix is:

[0049] z i = Transformer(E), z i ∈R N×d

[0050] The output feature matrix z i contains the global context information of the image;

[0051] S3.2 Pose Estimation Layer (SE_Layer) further predicts the preliminary skeletal point coordinates and connection graph of the driver based on the feature matrix z i

[0052] The feature matrix z i ∈R N×d generated by the Transformer encoder and the in-vehicle physical layout S car are input into the SE_Layer, through multi-modal fusion, S car is fused with z i , and input into the regression network g pose , to predict the preliminary skeletal point coordinates:

[0053] P i = g pose (concat(z i , S car )), P i = {(x i , y i )}

[0054] where P i = {(x i , y i )} represents the preliminary skeletal point coordinate set, including the two-dimensional position of each skeletal point, g pose is a regression function used to extract skeletal point coordinates from the feature matrix, concat represents concatenating the two kinds of information together, according to the skeletal point coordinate set P i , a skeletal point connection graph S i is constructed, which is generated through the topological relationship between skeletal points and describes the skeletal point features of the driver:

[0055] S i = (V, E), V = P i , E = {(v k , v j ) | v​k , v j ∈ V, and satisfy the connection relationship

[0056] The output result includes a preliminary set of bone point coordinates P i and a corresponding bone point connection graph S i ;

[0057] S3.3 Scene-aware Feature Integration Layer (SF_Layer) further optimizes the features of the bone points through in-vehicle environment perception mechanisms;

[0058] Input the preliminary set of bone point coordinates P i , the preliminary bone point connection graph S i , and the in-vehicle physical layout S car , respectively perform linear transformation on the bone point connection graph S i and the in-vehicle physical layout S car to map them to the same high-dimensional feature space:

[0059]

[0060] wherein, and are the weight matrices of linear transformation, and are the bias terms, and the relevance between the bone point features and the in-vehicle physical layout is calculated using the self-attention mechanism:

[0061]

[0062] wherein, d represents the feature dimension, S′ i (S′ car ) T reflects the degree of association between each bone point and the in-vehicle structure, softmax is the normalized attention weight, Attn i is the weighted influence result of the in-vehicle physical layout on the bone point features, and the corresponding bone point connection graph S is constructed according to the optimized set of bone point coordinates P

[0063]

[0064] The output optimized set of bone point coordinates P and the corresponding bone point connection graph S

[0065] S3.4 Spatio-Temporal Layer (ST_Layer) optimizes the temporal consistency and spatial coherence of the skeleton points by analyzing the changes of the skeleton points between consecutive frames of images;

[0066] Input the optimized skeleton point coordinate set of consecutive frames and the corresponding skeleton point connection diagram Integrate them into Stack the multi-frame feature data into a spatio-temporal feature block with dimensions T x N x d, where T represents the number of frames in the time dimension, and a 3D convolution is used to capture the time and spatial features, and optimize the dynamic coherence of the skeleton points:

[0067]

[0068] where W ST ∈R d′×d×k×k×T is a 3D convolution kernel, d' is the output feature dimension, k is the spatial dimension of the convolution kernel, T is the range of the time dimension, and z ST ∈R N×d′ is a high-dimensional feature matrix containing spatio-temporal context information, which considers the position changes of the skeleton points between multiple frames of images and captures the time coherence of the driver's actions; after spatio-temporal convolution processing, the skeleton point coordinates are dynamically optimized using dynamic weights, as follows:

[0069]

[0070] where, is the skeleton point coordinates of the t-th frame, a t is the dynamic weight, which is calculated based on the spatio-temporal feature correlation of each frame of skeleton points:

[0071]

[0072] where MLP is a multi-layer perceptron that extracts the correlation score f ST between each frame of skeleton point features and the overall dynamic optimization target from the spatio-temporal feature z t , which is used for subsequent weight calculation; the corresponding skeleton point connection diagram is constructed based on the dynamically optimized skeleton points

[0073]

[0074] The topological relationship here is the same as , but it is recalculated based on the dynamically optimized skeleton point positions.

[0075] Output the dynamically optimized skeleton point coordinate set ​and the corresponding skeleton point connection graph which retains the consistency in time and space;

[0076] S3.5 Proportion Adjustment Layer (PA_Layer) adjusts the dynamic optimized skeleton point set and the corresponding skeleton point connection graph corrects the human proportion to ensure compliance with human anatomy standards.

[0077] input the dynamic optimized skeleton point coordinate set and the corresponding skeleton point connection graph predict the physical size of the skeleton point from the features through the regression network:

[0078] h i = MLP(z ST )

[0079] According to the standard human proportion h max , calculate the proportion factor λ i of each skeleton point:

[0080]

[0081] λ i is a scaling factor representing the physical size of each skeleton point relative to the standard human proportion, which is used to ensure that the skeleton point prediction result can adapt to drivers of different body types, so the proportion factor λ i of each skeleton point is calculated independently. Adjust the skeleton point coordinates using the proportion factor:

[0082]

[0083] According to the corrected skeleton point coordinates , construct the corresponding skeleton point connection graph S final .

[0084] output the proportion-corrected skeleton point coordinate set and the corresponding skeleton point connection graph S final as the final two-dimensional skeleton point recognition result.

[0085] Further, the above step S4 specifically includes three-dimensional reconstruction of the two-dimensional skeleton point:

[0086] According to the two-dimensional skeleton point coordinate set , combine the internal structure information of the vehicle and the JOTR algorithm to complete the three-dimensional skeleton point reconstruction, finally generate the three-dimensional skeleton point connection graph and extract the body size parameters of the driver.

[0087] S4.1 Input the in-vehicle space data (such as seat height, width, depth, and in-vehicle length and width) for modeling, and use the modeling tool Blender to build the constraint structure of the three-dimensional space in the vehicle;

[0088] S4.2 Use the JOTR algorithm to combine the two-dimensional skeleton point coordinate set to calculate the three-dimensional coordinates of each skeleton point to provide input for subsequent three-dimensional reconstruction:

[0089]

[0090] initial three-dimensional skeleton points There is a certain error, so it is corrected through a self-supervised learning framework of a generative adversarial network (GAN), and the three-dimensional coordinates are projected to a two-dimensional plane to obtain a projection

[0091]

[0092] The projection function is calculated based on camera parameters and perspective relationships.

[0093] S4.3 The discriminator D feeds back the error between the points projected to the two-dimensional plane and the original two-dimensional points to the generator G, minimizing the loss function L self , and continuously optimizing the generator so that the three-dimensional coordinates output by the generator are more consistent with the two-dimensional projection of the real image points:

[0094]

[0095] After correction, the generator G outputs the corrected three-dimensional skeleton points:

[0096]

[0097] S4.4 According to the corrected three-dimensional skeleton point set generate a three-dimensional skeleton point connection diagram whose topological relationship is the same as the two-dimensional connection diagram S final , but is recalculated in three-dimensional space, calculating the Euclidean distance d ij between three-dimensional skeleton points, which is used to represent the limb length characteristics of the driver:

[0098]

[0099] where d ij represents the three-dimensional distance between the i th skeleton point and the j th skeleton point, and the distance feature set {d ij} is input to a multilayer perceptron (MLP) to map the height of the driver and body shape parameters ShapeParams:

[0100]

[0101] Final output more accurate driver height and its body shape parameters ShapeParams.

[0102] Further, the dynamic adjustment strategy of step S5 is as follows:

[0103] S5.1 Seat fore-aft adjustment: Ensure proper distance between driver and pedals, steering wheel for optimal control and comfort;

[0104] S5.2 Seat height adjustment: Ensure driver has optimal view, reducing interference from roof or other components;

[0105] S5.3 Seat back angle adjustment: Ensure adequate back support, reducing driver fatigue;

[0106] S5.4 Seat and steering wheel linkage adjustment: Ensure driver can comfortably control steering wheel, avoiding unnatural arm extension;

[0107] S5.5 Seat belt adjustment: Ensure seat belt accurately restrains driver, providing optimal safety protection.

[0108] Further, the above step S6 includes:

[0109] S6.1 Utilize three-dimensional skeletal point coordinates Extract driver's key part coordinate set P key :

[0110]

[0111] wherein, respectively represent the corrected three-dimensional coordinates of the head, chest and shoulders, represents the corrected three-dimensional coordinates of the key part k,

[0112] S6.2 Calculate driver body shape feature range according to body shape parameters ShapeParams:

[0113]

[0114] wherein, BodyWidth represents the horizontal distance between the left and right endpoints of the driver's shoulders, and are respectively the x-axis values of the corrected three-dimensional coordinates of the right shoulder and left shoulder; BodyHeight represents the vertical distance from the driver's head to the hips, and z-axis values of the corrected three-dimensional coordinates of the head and the hip, respectively;

[0115] S6.3 Calculate the optimal center position P key and coverage radius R deploy of the airbag deployment based on the key point coordinates P deploy of the driver:

[0116]

[0117] where P deploy is the center point of the airbag deployment, calculated based on the average coordinates of the driver's key points, and R deploy is the coverage radius of the airbag, determined by the maximum three-dimensional distance of the key points;

[0118] S6.4 Dynamically adjust the triggering conditions of the airbag in combination with the in-vehicle crash sensor data; the calculation formula of the trigger threshold T trigger is:

[0119] T trigger = w1F collision + w2θ impact + w3Δ pose

[0120] where F collision is the collision force, θ impact is the collision angle, Δ pose is the driver's sitting posture deviation, and w1, w2, w are weight parameters used to adjust the influencing factors of the triggering conditions;

[0121] The force P inflate of the airbag deployment is calculated according to the driver's body shape parameters and the severity of the collision:

[0122] P inflate = f(ShapeParams, Collision Severity)

[0123] S6.5 Dynamically adjust the airbag deployment direction and deployment strategy by real-time monitoring of the driver's sitting posture changes;

[0124] The adjusted airbag deployment angle α adjusted = α default + kΔ pose is represented as:

[0125] α adjusted = α default + k·Δ pose

[0126] where α default is the default deployment angle, k is the sitting posture correction coefficient, and ·Δpose to monitor the sitting posture deviation amount; obtain the deployment center point P of the airbag deploy , coverage range R deploy , trigger threshold T trigger and deployment intensity P inflate , the driver dynamically adjusted deployment angle α adjusted , the adjustment parameters of the airbag are provided, which can accurately protect the key parts of the driver when the airbag is deployed, and reduce the risk of secondary injury in the collision.

[0127] Compared with the prior art, the present application has the following advantages:

[0128] 1. Efficient motion capture and data labeling process: multiple frames of images are captured by dual-view cameras and adaptive screening strategy, and data labeling efficiency is improved by using CVAT semi-automatic labeling tool to ensure that the captured body feature data is more comprehensive and accurate, thereby enhancing the prediction performance of the model.

[0129] 2. Improved posture recognition accuracy in obstructed environment: the improved VitPose posture recognition algorithm combines the obstruction perception mechanism of the special scene in the car, which can accurately identify the obstructed bone points by the seat, steering wheel, etc., and realize more accurate two-dimensional bone point recognition.

[0130] 3. Enhanced accuracy of three-dimensional reconstruction: three-dimensional reconstruction is realized based on two-dimensional bone points using JOTR algorithm, combined with in-car space data, which improves the accuracy of driver body shape and height prediction, and overcomes the influence of in-car space limitations and obstructions on body shape measurement.

[0131] 4. Automatic driving position adjustment: the system automatically adjusts the seat and seat belt position according to the driver's body data, so that the driver can obtain the best safe driving posture, improve driving comfort and safety, and solve the inconvenience and insufficient adaptability of traditional manual adjustment.

[0132] 5. Precise deployment of airbag: through three-dimensional bone point and body shape parameter analysis, dynamically adjust the deployment position, coverage range and trigger strategy of the airbag, ensure the precise protection of the driver's head, chest and shoulder, etc. Key parts in the collision, reduce the risk of secondary injury, and significantly improve the passive safety performance of the vehicle.

[0133] 6. Privacy protection and local data processing: support for on-board deployment and local data storage, do not rely on cloud processing, ensure the user's autonomous control of personal data, effectively protect privacy and security. BRIEF DESCRIPTION OF DRAWINGS

[0134] Figure 1 is the flowchart of the two-dimensional bone point recognition system ICDPose constructed by the present application;

[0135] Figure 2 The final effect structure of the present application is shown in the schematic diagram. DETAILED DESCRIPTION

[0136] The technical solutions of the present application will be described in detail below, but the protection scope of the present application is not limited to the embodiments.

[0137] As shown in Figure 1 , Figure 2 , a driver physical feature recognition and seat and airbag automatic adjustment method of the present application comprises the following steps:

[0138] S1 image capture and data set construction captures a continuous image sequence of the whole process of the driver getting on the vehicle through the in-vehicle camera, analyzes the continuous image sequence using a motion recognition algorithm based on 2D CNN+LSTM, obtains the motion features of the driver to form a key frame data set, filters the key frame data set, and divides the filtered key frame data set into a training set and a test set according to a proportion;

[0139] S2 key skeleton point labeling uses an automatic labeling tool to label the skeleton points of each key frame image I i , generates a labeled skeleton point coordinate set P label ={(x label , y label )}, x label represents the horizontal coordinate of a certain skeleton point on the image plane, and y label represents the vertical coordinate of a certain key skeleton point on the image plane, and this labeling data is used for subsequent two-dimensional skeleton point recognition system training;

[0140] S3 two-dimensional skeleton point recognition system construction extracts features from the key frame image I i , generates a preliminary skeleton point coordinate set P i and a corresponding skeleton point connection graph S i ; combined with the spatial positions of fixed structures such as seats, steering wheels, rearview mirrors, etc. in the vehicle, an optimized result set and is generated through multi-modal fusion and self-attention mechanism, and the dynamic changes of the skeleton points of multiple images are analyzed, and a dynamic optimization result and is generated using spatio-temporal convolution, and the two-dimensional skeleton point coordinate set and the corresponding skeleton point connection graph S final are output through human body proportion correction;

[0141] S4 acquires in-vehicle space data, such as seat height, width, and depth, and interior space length and width, based on 3D reconstruction using 2D skeleton points. It then uses the modeling tool Blender to model the 3D in-vehicle space, utilizing a set of 2D skeleton point coordinates. Using the JOTR algorithm to calculate the coordinates of 3D skeleton points Using a self-supervised learning framework of generative adversarial networks to Perform correction to generate corrected 3D coordinates. The Euclidean distance d between 3D skeleton points is calculated based on the correction results. ij And mapped to the driver's height via a multilayer perceptron (MLP). Based on the body shape parameter ShapeParams, the driver's precise height is output. and body shape parameters ShapeParams;

[0142] The S5 seats automatically adjust to the driver's height. The ShapeParams parameter analyzes the position and sitting angle of the driver's head, shoulders, and waist, and dynamically adjusts the seat height, backrest angle, and seat cushion angle to match the driver's body shape needs. It adopts a dynamic adjustment strategy to monitor the driver's posture changes in real time during long-term driving and intelligently responds to optimize seat settings and reduce fatigue.

[0143] The S6 airbag automatic adjustment analyzes the driver's three-dimensional skeletal coordinates. The system dynamically optimizes the deployment location, deployment range, and triggering strategy of airbags using body shape parameters (ShapeParams) and real-time seating posture information. It also employs a real-time monitoring mechanism to ensure that airbags can accurately protect the driver's critical parts, such as the head, chest, and shoulders, reducing the risk of secondary injuries during a collision.

[0144] In a preferred embodiment of the present invention, image capture and dataset construction in S1 specifically include:

[0145] S1.1 2D convolutional neural networks extract spatial features, such as the outline of the human body, key limb parts and their motion trajectories, from each frame of a continuous image sequence.

[0146] S1.2 LSTM model is used to process temporal information in continuous image sequences, capture the continuous action features of the driver during the process of getting into the vehicle, and analyze the action transition features in a set of adjacent frames to identify the turning point of the action.

[0147] S1.3 Analyze the time curve of the motion amplitude change, mark the key frame at the moment of a sharp change in amplitude, and capture the start frame and stop frame of each motion;

[0148] S1.4 Develop an adaptive filtering strategy to select keyframe images I i The keyframe dataset {I} is composed of keyframes. i The selected keyframe dataset is divided into training and testing sets according to a certain ratio to ensure the model's generalization ability and accuracy.

[0149] As a preferred embodiment of the present invention, the adaptive filtering strategy includes the following:

[0150] S1.4.1) Motion smoothness quantization: Motion smoothness is quantified by calculating the optical flow between two adjacent frames;

[0151] Let image frame I t and I t+1 These are two consecutive frames, with optical flow F. t The calculation method is as follows:

[0152]

[0153] Where T is the time interval, F t It is the displacement vector between two frames, representing the movement of an object between frames; if the displacement exceeds the threshold, it is considered that the action has changed significantly and does not meet the smoothness requirements.

[0154] Calculate the mean square error (MSE) of optical flow between adjacent frames, let F t and F t+1 Calculate the root mean square error of the optical flow in adjacent frames:

[0155]

[0156] Select an indicator, if MSE(F) t F t+1 If the value is higher than the threshold of 0.15, the motion is considered not smooth and the frame is discarded.

[0157] S1.4.2) Visibility Quantization of Limb Parts: OpenPose is used to perform fast skeleton detection on the image to obtain preliminary visibility scores for key parts; the visibility V(p) of each part is calculated. i The detection confidence score and image quality are used for quantification, with a score range of [0,1]. Spatial information from fixed structures within the vehicle, such as seats and steering wheels, is used to determine whether a certain part is occluded. The occlusion area O of the vehicle structure and the total visible area V(p) of that part are also considered. i The following occlusion index is used for quantification:

[0158]

[0159] Among them, |V(p i|V(p) is the area of the part being occluded, |V(p) is the total visible area of the part, if the occlusion ratio is greater than the set threshold of 40%, the part is considered to be invisible; i i |V(p) is the area of the part being occluded, |V(p) is the total visible area of the part, if the occlusion ratio is greater than the set threshold of 40%, the part is considered to be invisible;

[0160] S1.4.3) Occlusion evaluation: Set the occlusion area percentage as the evaluation index to judge whether the key parts of the driver (such as upper limbs and legs) are greatly occluded by the fixed structure in the car (such as seat and steering wheel); process the image through image segmentation technology Mask R-CNN to extract the car structure region R car and the driver region R driver , calculate the intersection and overlapping region of the two regions; let |R car ∩R driver | be the intersection area of the car structure and the driver region, |R driver | be the total area of the driver region, then the occlusion degree is:

[0161]

[0162] If the occlusion ratio exceeds 40%, it is considered that the frame has a large occlusion problem.

[0163] As a preferred embodiment of the present application, the specific method of step S2 data labeling and skeleton point selection is:

[0164] Using CVAT automatic labeling tool, the selected each key frame image I i is labeled with skeleton points, and a dynamic labeling strategy is designed according to the special scene in the car and the continuous action of the driver from opening the door to holding the steering wheel. The specific labeling scheme is as follows:

[0165] Opening the door stage: key labeling of shoulder, elbow, wrist, knee, ankle and other skeleton points to capture the motion trajectory of the driver stretching his hand to pull the door, and ensure that the core posture features of the action are recorded. This stage is the only one to show the complete knee to ankle stage of the driver;

[0166] Sitting stage: key labeling of spine, knee and other key skeleton points to obtain the sitting angle and body rotation state of the driver.

[0167] Buckling the seat belt stage: focusing on capturing the subtle changes in the positions of shoulder, elbow and waist, and accurately recording the side action of the driver to provide data support for subsequent analysis;

[0168] Holding the steering wheel stage: key labeling of shoulder, elbow, wrist and neck to ensure accurate recording of the final driving posture;

[0169] The labeled key skeleton points include the following parts: (1) nose; (2) neck; (3) left shoulder; (4) right shoulder; (5) chest; (6) left elbow; (7) right elbow; (8) left hip; (9) right hip; (10) left knee; (11) right knee; (12) left ankle; (13) right ankle.

[0170] The connection information between the labeled key skeleton points will be used for subsequent skeleton proportion conversion and length calculation. These connections not only help to capture the detailed features of the driver's body structure, but also provide the necessary analysis information for the system to support more accurate pose estimation and body shape analysis.

[0171] Due to the spatial limitations of the in-vehicle environment, some skeleton points may be blocked by structures such as doors and seats. The system dynamically classifies and strategically processes this according to the technology supported by CVAT:

[0172] Fully visible points: refer to skeleton points that are completely exposed during the driver's actions, such as the head and shoulders. Such skeleton points are directly labeled.

[0173] Occluded points: for example, during actions such as opening the door or turning around, part of the arm or leg may be blocked by in-vehicle structures. When there are parts that are partially blocked but the general position of the skeleton point can still be inferred, click "Switch occluded property" to present it in a dashed line state to indicate that the key point is partially blocked.

[0174] Invisible points: such as the ankle being blocked by the seat after sitting down, click "Switch outside property" to present it in a hidden state to identify that the key point is invisible in this frame.

[0175] As a preferred embodiment of the present application, step S3 comprises:

[0176] S3.1 The feature extraction stage uses the VitPose network to perform feature extraction on each key frame image I i to generate a preliminary feature representation related to the skeleton points;

[0177] Input the key frame image I i , assuming it is an image of size HxWxC, where H and W are the height and width of the image, respectively, and C is the number of channels; divide the image into image blocks of size PxP, with P 2 ·C pixels in each image block, assuming a total of N image blocks, then the calculation formula for N is:

[0178]

[0179] Embed each image block into a d-dimensional feature vector space through linear transformation to obtain the embedding vector e ij∈R d , and the specific formula is as follows:

[0180] e ij =W patch P ij +b patch

[0181] wherein, is the weight of linear transformation, b patch is the bias term, and all embedding vectors e ij are stacked into a matrix E∈R N×d and input to the Transformer encoder, which captures the global context information of the image through the self-attention mechanism, and the core calculation formula is as follows:

[0182]

[0183] wherein, Q, K, and V are matrices of queries, keys, and values respectively, d k is a scaling factor of feature dimension, and after processing through multiple Transformer layers, the output feature matrix is:

[0184] z i =Transformer(E), z i ∈R N×d

[0185] The output feature matrix z i contains the global context information of the image;

[0186] S3.2 The pose estimation layer (SE_Layer) further predicts the preliminary skeletal point coordinates and connection diagram of the driver based on the feature matrix z i ;

[0187] The feature matrix z i ∈R N×d generated by the Transformer encoder and the in-vehicle physical layout S car are input to the SE_Layer, and through multi-modal fusion, S car is fused with z i and input to the regression network g pose , to predict the preliminary skeletal point coordinates:

[0188] P i =g pose (concat(z i , S car )), P i ={(x i , y i}

[0189] where P i = {(x i , y i )} represents the preliminary set of skeletal point coordinates, including the two-dimensional position of each skeletal point, g pose is a regression function used to extract skeletal point coordinates from the feature matrix, and concat represents concatenating two pieces of information together. According to the set of skeletal point coordinates P i , a skeletal point connection graph S i is constructed, which is generated through the topological relationship between skeletal points and describes the skeletal point features of the driver:

[0190] S i = (V, E), V = P i , E = {(v k , v j ) | v k , v j ∈ V and satisfy the connection relationship}

[0191] The output results include the preliminary set of skeletal point coordinates P i and the corresponding skeletal point connection graph S i ;

[0192] S3.3 Scene-aware Feature Integration Layer (SF_Layer) further optimizes the features of skeletal points through in-vehicle environment perception mechanisms;

[0193] The preliminary set of skeletal point coordinates P i , the preliminary skeletal point connection graph S i , and the in-vehicle physical layout S car are input, and linear transformations are performed on the skeletal point connection graph S i and the in-vehicle physical layout S car to map them to the same high-dimensional feature space:

[0194]

[0195] where W and W are weight matrices for linear transformation, and b and b are bias terms. The relevance between skeletal point features and in-vehicle physical layout is calculated using a self-attention mechanism:

[0196]

[0197] where d represents the feature dimension, S′ i (S′ car )T Reflecting the degree of association between each bone point and the structure inside the car, softmax is the normalized attention weight, Attn i is the weighted influence result of the physical layout inside the car on the bone point features, according to the optimized bone point coordinate set corresponding bone point connection diagram is constructed

[0198]

[0199] Output the optimized bone point coordinate set and the corresponding bone point connection diagram

[0200] S3.4 Spatio-Temporal Layer (ST_Layer) optimizes the temporal consistency and spatial coherence of the bone points by analyzing the changes of the bone points between consecutive multiple frames of images;

[0201] Input the optimized bone point coordinate set and the corresponding bone point connection diagram of consecutive multiple frames integrate them into Stack multiple frames of feature data into spatio-temporal feature blocks with dimensions T x N x d, where T represents the number of frames in the time dimension, and 3D convolution is used to capture time and spatial features to optimize the dynamic coherence of the bone points:

[0202]

[0203] where W ST ∈R d′×d×k×k×T is a 3D convolution kernel, d ′ is the output feature dimension, k is the spatial dimension of the convolution kernel, T is the range of the time dimension, and z ST ∈R N×d′ is a high-dimensional feature matrix containing spatio-temporal context information, which considers both the position changes of the bone points between multiple frames of images and the temporal coherence of the driver's actions; after spatio-temporal convolution processing, the bone point coordinates are dynamically optimized using dynamic weights, as follows:

[0204]

[0205] where, is the bone point coordinates of the t-th frame, and α t is the dynamic weight, which is calculated from the spatio-temporal feature correlation of each frame of bone points:

[0206]

[0207] Among them, MLP is a multilayer perceptron, which uses spatiotemporal features z ST Extract the correlation score f between the skeletal point features of each frame and the overall dynamic optimization objective. t This is used for subsequent weight calculations; based on the dynamically optimized skeletal points. Construct the corresponding skeletal point connection diagram

[0208]

[0209] The topological relationship here is The same, but recalculated based on the dynamically optimized bone point positions.

[0210] Output the dynamically optimized set of skeletal point coordinates Diagram connecting the corresponding skeletal points It maintains consistency in both time and space;

[0211] The S3.5 Proportion Adjustment Layer (PA_Layer) applies the dynamically optimized set of skeletal points. Diagram connecting the corresponding skeletal points Perform human proportion correction to ensure it conforms to human anatomical standards.

[0212] Input the dynamically optimized set of skeletal point coordinates Diagram connecting the corresponding skeletal points Predict the physical size of skeletal points from features using a regression network:

[0213] h i =MLP(z) ST )

[0214] According to standard human body proportions h max Calculate the scaling factor λ for each skeletal point. i :

[0215]

[0216] λ i This is a scaling factor representing the physical size of each skeletal point relative to the standard human body proportion. It is used to ensure that the skeletal point prediction results can be adapted to drivers of different body types. Therefore, the scaling factor λ for each skeletal point is... i It is calculated independently. The coordinates of the skeletal points are adjusted using a scaling factor:

[0217]

[0218] Based on the corrected skeletal point coordinates Construct the corresponding skeletal point connection diagram Sfinal .

[0219] The output includes a set of corrected skeletal point coordinates and a corresponding skeletal point connection graph S final as the final two-dimensional skeletal point recognition result.

[0220] As a preferred embodiment of the present application, step S4 specifically includes three-dimensional reconstruction of two-dimensional skeletal points based on:

[0221] According to the set of two-dimensional skeletal point coordinates Combine the internal structure information of the vehicle and the JOTR algorithm to complete the three-dimensional skeletal point reconstruction, and finally generate a three-dimensional skeletal point connection graph and extract the body size parameters of the driver.

[0222] S4.1 Input the in-vehicle space data (such as seat height, width, depth, and in-vehicle length and width) for modeling, and use the modeling tool Blender to build the constraint structure of the three-dimensional space in the vehicle;

[0223] S4.2 Use the JOTR algorithm in combination with the set of two-dimensional skeletal point coordinates to calculate the three-dimensional coordinates of each skeletal point to provide input for subsequent three-dimensional reconstruction:

[0224]

[0225] The initial three-dimensional skeletal points There is a certain error, so it is corrected by a self-supervised learning framework of a generative adversarial network (GAN), and the three-dimensional coordinates are projected to a two-dimensional plane to obtain a projection

[0226]

[0227] The projection function is calculated based on camera parameters and perspective relationships.

[0228] S4.3 The discriminator D feeds back the error between the points projected to the two-dimensional plane and the original two-dimensional points to the generator G, minimizing the loss function L self , constantly optimizing the generator to make the three-dimensional coordinates output by the generator more consistent with the two-dimensional projection of the real image points:

[0229]

[0230] After correction, the generator G outputs the corrected three-dimensional skeletal points:

[0231]

[0232] S4.4 According to the corrected three-dimensional skeleton point set Generate a three-dimensional skeleton point connection diagram Its topological relationship is the same as the two-dimensional connection diagram S final The same, but recalculated in three-dimensional space, calculating the Euclidean distance d ij between three-dimensional skeleton points, used to represent the limb length characteristics of the driver:

[0233]

[0234] Where d ij represents the three-dimensional distance between the i-th skeleton point and the j-th skeleton point, and the distance feature set {d ij} is input to a multi-layer perception (MLP) to map the height and body shape parameters ShapeParams of the driver:

[0235]

[0236] Finally output more accurate driver height and its body shape parameters ShapeParams.

[0237] According to the height and body shape prediction results of the driver, the system can automatically optimize and adjust multiple position parameters of the car seat, the position of the safety belt, and the deployment strategy of the airbag, etc. Based on the ISO 7250-1:2017 "Basic Human Body Measurement Technology Design" standard and SAE J826-2021 "Equipment for defining and measuring vehicle seat adjustment" data, the system formulates a dynamic adjustment scheme to achieve accurate matching of the driver's body shape requirements. At the same time, all data are processed and stored locally on the vehicle, effectively protecting the personal privacy and safety of the driver.

[0238] As a preferred embodiment of the present application, the dynamic adjustment strategy of step S5 is as follows:

[0239] S5.1 Seat forward and backward adjustment: Ensure the appropriate distance between the driver and the pedals, steering wheel, and achieve the best control and comfort.

[0240] Objective: Ensure the appropriate distance between the driver and the pedals, steering wheel, and achieve the best control and comfort.

[0241] Input parameters: Driver's leg length (vertical distance from hips to knees), knee to seat distance (straight line distance from knee to front edge of seat), pedal distance (distance from the bottom of the driver's foot to the accelerator and brake pedals).

[0242] Adjustment method:

[0243] ​Leg length data acquisition: Calculate the driver's thigh length and knee position to estimate the optimal leg angle (usually between 90°-120° at the knee).

[0244] Front and rear adjustment values:

[0245] For long-legged drivers (thigh length > 50 cm): move the seat forward by about 530 cm;

[0246] For short-legged drivers (thigh length < 45 cm): move the seat back by about 5-7 cm, and the distance between the knee and the front edge of the seat should be greater than 30 cm.

[0247] Dynamic adjustment: Based on the driver's foot movements, if discomfort in the knee is detected during sudden braking or acceleration, the seat can automatically fine-tune the front and rear positions by 1-2 cm.

[0248] S5.2 Seat height adjustment: Ensure the driver has the best view, reducing interference from the roof or other parts;

[0249] Objective: Ensure the driver has the best view, reducing interference from the roof or other parts.

[0250] Input parameters: height, safe distance between the roof and the head (vertical distance from the driver's head to the roof).

[0251] Adjustment method:

[0252] Height data acquisition: By detecting the driver's shoulder and head position, calculate the eye height and ensure the best angle of view with the windshield.

[0253] Height adjustment values:

[0254] For drivers taller than 180 cm: raise the seat by 5-8 cm to ensure that the eye position is parallel to the centerline of the window, and the head is more than 5 cm from the roof;

[0255] For drivers shorter than 160 cm: lower the seat by 5-7 cm to ensure that the eye position is not lower than the lower edge of the window, and the head is more than 7 cm from the roof.

[0256] Dynamic feedback: Real-time monitoring of the driver's line of sight changes through on-board cameras or sensors, and dynamic adjustment of the seat height (e.g., adjust 3-5 cm) to ensure that the view is not affected by changes in the vehicle's environment.

[0257] S5.3 Seat back angle adjustment: Ensure adequate back support to reduce driver fatigue;

[0258] Objective: Ensure adequate back support to reduce driver fatigue.

[0259] Input parameters: Driver's back curvature (natural bending angle between waist and chest), sitting angle (angle between back and seat).

[0260] Adjustment method:

[0261] Back data acquisition: Detect the skeletal point positions of the driver's waist and chest, calculate the angle between them, obtain the natural curvature of the back, and adjust the angle of the seat backrest.

[0262] Backrest angle adjustment value:

[0263] For slightly upright posture (back angle <100°): The seat backrest is reclined by 5-7°, making the back reach a natural bending angle;

[0264] For slightly relaxed posture (back angle >110°): The seat backrest is tilted forward by 5-7°, ensuring a comfortable angle (100°-110°) between the back and the seat.

[0265] Dynamic feedback: If the driver is in a relaxed posture for a long time, the seat will automatically detect and fine-tune the backrest angle (usually 1-3°) through sensors to reduce the burden on the back.

[0266] S5.4 Seat and steering wheel linkage adjustment: Ensure that the driver can comfortably operate the steering wheel, avoiding unnatural stretching of the arms;

[0267] Objective: Ensure that the driver can comfortably operate the steering wheel, avoiding unnatural stretching of the arms.

[0268] Input parameters: Arm length (distance from shoulder to wrist), distance from shoulder to steering wheel (based on skeletal point recognition data).

[0269] Adjustment method:

[0270] Arm length data acquisition: Through skeletal points, calculate the relative positions of the driver's shoulders and elbows to determine the appropriate arm angle.

[0271] Seat forward and backward adjustment value:

[0272] For long-armed drivers (arm length >60cm): Move the seat forward by 3-5cm to ensure that the elbow bending angle remains around 120°, facilitating flexible steering wheel operation;

[0273] For short-armed drivers (arm length <55cm): Move the seat backward by 3-5cm to ensure that the driver can naturally hold the steering wheel and both hands can comfortably operate it.

[0274] Steering wheel angle adjustment: If the seat position is adjusted, the height and angle of the steering wheel are automatically adjusted by 1-3° to maintain it within the ideal arm operating range.

[0275] S5.5 Seatbelt adjustment: Ensure the seatbelt can accurately bind the driver's body, providing the best safety protection.

[0276] Objective: Ensure the seatbelt can accurately bind the driver's body, providing the best safety protection.

[0277] Input parameters: Shoulder height (vertical distance from shoulder to ground), chest circumference and waist circumference (distance from chest to waist).

[0278] Adjustment method:

[0279] Shoulder strap adjustment: Automatically adjust the height of the shoulder strap of the seatbelt to ensure that the shoulder strap is aligned with the center of the driver's shoulders, usually 3-5 cm above the centerline of the shoulder.

[0280] Waist strap adjustment: Adjust the waist strap position according to the waist circumference data to ensure that the waist strap fits the pelvic area. If the waist circumference is large, the seat automatically adjusts the position to avoid the seatbelt being too tight.

[0281] Dynamic adjustment: During driving, if the driver's posture changes significantly (such as bending over, turning around, etc.), the system will automatically adjust the tightness and position of the seatbelt to ensure maximum protection for the driver in the event of a collision.

[0282] As a preferred embodiment of the present application, step S6 includes:

[0283] S6.1 Use three-dimensional skeletal point coordinates to extract the key part coordinate set P of the driver key :

[0284]

[0285] wherein, and respectively represent the corrected three-dimensional coordinates of the head, chest and shoulder,

[0286] S6.2 Calculate the driver's body shape feature range according to the body shape parameters ShapeParams:

[0287]

[0288] wherein, BodyWidth represents the horizontal distance between the left and right endpoints of the driver's shoulders, and are respectively the x-axis values of the corrected three-dimensional coordinates of the right shoulder and left shoulder; BodyHeight represents the vertical distance from the top of the driver's head to the hips, and respectively represent the z-axis values of the corrected three-dimensional coordinates of the head and hips;

[0289] S6.3 Based on the coordinates P of the driver's key body parts key Calculate the optimal center position P of the airbag deployment. deploy and coverage radius R deploy :

[0290]

[0291] Among them, P deploy R is calculated based on the average coordinates of the driver's key body parts, serving as the center point of airbag deployment. deploy The coverage radius of the airbag is determined by the maximum three-dimensional distance of the key parts;

[0292] S6.4 Combines data from in-vehicle collision sensors to dynamically adjust the airbag triggering conditions; trigger threshold T trigger The calculation formula is:

[0293] T trigger =w1F collision +w2θ impact +w3Δ pose

[0294] Among them, F collision For the collision force, θ impact Let Δ be the collision angle. pose The driver's posture deviation is represented by w1, w2, and w, which are weighting parameters used to adjust the influencing factors of the triggering conditions.

[0295] The force P of airbag deployment inflate Calculated based on the driver's body measurements and the severity of the collision:

[0296] P inflate =f(ShapeParams,Collision Severity)

[0297] S6.5 dynamically adjusts the airbag deployment direction and deployment strategy by monitoring changes in the driver's seating position in real time;

[0298] Adjusted airbag deployment angle α adjusted =α default +kΔ pose Represented as:

[0299] α adjusted =α default +k·Δ pose

[0300] Where, α default The default unfolding angle is given, k is the sitting posture correction coefficient, and ·Δ pose The monitored sitting posture deviation was used to determine the airbag deployment center point P. deploy Coverage area R deploy, trigger threshold T trigger and the unfolding force P inflate , the driver dynamically adjusted unfolding angle α adjusted , provide the adjustment parameters of the airbag, ensure that the airbag can accurately protect the key parts of the driver when unfolding, and reduce the risk of secondary injury when colliding.

Claims

1. A method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag, characterized in that, Includes the following steps: S1 Image Capture and Dataset Construction: A continuous image sequence of the driver's entire process of getting into the car is captured by an in-vehicle camera. The continuous image sequence is analyzed using a 2D CNN+LSTM-based action recognition algorithm to obtain the driver's action features and form a keyframe dataset. The keyframe dataset is then filtered and divided into training and test sets according to a certain ratio. S2 key skeleton point annotation utilizes an automatic annotation tool for each keyframe image. i Perform skeletal point annotation and generate a set P of annotated skeletal point coordinates. label ={(x label ,y label )},x label The y-coordinate represents the horizontal coordinate of a skeletal point on the image plane. label This indicates the vertical coordinates of a key skeletal point on the image plane. This labeled data is used for training the subsequent 2D skeletal point recognition system. The S3 2D skeleton point recognition system is built based on keyframe images I i Feature extraction is performed to generate a preliminary set of skeletal point coordinates P. i Diagram S connecting the corresponding skeletal points i Based on the vehicle's internal physical layout, an optimized result set is generated through multimodal fusion and self-attention mechanisms. and Analyze the dynamic changes of skeletal points in multiple frames of images and use spatiotemporal convolution to generate dynamic optimization results. and By correcting the human body proportions, a set of two-dimensional skeletal point coordinates is output. The diagram S connecting the corresponding skeletal points final ; S4 acquires in-vehicle space data based on 3D reconstruction using 2D skeleton points, and uses the modeling tool Blender to model the 3D space inside the vehicle; utilizing a set of 2D skeleton point coordinates... Using the JOTR algorithm to calculate the coordinates of 3D skeleton points Using a self-supervised learning framework of generative adversarial networks to Perform correction to generate corrected 3D coordinates. The Euclidean distance d between 3D skeleton points is calculated based on the correction results. ij It is then mapped to the driver's height and body shape using a multilayer perceptron (MLP) to output the driver's precise height. and body shape parameters ShapeParams; The S5 seats automatically adjust to the driver's height. ShapeParams analyzes the position and sitting angle of the driver's head, shoulders, and waist, and dynamically adjusts the seat height, backrest angle, and seat cushion angle to match the driver's body shape requirements. Employing a dynamic adjustment strategy, it monitors the driver's posture changes in real time during long-term driving and intelligently responds to optimize seat settings and reduce fatigue; The S6 airbag automatic adjustment analyzes the driver's three-dimensional skeletal coordinates. The system dynamically optimizes the deployment location, deployment range, and triggering strategy of airbags using body shape parameters (ShapeParams) and real-time seating posture information. A real-time monitoring mechanism is employed to ensure that the airbags can accurately protect the driver's critical parts when they deploy, reducing the risk of secondary injuries during a collision.

2. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, The specific steps of image capture and dataset construction in step S1 include: S1.1 2D convolutional neural networks extract spatial features from each frame of a continuous image sequence. S1.2 LSTM model is used to process temporal information in continuous image sequences, capture the continuous action features of the driver during the process of getting into the vehicle, and analyze the action transition features in a set of adjacent frames to identify the turning point of the action. S1.3 Analyze the time curve of the motion amplitude change, mark the key frame at the moment of a sharp change in amplitude, and capture the start frame and stop frame of each motion; S1.4 Develop an adaptive filtering strategy to select keyframe images I i The keyframe dataset {I} is composed of keyframes. i The selected keyframe dataset is divided into training and testing sets according to a certain ratio to ensure the model's generalization ability and accuracy.

3. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 2, characterized in that, The adaptive filtering strategy includes the following: S1.4.1) Motion smoothness quantization: Motion smoothness is quantified by calculating the optical flow between two adjacent frames; Let image frame I t and I t+1 These are two consecutive frames, with optical flow F. t The calculation method is as follows: Where T is the time interval, F t It is the displacement vector between two frames, representing the movement of an object between frames; if the displacement exceeds the threshold, it is considered that the action has changed significantly and does not meet the smoothness requirements. Calculate the mean square error (MSE) of optical flow between adjacent frames, let F t and F t+1 Calculate the root mean square error of the optical flow in adjacent frames: Select an indicator, if MSE(F) t ,F t+1 If the value is higher than the threshold of 0.15, the motion is considered not smooth and the next frame is discarded. S1.4.2) Visibility Quantization of Limb Parts: OpenPose is used to perform fast skeleton detection on the image to obtain preliminary visibility scores for key parts; the visibility V(p) of each part is calculated. i The detection confidence score and image quality are used for quantification, with a score range of [0,1]. Spatial information of the fixed structure inside the vehicle is used to determine whether a certain part is occluded. The occlusion area O of the structure inside the vehicle and the total visible area V(p) of that part are also considered. i The following occlusion index is used for quantification: Among them, |V(p i )∩O| represents the area that is obscured, |V(p) i )| represents the total visible area of ​​the part. If the occlusion ratio is greater than the set threshold of 40%, the part is considered invisible. S1.4.3) Occlusion Assessment: The percentage of occlusion area is used as an assessment metric to determine whether fixed structures inside the vehicle significantly obstruct the driver's key areas. The image is processed using Mask R-CNN image segmentation technology to extract the R-region of the vehicle's internal structures. car and driver area R driver Calculate the intersection and overlapping regions of the two regions; let |R| = 1. car ∩R driver |R represents the intersection area of ​​the vehicle interior structure and the driver's area. driver | represents the total area of ​​the driver's area, then the degree of obstruction is: If the occlusion rate exceeds 40%, the frame is considered to have a significant occlusion problem.

4. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, The specific method for data annotation and skeleton point selection in step S2 is as follows: S2.1 uses the CVAT automatic annotation tool to annotate each selected keyframe image I. i Skeleton point annotation was performed, and a dynamic annotation strategy was designed based on the specific in-vehicle scenario and the driver's continuous action from opening the car door to gripping the steering wheel. The annotation scheme is as follows: Opening the car door phase: Focus on marking the skeletal points of the shoulder, elbow, wrist, knee, and ankle to capture the trajectory of the driver reaching for the car door and ensure that the core postural features of the action are recorded. This phase is the only one that shows the driver's complete knee-to-ankle movement. Seated phase: Focus on marking key bone points of the spine and knees to obtain the driver's sitting angle and body rotation status; During the seatbelt fastening phase: the focus is on capturing subtle changes in the positions of the upper body skeletal points in the shoulders, elbows, and waist, accurately recording the driver's side movements to provide data support for subsequent analysis; Steering wheel grip phase: Focus on marking the skeletal points of the shoulders, elbows, wrists, and neck to ensure accurate recording of the final driving posture; The key skeletal points marked in S2.2 include the following parts: (1) nose; (2) neck; (3) left shoulder; (4) right shoulder; (5) chest; (6) left elbow; (7) right elbow; (8) left hip; (9) right hip; (10) left knee; (11) right knee; (12) left ankle; (13) right ankle.

5. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, Step S3 includes: S3.1 Feature Extraction Stage: Utilizing the VitPose network to extract features from each keyframe image I... i Perform feature extraction to generate preliminary feature representations related to skeletal points; Input keyframe image I i Assume the image is of size H×W×C, where H and W are the height and width of the image, respectively, and C is the number of channels; divide the image into image blocks of size P×P, with each image block containing P pixels. 2 •C, assuming there are a total of N image patches, then the formula for calculating N is: Each image patch is embedded into a d-dimensional feature vector space through a linear transformation, resulting in an embedding vector e for each image patch. ij ∈R d The specific formula is as follows: e ij =W patch P ij +b patch Among them, W patch ∈R d×(P2·C) It is the weight of the linear transformation, b patch It is a bias term that applies to all embedded vectors e ij Stacked into a matrix E∈R N×d The input is fed into the Transformer encoder, which captures global contextual information of the image through a self-attention mechanism. Its core calculation formula is: Where Q, K, and V are the matrices of query, key, and value, respectively, and d k This is a scaling factor for the feature dimension. After processing through multiple Transformer layers, the output feature matrix is: With i =Transformer(E),z i ∈R N×d The output feature matrix z i It contains global context information about the image; S3.2 Pose Estimation Layer: Based on Feature Matrix z i Further predict the initial skeletal coordinates and connection diagram of the driver; The feature matrix z generated by the Transformer encoder i ∈R N×d and the physical layout of the car car The input is fed into SE_Layer, and through multimodal fusion, S... car With z i The mixture is then fused and fed into the regression network g. pose In the process, the preliminary coordinates of the skeletal points are predicted: P i =g pose (concat(z i ,S car )),P i ={(x i ,y i )} Among them, P i ={(x i ,y i )} represents the initial set of skeletal point coordinates, including the two-dimensional position of each skeletal point, g pose It is a regression function used to extract skeletal point coordinates from the feature matrix. `concat` means concatenating the two sets of information, based on the set of skeletal point coordinates P. i Construct a skeletal point connection diagram S i It generates a description of the driver's skeletal point features through the topological relationships between skeletal points: S i =(V,E),V=P i E={(v k ,v j )∣v k ,v j ∈V, and satisfy the connection relationship} The output includes a preliminary set of skeletal point coordinates P. i The diagram S connecting the corresponding skeletal points i ; S3.3 Scene Perception and Feature Integration Layer: Further optimizes the features of skeletal points through the in-vehicle environment perception mechanism; Input the initial set of skeletal point coordinates P i Preliminary skeletal point connection diagram S i and the physical layout of the car car , connecting the skeletal points in the diagram S i and the physical layout of the car car Perform linear transformations on each, mapping them to the same high-dimensional feature space: S′ i =W Si S i +b Si ,S′ car =W Scar S car +b Scar Among them, W Si and W Scar It is the weight matrix of the linear transformation, b Si and b Scar This is a bias term that uses a self-attention mechanism to calculate the correlation between skeletal point features and the physical layout inside the vehicle: Where: d represents the feature dimension, S′ i (S′ car ) T Reflecting the degree of association between each skeletal point and the vehicle's interior structure, softmax represents the normalized attention weights, and Attn... i It is the weighted influence of the vehicle's internal physical layout on the skeletal point features, based on the optimized set of skeletal point coordinates. Construct the corresponding skeletal point connection diagram Output the optimized set of skeletal point coordinates Diagram connecting the corresponding skeletal points S3.4 Spatiotemporal Integration Layer: Optimizes the temporal consistency and spatial coherence of skeleton points by analyzing the changes in skeleton points between consecutive frames; Input the optimized set of skeletal point coordinates from multiple consecutive frames. Diagram connecting the corresponding skeletal points Integrate it into Multi-frame feature data Stacked into spatiotemporal feature blocks with dimensions T×N×d, where T represents the number of frames in the time dimension, 3D convolution is used to capture temporal and spatial features, optimizing the dynamic coherence of skeletal points: Among them, W ST ∈R d′×d×k×k×T It is a 3D convolution kernel, where d′ is the output feature dimension, k is the spatial dimension of the convolution kernel, T is the time dimension range, and z ST ∈R N×d′ It is a high-dimensional feature matrix containing spatiotemporal context information; after spatiotemporal convolution processing, dynamic weights are used to dynamically optimize the coordinates of the skeleton points, as shown in the following formula: in, α is the coordinate of the skeleton point in frame t. t It is a dynamic weight, calculated from the spatiotemporal correlation of the bone points in each frame: Among them, MLP is a multilayer perceptron, which uses spatiotemporal features z ST Extract the correlation score f between the skeletal point features of each frame and the overall dynamic optimization objective. t This is used for subsequent weight calculations; based on the dynamically optimized skeletal points. Construct the corresponding skeletal point connection diagram : Output the dynamically optimized set of skeletal point coordinates Diagram connecting the corresponding skeletal points It maintains consistency in both time and space; S3.5 scaling layer: for dynamically optimized skeletal point sets Diagram connecting the corresponding skeletal points Perform human proportion correction to ensure it conforms to human anatomical standards; Input the dynamically optimized set of skeletal point coordinates Diagram connecting the corresponding skeletal points Predict the physical size of skeletal points from features using a regression network: h i =MLP(z ST ) According to standard human body proportions h max Calculate the scaling factor λ for each skeletal point. i : λ i This is a scaling factor representing the physical size of each skeletal point relative to the standard human body proportion. It is used to ensure that the skeletal point prediction results can be adapted to drivers of different body types. The skeletal point coordinates are adjusted using the scaling factor. Based on the corrected skeletal point coordinates Construct the corresponding skeletal point connection diagram S final ; The output includes a set of scaled-corrected skeletal point coordinates. The diagram S connecting the corresponding skeletal points final This serves as the final result of two-dimensional skeletal point recognition.

6. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, The specific steps of step S4, which involves the three-dimensional reconstruction based on two-dimensional bone points, include: Based on the set of two-dimensional skeleton point coordinates By combining the vehicle's internal structural information with the JOTR algorithm, a 3D skeleton point reconstruction is completed, ultimately generating a 3D skeleton point connection map. And extract the driver's body shape parameters; S4.1 Input the interior space data to build a model, and use the modeling tool Blender to construct the constraint structure of the three-dimensional interior space; S4.2 Utilizes the JOTR algorithm combined with a set of 2D skeletal point coordinates Calculate the three-dimensional coordinates of each bone point Provides input for subsequent 3D reconstruction: Correction is performed using a self-supervised learning framework based on generative adversarial networks to transform the three-dimensional coordinates. Projecting onto a two-dimensional plane yields the projection. : S4.3 Discriminator D projects points onto the two-dimensional plane. and the original two-dimensional point Error feedback generator G between, minimizing loss function L self The generator is continuously optimized to make its output 3D coordinates more consistent with the 2D projection of real image points. After correction, generator G outputs the corrected 3D skeleton points: S4.4 Based on the corrected 3D skeleton point set Generate a 3D skeleton point connection map Its topological relationship and two-dimensional connection diagram S final The process is the same, but recalculated in three-dimensional space to calculate the Euclidean distance d between the three-dimensional skeleton points. ij Used to characterize the limb length features of a driver: Where, d ij Representing the 3D distance between the i-th and j-th bone points, the distance feature set {d} ij The input is fed into a multilayer perceptron (MLP) and mapped to the driver's height. Body shape parameters: Final output: driver's height And its body shape parameter ShapeParams.

7. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, The dynamic adjustment strategy in step S5 is as follows: S5.1 Seat fore-and-aft adjustment: Ensures a suitable distance between the driver and the pedals and steering wheel for optimal control and comfort; S5.2 Seat height adjustment: Ensures the driver has the best visibility and reduces interference from the roof or other components; S5.3 Seat back angle adjustment: Ensures sufficient back support and reduces driver fatigue; S5.4 Seat and steering wheel linkage adjustment: ensures that the driver can comfortably operate the steering wheel and avoids unnatural arm extension; S5.5 Seatbelt Adjustment: Ensures the seatbelt accurately restrains the driver's body, providing optimal safety protection.

8. The method for recognizing the physical characteristics of drivers and passengers and for automatically adjusting the seat and airbag according to claim 1, characterized in that, Step S6 includes: S6.1 Using 3D Skeleton Point Coordinates Extract the coordinate set P of the key parts of the driver key : in, These represent the corrected 3D coordinates of the head, chest, and shoulders, respectively. This represents the corrected three-dimensional coordinates of the key component k; S6.2 Calculate the driver's body shape characteristic range based on the body shape parameter ShapeParams: BodyWidth represents the horizontal distance between the left and right ends of the driver's shoulders. and These are the x-axis values ​​of the corrected 3D coordinates for the right and left shoulders, respectively; BodyHeight represents the vertical distance from the top of the driver's head to their hips. and These represent the z-axis values ​​of the corrected 3D coordinates for the head and hips, respectively. S6.3 Based on the coordinates P of the driver's key body parts key Calculate the center point P of the airbag deployment. deploy and the coverage radius R of the airbag deploy : Among them, P deploy R is calculated based on the average coordinates of the driver's key body parts, serving as the center point of airbag deployment. deploy The coverage radius of the airbag is determined by the maximum three-dimensional distance of the key parts; S6.4 Combines data from in-vehicle collision sensors to dynamically adjust the airbag triggering conditions; trigger threshold T trigger The calculation formula is: T trigger =w1F collision +w2θ impact +w3Δ pose Among them, F collision For the collision force, θ impact Let Δ be the collision angle. p ose is the driver's posture deviation, and w1, w2, w3 are weight parameters used to adjust the influencing factors of the triggering conditions; The force P of airbag deployment inflate Calculated based on the driver's body measurements and the severity of the collision: P inflate =f(ShapeParams,Collision Severity) S6.5 dynamically adjusts the airbag deployment direction and deployment strategy by monitoring changes in the driver's seating position in real time; Adjusted airbag deployment angle α adjusted Represented as: a adjusted =a default +kΔ pose Where, α default The default deployment angle is given, k is the seating posture correction coefficient, and Δpose is the driver's seating posture deviation; the center point P of airbag deployment is obtained. deploy The coverage radius R of the airbag deploy Trigger threshold T trigger The force P of airbag deployment inflate The airbag deployment angle α after dynamic adjustment by the driver adjusted It provides airbag adjustment parameters to ensure that the airbag can accurately protect the driver's critical parts when it deploys, reducing the risk of secondary injury during a collision.

Images