Intelligent perception method of passenger entry and exit behavior in transportation vehicles based on sensor fusion

Abstract

The present invention discloses an intelligent perception method of passenger entry and exit behavior of vehicles based on sensor fusion, which belongs to the field of passenger behavior perception, including: evaluating passenger behavior by reading local signals, establishing time-series spatial positioning coordinates, activating ultrasonic and infrared sensors for tracking and collection, and establishing a tracking data set; updating the target position based on ultrasonic data and establishing door distance parameters, performing motion perception analysis based on infrared data, and establishing fusion recognition results by fusion recognition; calling pressure and IMU data sets to generate real-time motion vectors, and performing fusion analysis on the fusion recognition results, local signals and real-time motion vectors to generate entry and exit behavior perception results. This application solves the technical problems of low accuracy and poor real-time performance of the perception of passenger entry and exit behavior of vehicles in the prior art, and achieves the technical effect of improving the accuracy and real-time performance of the perception of passenger entry and exit behavior of vehicles and realizing intelligent perception.

Description

Technical Field

[0001] The present invention relates to the field of passenger behavior perception, and in particular to an intelligent perception method for vehicle passenger entry and exit behavior based on sensor fusion. Background Art

[0002] With the continuous development of urban public transportation, the requirements for transportation operation management and service quality are also increasing. However, the existing passenger entry and exit behavior perception methods generally have the problems of low accuracy and poor real-time performance. Traditional perception methods mainly rely on a single type of sensor, such as infrared sensors or pressure sensors. These sensors perform well in specific scenarios, but when faced with complex and changeable actual operating environments, it is difficult to fully perceive passenger behavior and is easily affected by environmental interference, resulting in missed detection, misjudgment and other problems, affecting perception accuracy. In addition, existing perception methods often require a long data processing and analysis time, and cannot achieve real-time behavior perception and response, and have poor perception real-time performance. Summary of the invention

[0003] The present application provides a method for intelligently sensing the entry and exit behavior of passengers in a vehicle based on sensor fusion, aiming to solve the technical problems of low accuracy and poor real-time performance in the prior art in sensing the entry and exit behavior of passengers in a vehicle.

[0004] The present application discloses a method for intelligently perceiving the entry and exit behavior of passengers in a vehicle based on sensor fusion, comprising: reading local signals, evaluating the passenger's behavior perception based on the local signals, and establishing temporal spatial positioning coordinates based on the behavior perception evaluation results; activating ultrasonic sensors and infrared sensors using the temporal spatial positioning coordinates, and performing tracking and collection based on the temporal spatial positioning coordinates through the ultrasonic sensors and infrared sensors to establish a tracking data set; updating the position data of a target object based on the data collected by the ultrasonic sensor in the tracking data set, and establishing a door distance parameter based on the position data, performing motion perception analysis using the data collected by the infrared sensor in the tracking data set, fusing and identifying the motion perception analysis results and the door distance parameters, and establishing a fusion recognition result; calling the pressure data set and IMU data set of the target object to generate a real-time motion vector; and fusing and analyzing the fusion recognition result, the local signal, and the real-time motion vector to generate an entry and exit behavior perception result.

[0005] One or more technical solutions provided in this application have at least the following technical effects or advantages:

[0006] By reading local signals, the passenger's behavior perception evaluation is performed based on local signals, and the temporal space positioning coordinates are established based on the behavior perception evaluation results. The vehicle's own signals are used for preliminary judgment and positioning, providing a basis for subsequent precise perception; the ultrasonic sensor and infrared sensor are activated using the temporal space positioning coordinates, and the two sensors are used to perform tracking and collection based on the temporal space positioning coordinates to establish a tracking data set. Through the collaborative work of multiple sensors, more comprehensive and detailed passenger behavior data is obtained based on the preliminary positioning; the position data of the target object is updated based on the data collected by the ultrasonic sensor in the tracking data set, and the door distance parameters are established based on the position data. At the same time, the infrared sensor in the tracking data set is used to collect data for motion perception analysis, and the motion perception analysis results and door distance parameters are compared. Fusion recognition, establish fusion recognition results, and obtain more accurate passenger behavior recognition results through comprehensive utilization and fusion analysis of different sensor data; call the target object's pressure data set and IMU data set to generate real-time motion vectors. The pressure data reflects the passenger's position changes, and the IMU data reflects the passenger's motion status. The combination of the two can track the passenger's dynamic actions in real time; the fusion recognition results, local signals and real-time motion vectors are fused and analyzed to generate entry and exit behavior perception results, and the final passenger entry and exit behavior perception results are obtained to achieve a comprehensive, accurate and real-time intelligent perception technical solution, which solves the technical problems of low accuracy and poor real-time performance of vehicle passenger entry and exit behavior perception in the prior art, and improves the accuracy and real-time performance of vehicle passenger entry and exit behavior perception, and achieves the technical effect of intelligent perception.

[0007] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A flow chart of a method for intelligently sensing the entry and exit behavior of passengers in a vehicle based on sensor fusion is provided for an embodiment of the present application;

[0009] Figure 2 A flow chart of establishing a real-time motion vector in an intelligent perception method of vehicle passenger entry and exit behavior based on sensor fusion is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0010] The overall idea of ​​the technical solution provided by this application is as follows:

[0011] The embodiment of the present application provides a method for intelligent perception of passenger entry and exit behavior of vehicles based on sensor fusion, which comprehensively utilizes multi-source information and realizes comprehensive, accurate and real-time intelligent perception of passenger entry and exit behavior through multi-level and multi-dimensional perception and data fusion analysis.

[0012] First, by reading local signals such as door switches, the passenger's entry and exit behavior is preliminarily judged and the time-series spatial positioning coordinates are established. Then, based on the coordinates, the ultrasonic sensor and infrared sensor are activated for target tracking and behavior analysis to obtain information such as the distance between the passenger and the door and the characteristics of the movement. At the same time, the pressure sensor and inertial measurement unit provide the passenger's position change and motion state data. Through the fusion recognition and comprehensive analysis of these multi-source sensor data, accurate and complete passenger entry and exit behavior perception results are obtained, improving the accuracy and real-time performance of perception.

[0013] After introducing the basic principles of the present application, the non-limiting implementation methods of the present application will be specifically introduced below in conjunction with the drawings in the specification.

[0014] like Figure 1 As shown, the embodiment of the present application provides a method for intelligently sensing the entry and exit behavior of passengers in a vehicle based on sensor fusion, the method comprising:

[0015] S1: Read local signals, perform a behavior perception evaluation of passengers based on the local signals, and establish a temporal spatial positioning coordinate based on the behavior perception evaluation results.

[0016] Specifically, first, multiple high-definition cameras arranged at different positions in the vehicle are used to collect video image sequences of the vehicle environment in real time to form local signals. These cameras cover the main areas in the vehicle to ensure a more comprehensive observation perspective of the passengers' behavior. Next, the local signals are intelligently processed and analyzed, and individual passengers are accurately located and identified from the video image sequence through background modeling, target detection and tracking technologies, and key behavioral features such as their posture and movement trajectory are extracted. At the same time, the potential impact of factors such as lighting changes and occlusion interference on the perception results is comprehensively considered, the algorithm parameters are adaptively adjusted, and the credibility of perception is dynamically evaluated to improve the robustness of the behavior analysis. Based on the above analysis, quantitative evaluation indicators reflecting the behavioral status of passengers in the car are obtained to form the behavior perception evaluation results.

[0017] Afterwards, based on the behavior perception evaluation results, the spatial position coordinates of the passengers are extracted and associated with the timestamps of the video image acquisition to establish the temporal spatial positioning coordinates. The coordinates record the position change trajectory of the passengers at a certain time sampling frequency to form continuous spatiotemporal sequence data. Through the temporal spatial positioning coordinates, the macroscopic behavioral characteristics of the passengers such as the movement patterns and the staying areas in the carriage can be grasped, laying the foundation for subsequent tasks such as passenger tracking and abnormal behavior detection. At the same time, the temporal spatial positioning coordinates can also be used for spatiotemporal synchronization and registration with other sensor data to achieve the fusion and collaborative analysis of multi-source perception information.

[0018] S2: Use the temporal spatial positioning coordinates to activate the ultrasonic sensor and the infrared sensor, perform tracking collection based on the temporal spatial positioning coordinates through the ultrasonic sensor and the infrared sensor, and establish a tracking data set.

[0019] Specifically, the temporal spatial positioning coordinates provide the initial position and timestamp information of the passenger in the car. According to the coordinates, the working status and acquisition strategy of the ultrasonic sensor and infrared sensor are controlled to achieve accurate tracking of the target passenger. The ultrasonic sensor continuously measures the change in distance between the passenger and the sensor by emitting high-frequency sound waves and analyzing the time difference and intensity of its reflected echo. By arranging multiple ultrasonic sensors in the car and combining them with the temporal spatial positioning coordinates, the three-dimensional spatial position of the passenger can be estimated in real time, and the blind spots of visual perception can be supplemented. The infrared sensor receives infrared rays naturally radiated by the human body to form an infrared thermal image that reflects the distribution of thermal radiation of the passenger. Combined with the temporal spatial positioning coordinates, the outline and key parts of the passenger can be extracted and tracked on the thermal image, and their posture, gestures and other fine movement information can be analyzed.

[0020] During the tracking and acquisition process, the data of the two sensors complement and verify each other, improving the continuity and stability of target tracking. At the same time, the sensor's acquisition frequency and strategy are adaptively adjusted according to the changes in the temporal positioning coordinates, taking into account both real-time and energy consumption optimization. Afterwards, the original acquisition data of the ultrasonic sensor and infrared sensor, as well as the fused target tracking results, are aligned by timestamp to form a tracking data set, which contains multi-dimensional information such as the passenger's spatial position, posture and movement, and forms a spatiotemporal association with the temporal spatial positioning coordinates, providing rich data support for subsequent multimodal behavior analysis.

[0021] S3: Based on the data collected by the ultrasonic sensor in the tracking data set, the position data of the target object is updated, and the door distance parameter is established with the position data. The data collected by the infrared sensor in the tracking data set is used for motion perception analysis, and the motion perception analysis result and the door distance parameter are fused and recognized to establish a fusion recognition result.

[0022] Specifically, first, the spatial position of the target object is updated and optimized using the data from the ultrasonic sensor in the tracking data set. The ultrasonic sensor provides the distance measurement between the target object and the internal structure of the car at a high frequency. By combining the continuous distance data with the time-series spatial positioning coordinates, the real-time three-dimensional position coordinates of the target object are estimated. On this basis, the prior knowledge of the internal structure of the car, such as the position and size information of the door, is further introduced to establish the distance parameter between the target object and the door, quantitatively describing the degree to which the target object is close to or away from the door, and providing an important basis for judging the intention of passengers to get on and off the car.

[0023] Secondly, the collected data of the infrared sensor is extracted from the tracking data set to conduct a refined analysis of the target object's action behavior. The thermal image data provided by the infrared sensor contains human posture and key point information. By applying human posture estimation and key point detection algorithms, the skeleton model and joint point positions of the target object are extracted from the thermal image. Combined with the continuity constraints in the time domain and the structured prior in the spatial domain, the motion trajectory of the joint points is tracked and analyzed to achieve the decomposition, recognition and semantic description of the target object's actions. Through in-depth mining of infrared data, the specific actions of passengers such as standing, sitting, waving, etc. are accurately perceived, and the results of action perception analysis are obtained, laying the foundation for understanding their behavioral intentions.

[0024] Afterwards, the location data and door distance parameters obtained by the ultrasonic sensor are deeply fused and jointly judged with the motion perception analysis results perceived by the infrared sensor. From the perspective of multi-scale time and space, the location changes and motion changes of the target object are uniformly considered. By weighted fusion of decision evidence from different sensors, a confidence estimate of the passenger's intention to get on and off the bus is formed to obtain the fusion recognition result. The fusion recognition result is expressed as discrete states such as getting on and off the bus, or presented in the form of continuous probability values, providing a direct basis for subsequent behavior understanding.

[0025] S4: Call the pressure dataset and IMU dataset of the target object to generate a real-time motion vector.

[0026] Specifically, a pressure sensor array is installed on the passenger seat of the vehicle to collect the pressure distribution data of the passenger on the seat in real time, forming a pressure data set, reflecting the contact status between the passenger and the seat and the change in weight distribution, and providing information about the passenger's sitting posture and movement. By performing time-frequency domain analysis and feature extraction on the pressure data, the passenger's sitting, standing up, body tilting and other movements can be identified to determine the change in their state on the seat. At the same time, the IMU sensor equipped on the vehicle itself is used to collect inertial data such as acceleration and angular velocity during the vehicle's driving process in real time to form an IMU data set, reflecting the vehicle's motion state and posture changes, and providing information about the vehicle's driving environment and road conditions. By analyzing and extracting features from the IMU data, parameters such as the vehicle's acceleration, turning angle and degree of bumpiness can be estimated to determine the vehicle's driving stability and sudden situations.

[0027] The pressure dataset of the passenger seat and the IMU dataset of the vehicle are timestamped and correlated, taking into account the mutual influence between passenger behavior and vehicle motion. For example, when the vehicle brakes or turns sharply, the pressure distribution of the passenger on the seat will change significantly, and the IMU data will also show specific acceleration and angular velocity patterns. By cross-validating and jointly reasoning the two types of data, the posture and behavior changes of passengers in complex driving environments can be more accurately judged. After that, the motion features generated by the pressure data and IMU data are integrated to construct a real-time motion vector. This vector is represented by multi-dimensional numerical values, including parameters such as the pressure distribution of the passenger on the seat, the instantaneous acceleration and angular velocity of the vehicle, which characterizes the motion state of the passenger inside the vehicle, providing a more comprehensive and reliable basis for subsequent behavior perception.

[0028] S5: Fusing and analyzing the fusion recognition result, the local signal and the real-time motion vector to generate an entry and exit behavior perception result.

[0029] Specifically, the fusion recognition results provide key information such as the position, posture and movement of passengers in the car, reflecting the spatial relationship and interaction status between passengers and key objects such as doors and seats. Local signals, as the main basis for visual perception, provide intuitive evidence of passengers' entry and exit behaviors. By analyzing the image sequence in the local signal, key actions such as passengers approaching the door, reaching out to touch the door, and crossing the door can be captured, and the entry and exit behavior patterns of passengers can be inferred by combining the spatiotemporal context information. The real-time motion vector provides a quantitative description of the passenger's motion state, reflecting the dynamic characteristics of the passenger's displacement, velocity and acceleration in the car. By analyzing the changing trend and key threshold of the motion vector, it can be judged whether the passenger has obvious intentions such as getting up, moving and getting off the car, providing supplementary evidence for behavior perception.

[0030] When performing fusion analysis, the fused recognition results, local signals, and real-time motion vectors are synchronized and aligned in time and space to ensure that the passenger's state is represented at the same timestamp. Then, a multi-level feature fusion network is designed to map the fused recognition results, local signals, and real-time motion vectors to a shared feature space. In this space, the interaction and fusion of information of different dimensions are achieved through weighted summation, nonlinear transformation, and other operations. The fused feature vector contains a comprehensive representation of the passenger's behavior, taking into account both static posture and position information and dynamic movement and change trends. Based on the feature vector, an entry and exit behavior perception model is constructed, and the discrimination boundaries and decision rules of different behavior patterns are learned through training and optimization. After that, the fused feature vector is input into the entry and exit behavior perception model. After multi-level reasoning and decision-making, the confidence estimation and semantic description of the passenger's entry and exit behavior are generated to obtain the entry and exit behavior perception result. The confidence estimation represents the possibility of different behavior categories in the form of probability values, such as "getting on the bus" and "getting off the bus". The semantic description uses natural language to describe the passenger's behavior in a more specific and detailed manner, such as "the passenger is getting up and walking towards the door", "the passenger has stepped out of the door", etc., to achieve accurate perception of the passenger's entry and exit behavior.

[0031] Furthermore, the embodiment of the present application also includes:

[0032] Read the signal source information of the local signal, divide the collection range according to the signal source information, and establish the position perception obstacle based on the collection range division result;

[0033] Perform signal acquisition impact analysis through location-aware obstacles and local signals to generate location-aware impact factors;

[0034] Performing light collection evaluation based on the local signal and establishing a light perception influencing factor;

[0035] The location perception influencing factors and light perception influencing factors are used to evaluate the passengers' behavior perception and establish the behavior perception evaluation results.

[0036] In a feasible implementation, the signal source of the local signal, i.e., the in-vehicle camera, is divided into a collection range and a position perception obstacle is established. By analyzing the position, orientation, and field of view of the camera, the effective collection range of each camera is determined. On this basis, the entire vehicle compartment space is divided into different perception areas by comprehensively considering the structural layout and seat distribution of the interior of the vehicle compartment. For each perception area, the position perception obstacles such as obstructions and blind spots that may exist in the interior are evaluated, such as seat backs, armrests, luggage racks, etc. These obstacles will partially block the camera's field of view and affect the accuracy of behavior perception. By establishing a spatial distribution map of position perception obstacles, the difference in perception quality in different areas is quantified to provide a reference for subsequent perception evaluation. Then, the influence of position perception obstacles and local signals on signal acquisition is analyzed and quantified. Specifically, according to the position and collection range of the camera, the area in each frame of the image that may be affected by the position perception obstacle is determined. For these areas, local image features, such as texture, edge, color, etc., are extracted and compared with the features of the obstacle to estimate the degree of influence of the obstacle on the image quality. At the same time, considering the distance, angle, size and other attributes of obstacles, the position perception impact factor is established to quantitatively describe the degree of interference of obstacles on behavioral perception. The value range of the position perception impact factor is set according to the actual situation, for example, it is set between 0 and 1, where 0 means no impact at all and 1 means complete occlusion. By dynamically updating the position perception impact factor, the changes in the impact of the internal environment of the car on the perception quality are reflected in real time.

[0037] Subsequently, the acquisition quality of the local signal is evaluated from the perspective of lighting conditions. Specifically, the automatic exposure adjustment mechanism of the camera is used to obtain the exposure parameters of each frame of the image, such as shutter speed, aperture size, ISO sensitivity, etc. Combined with the lighting layout and light source distribution inside the car, the illumination intensity and uniformity of different areas are evaluated. For areas with insufficient or overexposed light, the image quality may be significantly reduced, resulting in increased errors in behavior perception. Therefore, a light perception influencing factor is established to quantitatively describe the degree of influence of lighting conditions on the perception quality. Similar to the position perception influencing factor, the value range of the light perception influencing factor is also set according to actual conditions. By monitoring the lighting changes inside the car in real time, the value of the influencing factor is dynamically adjusted to ensure the adaptability and robustness of the perception evaluation. Then, the position perception influencing factor and the light perception influencing factor are comprehensively used to evaluate the passenger's behavior perception. First, the two types of influencing factors are normalized to make their value ranges consistent. Then, a weighted fusion algorithm is designed to consider the importance and credibility of different influencing factors and calculate the comprehensive influence coefficient. The comprehensive influence coefficient reflects the overall quality and reliability of the local signal on the passenger's behavior perception under the current environmental conditions. Afterwards, the comprehensive impact coefficient is combined with the behavior perception result to generate the behavior perception evaluation result, which intuitively reflects the credibility of the current perception result.

[0038] By introducing position perception influencing factors and light perception influencing factors, the reliability and adaptability of the perception results are improved, thereby achieving more stable and accurate passenger behavior perception.

[0039] Furthermore, the embodiment of the present application also includes:

[0040] Performing signal analysis on the local signal to establish the proximity effect of the passenger;

[0041] The behavior perception evaluation is compensated by the proximity distance influence, a compensated behavior perception evaluation result is established, and the temporal space positioning coordinates are reestablished using the compensated behavior perception evaluation result.

[0042] In a preferred embodiment, first, the position and size information of the passenger is extracted from the local signal using image processing and computer vision algorithms, such as face detection and body detection. The distance between the passenger and the camera is estimated based on the focal length, resolution and other parameters of the camera. Generally speaking, the closer the distance between the passenger and the camera, the larger the pixel area of ​​the passenger in the image, and the higher the corresponding perceived quality. However, when the passenger is too close to the camera, part of the body area may exceed the camera's field of view, resulting in a decrease in perceived quality. Therefore, a proximity distance influence model is established to quantify the degree of influence of passenger distance on perceived quality. The model takes the form of a piecewise function or a continuous function. By calculating the distance between the passenger and the camera in real time and substituting it into the proximity distance influence model, the proximity distance influence of the passenger can be obtained, reflecting the degree of influence of the passenger distance on behavioral perception.

[0043] Then, the obtained behavior perception evaluation results are compensated by using the influence of the proximity distance to generate more accurate and reliable compensated behavior perception evaluation results. Specifically, the proximity distance influence is weighted and fused with the position perception influence factor and the light perception influence factor to obtain a comprehensive influence factor. The comprehensive influence factor reflects the comprehensive influence of multiple factors, including passenger distance, on behavior perception under current environmental conditions. Then, the comprehensive influence factor is used as a compensation coefficient to correct the original behavior perception evaluation results to obtain compensated behavior perception evaluation results, which comprehensively reflect the quality and credibility of passenger behavior perception and reduce the deviation introduced by the distance factor. Afterwards, the compensated behavior perception evaluation results are used to re-establish the temporal spatial positioning coordinates.

[0044] By introducing the influence of proximity and compensating for the behavior perception evaluation results, the accuracy and robustness of passenger behavior perception can be improved, adapting to the movement and position changes of passengers in the car, and providing a more reliable data basis for subsequent behavior analysis.

[0045] Furthermore, the embodiment of the present application also includes:

[0046] Use behavior perception evaluation results to establish data trust weights for local signals;

[0047] Perform weighted fusion of local signals and fusion recognition results according to the data trust weight to generate basic entry and exit behavior perception results;

[0048] The real-time motion vector is used to perform enhanced verification of the basic entry-exit behavior perception result, and the entry-exit behavior perception result is generated according to the enhanced verification result.

[0049] In a feasible implementation, first, the behavior perception evaluation result is quantified into a value between 0 and 1 as the data trust weight of the local signal of the frame. The higher the data trust weight, the higher the quality and reliability of the frame signal, and the more credible the corresponding perception result. On the contrary, if the data trust weight is low, it means that the frame signal may be subject to greater interference or distortion, and the uncertainty of the perception result is greater. By dynamically calculating the data trust weight of each frame of the local signal, the change of signal quality is reflected in real time, providing a weight reference for subsequent data fusion. Then, with the data trust weight as the weight, the local signal and the fusion recognition result are weighted and fused to obtain a comprehensive basic entry and exit behavior perception result. In the fusion process, the frame with a higher data trust weight contributes more to the final result, and the frame with a lower data trust weight has a relatively small impact. Through the weighted fusion strategy, the importance of different signal sources is adaptively adjusted, the interference of low-quality data is suppressed, and the reliability of the perception result is improved. At the same time, the fused result can also take into account the advantages of the local signal and the fusion recognition result to achieve complementarity and enhancement.

[0050] Afterwards, the basic in-and-out behavior perception results are enhanced and verified using the real-time motion vector to generate the final in-and-out behavior perception results. Enhanced verification is a perception result optimization method based on motion information, which aims to further improve the accuracy and stability of the perception results. Specifically, the real-time motion vector is aligned with the basic perception result in time and space, and the consistency between the passenger's motion state and the perception result is analyzed. If the two are highly consistent, such as the perception result is that the passenger is close to the door, and the motion vector shows that the passenger is moving toward the door, the enhanced verification passes and the basic perception result is confirmed. On the contrary, if there is an obvious contradiction or difference, such as the perception result is that the passenger is stationary, but the motion vector shows that the passenger is moving quickly, the enhanced verification fails, indicating that the basic perception result may be misjudged. In the case of verification failure, the basic perception result is corrected or reset by heuristic rules or decision trees. After enhanced verification, a more stable and reliable in-and-out behavior perception result is obtained as the final output.

[0051] Through data trust weights and enhanced verification mechanisms, the complementarity of multi-source heterogeneous data is fully utilized, effectively improving the accuracy and consistency of perception results. It can better cope with complex and changeable traffic scenarios and provide more reliable data support for real-time monitoring and analysis of passenger entry and exit behaviors.

[0052] Further, such as Figure 2 As shown, the embodiment of the present application also includes:

[0053] After aligning the timestamps of the pressure data set and the IMU data set, extracting the pressure peak value of the pressure data set, and converting the pressure peak value extraction result into acceleration to establish a first acceleration;

[0054] Extract acceleration data from IMU and establish a second acceleration;

[0055] The first acceleration and the second acceleration are integrated to establish a real-time velocity of the target object, and a real-time motion vector is established according to the real-time velocity.

[0056] In a feasible implementation, first, one of the pressure data set or the IMU data set is selected as a reference, and the timestamp of the other data set is interpolated or resampled so that the timestamps of the two data sets correspond one to one. After the timestamps are aligned, the pressure peak is extracted from the pressure data set, and the key time points in the passenger's movement process, such as heel landing and toe leaving the ground, are identified from the continuous pressure data. The extracted pressure peak point contains important information about the change of the passenger's movement state. In order to facilitate the subsequent fusion analysis, the pressure peak extraction result is converted into an acceleration dimension. The conversion process uses the time interval and peak value between the pressure peak points to estimate the corresponding acceleration value, forming an acceleration sequence corresponding to the timestamp of the pressure peak point, which is recorded as the first acceleration. At the same time, the acceleration data of the IMU data set is extracted to separate the influence of gravity acceleration and vehicle motion acceleration from the IMU data. Common methods include complementary filtering, Kalman filtering, etc., which estimate the passenger's movement acceleration by fusing accelerometer and gyroscope data, which is recorded as the second acceleration. The timestamp of the second acceleration is consistent with the original timestamp of the IMU data set, reflecting the high-frequency changes in the passenger's movement state.

[0057] Afterwards, the first acceleration and the second acceleration are fused and integrated to establish the real-time velocity and motion vector of the target object. Specifically, the first acceleration and the second acceleration are aligned by timestamp to form a multi-dimensional acceleration vector sequence. Then, the acceleration vector sequence is numerically integrated to obtain the corresponding velocity vector sequence, that is, the real-time velocity of the target object. According to the real-time velocity sequence, the displacement and motion trajectory of the target object in the spatial coordinate system are further calculated to form a real-time motion vector. The real-time motion vector is expressed in the form of a three-dimensional vector, which contains the velocity components and displacement components of the target object in three directions, and comprehensively describes the instantaneous motion state of the target object.

[0058] By fusing pressure data and IMU data, the complementary characteristics of the two types of sensors in motion perception are fully utilized. Pressure data can capture the interaction between passengers and vehicles, reflecting the passenger's force state and posture changes in the vehicle; while IMU data directly measures the passenger's own motion state, providing high-frequency, high-precision acceleration and angular velocity information. Through timestamp alignment and acceleration fusion, the present invention effectively integrates pressure data and IMU data to obtain a more comprehensive and accurate estimation of the passenger's motion state.

[0059] Furthermore, the embodiment of the present application also includes:

[0060] Activating the deep learning model, converting the infrared sensor collected data into thermal map data, and inputting the thermal map data into the deep learning model;

[0061] The deep learning model is used to perform motion perception analysis of the target object and output the motion perception analysis results.

[0062] In a preferred embodiment, first, a pre-trained deep learning model is activated to prepare for subsequent motion perception analysis. The deep learning model adopts architectures such as convolutional neural networks and recursive neural networks, and learns rich motion feature representations and recognition rules through training of large-scale data sets. The input data of the model needs to meet specific format and size requirements. To this end, the raw data collected by the infrared sensor is preprocessed and converted to convert the infrared image into thermal map data. The thermal map data maps the temperature information in the infrared image to the visible spectrum range in a color-coded manner to generate an intuitive and easy-to-understand visual representation. The conversion process uses methods such as linear mapping and logarithmic mapping, and is appropriately adjusted according to the dynamic range and display requirements of the temperature distribution. The thermal map data not only retains the spatial structure and texture information of the original infrared image, but also highlights the high temperature characteristics of the human body area, which is convenient for subsequent motion perception analysis. After the generated thermal map data is scaled, cropped and normalized according to the input requirements of the deep learning model, it is sent to the model for reasoning.

[0063] Subsequently, the deep learning model is used to perform motion perception analysis of the target object on the input heat map data and output the corresponding analysis results. The deep learning model automatically learns and depicts the motion patterns and rules contained in the heat map data through layer-by-layer feature extraction and abstraction. In the feature extraction process, the model uses the convolution layer to scan the local receptive field of the heat map data to extract low-level features such as edges, textures, and contours; then the feature map is downsampled through the pooling layer to reduce the data size and redundancy; and the local features are combined into a global feature vector through the fully connected layer to form a high-level semantic representation of the entire heat map data. On this basis, the model maps the extracted features to predefined action category labels, such as "standing", "sitting", "waving", etc., and includes the confidence score of each category, reflecting the model's ability to distinguish and recognize different actions. According to the confidence score, the category with the highest score is selected as the final motion perception analysis result, realizing the refined perception of passenger actions.

[0064] Furthermore, the embodiment of the present application also includes:

[0065] After the heat map data is processed by the preprocessing layer, local feature extraction is performed based on the low-level feature extraction layer to establish the local feature extraction result;

[0066] The global spatial feature extraction layer is used to extract the features of the preprocessed heat map data and establish the spatial feature extraction results;

[0067] Use the time series feature extraction level to extract time series features and establish a time series feature vector;

[0068] Use the multi-head attention layer to perform weighted attention processing on the time series feature vector and update the time series feature vector;

[0069] The local feature extraction results, spatial feature extraction results and updated time series feature vectors are fused to complete the motion perception analysis.

[0070] In a feasible implementation, first, the input heat map data is preprocessed, and then local features are extracted through the low-level feature extraction layer. The preprocessing layer is a built-in processing unit of the deep learning model, which is responsible for standardizing, normalizing, and enhancing the input data to meet the requirements of model training and reasoning, including image cropping, scaling, rotation, flipping, color space conversion, etc., in order to increase the diversity and robustness of the data and reduce the risk of overfitting. The preprocessed heat map data is subjected to local feature extraction through the low-level feature extraction layer. The low-level feature extraction layer consists of multiple convolutional layers and pooling layers, which perform sliding window operations on the heat map data through the convolution kernel to extract local features such as edges, textures, and contours. The convolution layer uses convolution kernels of different scales and directions to capture local patterns and structural information in the heat map data; the pooling layer downsamples the convolution output to reduce the size of the feature map while retaining significant activation responses. Through multiple convolutions and pooling, the low-level feature extraction layer gradually forms a multi-scale, multi-level description of the local area, and establishes a local feature representation of the heat map data. Subsequently, the global spatial feature extraction layer is used to extract global features from the preprocessed heat map data. The global spatial feature extraction layer uses structures such as full convolutional networks and pyramid pooling, and establishes a global semantic representation of the entire heat map data through multi-scale and multi-path feature fusion. Unlike the low-level feature extraction layer, the global spatial feature extraction layer focuses on the overall structure and contextual information in the heat map data, capturing the spatial relationship and interaction pattern between different regions. This layer gradually extracts global features with a higher level of abstraction through the combination of convolution kernels with different receptive fields and cross-scale feature fusion. Global features complement local features and carry the overall semantic information of heat map data, which is of great significance for understanding complex actions and interactions. The output of the global spatial feature extraction layer forms a high-level encoding of the spatial structure of the heat map data, providing rich spatial clues for subsequent action analysis.

[0071] After that, the temporal features of the heat map data are extracted using the temporal feature extraction layer. The temporal feature extraction layer models the evolution and change rules of the heat map data in the time dimension through structures such as recurrent neural networks and long short-term memory networks. This layer takes continuous heat map data frames as input sequences, and learns and characterizes the temporal dependencies between frames through recursive state transfer and gating mechanisms. At each time step, the temporal feature extraction layer updates the hidden state and generates a new feature representation based on the feature representation of the current frame and the hidden state of the previous moment. Through cyclic iteration, the temporal feature extraction layer can capture the dynamic change patterns in the heat map data, such as motion trajectory, speed change, etc. The output result of the temporal feature extraction layer is a time series feature vector, with one feature vector corresponding to each time step, reflecting the temporal features of the heat map data at different moments. Then, the multi-head attention layer is used to perform attention weighting on the time series feature vector. The multi-head attention layer is a neural network structure based on the self-attention mechanism. By calculating the similarity and weight between feature vectors at different time steps, it realizes the dynamic weighting and selection of temporal features. Specifically, the multi-head attention layer divides the time series feature vector into multiple subsequences, each of which corresponds to an attention head. In each attention head, an attention weight matrix is ​​obtained by calculating the dot product of the feature vector of the current time step and the feature vectors of other time steps. This matrix describes the correlation and importance between different time steps. Then, according to the attention weight matrix, the time series feature vector is weighted and summed to obtain a new feature vector, which highlights important time series information and suppresses minor time series information. The results of multiple attention heads are concatenated and linearly transformed to form an updated time series feature vector.

[0072] After that, the local feature extraction results, the global spatial feature extraction results and the updated time series feature vector are fused to complete the motion perception analysis. The feature fusion layer integrates and combines features of different levels and dimensions through operations such as connection, superposition, and gating to form a unified feature representation. The fused feature vector contains the local details, global semantics, and temporal evolution information of the heat map data, providing a comprehensive characterization of the action. After that, the fused feature vector is used to predict and identify the action category. According to the pattern and distribution of the feature vector, the probability scores of different action categories are obtained, and the category with the highest score is selected as the final motion perception result.

[0073] Through multi-level feature extraction and fusion methods, the hierarchical structure and representation capabilities of deep learning models are fully utilized to achieve a comprehensive understanding and analysis of heat map data. Through the collaborative work of low-level feature extraction layers, global spatial feature extraction layers, and temporal feature extraction layers, the method extracts rich action features from three dimensions: local, global, and temporal, capturing the spatial structure, contextual information, and dynamic change laws in heat map data, which can better cope with the changes and diversity of complex actions and improve the accuracy and robustness of action perception.

[0074] Furthermore, the embodiment of the present application also includes:

[0075] Establish a perception warning database, and match the warnings in the perception warning database through the entry and exit behavior perception results, and issue early warning alerts based on the warning matching results.

[0076] In a preferred embodiment, a perception warning database is established to store and manage warning rules and thresholds related to passenger entry and exit behaviors. The perception warning database adopts various forms such as relational databases and NoSQL databases, and is selected according to the requirements of data volume, query efficiency and scalability. Each record in the database corresponds to a warning rule, which contains attributes such as warning conditions, warning levels, and warning methods. Warning conditions can be defined through logical combination and threshold comparison based on various perception indicators such as the passenger's position, posture, movement, speed, and external information such as the vehicle's operating status and environmental factors. The warning level is divided into multiple levels according to the urgency and potential impact of the warning event, such as "prompt", "warning", "serious", etc. The warning method specifies the presentation form and transmission channel of the warning information, such as voice broadcast, graphical interface, SMS push, etc.

[0077] After obtaining the perception results of entry and exit behavior, they are matched and compared with the warning rules in the perception warning database. The matching process is done by writing SQL query statements or using a special rule engine to perform logical judgment and threshold comparison based on the attribute values ​​of the entry and exit behavior perception results and the warning conditions. For example, if the entry and exit behavior perception results show that the passenger attempts to open the door while the vehicle is driving and the speed exceeds a certain threshold, a "serious" level warning is triggered. For another example, if the serious display shows that the passenger stays near the door for a long time and the vehicle is about to arrive at the station, a "prompt" level warning is triggered. The warning matching result is a set of warning rules that meet the conditions, as well as the corresponding warning levels and warning methods. According to the warning matching results, the corresponding warning information is automatically generated and reported and transmitted through the warning method. The warning information contains event description, danger level, recommended operation and other contents, which are presented to the vehicle-mounted human-computer interaction system, passenger terminal, driver terminal, etc. in a clear and concise manner. For different levels of warnings, different reporting strategies are adopted, such as voice prompts, pop-up window warnings, flashing warning lights, etc., to attract the attention of relevant personnel. At the same time, the warning information can also be sent to the vehicle monitoring center, dispatch center and other back-end management systems through the wireless communication network to achieve remote monitoring and management.

[0078] By sensing the early warning database and early warning alarm, it can automatically identify and capture abnormal behaviors and potential risks of passengers entering and exiting, and give early warnings and prompts in a timely manner, which can achieve real-time and automated early warning response and improve the safety and efficiency of transportation operations.

[0079] In summary, the intelligent perception method of vehicle passenger entry and exit behavior based on sensor fusion provided in the embodiments of the present application has the following technical effects:

[0080] Read local signals, conduct passenger behavior perception evaluation based on local signals, establish temporal spatial positioning coordinates based on behavior perception evaluation results, preliminarily judge passenger entry and exit behaviors, and determine the temporal and spatial positions of target passengers, providing reference and basis for subsequent accurate perception. Use temporal spatial positioning coordinates to activate ultrasonic sensors and infrared sensors, conduct tracking and collection based on temporal spatial positioning coordinates through ultrasonic sensors and infrared sensors, establish tracking data sets, and on the basis of preliminary positioning, dispatch ultrasonic and infrared sensors in a targeted manner to track and collect data on target passengers, obtain more detailed distance, location, action and other information, and provide data support for subsequent behavior analysis. Update the location data of the target object based on the data collected by the ultrasonic sensor in the tracking data set, and establish the door distance parameter based on the location data. Use the data collected by the infrared sensor in the tracking data set for motion perception analysis, fuse the motion perception analysis results and the door distance parameters to establish fusion recognition results, and comprehensively utilize the distance measurement capability of ultrasonic sensors and the motion perception capability of infrared sensors. Through data fusion and comprehensive analysis, the distance relationship between passengers and doors and specific entry and exit behaviors are obtained to achieve more accurate behavior recognition. The pressure data set and IMU data set of the target object are called to generate real-time motion vectors, and the passenger's foot pressure data and inertial measurement data are introduced to track the position changes and movement status of passengers in real time, provide dynamic information for behavior perception, and improve the real-time and accuracy of perception. The fusion recognition results, local signals and real-time motion vectors are fused and analyzed to generate the entry and exit behavior perception results. The fusion recognition results of multi-source sensors, the local signals of the vehicle and the real-time movement status of the passengers are comprehensively considered, and the final passenger entry and exit behavior perception results are generated through cross-validation and comprehensive judgment of multi-dimensional information, realizing comprehensive, accurate and real-time intelligent perception.

[0081] Any step of the method described above can be stored as a computer instruction or program in an unlimited computer memory, and can be called and recognized by an unlimited computer processor to implement any method in the embodiments of the present application, without any unnecessary restrictions.

[0082] Furthermore, the first or second mentioned above may not only represent an order relationship, but may also represent a specific concept, and / or refer to multiple elements that can be selected individually or in full. Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present application and its equivalents, the present application intends to include these changes and variations.

Claims

1. An intelligent perception method for passenger entry and exit behavior of transportation vehicles based on sensor fusion, characterized in that: The method comprises: Reading local signals, performing a passenger's behavior perception evaluation based on the local signals, and establishing a temporal spatial positioning coordinate based on the behavior perception evaluation result; The ultrasonic sensor and the infrared sensor are activated by using the time-series spatial positioning coordinates, and the tracking collection based on the time-series spatial positioning coordinates is performed by the ultrasonic sensor and the infrared sensor to establish a tracking data set; Based on the data collected by the ultrasonic sensor in the tracking data set, the position data of the target object is updated, and the door distance parameter is established with the position data, and the motion perception analysis is performed using the data collected by the infrared sensor in the tracking data set, and the motion perception analysis result and the door distance parameter are fused and recognized to establish a fusion recognition result; Call the pressure data set and IMU data set of the target object to generate real-time motion vectors; The fusion recognition result, the local signal and the real-time motion vector are fused and analyzed to generate an entry and exit behavior perception result; The passenger's behavior perception evaluation based on the local signal further includes: Read the signal source information of the local signal, divide the collection range according to the signal source information, and establish the position perception obstacle based on the collection range division result; Perform signal acquisition impact analysis through location-aware obstacles and local signals to generate location-aware impact factors; Performing light collection evaluation based on the local signal and establishing a light perception influencing factor; Use the location perception influencing factors and light perception influencing factors to evaluate the passenger's behavior perception and establish the behavior perception evaluation results; The step of fusing and analyzing the fusion recognition result, the local signal and the real-time motion vector to generate the entry and exit behavior perception result also includes: Use behavior perception evaluation results to establish data trust weights for local signals; Perform weighted fusion of local signals and fusion recognition results according to the data trust weight to generate basic entry and exit behavior perception results; Performing enhanced verification of a basic entry / exit behavior perception result using the real-time motion vector, and generating an entry / exit behavior perception result according to the enhanced verification result; The calling of the pressure data set and the IMU data set of the target object to generate a real-time motion vector also includes: After aligning the timestamps of the pressure data set and the IMU data set, extracting the pressure peak value of the pressure data set, and converting the pressure peak value extraction result into acceleration to establish a first acceleration; Extract acceleration data from IMU and establish a second acceleration; The first acceleration and the second acceleration are integrated to establish a real-time velocity of the target object, and a real-time motion vector is established according to the real-time velocity.

2. The intelligent perception method for passenger entry and exit behavior of a vehicle based on sensor fusion as claimed in claim 1, characterized in that: The use of the position perception influencing factor and the light perception influencing factor to evaluate the passenger's behavior perception and establish the behavior perception evaluation result also includes: Performing signal analysis on the local signal to establish the proximity effect of the passenger; The behavior perception evaluation is compensated by the proximity distance influence, a compensated behavior perception evaluation result is established, and the temporal space positioning coordinates are reestablished using the compensated behavior perception evaluation result.

3. The intelligent perception method of vehicle passenger entry and exit behavior based on sensor fusion as claimed in claim 1, characterized in that: The method of using the infrared sensor in the tracking data set to collect data for motion perception analysis also includes: Activating the deep learning model, converting the infrared sensor collected data into thermal map data, and inputting the thermal map data into the deep learning model; The deep learning model is used to perform motion perception analysis of the target object and output the motion perception analysis results.

4. The intelligent perception method for passenger entry and exit behavior of a vehicle based on sensor fusion as claimed in claim 3, characterized in that: The use of the deep learning model to perform motion perception analysis of the target object also includes: After the heat map data is processed by the preprocessing layer, local feature extraction is performed based on the low-level feature extraction layer to establish the local feature extraction result; The global spatial feature extraction layer is used to extract the features of the preprocessed heat map data and establish the spatial feature extraction results; Use the time series feature extraction level to extract time series features and establish a time series feature vector; Use the multi-head attention layer to perform weighted attention processing on the time series feature vector and update the time series feature vector; The local feature extraction results, spatial feature extraction results and updated time series feature vectors are fused to complete the motion perception analysis.

5. The intelligent sensing method for passenger entry and exit behavior of a vehicle based on sensor fusion as claimed in claim 1, characterized in that: The method further comprises: Establish a perception warning database, and match the warnings in the perception warning database through the entry and exit behavior perception results, and issue early warning alerts based on the warning matching results.

Images