Vehicle welcome method and device, electronic equipment and storage medium

Abstract

The invention is suitable for the field of intelligent vehicle-mounted technology, and provides a vehicle welcome method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining multi-mode perception data in real time, the multi-mode perception data comprises environment data of a vehicle welcome area, basic motion data of a user, and visual feature data; according to the basic motion data and a trajectory prediction model, predicting a motion trajectory of the user, the trajectory prediction model being a pre-trained network model based on deep learning; determining a welcome strategy according to the motion track, the visual feature data and the environment data; and controlling the vehicle to execute the welcome strategy for welcome service. According to the method, the integrating degree of the welcome service and the actual demand of the user can be improved, and personalized and intelligent welcome interaction experience is provided for the user.

Description

Technical Field

[0001] The present application relates to the field of intelligent vehicle-mounted technology, and in particular to a vehicle welcoming method, device, electronic device and storage medium. Background Art

[0002] As a key driver of human-machine interaction in smart cars, vehicle welcome systems are widely used in scenarios such as daily commuting and business receptions. By automatically unlocking the doors, activating the welcome lighting, and adjusting the seat position, these systems provide users with a convenient keyless entry experience while enhancing the vehicle's technological awareness and brand value. With the advancement of intelligent connected technology, this feature has gradually penetrated from high-end models into the mainstream market and is playing a key role in increasing user engagement in emerging mobility sectors such as car-sharing.

[0003] However, existing vehicle welcome methods suffer from technical flaws such as high false trigger rates and significant welcome delays, which can lead to the welcome service not being fully aligned with actual user needs. Improving the alignment of welcome services with actual user needs and providing users with a personalized, intelligent welcome interaction experience is a technical challenge that needs to be addressed. Summary of the Invention

[0004] The embodiments of the present application provide a vehicle welcoming method, device, electronic device and storage medium, which can improve the fit between the welcoming service and the actual needs of users and provide users with a personalized and intelligent welcoming interactive experience.

[0005] In a first aspect, an embodiment of the present application provides a vehicle welcoming method, comprising:

[0006] Acquire multimodal perception data in real time, the multimodal perception data including environmental data of the vehicle's welcome area, basic motion data of the user, and visual feature data;

[0007] Predicting the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained, deep learning-based network model;

[0008] determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data;

[0009] The vehicle is controlled to execute the welcoming strategy to provide welcoming service.

[0010] In a possible implementation of the first aspect, the basic motion data includes the user's real-time position, motion speed, and motion acceleration;

[0011] The step of acquiring multimodal perception data in real time includes:

[0012] Real-time acquisition of user location provided by the ultra-wideband module and user speed and acceleration provided by the radar;

[0013] The user position, user speed and user acceleration are fused using a preset fusion algorithm to output optimized real-time position, motion speed and motion acceleration.

[0014] In a possible implementation of the first aspect, the step of predicting the user's motion trajectory based on the basic motion data and the trajectory prediction model includes:

[0015] Inputting the basic motion data and the cached historical motion parameter sequence into the trajectory prediction model;

[0016] The trajectory prediction model extracts the time-dependent characteristics of the user's motion based on the basic motion data and the historical motion parameter sequence, and based on the time-dependent characteristics, outputs the predicted position coordinates, velocity vector and acceleration vector in the future time window to determine the user's motion trajectory.

[0017] In a possible implementation of the first aspect, before the step of determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data, the method further includes:

[0018] authenticating the user based on the visual feature data to determine whether the user is an authorized greeter user;

[0019] If not, the door unlocking function is restricted and the security warning module is activated;

[0020] If so, a welcoming strategy is determined based on the motion trajectory, the visual feature data and the environmental data.

[0021] In a possible implementation of the first aspect, the step of determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data includes:

[0022] determining a behavior pattern of the user based on the motion trajectory and the visual feature data;

[0023] determining a target welcoming function based on the behavior pattern and the environmental data;

[0024] A welcoming strategy is determined according to the motion trajectory, the target welcoming function, and the environmental data.

[0025] In a possible implementation of the first aspect, the motion trajectory includes a path curvature and an acceleration vector, and the visual feature data includes gait features and human posture features; and the step of determining the user's behavior pattern based on the motion trajectory and the visual feature data includes:

[0026] Determining the instantaneous velocity change rate and the acceleration direction according to the acceleration vector;

[0027] The user's behavior pattern is determined according to the path curvature, the instantaneous speed change rate, the acceleration direction, the gait characteristics, and the human body posture characteristics.

[0028] In a possible implementation of the first aspect, the method further includes:

[0029] Record user feedback data on the implementation of the welcome strategy;

[0030] Adaptively learn and optimize the welcoming strategy based on the execution feedback data.

[0031] In a second aspect, an embodiment of the present application provides a vehicle welcoming device, comprising:

[0032] A data perception unit, configured to acquire multimodal perception data in real time, wherein the multimodal perception data includes environmental data of the vehicle's welcome area, basic motion data of the user, and visual feature data;

[0033] a trajectory prediction unit, configured to predict the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained network model based on deep learning;

[0034] a strategy determination unit, configured to determine a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data;

[0035] A welcome control unit is used to control the vehicle to execute the welcome strategy and provide welcome service.

[0036] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the vehicle welcoming method as described in the first aspect above is implemented.

[0037] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vehicle welcoming method as described in the first aspect above is implemented.

[0038] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the vehicle welcoming method as described in the first aspect above.

[0039] The embodiment of the present application adopts the collaborative processing and intelligent decision-making of multimodal perception data. By acquiring environmental data, basic motion data and visual feature data in real time, it establishes a full-dimensional perception capability of the user status and surrounding environment, providing a data foundation for precise services. The trajectory prediction model based on the deep learning model accurately predicts the user's motion trajectory, which can effectively ensure the synchronization of welcoming interaction. Then, through the strategy generation mechanism that integrates motion trajectory, visual feature data and environmental data, the welcoming strategy is dynamically determined according to the user's real-time behavior and environmental conditions, and differentiated welcoming services are performed for different users, thereby improving the fit between the welcoming service and the user's actual needs, and providing users with a personalized and intelligent welcoming interaction experience, thereby improving user experience and service quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0041] Figure 1 This is a flow chart of the implementation of the vehicle welcoming method provided in an embodiment of the present application;

[0042] Figure 2 This is a specific implementation flow chart of step S101 in the vehicle welcoming method provided in an embodiment of the present application;

[0043] Figure 3 This is a specific implementation flow chart of predicting the user's motion trajectory in the vehicle welcoming method provided in an embodiment of the present application;

[0044] Figure 4 This is a specific implementation flow chart of user identity authentication in the vehicle welcoming method provided in an embodiment of the present application;

[0045] Figure 5 This is a specific implementation flowchart of step S103 in the vehicle welcoming method provided in an embodiment of the present application;

[0046] Figure 6 This is a structural block diagram of a vehicle welcoming device provided in an embodiment of the present application;

[0047] Figure 7 Schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0049] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0050] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0051] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0052] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0053] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0054] By way of example and not limitation, the vehicle welcoming method provided in the embodiments of the present application can be applied to various types of electronic devices and servers, including, for example, vehicle-mounted terminals, mobile phones, tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), desktop computers, etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0055] Figure 1 The implementation process of the vehicle welcoming method provided by the embodiment of the present application is shown, and the method process includes steps S101 to S104. The specific implementation principles of each step are as follows:

[0056] Step S101: Acquire multimodal perception data in real time, where the multimodal perception data includes environmental data of a vehicle welcome area, basic motion data of a user, and visual feature data.

[0057] Multimodal sensory data refers to a diverse set of data collected from different dimensions using a variety of different types of sensors or sensing devices. In this embodiment, by collecting real-time environmental data from the vehicle's welcome area, basic user motion data, and visual feature data, a comprehensive description of the environmental state, user characteristics, and motion information within the vehicle's welcome area is provided.

[0058] The vehicle welcome area refers to a specific pre-set physical space range centered on the vehicle. This area is the effective range for the vehicle to actively perceive users and provide welcoming services. When the user enters the welcome area, the vehicle collects environmental data, basic motion data and visual feature data in the welcome area in real time through the deployed sensors or perception devices. Normally, the welcome area can cover a certain distance around the vehicle (for example, within 1-5 meters around the door). The specific range of the welcome area can be preset or dynamically adjusted according to the vehicle type, sensor layout, welcome strategy requirements and user needs. The setting of the welcome area ensures that the vehicle can perceive and prepare services in advance during the user's approach, avoiding invalid detection or false triggering.

[0059] In one possible implementation, to ensure the validity of multimodal perception data, the acquired multimodal perception data is cleaned and anomaly processed to filter out noise and outliers. Missing data can be supplemented using interpolation or predictive filling methods; outliers can be corrected using statistical methods and learning approaches.

[0060] As a possible implementation of the present application, the basic motion data includes the user's real-time location, motion speed, and motion acceleration; the visual feature data includes gait features and facial features; and the environmental data includes light intensity, ambient temperature and humidity, precipitation conditions, and obstacle distribution.

[0061] In one possible implementation, the sensory data acquired by multiple sensor types is spatially and temporally aligned to ensure that each type of data corresponds to the same user and environmental conditions at the same moment. This spatial and temporal alignment process transforms the multimodal sensory data into a unified sequence of spatial and temporal features, ensuring data validity.

[0062] Figure 2 A specific implementation process of step S101 in the vehicle welcoming method provided in an embodiment of the present application is shown and is described in detail as follows:

[0063] A1: The user's real-time location, speed, and acceleration are acquired through the integration of an ultra-wideband (UWB) module and radar. By acquiring basic motion data such as real-time location, speed, and acceleration, the user's dynamic location changes are quantified.

[0064] In one possible implementation, the user position provided by the UWB module and the user speed and acceleration provided by the radar are obtained in real time. The user position, speed, and acceleration are fused using a preset fusion algorithm to output the optimized real-time position, speed, and acceleration. The UWB module can be a positioning ultra-wideband module to accurately determine the user's position, and the radar can be a millimeter-wave radar, a laser radar, or an ultrasonic radar. The preset fusion algorithm can be a Kalman filter algorithm, a particle filter algorithm, a weighted average fusion algorithm, or the like. This embodiment does not impose any restrictions on the type, model, or fusion algorithm of the UWB module or radar.

[0065] In one possible implementation, the pre-set fusion algorithm is a Kalman filter. The UWB module acquires the user's position in the vehicle coordinate system in real time. Radar uses the Doppler effect to measure the user's speed, approach direction, and distance change. Acceleration is calculated by velocity decomposition. The two data sets are then fused using a Kalman filter. Position and velocity estimates are optimized using a state-space model to reduce noise interference from a single sensor. The final output is an optimized motion state vector, which includes real-time position, velocity, and acceleration.

[0066] The Kalman filter is used to fuse the user position provided by the UWB module and the user speed and acceleration provided by the radar. The method is divided into two stages: prediction stage and update stage.

[0067] During the prediction phase, the state-space model is used to estimate the current state using the previous state. The state-space model is the foundation of the Kalman filter algorithm. The state-space model is based on position and velocity. The user's previous position and velocity are combined with reasoning from the state-space model to predict the user's current position and velocity.

[0068] For example, the state space model is as follows (1):

[0069]

[0070] in, Represents the predicted state vector, including the predicted position p at the current k moment k and speed v k ; represents the state transfer matrix, which is used to describe the relationship between the state and time, and Δt represents the time step; Represents the control input matrix, which is used to describe the impact of acceleration on the state; U k is the acceleration detected by the radar at the current time k.

[0071] At the same time, the confidence of the predicted state vector is determined according to the following formula (2):

[0072]

[0073] in, is the uncertainty covariance matrix of the prediction, which represents the confidence of the predicted state vector; P k-1 represents the uncertainty covariance matrix of the state at the previous moment; A represents the state transfer matrix, which is used to describe the relationship between the state and time; A T represents the transposed matrix of the state transfer matrix A; Q represents the process noise covariance matrix, which is used to describe the uncertainty of the model.

[0074] In the updating stage, the observation data provided by the sensors (UWB module and radar) are fused to correct the prediction results.

[0075] For example, the UWB module provides the position p UWB , radar provides speed v Radar , the fusion formula of observation data is as follows (3) and (4):

[0076]

[0077] Among them, x k Represents the updated state vector, which contains the fused corrected position and velocity; represents the predicted state vector, z k Represents the observation vector (the position provided by UWB and the speed provided by radar); H represents the observation matrix, which is used to describe how the state vector is mapped to the sensor observation space, K k Represents the Kalman gain, which is used to determine the weighted ratio of the predicted value and the observed value. represents the confidence of the predicted state vector; R represents the observation noise covariance matrix, which is used to reflect the uncertainty of sensor measurement.

[0078] Update the state confidence according to the following formula (5):

[0079]

[0080] Among them, P k Represents the confidence of the updated state vector, I represents the identity matrix, which is a square matrix with elements on the diagonal being 1 and other elements being 0. The identity matrix is ​​used to ensure the consistency of the dimensions of the matrix calculation, H represents the observation matrix, K k represents the Kalman gain, Represents the confidence of the predicted state vector.

[0081] In this embodiment, a Kalman filter fuses the user's position, as provided by the UWB module, with the user's velocity and acceleration, as provided by the radar, ultimately outputting optimized real-time position, velocity, and acceleration. The Kalman filter algorithm effectively suppresses sensor noise by modeling user motion patterns in state space. It dynamically fuses heterogeneous data from the UWB module and radar, achieving synergistic gains in accuracy, robustness, and real-time performance, providing highly reliable state perception for the vehicle's welcome system.

[0082] A2: Obtain the gait features, facial features and body posture features of the user through an intelligent visual device. The intelligent visual device can be an intelligent camera or a depth sensor. The intelligent visual device collects a continuous video frame sequence of the user walking naturally, and extracts facial features and gait features from the continuous video frame sequence. Gait features include but are not limited to stride length and step frequency; facial features include but are not limited to facial features and expressions; body posture features include but are not limited to torso inclination, walking posture (whether carrying items, whether mobility is inconvenient), etc. Gait features, facial features and body posture features can be used to identify user identity, behavior patterns and personalized features.

[0083] In one possible implementation, image enhancement processing (denoising and contrast adjustment) is performed on the collected continuous video frame sequence; the extracted gait features are normalized, and the perspective difference is eliminated through human skeleton coordinate transformation.

[0084] In the embodiment of the present application, visual features and deep learning technology are combined to achieve contactless and imperceptible collection of user biometrics, without the user having to stop or actively cooperate. At the same time, the multi-dimensional biometric collection of gait and face helps to improve the accuracy of determining user behavior patterns, thereby improving the effectiveness of determining the welcoming strategy.

[0085] A3: Obtain the light intensity, ambient temperature and humidity, precipitation status, and obstacle distribution of the welcome area through environmental sensors.

[0086] In one possible implementation, environmental sensors include, but are not limited to, light sensors, temperature and humidity sensors, precipitation sensors, and obstacle detection sensors. The light sensor obtains the light intensity in the welcome area, the temperature and humidity sensor obtains the ambient temperature and humidity, the precipitation sensor obtains the precipitation status, and the obstacle detection sensor determines the distribution of obstacles. Environmental data such as light intensity, ambient temperature and humidity, precipitation status, and obstacle distribution collected by the environmental sensors reflects the user's external environmental conditions. This multi-dimensional environmental data is standardized and normalized to ensure the validity of the collected environmental data.

[0087] Collected environmental data is crucial in determining welcome strategies. Light intensity is used for intelligent lighting control, ambient temperature and humidity are used for in-car climate control, and precipitation levels are used to determine waterproof and anti-slip measures. Precipitation levels determine door opening angles and adjust the speed of the welcome pedal retraction and extension, while obstacle distribution aids in collision avoidance.

[0088] In the embodiments of the present application, accurate perception of multi-dimensional environmental data can help implement a scene-adaptive welcoming strategy, significantly improving safety and user experience.

[0089] Step S102: Predicting the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained network model based on deep learning.

[0090] The motion trajectory includes the predicted future position coordinates, velocity vector, acceleration vector, etc. Based on the future position coordinates, velocity vector, acceleration vector, it reflects the user's movement route, arrival time, and motion trend (such as whether to accelerate and approach, whether to bypass obstacles).

[0091] In the embodiments of this application, a deep learning-based network model is used to model the user's motion trajectory, enabling high-precision, real-time prediction of future motion, providing an important basis for determining subsequent welcoming strategies. The trajectory prediction model can be a long short-term memory network model or a Transformer-based network model. This embodiment does not impose any restrictions on the architecture or type of the trajectory prediction model.

[0092] As a possible implementation of this application, Figure 3 As shown, the step of predicting the user's motion trajectory based on the basic motion data and the trajectory prediction model includes:

[0093] B1: Inputting the basic motion data and the cached historical motion parameter sequence into a trajectory prediction model.

[0094] After feature concatenation of the real-time basic motion data and the cached historical motion parameter sequence, the data is input into the trajectory prediction model. In one possible implementation, the trajectory prediction model may be a long short-term memory network model.

[0095] Basic motion data is real-time motion data, such as the current position, velocity, and acceleration. Historical motion parameter sequences are motion data acquired over a period of time, such as the position, velocity vector, and acceleration every 0.1 seconds over the past 10 seconds. The specific duration of the past period can be customized based on user habits, welcome scenarios, and other factors. The cache of historical motion parameter sequences can be used to model the inertial patterns of user motion. The cache of historical motion parameter sequences is dynamically updated.

[0096] B2: The trajectory prediction model extracts the time-dependent characteristics of the user's motion based on the basic motion data and the historical motion parameter sequence, and based on the time-dependent characteristics, outputs the predicted position coordinates, velocity vector and acceleration vector in the future time window to determine the user's motion trajectory.

[0097] In an embodiment of the present application, the trajectory prediction model combines real-time basic motion data with a historical motion parameter sequence to achieve time-dependent modeling of the user's motion, effectively adapting to the dynamic changes of the user's motion, capturing dependencies in long time series through the memory and forget gate mechanisms, automatically filtering out irrelevant historical motion data (such as forgetting early motion data during long periods of stillness), retaining key motion data (such as recent acceleration), and then outputting the route and speed vector of the user approaching the vehicle in a short period of time in the future, thereby providing high-precision motion trajectory predictions and avoiding the one-sidedness of predictions based solely on current basic motion data.

[0098] Step S103: determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data.

[0099] In this embodiment, a refined welcoming strategy adapted to user needs and environmental conditions is generated by dynamically fusing motion trajectories, visual feature data, and environmental data.

[0100] In one possible implementation, the three types of data, motion trajectory, visual feature data, and environmental data, are mapped to a unified time axis through timestamp synchronization for time alignment to eliminate time sequence misalignment; the environmental data includes obstacle distribution, and the motion trajectory based on the vehicle coordinate system, the visual feature data based on the world coordinate system, and the obstacle distribution based on the grid map are spatially aligned through coordinate transformation.

[0101] As a possible implementation of this application, Figure 4As shown, before the step of determining the welcoming strategy based on the motion trajectory, the visual feature data and the environmental data, the method further includes:

[0102] C1: Authenticate the user based on the visual feature data to determine whether the user is an authorized welcome user. An authorized welcome user is a user who has been recognized by the vehicle welcome system and allowed to trigger a specific welcome service. Authorized welcome users for different vehicles can be pre-defined based on user needs.

[0103] In one possible embodiment, the identity of the user approaching the vehicle is authenticated based on facial features or gait features. When the matching degree between the facial features and pre-stored facial features is higher than or equal to a first threshold, or when the matching degree between the gait features and pre-stored gait features is higher than or equal to a second threshold, the user is determined to be a welcome authorized user. When the matching degree between the facial features and pre-stored facial features is lower than the first threshold, or when the matching degree between the gait features and pre-stored gait features is lower than the second threshold, the user is determined not to be a welcome authorized user.

[0104] In one possible implementation, the user's identity is authenticated based on facial and gait features to determine whether the user is an authorized greeter user. Specifically, a first-level verification is initiated based on facial features. When the degree of match between the facial features and pre-stored facial features is greater than or equal to a first threshold, the user is determined to be an authorized greeter user. When the degree of match between the facial features and pre-stored facial features is less than the first threshold, a second-level verification is initiated based on gait features. When the degree of match between the gait features and pre-stored gait features is greater than or equal to a second threshold, the user is determined to be an authorized greeter user; otherwise, the user is not an authorized greeter user.

[0105] C2: If not, the door unlocking function is restricted and the security warning module is activated. The door unlocking function can be implemented through the vehicle's ECU (Electronic Control Unit) or access control system. After activation, the security warning module is used to detect in real time whether the vehicle has been pried, hit, or moved abnormally.

[0106] When the user approaching the vehicle is not an authorized user, the safety protection mechanism is automatically triggered, immediately restricting the door unlocking function, cutting off unauthorized access to the vehicle at the source, and activating the security warning module at the same time to avoid unauthorized operations and provide real-time risk feedback, ensuring vehicle safety and reducing the risk of property loss.

[0107] C3: If yes, determine a welcoming strategy based on the motion trajectory, the visual feature data and the environmental data.

[0108] In the embodiment of the present application, before determining the welcoming strategy, the user identity is first identified through visual feature data to ensure that the service object is trustworthy, avoid false triggering of the welcoming, and improve security.

[0109] As a possible implementation of this application, Figure 5 A specific implementation process of step S103 in the vehicle welcoming method provided in an embodiment of the present application is shown and is described in detail as follows:

[0110] D1: Determine the user's behavior pattern based on the motion trajectory and the visual feature data. A behavior pattern refers to a combination of user behavior features.

[0111] The user's behavior pattern is comprehensively determined based on the movement trajectory, facial features, gait characteristics, etc. The determination of the behavior pattern helps to accurately identify the user's intention.

[0112] In one possible implementation, the user's behavior modes include, but are not limited to, a normal walking approach mode, a fast approach mode, an item carrying mode, an emergency running mode, a stop and observe mode, and an accompanying others mode.

[0113] In normal walking approach mode, the movement trajectory is a straight line or a gentle curve approaching the vehicle, the speed is uniform, there is no sudden acceleration, deceleration or detour, the stride is stable, the step frequency is medium, the body center of gravity is stable, and the arms swing naturally.

[0114] In the rapid approach mode, the motion trajectory becomes linear, the speed is significantly accelerated, the stride length and frequency are increased, the arm swing amplitude increases, and the center of gravity fluctuates more obviously.

[0115] In carrying mode, the speed is slower, the body leans forward or sideways, the path may sway slightly due to the load, the stride is shortened, the step frequency is reduced, the arm swing amplitude is reduced or fixed (such as when carrying heavy objects), and the torso stability is reduced.

[0116] In emergency running mode, the rider sprints in a straight line at high speed, with a straight path, which may be accompanied by sharp turns (such as avoiding obstacles). The stride is large, the step frequency is extremely high, the body leans forward sharply, the arm swing frequency is fast and large, and the center of gravity fluctuates significantly.

[0117] In the stop-and-watch mode, the driver moves back and forth around the vehicle (such as to inspect the vehicle or look for a parking space), at a low speed, with multiple changes in direction, an irregular stride, and sometimes fast and sometimes slow steps, accompanied by pauses (such as stopping to check the vehicle's condition).

[0118] In the accompanying others mode, users maintain a fixed distance from other users, synchronize their movement routes (such as walking side by side or following each other), match their stride length and cadence with their companions, and their body direction may be towards their companions (such as turning their head to communicate).

[0119] In one possible implementation, the motion trajectory includes path curvature and acceleration vector, and the visual feature data includes gait features and human posture features. Based on the acceleration vector, the instantaneous velocity change rate and acceleration direction are determined; based on the path curvature, instantaneous velocity change rate, acceleration direction, gait features, and human posture features, the user's behavior pattern is determined.

[0120] Path curvature can be calculated based on the position coordinate sequence output by the trajectory prediction model. The position coordinate sequence is a chronological sequence of the user's position coordinates. The position sequence is processed using a difference method and combined with the curvature formula to calculate the path curvature of the motion trajectory. Path curvature accurately quantifies the degree of curvature of the user's motion trajectory. For example, in normal walking approach mode, the trajectory is typically straight or gently curved, with a small curvature value. In contrast, in emergency running mode, sharp turns may occur, resulting in significantly larger curvature values.

[0121] The instantaneous rate of change of velocity is the magnitude of acceleration, which can be obtained by calculating the modulus of the acceleration vector. The direction of acceleration is the direction of the acceleration vector, which can be determined by calculating the angle between the acceleration vector and the coordinate axis. The instantaneous rate of change of velocity accurately reflects the speed of the user's speed change, and the direction of acceleration reflects the direction of the change in the user's movement speed. For example, in the rapid approach mode, the speed increases significantly, and the instantaneous rate of change of velocity increases significantly; in the emergency running mode, there may be drastic speed changes such as high-speed straight sprinting and sharp turns, and the instantaneous rate of change of velocity will fluctuate greatly; in the normal walking approach mode, the acceleration direction is relatively stable; in the emergency running mode, the acceleration direction may change frequently. Determining the instantaneous rate of change of velocity and the acceleration direction helps to identify sudden changes in the user's motion state.

[0122] By integrating and analyzing different features such as path curvature, instantaneous velocity change rate, acceleration direction, gait characteristics, and human posture characteristics, the user's behavior pattern is determined. In one possible implementation, a comprehensive analysis model based on machine learning algorithms or rule-matching logic is established, using path curvature, instantaneous velocity change rate, acceleration direction, gait characteristics, and human posture characteristics as model inputs. This comprehensive analysis model performs a fusion analysis and outputs the behavior pattern corresponding to the input.

[0123] In an embodiment of the present application, by integrating the geometric characteristics of the motion trajectory (path curvature), the motion state change characteristics (instantaneous velocity change rate and acceleration direction) and the human body's own motion characteristics (gait characteristics and human posture characteristics), the user's behavior state is comprehensively described from multiple angles, which can effectively improve the accuracy of user behavior pattern recognition and achieve refined classification of user behavior patterns, providing an accurate basis for subsequent personalized welcoming services.

[0124] D2: determining a target welcoming function according to the behavior mode and the environment data.

[0125] Different environments and different behavior modes match different welcoming functions. Welcoming functions include but are not limited to unlocking vehicle doors, unlocking the trunk, adjusting the seat position, turning on the welcome light, and popping up the waterproof baffle inside the door. A set of matching logic is preset, which includes the mapping rules of behavior mode, environment data and target welcoming function. According to the preset matching logic, the target welcoming function corresponding to the user's behavior mode in the current environment is determined. The matching logic can be adjusted adaptively according to user needs and actual application situations, and can be continuously learned and optimized to adapt to the individual preferences and behavior habits of different users.

[0126] For example, when the user is in a normal walking approach mode, the target welcoming function includes preferentially unlocking the driver's door, delaying unlocking other doors, and automatically adjusting the driver's seat. When the user is in a fast approach mode, the target welcoming function includes unlocking the door in advance. When the user is in a carrying items mode, the target welcoming function includes synchronously unlocking the driver's door and the trunk. When the user is in an emergency running mode, the target welcoming function includes immediately unlocking all doors and starting the engine. When the user is in an accompanying others mode, the target welcoming function includes unlocking the corresponding door according to the companion (such as synchronously unlocking the driver's door, the front passenger's door, and the rear left door). When the user is in a stay and observe mode, the target welcoming function is temporarily empty. In any behavior mode, if it is determined according to the precipitation state in the environment data that it is currently raining, the target welcoming function further includes popping up the waterproof baffle inside the door.

[0127] D3: determining a welcoming strategy according to the motion trajectory, the target welcoming function, and the environment data.

[0128] The motion trajectory, the target welcoming function, and the environment data are taken as inputs, and a comprehensive decision system is used for analysis and processing to determine the welcoming strategy. Determining the welcoming strategy includes determining the triggering time, execution order, and door opening amplitude of the target welcoming function. For example, unlocking the door and turning on the welcome light first, adjusting the seat position when the user approaches to a certain distance, popping up the waterproof baffle when the user reaches the door if precipitation is detected, etc., to ensure the continuity and adaptability of the welcoming process.

[0129] The predicted motion trajectory includes the user's position in a future period of time (such as 1-3 seconds), the user's arrival time at the door is predicted based on the position in the future period of time, and the triggering time of the target welcoming function is planned; the door opening amplitude is dynamically adjusted according to the precipitation state. For example, when continuous precipitation is detected, the door opening angle is determined to be reduced by 30%-50% compared to the default door opening amplitude.

[0130] In an embodiment of the present application, the user's behavior pattern is determined by motion trajectory and visual feature data, and the target welcome function is determined in combination with environmental data. Finally, a welcome strategy is generated based on the motion trajectory, target welcome function and environmental data, thereby achieving an organic coordination of user behavior intention, scene constraints and service execution timing. The combination of behavior pattern and environmental data makes the target welcome function both in line with user needs and adapt to real scenes, which helps to improve the accuracy of the welcome strategy determination. The precise planning of the function activation timing by motion trajectory can ensure seamless connection between service and user actions, reduce waiting time, and ultimately make the determined welcome strategy have both depth of intention understanding, breadth of scene adaptation and execution timing accuracy, effectively promoting the evolution of intelligent welcome service from passive response to active prediction, and providing users with personalized and intelligent welcome interaction experience.

[0131] Step S104: controlling the vehicle to execute the welcoming strategy to provide welcoming service.

[0132] The on-board control device breaks down the welcoming strategy into precise vehicle actions, generates corresponding underlying control instructions, and drives the vehicle actuators corresponding to each action to execute the underlying control instructions, thereby realizing intelligent welcoming.

[0133] For example, the door unlocking command and the welcome light control command are determined according to the welcome strategy, and the control signal corresponding to the welcome strategy is sent to the door actuator and the light controller via the vehicle controller area network (CAN bus) to achieve door unlocking and lighting of the welcome light.

[0134] In the embodiment of the present application, by acquiring environmental data, basic motion data and visual feature data in real time, an all-round perception capability of the user status and surrounding environment is established, providing a data basis for precise services. The trajectory prediction model based on the deep learning model accurately predicts the user's motion trajectory, which can effectively ensure the synchronization of the welcoming interaction. Then, through the strategy generation mechanism that integrates the motion trajectory, visual feature data and environmental data, the welcoming strategy is dynamically determined according to the user's real-time behavior and environmental conditions, and differentiated welcoming services are performed for different users, thereby improving the fit between the welcoming service and the user's actual needs, and providing users with a personalized and intelligent welcoming interaction experience, thereby improving user experience and service quality.

[0135] As a possible implementation of the present application, user feedback data on the welcome strategy is recorded; based on the feedback data, the welcome strategy is adaptively learned and optimized. The adaptive learning optimization includes optimizing the matching logic between the behavior pattern and the welcome function.

[0136] Execution feedback data includes user operation behavior and feedback. The adaptive learning module continuously collects user operation behavior and user feedback on the type and triggering timing of the target welcome function (which can be determined through satisfaction surveys or usage records), automatically adjusting and optimizing the welcome strategy. For example, after the user gets on the vehicle, the system continuously collects user operation behavior and collects feedback on the user's satisfaction with the welcome service and the reasons for dissatisfaction through the in-vehicle interactive interface. Based on the operation behavior and feedback, the system further optimizes the welcome function, such as the timing and width of the door opening.

[0137] In one possible implementation, adaptive learning can use a reinforcement learning algorithm to continuously optimize the welcome strategy by defining states (user location, speed, environmental parameters), actions (different welcome strategies), and reward functions (such as strategy accuracy, safety, and user satisfaction).

[0138] For example, the user's feedback data on the execution of the welcome strategy is recorded. When it is detected that the number of times the user manually corrects the door opening angle exceeds a preset frequency, the welcome strategy is adjusted through a reinforcement learning algorithm so that the door opening angle in the welcome strategy tends to be consistent with the user's preference.

[0139] In the embodiment of the present application, by collecting user operation behavior and user feedback to dynamically adjust strategy parameters, relying on manual experience to adjust the strategy can be avoided, which can further realize the personalization of the welcoming service and ensure the correctness of the optimization direction.

[0140] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0141] Corresponding to the vehicle welcoming method described in the above embodiment, Figure 6 A structural block diagram of a vehicle welcoming device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0142] Reference Figure 6 The vehicle welcoming device includes: a data sensing unit 61, a trajectory prediction unit 62, a strategy determination unit 63, and a welcoming control unit 64, wherein:

[0143] A data sensing unit 61 is configured to acquire multimodal sensing data in real time, wherein the multimodal sensing data includes environmental data of the vehicle's welcome area, basic motion data of the user, and visual feature data;

[0144] a trajectory prediction unit 62 for predicting the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained network model based on deep learning;

[0145] a strategy determination unit 63, configured to determine a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data;

[0146] The welcome control unit 64 is used to control the vehicle to execute the welcome strategy and provide welcome service.

[0147] As a possible implementation of the present application, the basic motion data includes the user's real-time position, motion speed, and motion acceleration; the visual feature data includes gait features and facial features; the environmental data includes light intensity, ambient temperature and humidity, precipitation status, and obstacle distribution; the data sensing unit 61 specifically includes:

[0148] A basic motion data acquisition module is used to obtain the user's real-time position, motion speed and motion acceleration through the fusion of the ultra-wideband module and radar;

[0149] A visual feature acquisition module, configured to acquire the user's gait features and facial features through an intelligent visual device;

[0150] The environmental data acquisition module is used to obtain the light intensity, ambient temperature and humidity, precipitation status and obstacle distribution of the welcome area through environmental sensors.

[0151] As a possible implementation of the present application, the data sensing unit 61 is specifically configured to:

[0152] The user position provided by the ultra-wideband module and the user speed and user acceleration provided by the radar are obtained in real time; the user position, the user speed and the user acceleration are fused using a preset fusion algorithm to output the optimized real-time position, movement speed and movement acceleration.

[0153] As a possible implementation of the present application, the trajectory prediction unit 62 is specifically configured to:

[0154] Inputting the basic motion data and the cached historical motion parameter sequence into a trajectory prediction model;

[0155] The trajectory prediction model extracts the time-dependent characteristics of the user's motion based on the basic motion data and the historical motion parameter sequence, and based on the time-dependent characteristics, outputs the predicted position coordinates, velocity vector and acceleration vector in the future time window to determine the user's motion trajectory.

[0156] As a possible implementation of the present application, the vehicle welcoming device further includes:

[0157] An identity verification unit, configured to verify the identity of the user based on the visual feature data and determine whether the user is an authorized user;

[0158] a security warning unit, configured to limit a car door unlocking function and activate a security warning module if not;

[0159] The policy determination unit 63 is further configured to determine a welcoming policy according to the motion trajectory, the visual feature data, and the environment data if yes.

[0160] As a possible implementation form of the present application, the policy determination unit 63 comprises:

[0161] a behavior pattern determination module, configured to determine a behavior pattern of the user according to the motion trajectory and the visual feature data;

[0162] a welcoming function determination module, configured to determine a target welcoming function according to the behavior pattern and the environment data;

[0163] a welcoming policy determination module, configured to determine a welcoming policy according to the motion trajectory, the target welcoming function, and the environment data.

[0164] As a possible implementation form of the present application, the motion trajectory comprises a path curvature and an acceleration vector, and the visual feature data comprises gait features and human posture features; the behavior pattern determination module is specifically configured to:

[0165] determine an instantaneous speed change rate and an acceleration direction according to the acceleration vector;

[0166] determine the behavior pattern of the user according to the path curvature, the instantaneous speed change rate, the acceleration direction, the gait features, and the human posture features.

[0167] As a possible implementation form of the present application, the vehicle welcoming device further comprises:

[0168] a feedback recording unit, configured to record execution feedback data of a user on a welcoming policy;

[0169] a policy optimization unit, configured to perform adaptive learning optimization on the welcoming policy according to the execution feedback data.

[0170] From the above, in the embodiments of the present application, by acquiring the environment data, the basic motion data and the visual feature data in real time, the all-around perception ability of the user state and the surrounding environment is established, the data foundation for the accurate service is provided, the motion trajectory of the user is accurately predicted based on the trajectory prediction model of the deep learning model, the welcome interaction can be effectively guaranteed, then the strategy generation mechanism of the motion trajectory, the visual feature data and the environment data is fused, the welcome strategy is dynamically determined according to the real-time behavior of the user and the environment condition, the differentiated welcome service is executed for different users, the fitting degree of the welcome service and the actual demand of the user is improved, the personalized and intelligent welcome interaction experience is provided for the user, and thus the user experience and the service quality are improved.

[0171] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and the specific functions and the brought technical effects can be referred to the method embodiments part, which will not be described here.

[0172] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the vehicle welcome method as Figures 1 to 5 indicated.

[0173] The embodiments of the present application further provide an electronic device, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the steps of any one of the vehicle welcome method as Figures 1 to 5 indicated when executing the computer program.

[0174] The embodiments of the present application further provide a computer program product, which, when running on an electronic device, causes the electronic device to implement the steps of any one of the vehicle welcome method as Figures 1 to 5 indicated.

[0175] Figure 7 is a schematic diagram of an electronic device provided by an embodiment of the present application. As shown in Figure 7 the electronic device 7 of this embodiment includes a processor 70, a memory 71 and a computer program 72 stored in the memory 71 and executable on the processor 70. The processor 70 implements the steps in the above vehicle welcome method embodiments when executing the computer program 72, such as the steps S101 to S104 as shown in Figure 1 . Alternatively, the processor 70 implements the functions of the modules / units in the above apparatus embodiments when executing the computer program 72, such as the functions of the units 61 to 64 as shown in Figure 6 .

[0176] Exemplarily, the computer program 72 may be divided into one or more modules / units, which are stored in the memory 71 and executed by the processor 70 to implement the present application. The one or more modules / units may be a series of computer-readable instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program 72 in the electronic device 7.

[0177] The electronic device 7 may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that Figure 7 It is only an example of the electronic device 7 and does not constitute a limitation of the electronic device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device 7 may also include input and output devices, network access devices, buses, etc.

[0178] The processor 70 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0179] The memory 71 may be an internal storage unit of the electronic device 7, such as a hard disk or memory of the electronic device 7. The memory 71 may also be an external storage device of the electronic device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 7. Furthermore, the memory 71 may include both an internal storage unit of the electronic device 7 and an external storage device. The memory 71 is used to store the computer program and other programs and data required by the electronic device. The memory 71 may also be used to temporarily store data that has been output or is about to be output.

[0180] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0181] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0182] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable medium cannot be an electric carrier signal or a telecommunication signal.

[0183] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0184] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A vehicle welcoming method, characterized in that: include: Acquire multimodal perception data in real time, the multimodal perception data including environmental data of the vehicle's welcome area, basic motion data of the user, and visual feature data; Predicting the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained, deep learning-based network model; determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data; The vehicle is controlled to execute the welcoming strategy to provide welcoming service.

2. The method according to claim 1, characterized in that The basic motion data includes the user's real-time position, motion speed, and motion acceleration; The step of acquiring multimodal perception data in real time includes: Real-time acquisition of user location provided by the ultra-wideband module and user speed and acceleration provided by the radar; The user position, user speed and user acceleration are fused using a preset fusion algorithm to output the optimized real-time position, the movement speed and the movement acceleration.

3. The method according to claim 1, characterized in that The step of predicting the user's motion trajectory based on the basic motion data and the trajectory prediction model includes: Inputting the basic motion data and the cached historical motion parameter sequence into a trajectory prediction model; The trajectory prediction model extracts the time-dependent characteristics of the user's motion based on the basic motion data and the historical motion parameter sequence, and based on the time-dependent characteristics, outputs the predicted position coordinates, velocity vector and acceleration vector in the future time window to determine the user's motion trajectory.

4. The method according to claim 1, wherein Before the step of determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data, the method further includes: authenticating the user based on the visual feature data to determine whether the user is an authorized greeter user; If not, the door unlocking function is restricted and the security warning module is activated; If so, a welcoming strategy is determined based on the motion trajectory, the visual feature data and the environmental data.

5. The method according to claim 1, wherein The step of determining a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data includes: determining a behavior pattern of the user based on the motion trajectory and the visual feature data; determining a target welcoming function based on the behavior pattern and the environmental data; A welcoming strategy is determined according to the motion trajectory, the target welcoming function, and the environmental data.

6. The method according to claim 5, characterized in that The motion trajectory includes path curvature and acceleration vector, and the visual feature data includes gait features and human posture features; The step of determining the user's behavior pattern based on the motion trajectory and the visual feature data includes: Determining the instantaneous velocity change rate and the acceleration direction according to the acceleration vector; The user's behavior pattern is determined according to the path curvature, the instantaneous speed change rate, the acceleration direction, the gait characteristics, and the human body posture characteristics.

7. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Record user feedback data on the implementation of the welcome strategy; Adaptively learn and optimize the welcoming strategy based on the execution feedback data.

8. A vehicle welcoming device, characterized in that: include: A data perception unit, configured to acquire multimodal perception data in real time, wherein the multimodal perception data includes environmental data of the vehicle's welcome area, basic motion data of the user, and visual feature data; a trajectory prediction unit, configured to predict the user's motion trajectory based on the basic motion data and a trajectory prediction model, wherein the trajectory prediction model is a pre-trained network model based on deep learning; a strategy determination unit, configured to determine a welcoming strategy based on the motion trajectory, the visual feature data, and the environmental data; A welcome control unit is used to control the vehicle to execute the welcome strategy and provide welcome service.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the vehicle welcoming method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the vehicle welcoming method according to any one of claims 1 to 7 is implemented.

Images