Method and device for identifying getting-off behavior and vehicle-mounted terminal

Abstract

The invention provides a getting-off behavior recognition method and device and a vehicle-mounted terminal. The method comprises the steps that to-be-recognized behavior images of continuous frames are extracted; inputting the to-be-recognized behavior image into a human shape region recognition model which is trained in advance or in real time, and extracting a human shape region in the to-be-recognized behavior image; whether the target user has a preliminary getting-off behavior is calculated through the multiple human shape areas, if it is determined that the target user has the preliminary getting-off behavior, the final getting-off result of the target user is determined in combination with the image data outside the vehicle, and the vehicle door state information is combined with image processing; according to the method, whether the passenger has the getting-off action or not and whether the getting-off action is completed or not can be identified more accurately, so that vehicle starting caused by misoperation of a driver is avoided, the safety is improved, and the use experience of a user is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of vehicle networking, and in particular to a method and device for identifying getting off a vehicle, and a vehicle-mounted terminal. Background Art

[0002] With the development of the automobile industry, the safety of vehicles has attracted more and more attention. In the process of parking the vehicle and getting off the passengers, there is a problem that the driver is not paying attention and starts the vehicle, causing personal injury to the passengers getting off the vehicle.

[0003] The existing solutions mainly detect the status of seats and doors by installing various sensors on the vehicle, such as seat pressure sensors, door status sensors, etc. However, there are still some problems with the existing technology in practical applications. First, the existing sensor technology is not accurate enough and may cause misjudgment. For example, objects on the seat may be misjudged as passengers, resulting in the vehicle starting incorrectly. Second, the existing technical solutions can often only detect the status of seats and doors, but cannot detect the real status of passengers, such as whether the passengers have completely got off the vehicle. Summary of the invention

[0004] The present application provides a method, device and vehicle-mounted terminal for identifying getting off a vehicle to solve the above-mentioned technical problems.

[0005] An embodiment of the present application provides a method for identifying getting-off behavior, the method comprising: acquiring a real-time video stream inside the vehicle, and extracting a plurality of continuous frame images from the real-time video stream inside the vehicle as behavior images to be identified; inputting the behavior images to be identified into a human shape region recognition model that has been trained in advance or in real time, and extracting the human shape region in the behavior images to be identified; identifying whether a target user has a preliminary getting-off behavior through the human shape region, and when the target user has a preliminary getting-off behavior, determining the final getting-off result of the target user in combination with image data outside the vehicle.

[0006] In one embodiment of the present application, identifying whether the target user has a preliminary behavior of getting off the vehicle is performed through the human-shaped area, including: calculating the horizontal optical flow vector and the vertical optical flow vector of each pixel point in the human-shaped area; if the absolute value of the horizontal optical flow vector is greater than the preset optical flow value, or the absolute value of the vertical optical flow vector is greater than the preset optical flow value, then the horizontal optical flow vector greater than the preset optical flow value is recorded as a preliminary screening horizontal optical flow vector, and the vertical optical flow vector greater than the preset optical flow value is recorded as a preliminary screening vertical optical flow vector; calculating the covariance matrix composed of the preliminary screening horizontal optical flow vector and the preliminary screening vertical optical flow vector, and performing eigenvalue decomposition on the covariance matrix to obtain an eigenvector, the eigenvector represents the direction of the optical flow vector field, and the optical flow vector field is a two-dimensional vector field that describes the movement of pixel points in the human-shaped area over time; if the direction of the eigenvector is consistent with the door opening direction, it is determined that the target user has a preliminary behavior of getting off the vehicle.

[0007] In one embodiment of the present application, the final getting-off result of the target user is determined based on the external image data, including: extracting a human-shaped area from the external image data, recorded as a tracking human-shaped area; if a complete human portrait is identified through the tracking human-shaped area, and it is determined that the tracking user in the tracking pedestrian area and the target user in the human-shaped area are the same user, then the final getting-off result of the target user is determined to be completed getting off; if a partial human portrait is identified through the tracking human-shaped area, and a complete human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be getting off and returning; if a partial human portrait is identified through the tracking human-shaped area, and a partial human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be getting off and returning.

[0008] In one embodiment of the present application, the process of training and generating the human shape area recognition model includes: obtaining input data; forward propagating the input data through the backbone network in the initial human shape area recognition model, calculating the output of each network layer, and obtaining the final prediction result; calculating the difference between the prediction result and the true label through a binary cross entropy loss function to obtain a loss value, wherein the true label represents the correct result corresponding to the input data; calculating the gradient according to the loss function, and back-propagating it to the weights and bias values ​​in the network layer to reduce the loss function value; iteratively training the initial human shape area recognition model until the loss function value stabilizes within a preset function value range to obtain a human shape area recognition model.

[0009] In one embodiment of the present application, the process of obtaining the input data includes: obtaining sample in-vehicle image data, and annotating the in-vehicle image data with portrait features to obtain a portrait feature image and a non-portrait feature image; adjusting the portrait feature image and the non-portrait feature image to a preset size, and normalizing the pixel values ​​of the resized portrait feature image and the pixel values ​​of the non-portrait feature image to a preset pixel value range to obtain a preprocessed portrait feature image and a preprocessed non-portrait feature image, and using the preprocessed portrait feature image and the preprocessed non-portrait feature image as input data for the initial human area recognition model.

[0010] In one embodiment of the present application, after completing the identification of the target user's getting off the vehicle, it also includes: if it is determined that the target user is getting off the vehicle, sending a warning message to the in-vehicle display device; when it is detected that the current gear position of the vehicle is not in the parking gear position, obtaining throttle braking information; if it is determined that the throttle braking information is the first throttle braking, limiting this throttle braking; if it is determined that the throttle braking information is not the first throttle braking, responding to the throttle braking.

[0011] In one embodiment of the present application, obtaining a real-time video stream inside the vehicle includes: after the vehicle stops, detecting the switch state of the door; if the switch state of any door is detected to be open, detecting the pressure signal of each seat through the pressure sensor on the seat in the vehicle; if the pressure signal is less than a preset pressure threshold, collecting the real-time video stream inside the vehicle through the in-vehicle imaging device.

[0012] In one embodiment of the present application, multiple behavior images to be identified are extracted from the real-time in-vehicle video stream, including: extracting multiple continuous frames of original in-vehicle behavior images from the real-time in-vehicle video stream; performing data enhancement on the original in-vehicle behavior images, and performing mean filtering on the data-enhanced in-vehicle behavior images to obtain preprocessed in-vehicle behavior images; the data enhancement includes: contrast enhancement and clarity enhancement; normalizing the pixel values ​​of the preprocessed in-vehicle behavior images to a preset pixel value range to obtain the behavior images to be identified.

[0013] The embodiment of the present application provides a device for identifying getting off the vehicle behavior, and the device includes: an image acquisition module, which is used to obtain a real-time video stream inside the vehicle, and extract multiple continuous frame images from the real-time video stream inside the vehicle as the behavior image to be identified; a human shape area recognition module, which is used to input the behavior image to be identified into a pre- or real-time trained human shape area recognition model, and extract the human shape area in the behavior image to be identified; an image processing module, which is used to identify whether the target user has a preliminary getting off the vehicle behavior through the human shape area, and when the target user has a preliminary getting off the vehicle behavior, determine the final getting off result of the target user in combination with the image data outside the vehicle.

[0014] An embodiment of the present application provides a vehicle-mounted terminal, including a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute a computer program stored in the memory to implement the above-mentioned getting-off behavior identification method.

[0015] The beneficial effects of the present application are as follows: the present invention extracts the behavior images to be identified of continuous frames; and inputs the behavior images to be identified into a pre- or real-time trained human shape region recognition model to extract the human shape region in the behavior images to be identified; calculates whether the target user has a preliminary getting-off behavior through multiple human shape regions, and if it is determined that the target user has a preliminary getting-off behavior, determines the final getting-off result of the target user in combination with the external image data; and more accurately identifies whether the passenger has the action of getting off the vehicle and whether the getting-off action is completed by combining the door status information with image processing, thereby avoiding the vehicle starting due to the driver's misoperation and improving safety. Compared with the prior art that can only detect the status of seats and doors, the present invention can more accurately detect the passenger's getting-off behavior through multiple parameters, further improving safety and enhancing the user experience.

[0016] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0018] Figure 1 is a flow chart of a method for identifying a vehicle getting off behavior shown in an exemplary embodiment of the present application;

[0019] Figure 2 is a structural block diagram of a vehicle getting off behavior recognition device shown in an exemplary embodiment of the present application;

[0020] Figure 3 It is a schematic diagram of the hardware structure of a vehicle-mounted terminal shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will describe the implementation methods of the present application with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, not for limiting the scope of protection of the present application.

[0022] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0023] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application. However, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present application difficult to understand.

[0024] The embodiments of the present application respectively propose a method for identifying getting off a vehicle, a device for identifying getting off a vehicle, and a vehicle-mounted terminal, and these embodiments will be described in detail below.

[0025] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a method for identifying a vehicle getting off a vehicle according to an embodiment of the present application. Figure 1 As shown, the method at least includes steps S110 to S130, which are described in detail as follows:

[0026] Step S110, obtaining a real-time video stream in the vehicle, and extracting a plurality of continuous frame images from the real-time video stream in the vehicle as behavior images to be identified.

[0027] In one embodiment of the present application, obtaining a real-time video stream inside a vehicle includes: after the vehicle stops, detecting the switch state of the vehicle doors; if the switch state of any door is detected to be open, detecting the pressure signal of each seat through a pressure sensor on the seat in the vehicle; if the pressure signal is less than a preset pressure threshold, collecting the real-time video stream inside the vehicle through an in-vehicle imaging device.

[0028] As an example, the real-time video stream inside the car is collected by the in-car imaging device. The real-time video stream inside the car can be obtained through the camera equipment arranged inside the vehicle. Specifically, the camera equipment inside the car includes a front camera and a rear camera. The front camera is used to capture the driver's facial expressions and behaviors, and the rear camera is used to monitor the facial expressions and behaviors of the rear passengers. The real-time video stream inside the car is obtained through the front camera and the rear camera, and multiple behavior images to be identified are extracted from the real-time video stream inside the car.

[0029] In one embodiment of the present application, a plurality of behavior images to be identified are extracted from a real-time in-vehicle video stream, including: extracting a plurality of original in-vehicle behavior images of continuous frames from a real-time in-vehicle video stream; performing data enhancement on the original in-vehicle behavior images, and performing mean filtering on the in-vehicle behavior images after data enhancement to obtain pre-processed in-vehicle behavior images; data enhancement includes: contrast enhancement and clarity enhancement; normalizing the pixel values ​​of the pre-processed in-vehicle behavior images to a preset pixel value range to obtain the behavior images to be identified. Through data processing, the behavior images to be identified finally obtained have higher quality and standardization, which provides better input for subsequent behavior recognition algorithms and can improve the accuracy and efficiency of behavior recognition. For example, in the identification of passenger getting off the vehicle, clear, noise-free and standardized images can enable the algorithm to more accurately judge the characteristics of the passenger's movement direction, speed, etc., so as to more reliably determine whether the passenger has got off the vehicle. At the same time, high-quality images can also reduce the calculation time of the algorithm and improve the detection accuracy and detection speed.

[0030] Step S120 , inputting the behavior image to be identified into a human shape region recognition model that has been trained in advance or in real time, and extracting the human shape region in the behavior image to be identified.

[0031] In one embodiment of the present application, the process of training and generating a humanoid region recognition model includes: obtaining input data; forward propagating the input data through the backbone network in the initial humanoid region recognition model, calculating the output of each network layer, and obtaining the final prediction result; calculating the difference between the prediction result and the true label through the binary cross entropy loss function to obtain the loss value, and the true label represents the correct result corresponding to the input data; calculating the gradient according to the loss function, and back-propagating to the weights and bias values ​​in the network layer to reduce the loss function value; iteratively training the initial humanoid region recognition model until the loss function value stabilizes in the preset function value range to obtain the humanoid region recognition model. The humanoid region recognition model can automatically extract features related to the human body shape from the input data, calculate the output of each network layer, and finally obtain the prediction result, thereby improving the accuracy of humanoid region recognition, and calculating the difference between the prediction result and the true label through the binary cross entropy loss function, which can clearly measure the gap between the humanoid region predicted by the model and the actual humanoid region. And the model can automatically adjust the weights and bias values ​​in the network layer so that the model can better fit the training data in the next iteration and improve the accuracy of humanoid region recognition. The human shape area recognition model can achieve a stable performance state after multiple trainings, providing reliable support for subsequent applications.

[0032] As an example, the humanoid region recognition model is obtained based on the training of the CNN model. The CNN model (Convolutional Neural Network) is a deep learning model specifically used to process data with a similar grid structure. It simulates the hierarchical structure of the human visual system and performs layer-by-layer abstraction and feature extraction on the input data. It is widely used in image recognition, video analysis, natural language processing and other fields. The embodiment of the present application trains the CNN model based on the CNN model with a large amount of sample image data to obtain a humanoid region recognition model. The humanoid region recognition model can identify the humanoid region of the behavior image to be identified, thereby improving the accuracy of humanoid detection.

[0033] As an example, if a certain frame of the behavior image to be recognized does not recognize a human shape area, the next frame will continue to be recognized until the behavior image to be recognized is completely recognized. If a human shape area is not recognized in the behavior image to be recognized, the real-time video stream inside the vehicle is obtained, and multiple continuous frame images are extracted from the real-time video stream inside the vehicle as the behavior image to be recognized until a human shape area is recognized.

[0034] In one embodiment of the present application, obtaining input data includes: obtaining sample in-vehicle image data, and annotating the in-vehicle image data with portrait features to obtain a portrait feature image and a non-portrait feature image; adjusting the portrait feature image and the non-portrait feature image to a preset size, and normalizing the pixel values ​​of the resized portrait feature image and the pixel values ​​of the non-portrait feature image to a preset pixel value range to obtain a preprocessed portrait feature image and a preprocessed non-portrait feature image, and using the preprocessed portrait feature image and the preprocessed non-portrait feature image as input data for an initial human area recognition model.

[0035] As an example, sample in-car image data is obtained, and portrait feature annotation is performed on the in-car image data to obtain portrait feature images and non-portrait feature images, and to ensure data diversity, different lighting conditions, human body postures, etc. are included. All portrait feature images and non-portrait feature images are adjusted to the uniform size required for network input, and the pixel values ​​of portrait feature images and non-portrait feature images are normalized to between 0 and 1, as input data for the initial human shape area recognition model. As an implementation method, the initial human shape area recognition model is a CNN model, which is usually composed of an input layer, a convolution layer, an activation function, a pooling layer, a fully connected layer, and an output layer, wherein the input layer is used to receive preprocessed image data, and the convolution layer, that is, using a convolution kernel to extract image features, can be specifically:

[0036]

[0037] Where I is the input image, F is the output of the convolution layer, K is the convolution kernel, i, j represent the position in the output image, and m, n represent the position in the convolution kernel. The pooling layer is used to reduce the feature dimension by using the maximum pooling. The specific implementation methods include:

[0038] P(i,j)=max m,n I(i·sm,j·sn)

[0039] Where P is the output of the pooling layer and s is the stride. The fully connected layer is used to convert the feature maps extracted by the convolutional layer and the pooling layer into the final output:

[0040] O=f(W·P+b),

[0041] Where W is the weight matrix, b is the bias term, f is the activation function, and P is the output of the pooling layer. The convolution layer extracts local features from the input data and generates a feature map. The activation function introduces nonlinear factors to enhance the expressiveness of the model. The pooling layer reduces the spatial dimension of the feature map and reduces the amount of calculation. The fully connected layer synthesizes the previously extracted features and generates the final classification or regression results through the output layer. As an implementation method: select Adam optimizer for training and configure its parameters, including learning rate (0.001), first-order moment decay rate (0.9), second-order moment decay rate (0.999), etc. First, the input data is forward propagated through the network, and the output of each layer is calculated until the final prediction result is obtained. Then, the binary cross entropy loss function is used to calculate the difference between the prediction result and the true label to obtain the loss value. Then, the gradient is calculated according to the loss function, and back-propagated to each parameter in the network to update the weight and bias values. Then, the Adam optimizer is used to iteratively train the model, and the weight and bias parameters are continuously adjusted to reduce the loss function value. Finally, iterative training is performed: the process of forward propagation, loss calculation, back-propagation and parameter update is repeated. When the loss function value no longer decreases significantly, the training is stopped to obtain a trained CNN model, namely, a humanoid area recognition model. The input data is forward propagated through the backbone network in the initial humanoid region recognition model, which can automatically extract features related to human body shape from the input data, calculate the output of each network layer, and finally obtain the prediction result. In the process, the accuracy of humanoid region recognition is improved, and the difference between the predicted result and the true label is calculated through the binary cross entropy loss function to obtain the loss value. The binary cross entropy loss function is suitable for binary classification problems. In humanoid region recognition, the gap between the humanoid region predicted by the model and the actual humanoid region can be clearly measured. The smaller the loss value, the closer the model's prediction result is to the true label, and the better the model's performance. Through the back propagation algorithm, the model can automatically adjust the weights and bias values ​​in the network layer, so that the model can better fit the training data in the next iteration and improve the accuracy of humanoid region recognition. Iteratively train the initial humanoid region recognition model until the loss function value stabilizes in the preset function value range, ensuring that the model can reach a stable performance state after multiple trainings, providing reliable support for subsequent applications.

[0042] Step S130, identifying whether the target user has a preliminary getting-off behavior through the human-shaped area, and when the target user has a preliminary getting-off behavior, determining the final getting-off result of the target user in combination with the vehicle exterior image data.

[0043] In one embodiment of the present application, identifying whether a target user has a preliminary behavior of getting off the vehicle is performed through a humanoid area, including: calculating a horizontal optical flow vector and a vertical optical flow vector for each pixel in the humanoid area; if the absolute value of the horizontal optical flow vector is greater than a preset optical flow value, or the absolute value of the vertical optical flow vector is greater than the preset optical flow value, then the horizontal optical flow vector greater than the preset optical flow value is recorded as a preliminary screening horizontal optical flow vector, and the vertical optical flow vector greater than the preset optical flow value is recorded as a preliminary screening vertical optical flow vector; calculating a covariance matrix composed of the preliminary screening horizontal optical flow vector and the preliminary screening vertical optical flow vector, and performing eigenvalue decomposition on the covariance matrix to obtain an eigenvector, the eigenvector represents the direction of the optical flow vector field, and the optical flow vector field is a two-dimensional vector field that describes the movement of pixels in the humanoid area over time; if the direction of the eigenvector is consistent with the door opening direction, it is determined that the target user has a preliminary behavior of getting off the vehicle. By calculating the horizontal and vertical optical flow vectors of each pixel in the humanoid area, the movement changes of the pixels in the humanoid area in the time dimension can be captured in detail. The optical flow vector can reflect the moving speed and direction of the object. By calculating the horizontal and vertical directions separately, the movement state of the object in the humanoid area can be fully understood, and the preset optical flow value is used as the standard for screening, which can exclude those tiny optical flow vector changes that may be caused by noise or irrelevant movement. Only the horizontal and vertical optical flow vectors with absolute values ​​greater than the preset optical flow value are paid attention to, and then the pixels with significant movement are more accurately focused on, which improves the pertinence of the judgment of the target user's behavior. The covariance matrix can reflect the correlation of the optical flow vector in different directions, and the eigenvector represents the main direction of the optical flow vector field. Through the covariance matrix, the movement trend of the pixels in the humanoid area can be grasped as a whole, not just the movement of a single pixel. As a directional description of the optical flow vector field, the eigenvector can more accurately determine whether the target user has got off the bus.

[0044] As an example, whether the target user has a preliminary getting-off behavior is calculated through continuous human-shaped areas. Specifically, the following algorithm is used to calculate whether the target user has a preliminary getting-off behavior:

[0045] I(x,y,t)·(1+V x )+V y I(x,y,t)+I(x,y,t)=0

[0046] Where I(x,y,t) is the image brightness at position (x,y) and time t, and V x is the horizontal component of the optical flow vector, V y is the longitudinal component of the optical flow vector, I x is the spatial derivative of brightness, I t are the time derivatives of brightness, respectively;

[0047] In the above embodiment, if the preset optical flow value is θ, when |V x |>θ|||V y |>θ, it is determined that the target user has moved. For example, θ is 10 frames, when |V x |>10 frames or |V y |>10 frames, it is considered that the passenger has moved. After determining that the passenger has moved, the covariance matrix of the optical flow vectors of all pixels in the humanoid area is calculated, and the covariance matrix is ​​eigen-decomposed to obtain the eigenvalues ​​and corresponding eigenvectors. If within the preset time period, the angle between the main eigenvector and the door opening direction is calculated, for example, the main eigenvector is the eigenvector corresponding to the maximum eigenvalue. If the angle between the main eigenvector and the door opening direction is less than the preset angle threshold (such as 30 degrees) and lasts for a certain time (such as 2 seconds), it is judged that the passenger has got off the bus. As a specific example, if the angle between the main eigenvector and the door opening direction is less than 30 degrees) and lasts for 2 seconds, it is judged that the passenger has got off the bus. By determining the direction of the door opening through sensors or preset information and comparing it with the main eigenvector, it is possible to accurately judge whether the passenger has gotten off the bus. The judgment method based on the vector angle and duration can effectively filter out misjudgments and improve the accuracy of judgment.

[0048] In one embodiment of the present application, the final getting-off result of the target user is determined based on the image data outside the vehicle, including: extracting the human-shaped area of ​​the image data outside the vehicle, recorded as the tracking human-shaped area; if a complete human portrait is identified by tracking the human-shaped area, and it is determined that the tracking user in the tracking pedestrian area and the target user in the human-shaped area are the same user, then the final getting-off result of the target user is determined to be the getting-off completion; if a partial human portrait is identified by tracking the human-shaped area, and the complete human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be the getting-off turnaround; if a partial human portrait is identified by tracking the human-shaped area, and the partial human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be the getting-off. Different getting-off situations correspond to different judgment criteria, which can more accurately determine the passenger's getting-off status, and can adapt to various different scenarios. Whether the passenger gets off quickly, gets off slowly, or returns to the vehicle midway, accurate judgments can be made, and corresponding safety measures can be taken according to the accurate judgment of the getting-off status, thereby improving the user experience.

[0049] As an example, the target user is a rear passenger. When the rear passenger is detected to have a preliminary get-off behavior, the human-shaped area in the rear passenger's exterior image data is extracted and recorded as the tracked human-shaped area. It is determined that the final get-off result of the rear passenger is as follows:

[0050] Case 1: When the pressure signal value of the rear seat is detected to be greater than the preset pressure value, the rear passenger human shape area on the seat is extracted. When it is determined that the rear passenger has a preliminary get-off behavior, if the rear passenger is not recognized in the car at this time, and a complete human portrait is recognized in the tracking human shape area, it is determined whether the complete human portrait outside the car is the same person as the rear passenger. As an implementation method, the Dlib library can be used to extract the facial feature descriptors of the two faces and calculate the Euclidean distance of the two descriptors. When the Euclidean distance is less than the set distance threshold, it indicates that they are the same person and it is considered that the rear passenger has gotten off the car.

[0051] Case 2: When the pressure signal value of the rear seat is detected to be greater than the preset pressure value, the rear passenger human shape area on the seat is extracted. When it is determined that the rear passenger has a preliminary behavior of getting off the vehicle, but a partial human image is recognized in the tracking human shape area, it is considered that the rear passenger is getting off and turning back;

[0052] Case 3: When the pressure signal value of the rear seat is detected to be greater than the preset pressure value, the rear passenger human area on the seat is extracted. When it is determined that the rear passenger has a preliminary getting-off behavior, if part of the human image is recognized in the tracking human area and part of the human image is recognized in the behavior image to be recognized, it is determined that the target user is getting off. When it is determined that the rear passenger is getting off, at this time, in order to prevent the passenger from being injured, a large screen reminder "passenger is getting off" is used. If the vehicle is not in P gear, VDC does not respond to the driver's first accelerator action, but does not restrict subsequent accelerator actions. Different getting-off situations correspond to different judgment criteria, which can more accurately determine the passenger's getting-off status and can adapt to various scenarios. Whether the passenger gets off quickly, gets off slowly, or returns to the car midway, accurate judgment can be made, and corresponding safety measures can be taken based on accurate judgment of the getting-off status, thereby improving the user experience.

[0053] As an example, the outside vehicle image data is collected by an outside vehicle camera device. Specifically, the outside vehicle camera device includes a door camera and a panoramic camera. The outside vehicle camera device is used to record the passengers getting on and off the vehicle, and the panoramic camera is used to record the overall situation of the passengers outside the vehicle. The outside vehicle image data is collected by the door camera and the panoramic camera, and the final getting off result of the target user is determined based on the outside vehicle image data to complete the identification of the target user's getting off behavior.

[0054] In one embodiment of the present application, after the target user's getting off behavior is identified, it also includes: if it is determined that the target user is getting off the vehicle, a warning message is sent to the in-vehicle display device; when it is detected that the current gear of the vehicle is not in the parking gear, the throttle brake information is obtained; if it is determined that the throttle brake information is the first throttle brake, the throttle brake is limited, and if it is determined that the throttle brake information is not the first throttle brake, the throttle brake is responded. By disabling the throttle brake, the vehicle is prevented from starting due to the driver's misoperation, and the safety of passengers is guaranteed.

[0055] exist Figure 1 In the technical scheme shown, continuous frames of images of behaviors to be identified are extracted; and the images of behaviors to be identified are input into a pre- or real-time trained humanoid region recognition model to extract the humanoid region in the images of behaviors to be identified; multiple humanoid regions are used to calculate whether the target user has a preliminary getting-off behavior, and when the target user has a preliminary getting-off behavior, the final getting-off result of the target user is determined in combination with the external image data; by combining the seat pressure signal and door status information with image processing, it is more accurately identified whether the passenger has the action of getting off the vehicle and whether the getting-off action is completed, thereby avoiding the vehicle starting due to driver's misoperation and improving safety; and compared with the prior art that can only detect the status of seats and doors, the present invention can more accurately detect the passenger's getting-off behavior through multiple parameters, further improving safety and enhancing the user experience.

[0056] Figure 2 FIG. 1 shows a block diagram of a vehicle getting off behavior recognition device according to an exemplary embodiment of the present application. Figure 2 As shown, according to an embodiment of the present application, the device 200 for identifying the getting-off behavior includes: an image acquisition module 210, a humanoid region recognition module 220, and an image processing module 230. The image acquisition module is used to obtain the real-time video stream in the vehicle, and extract multiple continuous frame images from the real-time video stream in the vehicle as the behavior image to be identified; the humanoid region recognition module is used to input the behavior image to be identified into the humanoid region recognition model trained in advance or in real time, and extract the humanoid region in the behavior image to be identified; the image processing module is used to identify whether the target user has a preliminary getting-off behavior through the humanoid region, and when the target user has a preliminary getting-off behavior, determine the final getting-off result of the target user in combination with the image data outside the vehicle. By combining the pressure signal of the seat, the door status information and the image processing, it is more accurately identified whether the passenger has the action of getting off the vehicle, and whether the getting-off action is completed, thereby avoiding the vehicle starting caused by the driver's misoperation, improving safety, and compared with the prior art that can only detect the status of the seat and the door, the present invention can more accurately detect the passenger's getting-off behavior through multiple parameters, further improving safety and enhancing the user's experience.

[0057] In one embodiment of the present application, the getting-off behavior recognition device also includes a calculation module 240, which is used to calculate the horizontal optical flow vector and the vertical optical flow vector of each pixel point in the human-shaped area; if the absolute value of the horizontal optical flow vector is greater than the preset optical flow value, or the absolute value of the vertical optical flow vector is greater than the preset optical flow value, then the horizontal optical flow vector greater than the preset optical flow value is recorded as the preliminary screening horizontal optical flow vector, and the vertical optical flow vector greater than the preset optical flow value is recorded as the preliminary screening vertical optical flow vector; calculate the covariance matrix composed of the preliminary screening horizontal optical flow vector and the preliminary screening vertical optical flow vector, and perform eigenvalue decomposition on the covariance matrix to obtain an eigenvector, the eigenvector represents the direction of the optical flow vector field, and the optical flow vector field is a two-dimensional vector field that describes the movement of pixel points in the human-shaped area over time; if the direction of the eigenvector is consistent with the door opening direction, it is determined that the target user has a preliminary getting-off behavior. By calculating the horizontal and vertical optical flow vectors of each pixel in the humanoid area, the movement changes of the pixels in the humanoid area in the time dimension can be captured in detail. The optical flow vector can reflect the moving speed and direction of the object. By calculating the horizontal and vertical directions separately, the movement state of the object in the humanoid area can be fully understood, and the preset optical flow value is used as the standard for screening, which can exclude those tiny optical flow vector changes that may be caused by noise or irrelevant movement. Only the horizontal and vertical optical flow vectors with absolute values ​​greater than the preset optical flow value are paid attention to, and then the pixels with significant movement are more accurately focused on, which improves the pertinence of the judgment of the target user's behavior. The covariance matrix can reflect the correlation of the optical flow vector in different directions, and the eigenvector represents the main direction of the optical flow vector field. Through the covariance matrix, the movement trend of the pixels in the humanoid area can be grasped as a whole, not just the movement of a single pixel. As a directional description of the optical flow vector field, the eigenvector can more accurately determine whether the target user has got off the bus.

[0058] In one embodiment of the present application, the image processing module is used to extract the human-shaped area of ​​the image data outside the vehicle, which is recorded as the tracking human-shaped area; if a complete human portrait is identified by tracking the human-shaped area, and it is determined that the tracking user in the tracking pedestrian area and the target user in the human-shaped area are the same user, then the final getting-off result of the target user is determined to be completed getting off; if a partial human portrait is identified by tracking the human-shaped area, and a complete human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be getting off and turning back; if a partial human portrait is identified by tracking the human-shaped area, and a partial human portrait is identified in the human-shaped area, then the final getting-off result of the target user is determined to be getting off. Different getting-off situations correspond to different judgment criteria, which can more accurately determine the passenger's getting-off status and can adapt to various different scenarios. Whether the passenger gets off quickly, gets off slowly, or returns to the vehicle midway, accurate judgments can be made, and corresponding safety measures can be taken according to accurate judgments of the getting-off status, thereby improving the user experience.

[0059] In one embodiment of the present application, the vehicle getting off behavior recognition device further includes a model training module 250, which is used to obtain input data; forward propagate the input data through the backbone network in the initial humanoid region recognition model, calculate the output of each network layer, and obtain the final prediction result; calculate the difference between the prediction result and the true label through the binary cross entropy loss function to obtain the loss value, and the true label represents the correct result corresponding to the input data; calculate the gradient according to the loss function, and back-propagate to the weights and bias values ​​in the network layer to reduce the loss function value; iteratively train the initial humanoid region recognition model until the loss function value stabilizes in the preset function value range to obtain the humanoid region recognition model. The humanoid region recognition model can automatically extract features related to the human body shape from the input data, calculate the output of each network layer, and finally obtain the prediction result, thereby improving the accuracy of humanoid region recognition, and by calculating the difference between the prediction result and the true label through the binary cross entropy loss function, the gap between the humanoid region predicted by the model and the actual humanoid region can be clearly measured. The model can also automatically adjust the weights and bias values ​​in the network layer so that the model can better fit the training data in the next iteration and improve the accuracy of humanoid region recognition. After multiple trainings, the humanoid region recognition model can achieve a stable performance state, providing reliable support for subsequent applications.

[0060] In one embodiment of the present application, the model training module 250 is used to obtain sample in-vehicle image data, and to perform portrait feature annotation on the in-vehicle image data to obtain a portrait feature image and a non-portrait feature image; adjust the portrait feature image and the non-portrait feature image to a preset size, and normalize the pixel values ​​of the resized portrait feature image and the pixel values ​​of the non-portrait feature image to a preset pixel value range to obtain a preprocessed portrait feature image and a preprocessed non-portrait feature image, and use the preprocessed portrait feature image and the preprocessed non-portrait feature image as input data for the initial human area recognition model.

[0061] In one embodiment of the present application, the vehicle getting-off behavior recognition device further includes a braking module 260, which is used to send a warning message to the in-vehicle display device if it is determined that the target user is getting off the vehicle after completing the target user's getting-off behavior recognition; obtain throttle braking information when it is detected that the current gear of the vehicle is not in the parking gear; if it is determined that the throttle braking information is the first throttle braking, limit the current throttle braking; if it is determined that the throttle braking information is not the first throttle braking, respond to throttle braking. By disabling throttle braking, the vehicle is prevented from starting due to driver's misoperation, ensuring the safety of passengers.

[0062] In one embodiment of the present application, the image acquisition module is used to detect the switch status of the vehicle doors after the vehicle stops; if the switch status of any door is detected to be open, the pressure signal of each seat is detected by the pressure sensor on the seat in the vehicle; if the pressure signal is less than a preset pressure threshold, the real-time video stream in the vehicle is collected by the in-vehicle imaging device.

[0063] In one embodiment of the present application, the image acquisition module is used to extract multiple continuous frames of original in-vehicle behavior images from the real-time in-vehicle video stream; perform data enhancement on the original in-vehicle behavior images, and perform mean filtering on the in-vehicle behavior images after data enhancement to obtain pre-processed in-vehicle behavior images; data enhancement includes: contrast enhancement and clarity enhancement; normalize the pixel values ​​of the pre-processed in-vehicle behavior images to a preset pixel value range to obtain the behavior images to be identified. By performing data processing on the behavior images to be identified, the behavior images to be identified finally obtained have higher quality and standardization, which provides better input for the subsequent behavior recognition algorithm and can improve the accuracy and efficiency of behavior recognition.

[0064] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0065] See also Figure 3 , Figure 3 is a structural diagram of a vehicle-mounted terminal shown in an exemplary embodiment of the present application. It should be noted that: Figure 3 The vehicle-mounted terminal 300 shown is only an example and should not bring any limitation to the functions and usage area of ​​the embodiments of the present application.

[0066] like Figure 3 As shown, the vehicle terminal 300 includes a processor 301, a memory 302 and a communication bus 303; the communication bus 303 is used to connect the processor 301 and the memory 302; the processor 301 is used to execute the computer program stored in the memory 302 to implement one or more methods in the above embodiments.

[0067] The vehicle-mounted terminal provided in the present application includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic device executes the various steps of the above method.

[0068] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0069] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0070] The above embodiments are merely illustrative of the principles and effects of the present application, and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A method for identifying getting off a vehicle, characterized in that: The method comprises: Acquire a real-time video stream in the vehicle, and extract a plurality of continuous frame images from the real-time video stream in the vehicle as behavior images to be identified; Inputting the behavior image to be identified into a human shape region recognition model that has been trained in advance or in real time, and extracting the human shape region in the behavior image to be identified; It is identified through the human-shaped area whether the target user has a preliminary getting-off behavior, and when the target user has a preliminary getting-off behavior, the final getting-off result of the target user is determined in combination with the vehicle exterior image data.

2. The method for identifying getting off a vehicle according to claim 1, characterized in that: Identifying whether the target user has preliminary getting-off behavior through the human-shaped area includes: Calculate the horizontal optical flow vector and the vertical optical flow vector of each pixel point in the human-shaped area; If the absolute value of the horizontal optical flow vector is greater than the preset optical flow value, or the absolute value of the vertical optical flow vector is greater than the preset optical flow value, the horizontal optical flow vector greater than the preset optical flow value is recorded as the preliminary screening horizontal optical flow vector, and the vertical optical flow vector greater than the preset optical flow value is recorded as the preliminary screening vertical optical flow vector; Calculate the covariance matrix composed of the preliminary screening horizontal axis optical flow vector and the preliminary screening vertical axis optical flow vector, and perform eigenvalue decomposition on the covariance matrix to obtain eigenvectors, wherein the eigenvectors represent the direction of the optical flow vector field, and the optical flow vector field is a two-dimensional vector field that describes the movement of pixel points in the humanoid area over time; If the direction of the feature vector is consistent with the door opening direction, it is recognized that the target user has a preliminary behavior of getting off the vehicle.

3. The method for identifying getting off a vehicle according to claim 1, characterized in that: Determining the final exit result of the target user according to the vehicle exterior image data includes: Extracting a human-shaped area of ​​the vehicle exterior image data, and recording it as a tracking human-shaped area; If a complete human portrait is recognized through the tracking human-shaped area, and it is determined that the tracking user in the tracking pedestrian area and the target user in the human-shaped area are the same user, then it is determined that the final getting-off result of the target user is completed; If a partial human portrait is identified through the tracking human-shaped area, and a complete human portrait is identified in the human-shaped area, it is determined that the final getting-off result of the target user is getting off and returning; If a portion of the human figure is recognized through the tracking human figure area and a portion of the human figure is recognized in the human figure area, it is determined that the final getting-off result of the target user is getting-off.

4. The method for identifying getting off a vehicle according to claim 1, characterized in that: The process of training and generating the humanoid region recognition model includes: Get input data; The input data is forward propagated through the backbone network in the initial human shape region recognition model, the output of each network layer is calculated, and the final prediction result is obtained; Calculate the difference between the predicted result and the true label by using a binary cross entropy loss function to obtain a loss value, wherein the true label represents the correct result corresponding to the input data; Calculate the gradient according to the loss function and back-propagate to the weights and bias values ​​in the network layer to reduce the loss function value; The initial human shape area recognition model is iteratively trained until the loss function value is stabilized within a preset function value range to obtain a human shape area recognition model.

5. The method for identifying getting off a vehicle according to claim 4, characterized in that: The process of obtaining the input data includes: Acquire sample in-vehicle image data, and annotate the in-vehicle image data with portrait features to obtain a portrait feature image and a non-portrait feature image; Adjusting the portrait feature image and the non-portrait feature image to a preset size, and normalizing the pixel values ​​of the portrait feature image and the non-portrait feature image after the size adjustment to a preset pixel value interval, to obtain a pre-processed portrait feature image and a pre-processed non-portrait feature image; The preprocessed portrait feature image and the preprocessed non-portrait feature image are used as input data of the initial human shape region recognition model.

6. The method for identifying getting off a vehicle according to claim 3, characterized in that: After the target user's getting-off behavior is identified, the method further includes: If it is determined that the target user is getting off the vehicle, a warning message is sent to a display device in the vehicle; When it is detected that the current gear position of the vehicle is not in the parking gear position, the accelerator brake information is obtained; If it is determined that the throttle braking information is the first throttle braking, the current throttle braking is restricted; if it is determined that the throttle braking information is not the first throttle braking, the throttle braking is responded to.

7. The method for identifying getting off a vehicle according to claim 1, characterized in that: Get real-time in-car video streaming, including: After the vehicle stops, detect the switch status of the door; If it is detected that the switch state of any door is open, the pressure signal of each seat is detected by the pressure sensor on the seat in the car; If the pressure signal is less than a preset pressure threshold, a real-time video stream inside the vehicle is collected through an in-vehicle imaging device.

8. The method for identifying getting off a vehicle according to claim 1, characterized in that: Extracting multiple behavior images to be identified from the real-time in-vehicle video stream, including: Extracting a plurality of continuous frames of original in-vehicle behavior images from the in-vehicle real-time video stream; Performing data enhancement on the original in-vehicle behavior image, and performing mean filtering on the in-vehicle behavior image after data enhancement to obtain a preprocessed in-vehicle behavior image; the data enhancement includes: contrast enhancement and clarity enhancement; The pixel values ​​of the preprocessed in-vehicle behavior image are normalized to a preset pixel value range to obtain a behavior image to be identified.

9. A vehicle getting off behavior recognition device, characterized in that: The device comprises: An image acquisition module is used to obtain a real-time video stream in the vehicle and extract a plurality of continuous frame images from the real-time video stream in the vehicle as the behavior images to be identified; A humanoid region recognition module is used to input the behavior image to be recognized into a pre-trained or real-time trained humanoid region recognition model to extract the humanoid region in the behavior image to be recognized; The image processing module is used to identify whether the target user has a preliminary getting-off behavior through the human-shaped area, and when the target user has a preliminary getting-off behavior, determine the final getting-off result of the target user in combination with the external vehicle image data.

10. A vehicle-mounted terminal, characterized in that: It comprises a processor, a memory and a communication bus; the communication bus is used to connect the processor and the memory; the processor is used to execute the computer program stored in the memory to implement the method for identifying getting off the vehicle as described in any one of claims 1 to 8.

Images