Perception safety protection system for pedestrians outside vehicle

Abstract

The invention relates to the field of vehicle-mounted technologies, in particular to a pedestrian sensing safety protection system outside a vehicle. The system comprises an image acquisition unit, an outside-vehicle sensing unit, an emotion sensing unit and a human-vehicle interaction unit. Corresponding language features are established for different emotion types through the human-vehicle interaction unit, the language features comprise language information and mood and intonation emotion colors, the emotion of the person in the vehicle in the emotion perception unit is received, the corresponding language features are called out, the gesture meaning in the outside-vehicle perception unit is interactively transmitted to the person in the vehicle, and the person in the vehicle is more interesting. According to the method, people in the automobile can find out conditions outside the automobile in time to make a response, a driver is assisted to make a driving decision, powerful environment analysis support is provided for the automatic driving automobile, interaction is conducted according to the emotion of the people in the automobile, the people in the automobile are prevented from being discolored, and the comfort level is improved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle-mounted technology, and in particular to a safety protection system for sensing pedestrians outside a vehicle. Background Art

[0002] With the continuous development of technologies such as artificial intelligence and the Internet of Things, in-vehicle intelligent systems also need to be constantly updated and upgraded to improve their intelligence level. With the gradual penetration of the domestic Internet of Vehicles, consumers have put forward higher and higher requirements for automobile safety, operational convenience, entertainment, etc., which has led to an increase in the market demand for in-vehicle intelligent terminals. Based on mobile communication technology, the shipment volume of in-vehicle terminals with built-in mobile communication modules has grown rapidly;

[0003] However, the existing in-vehicle system interaction experience is not intelligent enough and cannot effectively process complex commands. The perception of non-motor vehicles outside the vehicle is relatively simple. For example, it is difficult to recognize the lane change gestures of the cyclist in front of the vehicle in response to the traffic command of the traffic police. As a result, if the driver does not pay attention to the traffic command or lane change gestures of the police outside the vehicle, dangerous accidents are likely to occur, and the safety performance is limited.

[0004] In addition, the driver's emotions have a certain impact on driving safety. If the driver is in a bad mood, he or she is prone to excessive behavior and cause dangerous accidents. The existing vehicle-mounted system is not convenient for two-way evaluation of the driver's emotional intensity, resulting in the inability to warn and relieve emotions when bad emotions appear, and the driving experience is limited. In view of this, we propose an off-vehicle pedestrian perception safety protection system that can improve driving safety, reduce traffic congestion, reduce environmental pollution and other social problems through on-board intelligent AI unit technology, and bring greater benefits to society. Summary of the invention

[0005] The purpose of the present invention is to provide a safety protection system for sensing pedestrians outside a vehicle to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above-mentioned purpose, the present invention provides a safety protection system for pedestrian perception outside a vehicle, comprising an image acquisition unit, an outside vehicle perception unit, an emotion perception unit and a human-vehicle interaction unit;

[0007] The image acquisition unit is used to acquire image data outside the vehicle and image data inside the vehicle;

[0008] The vehicle-exterior perception unit associates gestures with gesture meanings through gesture history data, trains an analysis model, calculates image depth information using binocular vision principles, obtains the three-dimensional structure of the vehicle-exterior image data scene in the image acquisition unit, perceives the gesture area, which includes roads, pedestrians and bicycles, extracts feature points of the gesture area and inputs them into the analysis model, and outputs the gesture meaning;

[0009] The emotion perception unit associates the facial muscle movement state with the emotion through the emotion history data, trains the emotion model to receive the in-vehicle image data from the image acquisition unit, captures the user's expression, analyzes the characteristics of the user's facial muscle movement state, inputs the emotion model, and outputs the emotion of the person in the vehicle;

[0010] The human-vehicle interaction unit is used to establish corresponding language features for different emotion types, and the language features include language information and tone of voice and emotional color. It receives the emotions of the people in the car from the emotional perception unit, calls out the corresponding language features, and interactively transmits the gesture meanings in the external perception unit to the people in the car.

[0011] As a further improvement of the technical solution, the image acquisition unit includes two sets of binocular cameras and a light adaptation module respectively installed facing outside the vehicle and inside the vehicle;

[0012] The binocular camera is composed of two parallel cameras, which are aimed at the target scene to shoot at the same time, find the corresponding points of the same object in the two images, calculate the pixel offset between the corresponding points, calculate the depth information of the object using the disparity value and the baseline length, and construct a three-dimensional model of the scene based on the obtained depth information;

[0013] The light adaptation module is used to detect the brightness level of the current environment using the photosensor built into the binocular camera, calculate the exposure parameters that need to be adjusted based on the difference between the detected brightness value and the target brightness value, and adjust the gain of the image sensor until the target brightness value is reached.

[0014] As a further improvement of the present technical solution, the meaning of the gestures includes traffic police gestures and the lane changing gestures of the bicycle driver in front, and the traffic police gestures include traffic police lane changing gestures, traffic police stop gestures, traffic police straight going gestures, traffic police turn gestures, traffic police slow down gestures and traffic police pull over gestures.

[0015] As a further improvement of the technical solution, the vehicle exterior perception unit includes an analysis model training module, an image data division module and a gesture feature extraction module;

[0016] The analysis model training module is used to collect a gesture history data set containing various gesture feature points, annotate each gesture sample with its corresponding gesture meaning, and use a convolutional neural network to train the analysis model;

[0017] The image data division module is used to receive the three-dimensional model of the outside of the vehicle output by the binocular camera, convert the three-dimensional model into a binary image, wherein the gesture area is marked as the foreground and the background is marked as black, and the range of the gesture is determined by using a connected domain to divide the gesture area;

[0018] The gesture feature extraction module is used to extract key feature points of the gesture area from the image data division module using a feature point detection algorithm, and match the key feature points with the feature points in the analysis model training module. If the match is successful, it is transmitted to the analysis model and the gesture meaning is output. If the match fails, an error signal is issued.

[0019] As a further improvement of the present technical solution, the vehicle-exterior perception unit also includes a gesture classification control module, which is used to receive the gesture meaning output by the gesture feature extraction module. If the gesture meaning is classified as a traffic police gesture, it is directly transmitted to the human-vehicle interaction unit. If the gesture meaning is classified as a lane change gesture of the bicycle driver in front, a control signal is transmitted to the vehicle control system to control the strength and time of the brake.

[0020] As a further improvement of the present technical solution, the emotion perception unit includes an emotion model training module, an expression capture module and an emotion output module;

[0021] The emotion model training module is used to collect feature points containing various facial muscle movement states as an emotion history data set, label each sample with its corresponding emotion type, and use a convolutional neural network to train the emotion model;

[0022] The expression capture module is used to receive the three-dimensional model of the interior of the vehicle output by the binocular camera, use the face recognition library to detect the face area in the image, and use the facial key point positioning algorithm to locate the key points of the face, and extract the key points to analyze the movement state characteristics of the facial muscles;

[0023] The emotion output module is used to convert the motion state features in the expression capture module into a numerical vector, and input it into the emotion model to predict the emotion type.

[0024] As a further improvement of the technical solution, the convolutional neural network training emotion model and the convolutional neural network training analysis model both include the following steps:

[0025] Normalize the input data and enhance the data;

[0026] For the convolution kernel of each layer and the weight matrix of the fully connected layer, the dot product of the input image and the convolution kernel is calculated in the convolution layer, the bias term is added, and then the output is output through the activation function. In the pooling layer, the output of the convolution layer is sampled. Commonly used pooling methods are maximum pooling and average pooling. In the fully connected layer, the output of the pooling layer is flattened into a one-dimensional vector, multiplied by the weight matrix of the fully connected layer, the bias term is added, and then the output is output through the activation function. The training hyperparameters are set, the data set is divided into training set, validation set and test set, the training set data is used for model training, and the analysis model and emotion model are output respectively.

[0027] As a further improvement of the technical solution, the human-vehicle interaction unit includes a database establishment module and an interaction control module;

[0028] The database establishment module is used to establish a language feature library corresponding to different emotion types, language feature library = (language information, tone and intonation, emotional color) → (emotion type), receive the emotion type input language feature library of the emotion output module, and output the control type in accordance with the language information, tone and intonation and emotional color;

[0029] The interactive control module is used to receive the control type that conforms to the language information, tone and emotional color in the database establishment module and maps it to the gesture meaning in the gesture feature extraction module, output text information corresponding to the gesture meaning, use speech synthesis technology to convert the text information into speech, and broadcast it through the vehicle audio.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] In the off-vehicle pedestrian perception safety protection system, corresponding language features are established for different emotion types through the human-vehicle interaction unit. The language features include language information and tone of voice and emotional color. The emotions of the occupants in the vehicle are received in the emotional perception unit, and the corresponding language features are called out. The meaning of gestures in the off-vehicle perception unit is interactively transmitted to the occupants in the vehicle. This is not only conducive to the occupants in the vehicle to promptly detect the situation outside the vehicle and respond, assist the driver in making driving decisions, and provide strong environmental analysis support for the self-driving car, but also to interact with the emotions of the occupants in the vehicle to avoid causing disgust among the occupants and improve comfort. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall structural principle block diagram of the present invention;

[0033] Figure 2 This is a block diagram of the structural principles of the vehicle exterior sensing unit and the emotion sensing unit of the present invention.

[0034] The meaning of each number in the figure is:

[0035] 100, image acquisition unit; 110, binocular camera; 120, light adaptation module;

[0036] 200, vehicle exterior perception unit; 210, analysis model training module; 220, image data division module; 230, gesture feature extraction module;

[0037] 300, emotion perception unit; 310, emotion model training module; 320, expression capture module; 330, emotion output module;

[0038] 400, human-vehicle interaction unit; 410, database establishment module; 420, interaction control module. DETAILED DESCRIPTION

[0039] The following will be combined with the accompanying drawings in the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] Example

[0041] With the continuous development of technologies such as artificial intelligence and the Internet of Things, in-vehicle intelligent systems also need to be continuously updated and upgraded to improve their intelligence level. As the domestic Internet of Vehicles gradually penetrates, consumers have higher and higher requirements for car safety, operational convenience, and entertainment, which has led to an increase in the market demand for in-vehicle intelligent terminals. Based on mobile communication technology, the shipment volume of in-vehicle terminals with built-in mobile communication modules has grown rapidly. Therefore, please refer to Figure 1-Figure 2 As shown, this embodiment provides a safety protection system for pedestrian perception outside a vehicle, including an image acquisition unit 100, an outside vehicle perception unit 200, an emotion perception unit 300 and a human-vehicle interaction unit 400;

[0042] The image acquisition unit 100 is used to acquire image data outside the vehicle and image data inside the vehicle;

[0043] Furthermore, the image acquisition unit 100 includes two sets of binocular cameras 110 and a light adaptation module 120 which are respectively installed facing outside the vehicle and inside the vehicle;

[0044] The binocular camera 110 is composed of two parallel cameras. The two cameras are aimed at the target scene to shoot at the same time, find the corresponding points of the same object in the two images, calculate the pixel offset between the corresponding points, and use the disparity value and the baseline length to calculate the depth information of the object. According to the obtained depth information, a three-dimensional model of the scene is constructed, wherein: the distance between the two cameras is called the baseline. When the two cameras shoot the same scene at the same time, due to the different viewpoints, the images obtained will have a certain difference. This difference is called the disparity value. By calculating these disparity values, the distance of the object relative to the camera is calculated, thereby constructing the three-dimensional structure of the scene;

[0045] The light adaptation module 120 is used to detect the brightness level of the current environment with the built-in photosensitive element in the binocular camera 110, calculate the exposure parameters that need to be adjusted according to the difference between the detected brightness value and the target brightness value, and adjust the gain of the image sensor until the target brightness value is reached. The photosensitive element detects the lighting conditions of the current environment, usually indirectly by measuring the number of photons reaching the sensor. In a dark environment, the sensor receives fewer photons, while in a well-lit environment, the number of photons received is greater. The output signal of the sensor (such as voltage or current) is proportional to the number of photons received. The current brightness level is compared with a preset target brightness value. The target brightness value may be set based on application requirements. For example, when collecting images, it avoids unclear images caused by too bright or too dark, which is conducive to the binocular camera 110 to adapt to various lighting conditions and provide stable and high-quality image capture.

[0046] The vehicle-exterior sensing unit 200 associates gestures with gesture meanings through gesture history data, trains an analysis model, calculates image depth information using binocular vision principles, obtains the three-dimensional structure of the vehicle-exterior image data scene in the image acquisition unit 100, senses the gesture area, which includes roads, pedestrians, and bicycles, extracts feature points of the gesture area, inputs them into the analysis model, and outputs the gesture meaning;

[0047] Specifically, the meaning of the hand gestures includes the traffic police hand gestures and the lane change gestures of the cyclist in front. The traffic police hand gestures include the traffic police lane change gestures, the traffic police stop gestures, the traffic police go straight gestures, the traffic police turn gestures, the traffic police slow down gestures and the traffic police pull over gestures.

[0048] Traffic police lane change gesture: right arm extended forward, palm facing left, right arm swung horizontally to the left, vehicles should clear the designated lane and slow down;

[0049] Traffic police stop sign: left arm extended straight forward and upward, palm facing forward, no vehicles in front are allowed to pass;

[0050] Traffic police's straight ahead gesture: The traffic police extends his left arm to the left, palm facing forward; extends his right arm to the right, palm facing forward, and swings it to the left, allowing vehicles going straight to the right of the traffic police to pass;

[0051] Traffic police turning gesture: The traffic police extends his right arm (left arm) forward with his palm facing forward, and the vehicles in front of the traffic police stop passing; the left arm (right arm) and the palm are swung straight to the right (left) front, with the palm facing right (left), allowing the vehicles on the left (right) of the traffic police to turn left (right), and they can turn around without hindering the passage of the vehicles being released;

[0052] Traffic police slow down gesture: The traffic police extends his right arm to the right front, with the palm facing down, and swings his right arm and palm straight down. Vehicles should slow down;

[0053] Traffic police pull-over gesture: The traffic police extends his left arm forward and upward, with the palm facing forward, and extends his right arm forward and downward, with the palm facing left; the right arm swings horizontally to the left, and the vehicle should pull over;

[0054] Lane changing gesture for the cyclist ahead: When turning left (right), extend your left hand (right hand) horizontally with your palm facing forward, indicating a left turn (right turn) or a left lane change (right lane change).

[0055] Then, the vehicle exterior perception unit 200 includes an analysis model training module 210, an image data division module 220, and a gesture feature extraction module 230;

[0056] The analysis model training module 210 is used to collect a gesture history data set containing various gesture feature points, annotate each gesture sample with its corresponding gesture meaning, and use a convolutional neural network to train an analysis model;

[0057] The image data segmentation module 220 is used to receive the three-dimensional model of the outside of the vehicle output by the binocular camera 110, and convert the three-dimensional model into a binary image, in which the gesture area is marked as the foreground and the background is marked as black. The connected domain is used to determine the range of the gesture and divide the gesture area. When the connected domain is used, all the connected domains in the binary image are found. Once the connected domain is detected, it is implemented in the following ways to help us accurately recognize the gesture and provide a basis for further gesture meaning recognition:

[0058] ① Area filtering: Gestures are usually neither too large nor too small. By setting a reasonable area range, connected domains that are obviously not gestures can be filtered out;

[0059] ② Shape analysis: Gestures usually have certain shape characteristics, such as circles or ellipses, and are filtered by calculating the aspect ratio, roundness and other features of the connected domain;

[0060] ③ Position information: Considering that gestures usually appear near the human body, position information (such as the center position of the connected domain) is used

[0061] To further filter;

[0062] The gesture feature extraction module 230 is used to extract key feature points of the gesture area from the image data division module 220 using a feature point detection algorithm, and match the key feature points with the feature points in the analysis model training module 210. If the match is successful, the key feature points are transmitted to the analysis model and the gesture meaning is output. If the match fails, an error signal is issued. The feature point detection algorithm is used to extract unique key points from the image. These points usually have good repeatability and stability. Commonly used feature point detection algorithms include but are not limited to SIFT (Scale-Invariant Feature Transform, SURF (Speeded Up Robust Features, ORB (Oriented FAST and Rotated BRIEF, etc.;

[0063] Moreover, feature point matching is the process of pairing feature points in two images. Usually, the nearest neighbor distance ratio (NNDR) and other methods are used to select the most appropriate matching pair, and feature point descriptors are used for matching to generate matching candidate pairs, and high-quality matching pairs are screened out according to certain standards.

[0064] Since different people outside the vehicle correspond to different gestures, such as the gestures of the traffic police and the gestures of the cyclist are issued at different times, the corresponding countermeasures are different. In particular, when the cyclist suddenly issues a turning gesture that is consistent with the driving direction, if the vehicle speed cannot be controlled in time, a traffic accident is likely to occur. Therefore, the vehicle-outside perception unit 200 also includes a gesture classification control module. The gesture classification control module is used to receive the meaning of the gesture output by the gesture feature extraction module 230. If the meaning of the gesture is classified as a traffic police gesture, it is directly transmitted to the human-vehicle interaction unit 400. If the meaning of the gesture is classified as a lane change gesture of the cyclist in front, a control signal is transmitted to the vehicle control system to control the strength and time of the brake to achieve smooth deceleration and emergency braking, so as to avoid the situation where the lane the cyclist wants to take conflicts with the current vehicle driving direction and cannot respond in time. This is beneficial to control the vehicle and improve safety.

[0065] The emotion perception unit 300 associates the facial muscle movement state with the emotion through the emotion history data, trains the emotion model, receives the in-vehicle image data from the image acquisition unit 100, captures the user's expression, and analyzes the characteristics of the user's facial muscle movement state to input into the emotion model, and outputs the emotion of the person in the vehicle;

[0066] Furthermore, the emotion perception unit 300 includes an emotion model training module 310 , an expression capture module 320 , and an emotion output module 330 ;

[0067] The emotion model training module 310 is used to collect feature points including various facial muscle movement states as an emotion history data set, label each sample with its corresponding emotion type, and train the emotion model using a convolutional neural network;

[0068] The expression capture module 320 is used to receive the three-dimensional model of the interior of the vehicle output by the binocular camera 110, use a face recognition library, such as OpenCV or Dlib, to detect the face area in the image, and use a facial key point positioning algorithm, such as shape_predictor of Dlib, to locate the key points of the face, such as eyes, nose, mouth, eyebrows, etc., extract the key points to analyze the movement state characteristics of facial muscles, analyze the distance, angle, change trend and other characteristics between the facial key points, so as to describe the movement state of facial muscles, for example:

[0069] Changes in the curvature of the mouth corners: reflects emotions such as happiness and sadness;

[0070] The degree of eyebrow raising: reflects emotions such as surprise and anger;

[0071] The degree of eye opening and closing: reflects emotions such as fatigue and excitement;

[0072] The degree of opening and closing of the mouth: reflects emotions such as surprise and fear;

[0073] The emotion output module 330 is used to convert the motion state features in the expression capture module 320 into a numerical vector, and input it into the emotion model to predict the emotion type.

[0074] Specifically, the convolutional neural network training emotion model and the convolutional neural network training analysis model both include the following steps:

[0075] Normalize input data. For example, the pixel values ​​of an image are usually normalized to [0, 1] or [-1, 1]. Enhance the data by performing rotation, flipping, shearing, etc. to increase the generalization ability of the model.

[0076] For the convolution kernel of each layer and the weight matrix of the fully connected layer, the dot product of the input image and the convolution kernel is calculated in the convolution layer, the bias term is added, and then the output is output through the activation function (such as ReLU). In the pooling layer, the output of the convolution layer is downsampled. Commonly used pooling methods are maximum pooling and average pooling. In the fully connected layer, the output of the pooling layer is flattened into a one-dimensional vector, multiplied by the weight matrix of the fully connected layer, the bias term is added, and then the output is output through the activation function. The training hyperparameters are set, such as learning rate, batch size, number of iterations, etc. The data set is divided into training set, validation set and test set, and the training set data is used for model training, and the analysis model and emotion model are output respectively.

[0077] The human-vehicle interaction unit 400 is used to establish corresponding language features for different emotion types. The language features include language information and tone of voice and emotional color. It receives the emotions of the people in the car from the emotion perception unit 300, calls out the corresponding language features, and interactively transmits the meaning of gestures in the outside-vehicle perception unit 200 to the people in the car. This is not only beneficial for the people in the car to promptly discover the situation outside the car and respond, assist the driver in making driving decisions, and provide strong environmental analysis support for the self-driving car, but also interacts with the emotions of the people in the car to avoid causing disgust among the people in the car and improve comfort.

[0078] In addition, the human-vehicle interaction unit 400 includes a database establishment module 410 and an interaction control module 420;

[0079] The database establishment module 410 is used to establish a language feature library corresponding to different emotion types, language feature library = (language information, tone of voice, emotional color) → (emotion type), receive the emotion type input language feature library of the emotion output module 330, and output the control type in accordance with the language information, tone of voice and emotional color, wherein:

[0080] First, language information, choose different language expressions for different types of emotions, for example:

[0081] Happy: Use positive, exciting words, such as "Great! The vehicle changed lanes", "Great! The vehicle changed lanes";

[0082] Sadness: Use comforting, sympathetic words, such as "It's OK, the vehicle will change lanes" and "Don't worry, the vehicle will change lanes";

[0083] Angry: Use calm, neutral words, such as "Please note, the vehicle is changing lanes" or "Be careful, the vehicle is changing lanes";

[0084] Anxiety: Use calming, relaxing words, such as "Relax, the vehicle is changing lanes" and "Take your time, the vehicle is changing lanes";

[0085] Second, tone and intonation: Choose different tones and intonations for different emotional types, for example:

[0086] Happy: Use a light, lively tone of voice;

[0087] Sadness: Use a low, gentle tone of voice;

[0088] Angry: Use a serious, calm tone;

[0089] Anxiety: Use a soft, soothing tone of voice;

[0090] Third, emotional color: Choose different emotional colors for different emotional types, for example:

[0091] Happy: Use positive, positive emotions

[0092] Sadness: Use negative, negative emotions

[0093] Anger: Use neutral, objective emotional coloring

[0094] Anxiety: Use gentle, comforting emotional colors;

[0095] The interactive control module 420 is used to receive the control type that conforms to the language information, tone and emotional color in the database establishment module 410 and maps it to the gesture meaning in the gesture feature extraction module 230, output the text information corresponding to the gesture meaning, use the speech synthesis technology to convert the text information into speech, and broadcast it through the car audio, which is conducive to the people in the car to comfortably receive the gesture content.

[0096] The human-vehicle interaction unit 400 establishes corresponding language features for different emotion types, receives the emotions of the people in the car in the emotion perception unit 300, calls out the corresponding language features, and interactively transmits the meaning of the gestures in the outside-vehicle perception unit 200 to the people in the car. This is not only beneficial for the people in the car to promptly discover the situation outside the car and respond, assist the driver in making driving decisions, and provide strong environmental analysis support for the autonomous driving car, but also interacts with the emotions of the people in the car to avoid causing disgust among the people in the car and improve comfort.

[0097] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments. The above embodiments and descriptions are only preferred examples of the present invention and are not intended to limit the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A pedestrian sensing safety protection system outside a vehicle, characterized by: It comprises an image acquisition unit (100), an external vehicle sensing unit (200), an emotion sensing unit (300) and a human-vehicle interaction unit (400); The image acquisition unit (100) is used to acquire image data outside the vehicle and image data inside the vehicle; The vehicle-exterior sensing unit (200) associates gestures with gesture meanings through gesture history data, trains an analysis model, calculates image depth information using binocular vision principles, obtains the three-dimensional structure of the vehicle-exterior image data scene in the image acquisition unit (100), senses a gesture area, the gesture area includes roads, pedestrians and bicycles, extracts feature points of the gesture area and inputs them into the analysis model, and outputs the gesture meaning; The emotion perception unit (300) associates the facial muscle movement state with the emotion through the emotion history data, trains the emotion model, receives the in-vehicle image data from the image acquisition unit (100), captures the user's expression, analyzes the characteristics of the user's facial muscle movement state, inputs the characteristics into the emotion model, and outputs the emotion of the person in the vehicle; The vehicle-to-human interaction unit (400) establishes corresponding language features for different emotion types, wherein the language features include language information and tone of voice and emotional color, receives the emotions of the vehicle occupants from the emotion perception unit (300), calls out the corresponding language features, and interactively transmits the meaning of the gestures in the vehicle-external perception unit (200) to the vehicle occupants.

2. The pedestrian sensing safety protection system according to claim 1, characterized in that: The image acquisition unit (100) comprises two sets of binocular cameras (110) respectively installed facing outside the vehicle and inside the vehicle, and a light adaptation module (120); The binocular camera (110) is composed of two cameras placed in parallel, the two cameras are aimed at the target scene to shoot at the same time, find the corresponding points of the same object in the two images, calculate the pixel offset between the corresponding points, calculate the depth information of the object using the disparity value and the baseline length, and construct a three-dimensional model of the scene based on the obtained depth information; The light adaptation module (120) is used to detect the brightness level of the current environment using a photosensitive element built into the binocular camera (110), calculate the exposure parameter that needs to be adjusted based on the difference between the detected brightness value and the target brightness value, and adjust the gain of the image sensor until the target brightness value is reached.

3. The pedestrian sensing safety protection system according to claim 2 is characterized in that: The meanings of the gestures include traffic police gestures and lane-changing gestures of the bicycle driver in front, and the traffic police gestures include traffic police lane-changing gestures, traffic police stop-traffic gestures, traffic police straight-ahead gestures, traffic police turn gestures, traffic police slow-down gestures and traffic police pull-over gestures.

4. The pedestrian sensing safety protection system according to claim 3 is characterized in that: The vehicle exterior perception unit (200) comprises an analysis model training module (210), an image data division module (220) and a gesture feature extraction module (230); The analysis model training module (210) is used to collect a gesture history data set containing various gesture feature points, annotate each gesture sample with its corresponding gesture meaning, and use a convolutional neural network to train the analysis model; The image data division module (220) is used to receive the three-dimensional model of the outside of the vehicle output by the binocular camera (110), convert the three-dimensional model into a binary image, wherein the gesture area is marked as the foreground and the background is marked as black, and the range of the gesture is determined by using a connected domain to divide the gesture area; The gesture feature extraction module (230) is used to extract key feature points of the gesture area from the image data division module (220) using a feature point detection algorithm, and match the key feature points with the feature points in the analysis model training module (210); if the match is successful, the key feature points are transmitted to the analysis model and the gesture meaning is output; if the match fails, an error signal is issued.

5. The pedestrian sensing safety protection system according to claim 4 is characterized in that: The vehicle exterior sensing unit (200) further comprises a gesture classification control module, the gesture classification control module being used to receive the meaning of the gesture output by the gesture feature extraction module (230), and if the meaning of the gesture is classified as a traffic police gesture, directly transmitting it to the vehicle-to-human interaction unit (400); if the meaning of the gesture is classified as a lane-changing gesture of a bicycle driver ahead, transmitting a control signal to the vehicle control system to control the force and time of the brake.

6. The pedestrian sensing safety protection system according to claim 4, characterized in that: The emotion perception unit (300) comprises an emotion model training module (310), an expression capture module (320) and an emotion output module (330); The emotion model training module (310) is used to collect feature points containing various facial muscle movement states as an emotion history data set, label each sample with its corresponding emotion type, and train the emotion model using a convolutional neural network; The expression capture module (320) is used to receive the three-dimensional model of the interior of the vehicle output by the binocular camera (110), detect the face area in the image using a face recognition library, and use a facial key point positioning algorithm to locate the key points of the face, and extract the key points to analyze the movement state characteristics of the facial muscles; The emotion output module (330) is used to convert the motion state features in the expression capture module (320) into a numerical vector, and input it into the emotion model to predict the emotion type.

7. The pedestrian sensing safety protection system according to claim 6, characterized in that: The convolutional neural network training emotion model and the convolutional neural network training analysis model both include the following steps: Normalize the input data and enhance the data; For the convolution kernel of each layer and the weight matrix of the fully connected layer, the dot product of the input image and the convolution kernel is calculated in the convolution layer, the bias term is added, and then the output is output through the activation function. In the pooling layer, the output of the convolution layer is sampled. Commonly used pooling methods are maximum pooling and average pooling. In the fully connected layer, the output of the pooling layer is flattened into a one-dimensional vector, multiplied by the weight matrix of the fully connected layer, the bias term is added, and then the output is output through the activation function. The training hyperparameters are set, the data set is divided into training set, validation set and test set, the training set data is used for model training, and the analysis model and emotion model are output respectively.

8. The pedestrian sensing safety protection system according to claim 6, characterized in that: The human-vehicle interaction unit (400) comprises a database establishment module (410) and an interaction control module (420); The database establishment module (410) is used to establish a language feature library corresponding to different emotion types, language feature library = (language information, tone of voice, emotional color) → (emotion type), receive the emotion type input language feature library of the emotion output module (330), and output the control type in accordance with the language information, tone of voice and emotional color; The interactive control module (420) is used to receive the control type in the database establishment module (410) that matches the language information, tone of voice and emotional color and is mapped to the gesture meaning in the gesture feature extraction module (230), output text information corresponding to the gesture meaning, use speech synthesis technology to convert the text information into speech, and broadcast it through the vehicle audio.

Images