A safety-graded in-vehicle driver-passenger conflict recognition method

Abstract

The present invention relates to the technical field of online car-hailing, and specifically discloses a method for identifying in-vehicle driver-passenger conflicts with security grading, comprising the following steps: S1, collecting video and sound data in the vehicle through Internet of Vehicles equipment; S2, sound data samples and encoding processing; S3, building a machine learning emotion recognition model; S4, training the machine learning emotion recognition model; S5, video data samples and encoding processing; S6, building a machine learning action recognition model; S7, training the machine learning action recognition model; S8, making security grading according to emotion recognition and action recognition; The present invention provides a method for identifying in-vehicle driver-passenger conflicts with security grading to solve the problems of in-vehicle physical conflicts and security grading. The method uses emotion recognition and action recognition to improve the accuracy of identifying conflicts. The advantage of this method is that safety can be graded, and different strategies can be formulated through different levels.

Description

Technical Field

[0001] The present invention relates to the technical field of online car-hailing, and in particular to a safety-graded in-vehicle driver-passenger conflict identification method. Background Art

[0002] Currently, there are physical conflicts between online ride-hailing drivers and passengers, which escalate into vicious incidents, thus affecting the travel experience and individual safety. The mainstream safety detection methods mainly rely on reporting during and after the incident and the intervention of the online ride-hailing platform. These are passive perception methods for the online ride-hailing platform, which cannot perceive the malicious incidents in the first time, increasing the degree of harm. Summary of the invention

[0003] The purpose of the present invention is to provide a safety-graded in-vehicle driver-passenger conflict identification method to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a safety graded in-vehicle driver-passenger conflict identification method, comprising the following steps:

[0005] S1. Collecting video and sound data in the car through the Internet of Vehicles equipment: Specifically, the Internet of Vehicles equipment is installed near the rearview mirror of the car. The Internet of Vehicles equipment integrates a camera and a recorder. It is turned on when the car is started. The video in the car is collected through the camera, and the sound data in the car is collected through the recorder. The collected video data and sound data are transmitted back to the online car-hailing platform in real time through the network;

[0006] S2. Sound data samples and coding processing: emotions are divided into 10 categories, among which categories 1-4 are like, happiness, gratitude and neutral, and categories 5-10 are complaint, disgust, anger, fear, sadness and pain. One-hot encoding is performed on categories 1-10, that is, category 1 is encoded as 1, 0, 0, 0, 0, 0, 0, 0, 0, category 2 is encoded as 0, 1, 0, 0, 0, 0, 0, 0, 0, and so on. Category 10 is encoded as 0, 0, 0, 0, 0, 0, 0, 0, 1;

[0007] S3. Build a machine learning emotion recognition model: (1) Convert the encoding vector Embedding into a word vector, set the vocabulary parameter to dic, and set the output dimension to embedding_dim, where embedding_dim = 128. Then each integer ID in (3, 10, 201, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) is encoded as a 128-dimensional word vector. The text encoding vector of length MaxLen is converted into MaxLen word vectors of length embedding_dim. MaxLen word vectors are sentence vectors. A text and a label constitute a sample. The final encoding length of the sample is: MaxLen*embedding_dim+10.

[0008] (2) Build an LSTM network: Set the input feature dimension input_dim to embedding_dim, the time step time_steps to MaxLen, the output layer feature dimension output_dim to 256, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by the LSTM is a feature vector rather than a group of feature vectors. The time_steps*embedding_dim features of the sample are the input of time_steps LSTM units, and each unit inputs embedding_dim features, that is, a word vector;

[0009] (3) Add a Dense linear layer to the output of the LSTM network. The output of the LSTM network is a 256-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 10-dimensional vector, denoted as (z1, z2…z10).

[0010] (4) Add a softmax activation function to the output of the dense linear layer, where softmax is a normalization function and the output is a 10-dimensional vector (y1, y2…y10). y1+y2+…y10=1, y1, y2,…y10 are the probabilities of category 1, category 2…category 10 respectively;

[0011] S4. Training the machine learning emotion recognition model: (1) Setting the loss function: The collected text and annotations are input into the emotion recognition model, with the label value (p1, p2…p10) and the value calculated by the forward network each time being (y1, y2…y10). The loss function is set as:

[0012] (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation;

[0013] S5, video data samples and encoding processing: the historical video data is manually marked in two categories, the body movement contact is marked as 1, and the body non-movement contact is marked as 0. The sound sample s and the video sample v associated with s are defined as the video data when the sound data is collected at the same time. The video v is processed by image processing, 4 frames are selected at uniform intervals per second, and 6 seconds of data, a total of 24 frames of images, are selected as a video sample;

[0014] S6. Build a machine learning action recognition model: (1) Build an LSTM network, set the input feature dimension input_dim to 2048, the time step time_steps to 24, the output layer feature dimension output_dim to 512, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by LSTM is a feature vector rather than a group of feature vectors. The 24*2048 features of the sample are the input of 24 LSTM units, and each unit inputs 2048 features;

[0015] (2) Add a Dense linear layer to the output of the LSTM network: The output of the LSTM network is a 512-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 1-dimensional vector.

[0016] (3) Add a sigmoid activation function to the output of the Dense linear layer:

[0017] S7. Training the machine learning action recognition model: (1) Setting the loss function: the label value is p, the value calculated by the forward network each time is y, and the loss function is set to:

[0018] binary_crossentropy=p*log(y)+(1-p)*log(1-y);

[0019] (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation;

[0020] S8. Make security classification based on emotion recognition and action recognition.

[0021] Preferably, the processing of the sound data sample in step S2 comprises the following steps:

[0022] S21. Collect complaints about disputes between drivers and passengers in the history of online ride-hailing platforms, retrieve the corresponding audio, manually determine the emotions into 5-10 categories, convert them into text, and manually mark and classify the text into 5-10 categories;

[0023] S22, collecting audio corresponding to orders in which passengers gave drivers high ratings in the history of the online car-hailing platform, manually determining the emotions as categories 1-3, converting them into text, and manually marking and classifying the text into categories 1-3;

[0024] S23, collecting the sounds corresponding to the unevaluated orders in the history of the online car-hailing platform, manually determining the emotions into 4 categories, converting them into text, and manually marking and classifying the text into 4 categories;

[0025] Among them, the processing of the sound in steps S21-23 adopts the Speech 2 Text method, and the sound-to-text conversion is achieved through the API conversion interface provided by the public cloud service.

[0026] Preferably, after manually collecting and annotating data, n texts and corresponding annotations are collected, Jieba open source word segmenter is used for word segmentation, the frequency of occurrence of words in the n texts is counted, and each word is converted into an integer ID according to the frequency. The ID of a word with a high frequency is smaller than that of a word with a low frequency. The integer ID starts from 1, and the maximum value is the number of different words, recorded as MaxWords, and all different words are put into the vocabulary dic.

[0027] Preferably, if each frame of the image in step S5 is a 3*400*400 image, CNN is used to encode the image, which specifically includes the following steps:

[0028] S51, convolution layer Conv2D processing, using 96 convolution kernels of size 3*3, activation function relu, then input 3*400*400 data, output 96*400*400 data;

[0029] S52, pooling layer MaxPooling2D processing, pooling size 2*2, then the input is 96*400*400, and the output is 96*200*200;

[0030] S53, forgetting layer DropOut processing, forgetting rate 0.2;

[0031] S54, convolution layer Conv2D processing, using 32 convolution kernels of size 3*3, activation function relu, then input 96*200*200 data, output 32*200*200 data;

[0032] S55, pooling layer MaxPooling2D processing, pooling size 4*4, then the input is 32*200*200, and the output is 32*50*50;

[0033] S56, forgetting layer DropOut processing, forgetting rate 0.25;

[0034] S57, Flatten layer processing, input is 32*50*50, output is 80000;

[0035] S58, linear fully connected Dense layer, input is 80,000 features, output is 2048 features;

[0036] These 2048 features are used as the encoding of one frame of image, and the feature encoding of 24 images is 24*2048 in total.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a safety-graded in-vehicle driver-passenger conflict identification method to solve the problems of in-vehicle physical conflicts and safety grading. The method uses emotion recognition and motion recognition to improve the accuracy of conflict identification. The advantage of this method is that safety is graded, and different strategies can be formulated through different levels, so that users in the car can be dissuaded by shouting, position detection and automatic alarm through the Internet of Vehicles, and for serious conflicts, vehicle speed limit and speed reduction are adopted to achieve the purpose of enhancing the safety of people in the car. Compared with traditional conflict detection methods, it has the advantages of safety grading, combining emotion and motion recognition, etc., which improves recognition accuracy and strengthens safety management and control. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a structural flowchart of the overall steps of the present invention. DETAILED DESCRIPTION

[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] See also Figure 1 The present invention provides a technical solution: a safety-graded in-vehicle driver-passenger conflict identification method, comprising the following steps:

[0041] S1. Collecting video and sound data in the car through the Internet of Vehicles equipment: Specifically, the Internet of Vehicles equipment is installed near the rearview mirror of the car. The Internet of Vehicles equipment integrates a camera and a recorder. It is turned on when the car is started. The video in the car is collected through the camera, and the sound data in the car is collected through the recorder. The collected video data and sound data are transmitted back to the online car-hailing platform in real time through the network;

[0042] S2. Sound data samples and coding processing: emotions are divided into 10 categories, among which categories 1-4 are like, happiness, gratitude and neutral, and categories 5-10 are complaint, disgust, anger, fear, sadness and pain. One-hot encoding is performed on categories 1-10, that is, category 1 is encoded as 1, 0, 0, 0, 0, 0, 0, 0, 0, category 2 is encoded as 0, 1, 0, 0, 0, 0, 0, 0, 0, and so on. Category 10 is encoded as 0, 0, 0, 0, 0, 0, 0, 0, 1;

[0043] S3. Build a machine learning emotion recognition model: (1) Convert the encoding vector Embedding into a word vector, set the vocabulary parameter to dic, and set the output dimension to embedding_dim, where embedding_dim = 128. Then each integer ID in (3, 10, 201, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) is encoded as a 128-dimensional word vector. The text encoding vector of length MaxLen is converted into MaxLen word vectors of length embedding_dim. MaxLen word vectors are sentence vectors. A text and a label constitute a sample. The final encoding length of the sample is: MaxLen*embedding_dim+10.

[0044] (2) Build an LSTM network: Set the input feature dimension input_dim to embedding_dim, the time step time_steps to MaxLen, the output layer feature dimension output_dim to 256, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by the LSTM is a feature vector rather than a group of feature vectors. The time_steps*embedding_dim features of the sample are the input of time_steps LSTM units, and each unit inputs embedding_dim features, that is, a word vector;

[0045] (3) Add a Dense linear layer to the output of the LSTM network. The output of the LSTM network is a 256-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 10-dimensional vector, denoted as (z1, z2…z10).

[0046] (4) Add a softmax activation function to the output of the dense linear layer, where softmax is a normalization function and the output is a 10-dimensional vector (y1, y2…y10). y1+y2+…y10=1, y1, y2,…y10 are the probabilities of category 1, category 2…category 10 respectively;

[0047] S4. Training the machine learning emotion recognition model: (1) Setting the loss function: The collected text and annotations are input into the emotion recognition model, with the label value (p1, p2…p10) and the value calculated by the forward network each time being (y1, y2…y10). The loss function is set as:

[0048] (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation;

[0049] S5, video data samples and encoding processing: the historical video data is manually marked in two categories, the body movement contact is marked as 1, and the body non-movement contact is marked as 0. The sound sample s and the video sample v associated with s are defined as the video data when the sound data is collected at the same time. The video v is processed by image processing, 4 frames are selected at uniform intervals per second, and 6 seconds of data, a total of 24 frames of images, are selected as a video sample;

[0050] S6. Build a machine learning action recognition model: (1) Build an LSTM network, set the input feature dimension input_dim to 2048, the time step time_steps to 24, the output layer feature dimension output_dim to 512, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by LSTM is a feature vector rather than a group of feature vectors. The 24*2048 features of the sample are the input of 24 LSTM units, and each unit inputs 2048 features;

[0051] (2) Add a Dense linear layer to the output of the LSTM network: The output of the LSTM network is a 512-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 1-dimensional vector.

[0052] (3) Add a sigmoid activation function to the output of the Dense linear layer:

[0053] S7. Training the machine learning action recognition model: (1) Setting the loss function: the label value is p, the value calculated by the forward network each time is y, and the loss function is set to:

[0054] binary_crossentropy=p*log(y)+(1-p)*log(1-y);

[0055] (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation;

[0056] S8. Make security classification based on emotion recognition and action recognition.

[0057] Furthermore, the processing of the sound data sample in step S2 includes the following steps:

[0058] S21. Collect complaints about disputes between drivers and passengers in the history of online ride-hailing platforms, retrieve the corresponding audio, manually determine the emotions into 5-10 categories, convert them into text, and manually mark and classify the text into 5-10 categories;

[0059] S22, collecting audio corresponding to orders in which passengers gave drivers high ratings in the history of the online car-hailing platform, manually determining the emotions as categories 1-3, converting them into text, and manually marking and classifying the text into categories 1-3;

[0060] S23, collecting the sounds corresponding to the unevaluated orders in the history of the online car-hailing platform, manually determining the emotions into 4 categories, converting them into text, and manually marking and classifying the text into 4 categories;

[0061] Among them, the processing of the sound in steps S21-23 adopts the Speech 2 Text method, and the sound-to-text conversion is achieved through the API conversion interface provided by the public cloud service.

[0062] Furthermore, after manually collecting and annotating data, n texts and corresponding annotations are collected, and Jieba open source word segmenter is used for word segmentation. The frequency of words in the n texts is counted, and each word is converted into an integer ID according to the frequency. The ID of the word with high frequency is smaller than the word with low frequency. The integer ID starts from 1, and the maximum value is the number of different words recorded as MaxWords. All different words are put into the vocabulary table dic;

[0063] Specifically, for the text "I am very happy today", the word segmentation is "I" (code 3), "today" (code 10), "very" (code 201), "happy" (code 28), and the corresponding encoding vector is (3, 10, 201, 28);

[0064] Set the maximum word length of the text as MaxLen. If the length exceeds, the text is truncated. If it is less than the maximum length, it is padded with 0. If MaxLen=20, the encoding vector of "I am very happy today" is (3, 10, 201, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0;

[0065] The label corresponding to (3, 10, 201, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) is 2. The one-hot encoding of label 2 is (0, 1, 0, 0, 0, 0, 0, 0, 0), and the label is denoted by (p1, p2…p10);

[0066] Furthermore, if each frame image in step S5 is a 3*400*400 image, CNN is used to encode the image, which specifically includes the following steps:

[0067] S51, convolution layer Conv2D processing, using 96 convolution kernels of size 3*3, activation function relu, then input 3*400*400 data, output 96*400*400 data;

[0068] S52, pooling layer MaxPooling2D processing, pooling size 2*2, then the input is 96*400*400, and the output is 96*200*200;

[0069] S53, forgetting layer DropOut processing, forgetting rate 0.2;

[0070] S54, convolution layer Conv2D processing, using 32 convolution kernels of size 3*3, activation function relu, then input 96*200*200 data, output 32*200*200 data;

[0071] S55, pooling layer MaxPooling2D processing, pooling size 4*4, then the input is 32*200*200, and the output is 32*50*50;

[0072] S56, forgetting layer DropOut processing, forgetting rate 0.25;

[0073] S57, Flatten layer processing, input is 32*50*50, output is 80000;

[0074] S58, linear fully connected Dense layer, input is 80,000 features, output is 2048 features;

[0075] These 2048 features are used as the encoding of one frame of image, and the feature encoding of 24 images is 24*2048 in total.

[0076] The emotions in step S2 are divided into 10 categories, as shown in Table 1:

[0077]

[0078]

[0079] Table 1

[0080] The security classification in step S8 is shown in Table 2:

[0081]

[0082] Table 2

[0083] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A safety-graded in-vehicle driver-passenger conflict identification method, characterized in that: The following steps are involved: S1. Collecting video and sound data in the car through the Internet of Vehicles equipment: Specifically, the Internet of Vehicles equipment is installed near the rearview mirror of the car. The Internet of Vehicles equipment integrates a camera and a recorder. It is turned on when the car is started. The video in the car is collected through the camera, and the sound data in the car is collected through the recorder. The collected video data and sound data are transmitted back to the online car-hailing platform in real time through the network; S2. Sound data samples and coding processing: emotions are divided into 10 categories, among which categories 1-4 are like, happiness, gratitude and neutral, and categories 5-10 are complaint, disgust, anger, fear, sadness and pain. One-hot encoding is performed on categories 1-10, that is, category 1 is encoded as 1, 0, 0, 0, 0, 0, 0, 0, 0, category 2 is encoded as 0, 1, 0, 0, 0, 0, 0, 0, 0, and so on. Category 10 is encoded as 0, 0, 0, 0, 0, 0, 0, 0, 1; S3. Build a machine learning emotion recognition model: (1) Convert the encoding vector Embedding into a word vector, set the vocabulary parameter to dic, and set the output dimension to embedding_dim, where embedding_dim = 128. Then each integer ID in (3, 10, 201, 28, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) is encoded as a 128-dimensional word vector. The text encoding vector of length MaxLen is converted into MaxLen word vectors of length embedding_dim. MaxLen word vectors are sentence vectors. A text and a label constitute a sample. The final encoding length of the sample is: MaxLen*embedding_dim+10. (2) Build an LSTM network: Set the input feature dimension input_dim to embedding_dim, the time step time_steps to MaxLen, the output layer feature dimension output_dim to 256, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by the LSTM is a feature vector rather than a group of feature vectors. The time_steps*embedding_dim features of the sample are the input of time_steps LSTM units, and each unit inputs embedding_dim features, that is, a word vector; (3) Add a Dense linear layer to the output of the LSTM network. The output of the LSTM network is a 256-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 10-dimensional vector, denoted as (z1, z2…z10). (4) Add a softmax activation function to the output of the dense linear layer, where softmax is a normalization function and the output is a 10-dimensional vector (y1, y2…y10). y1+y2+…y10=1, y1, y2,…y10 are the probabilities of category 1, category 2…category 10 respectively; S4. Training the machine learning emotion recognition model: (1) Setting the loss function: The collected text and annotations are input into the emotion recognition model, with the label value (p1, p2…p10) and the value calculated by the forward network each time being (y1, y2…y10). The loss function is set as: (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation; S5, video data samples and encoding processing: the historical video data is manually marked in two categories, the body movement contact is marked as 1, and the body non-movement contact is marked as 0. The sound sample s and the video sample v associated with s are defined as the video data when the sound data is collected at the same time. The video v is processed by image processing, 4 frames are selected at uniform intervals per second, and 6 seconds of data, a total of 24 frames of images, are selected as a video sample; S6. Build a machine learning action recognition model: (1) Build an LSTM network, set the input feature dimension input_dim to 2048, the time step time_steps to 24, the output layer feature dimension output_dim to 512, the number of hidden layers to 1, the epoch to 120, and whether to return a time series return_sequence to false to ensure that the last time series returned by LSTM is a feature vector rather than a group of feature vectors. The 24*2048 features of the sample are the input of 24 LSTM units, and each unit inputs 2048 features; (2) Add a Dense linear layer to the output of the LSTM network: The output of the LSTM network is a 512-dimensional vector, which is used as the input of the Dense linear layer. The output of the Dense linear layer is a 1-dimensional vector. (3) Add a sigmoid activation function to the output of the Dense linear layer: S7. Training the machine learning action recognition model: (1) Setting the loss function: the label value is p, the value calculated by the forward network each time is y, and the loss function is set to: binary_crossentropy=p*log(y)+(1-p)*log(1-y); (2) Adopting Adam algorithm to train the model: Adam optimization algorithm is an extension of stochastic gradient descent method, which is widely used in deep learning in computer vision and natural language processing. Among them, the relevant parameters of Adam are set as follows: learning rate alpha = 0.001, exponential decay rate of the first estimate beta1 = 0.9, exponential decay rate of the second estimate beta2 = 0.999, and the minimum parameter is a very small number eepsilon = 10E-8 to prevent division by zero in the implementation; S8. Make security classification based on emotion recognition and action recognition.

2. The method for identifying in-vehicle driver-passenger conflicts according to claim 1, characterized in that: The processing of the sound data samples in step S2 includes the following steps: S21. Collect complaints about disputes between drivers and passengers in the history of online ride-hailing platforms, retrieve the corresponding audio, manually determine the emotions into 5-10 categories, convert them into text, and manually mark and classify the text into 5-10 categories; S22, collecting audio corresponding to orders in which passengers gave drivers high ratings in the history of the online car-hailing platform, manually determining the emotions as categories 1-3, converting them into text, and manually marking and classifying the text into categories 1-3; S23, collecting the sounds corresponding to the unevaluated orders in the history of the online car-hailing platform, manually determining the emotions into 4 categories, converting them into text, and manually marking and classifying the text into 4 categories; Among them, the processing of the sound in steps S21-23 adopts the Speech 2 Text method, and the sound-to-text conversion is achieved through the API conversion interface provided by the public cloud service.

3. The method for identifying in-vehicle driver-passenger conflicts according to claim 1, characterized in that: After manually collecting and annotating data, n texts and corresponding annotations are collected. Jieba open source word segmenter is used for word segmentation. The frequency of words in the n texts is counted, and each word is converted into an integer ID according to the frequency. The ID of words with high frequency is smaller than that of words with low frequency. The integer ID starts from 1, and the maximum value is the number of different words, recorded as MaxWords. All different words are put into the vocabulary table dic.

4. The method for identifying in-vehicle driver-passenger conflicts with safety classification according to claim 1, characterized in that: In step S5, if each frame image is a 3*400*400 image, CNN is used to encode the image, which specifically includes the following steps: S51, convolution layer Conv2D processing, using 96 convolution kernels of size 3*3, activation function relu, then input 3*400*400 data, output 96*400*400 data; S52, pooling layer MaxPooling2D processing, pooling size 2*2, then the input is 96*400*400, and the output is 96*200*200; S53, forgetting layer DropOut processing, forgetting rate 0.2; S54, convolution layer Conv2D processing, using 32 convolution kernels of size 3*3, activation function relu, then input 96*200*200 data, output 32*200*200 data; S55, pooling layer MaxPooling2D processing, pooling size 4*4, then the input is 32*200*200, and the output is 32*50*50; S56, forgetting layer DropOut processing, forgetting rate 0.25; S57, Flatten layer processing, input is 32*50*50, output is 80000; S58, linear fully connected Dense layer, input is 80,000 features, output is 2048 features; These 2048 features are used as the encoding of one frame of image, and the feature encoding of 24 images is 24*2048 in total.

Images