A method, device, computer equipment and storage medium for controlling public transportation

Abstract

The embodiment of the present invention provides a method, device, computer equipment and storage medium for driving control of a public bus. The method includes: when the bus is traveling along a designated route, detecting the time a passenger stays in an area adjacent to a driver's seat; if the time exceeds a preset first threshold, collecting video data and first voice data of the passenger, collecting second voice data of the driver in the driver's seat, detecting the emotion expressed by the passenger in the video data and / or the first voice data, and if the emotion is negative, detecting the direction of the passenger's line of sight in the video data, matching the first voice data with the second voice data in natural language, and if the line of sight points to the driver and the first voice data matches the second voice data successfully, performing an alarm operation on the passenger and performing a driving control operation on the bus; based on the passenger's line of sight and the conversation between the passenger and the driver, the accuracy of detecting whether a conflict occurs between the passenger and the driver can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of public transportation, and in particular to a method, apparatus, computer equipment and storage medium for controlling the travel of a public transportation. Background Art

[0002] Buses are one of the public transportation means for users to travel. Buses are driven by specific drivers, and passengers come and go frequently. It is easy for them to have contradictions and conflicts with the drivers during stops, boarding, getting off, and carrying goods or pets, which will seriously affect the driving safety of the bus.

[0003] To this end, video surveillance is carried out on the bus to detect whether passengers have aggressive behavior towards the driver, so as to determine whether there is a conflict between the passenger and the driver. However, in the bus environment, the passengers are relatively dense, and the accuracy of detecting aggressive behavior through video is low, resulting in low accuracy in determining whether there is a conflict between the passenger and the driver. Therefore, the video is usually further reviewed manually, resulting in high costs for manual review. Summary of the invention

[0004] The embodiments of the present invention provide a method, apparatus, computer device and storage medium for controlling the operation of a public bus, so as to solve the problem of how to improve the accuracy of detecting whether a conflict occurs between a passenger and a driver, thereby reducing the cost of manual review.

[0005] In a first aspect, an embodiment of the present invention provides a method for controlling the travel of a public bus, comprising:

[0006] When the bus is traveling along a designated route, the time that the passenger stays in the area adjacent to the driver's seat is detected;

[0007] If the time exceeds a preset first threshold, collecting video data and first voice data of the passenger;

[0008] collecting second voice data from the driver in the driving seat;

[0009] detecting an emotion expressed by the passenger in the video data and / or the first voice data;

[0010] If the emotion is a negative emotion, detecting the sight direction of the passenger in the video data;

[0011] Performing natural language matching on the first voice data and the second voice data;

[0012] If the line of sight is directed toward the driver and the first voice data matches the second voice data successfully, an alarm operation is performed on the passenger and a driving control operation is performed on the bus.

[0013] Optionally, the video data has multiple frames of image data, and detecting the sight direction of the passenger in the video data includes:

[0014] Extracting the image data of the area where the eyeball of the passenger is located from the image data;

[0015] extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data;

[0016] Calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data;

[0017] The posture is converted into the viewing direction of the passenger.

[0018] Optionally, extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data includes:

[0019] Encoding the regional image data in a first encoder to obtain first feature image data;

[0020] encoding the first feature image data in a second encoder to obtain second feature image data;

[0021] Decoding the second feature image data in a first decoder to obtain third feature image data;

[0022] Decoding the third characteristic image data and the first characteristic image data in a second decoder to obtain fourth characteristic image data;

[0023] The fourth feature image data is processed in a residual network to obtain iris image data representing the iris and eyeball image data representing the eyeball.

[0024] Optionally, the calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data comprises:

[0025] Processing the iris image data and the eyeball image data in a first dense block to obtain first reference image data;

[0026] performing a first batch of normalization and a first pooling operation on the first reference image data;

[0027] Processing the first reference image data in a second dense block to obtain first reference image data;

[0028] performing a second batch of normalization and a second pooling operation on the second reference image data;

[0029] Processing the second reference image data in a third dense block to obtain second reference image data;

[0030] The second reference image data is processed in a fully connected layer to obtain a yaw angle and a pitch angle to represent the posture of the passenger's iris relative to the eyeball.

[0031] Optionally, converting the posture into the sight direction of the passenger includes:

[0032] If the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinates of the passenger's iris;

[0033] The yaw angle, the pitch angle and the center coordinates are combined to form the sight direction of the passenger.

[0034] Optionally, the matching the first voice data with the second voice data in natural language includes:

[0035] Performing speech recognition on the first speech data to obtain first text information;

[0036] Performing voice recognition on the second voice data to obtain second text information;

[0037] Using the first text information as a first sentence, a second sentence of a dialogue with the sentence in natural language;

[0038] identifying a similarity between the second sentence and the second text information;

[0039] If the similarity is greater than a preset second threshold, it is determined that the first voice data and the second voice data are matched successfully.

[0040] Optionally, the performing a driving control operation on the bus includes:

[0041] Acquire multiple trajectories recorded when the bus travels along the route as original trajectories;

[0042] Calculate an average value of the multiple original trajectories as a target trajectory;

[0043] Performing a positioning operation on the bus to record the current driving trajectory;

[0044] Calculating the degree to which the driving trajectory deviates from the target trajectory;

[0045] If the degree is greater than a preset third threshold, the double flashing lights of the bus are activated and the speed of the bus is reduced until it stops.

[0046] In a second aspect, an embodiment of the present invention further provides a bus driving control device, comprising:

[0047] A stay time detection module, used to detect the time a passenger stays in an area adjacent to the driver's seat when the bus travels along a designated route;

[0048] A passenger data collection module, configured to collect video data and first voice data of the passenger if the time exceeds a preset first threshold;

[0049] A driver data collection module, used for collecting second voice data of the driver in the driving position;

[0050] a passenger emotion detection module, configured to detect the emotion expressed by the passenger in the video data and / or the first voice data;

[0051] a sight line direction detection module, configured to detect the sight line direction of the passenger in the video data if the emotion is a negative emotion;

[0052] A voice matching module, used for matching the first voice data with the second voice data in natural language;

[0053] A control operation execution module is used to execute an alarm operation on the passenger and a driving control operation on the bus if the line of sight is directed toward the driver and the first voice data matches the second voice data successfully.

[0054] Optionally, the video data has multiple frames of image data, and the sight direction detection module is further used to:

[0055] Extracting the image data of the area where the eyeball of the passenger is located from the image data;

[0056] extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data;

[0057] Calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data;

[0058] The posture is converted into the viewing direction of the passenger.

[0059] Optionally, the sight direction detection module is further used for:

[0060] Encoding the regional image data in a first encoder to obtain first feature image data;

[0061] encoding the first feature image data in a second encoder to obtain second feature image data;

[0062] Decoding the second feature image data in a first decoder to obtain third feature image data;

[0063] Decoding the third characteristic image data and the first characteristic image data in a second decoder to obtain fourth characteristic image data;

[0064] The fourth feature image data is processed in a residual network to obtain iris image data representing the iris and eyeball image data representing the eyeball.

[0065] Optionally, the sight direction detection module is further used for:

[0066] Processing the iris image data and the eyeball image data in a first dense block to obtain first reference image data;

[0067] performing a first batch of normalization and a first pooling operation on the first reference image data;

[0068] Processing the first reference image data in a second dense block to obtain first reference image data;

[0069] performing a second batch of normalization and a second pooling operation on the second reference image data;

[0070] Processing the second reference image data in a third dense block to obtain second reference image data;

[0071] The second reference image data is processed in a fully connected layer to obtain a yaw angle and a pitch angle to represent the posture of the passenger's iris relative to the eyeball.

[0072] Optionally, the sight direction detection module is further used for:

[0073] If the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinates of the passenger's iris;

[0074] The yaw angle, the pitch angle and the center coordinates are combined to form the sight direction of the passenger.

[0075] Optionally, the voice matching module is further used for:

[0076] Performing speech recognition on the first speech data to obtain first text information;

[0077] Performing voice recognition on the second voice data to obtain second text information;

[0078] Using the first text information as a first sentence, a second sentence of a dialogue with the sentence in natural language;

[0079] identifying a similarity between the second sentence and the second text information;

[0080] If the similarity is greater than a preset second threshold, it is determined that the first voice data and the second voice data are matched successfully.

[0081] Optionally, the control operation execution module is further used to:

[0082] Acquire multiple trajectories recorded when the bus travels along the route as original trajectories;

[0083] Calculate an average value of the multiple original trajectories as a target trajectory;

[0084] Performing a positioning operation on the bus to record the current driving trajectory;

[0085] Calculating the degree to which the driving trajectory deviates from the target trajectory;

[0086] If the degree is greater than a preset third threshold, the double flashing lights of the bus are activated and the speed of the bus is reduced until it stops.

[0087] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0088] one or more processors;

[0089] a memory for storing one or more programs,

[0090] When the one or more programs are executed by the one or more processors, the one or more processors implement the bus driving control method as described in the first aspect.

[0091] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the method for controlling the driving of a public bus as described in the first aspect is implemented.

[0092] In this embodiment, when the bus travels along a designated route, the time a passenger stays in an area adjacent to the driver's seat is detected. If the time exceeds a preset first threshold, video data and first voice data are collected from the passenger, and second voice data is collected from the driver in the driver's seat. The emotion expressed by the passenger is detected in the video data and / or the first voice data. If the emotion is negative, the direction of the passenger's gaze is detected in the video data, and the first voice data is matched with the second voice data in natural language. If the gaze direction points to the driver and the first voice data matches the second voice data successfully, an alarm operation is performed on the passenger and a driving control operation is performed on the bus. The passengers are screened by the time they stay around the driver and the emotions they express. This can reduce the amount of calculation while ensuring the accuracy of detection. The direction of the passenger's gaze and the conversation between the passenger and the driver are used as the basis for strong correlation between the conflict, which can improve the accuracy of detecting whether a conflict occurs between the passenger and the driver, thereby reducing the number of manual reviews and the cost of manual reviews. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A flowchart of a method for controlling the operation of a public transport provided in the first embodiment of the present invention;

[0094] Figure 2 A schematic diagram of the structure of a network for extracting features provided in the first embodiment of the present invention;

[0095] Figure 3 is a schematic diagram of the structure of a network for detecting posture provided by Embodiment 1 of the present invention;

[0096] Figure 4 is a cross-sectional schematic diagram of an eyeball provided in Embodiment 1 of the present invention;

[0097] Figure 5 A schematic diagram of the structure of a bus driving control device provided in Embodiment 2 of the present invention;

[0098] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0099] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0100] Embodiment 1

[0101] Figure 1This is a flow chart of a method for controlling the operation of a public bus provided in the first embodiment of the present invention. This embodiment can be applied to the situation according to the sight of passengers, the emotions expressed, the language communication with the driver, etc. The method can be executed by a driving control device of a public bus. The driving control device of the public bus can be implemented by software and / or hardware and can be configured in a computer device. Specifically, it includes the following steps:

[0102] Step 101: When the bus travels along a designated route, the time a passenger stays in an area adjacent to the driver's seat is detected.

[0103] A bus (also called a bus or a public bus) is a motor vehicle that runs on a fixed route on a city road. It has multiple stops on the route, with or without fixed departure times, and has a dedicated route number (such as No. 960, No. 195A, No. 35, etc.). It stops at stops to let passengers get on and off, and carries passengers on their trips. It is generally square in shape, has windows, and is equipped with seats.

[0104] In urban areas, buses generally travel at speeds of 25-50 km / h, while in suburban areas, buses can travel at speeds of up to 80 km / h.

[0105] In this embodiment, the bus is assigned one or more route numbers, which usually do not change. Therefore, a relationship between the bus (indicated by a number or other identifier) ​​and the route number can be established in the server. After the bus starts traveling, it can query the server for the currently used route number, thereby querying the route corresponding to the route number as the route the bus is currently traveling.

[0106] One or more cameras can be deployed inside the bus. These cameras can be fixed-direction and can cover the entire environment inside the bus when running simultaneously. These cameras can also be equipped with a pan / tilt and can rotate, and the rotation range can cover the entire environment inside the bus.

[0107] After the bus starts, when it travels along the designated route, the camera can be activated to continuously collect video data of the environment inside the bus and record the actions of passengers on the bus.

[0108] In this embodiment, the coordinates of the driving position and the area adjacent to the driving position in the coordinate system of the camera can be pre-calibrated, wherein the driving position is the position where the driver controls the bus, and the area adjacent to the driving position is the position that may cause interference to the driver. For different models of buses, the area adjacent to the driving position is set differently, and this embodiment does not limit this.

[0109] When a passenger is detected in the bus, CenterNet or a network improved based on CenterNet can be used to calculate the passenger's position in the bus, and whether the passenger's position is located in the area adjacent to the driver's seat can be determined through geometric relationships. If the passenger is located in the area adjacent to the driver's seat, a timer can be started for the passenger to calculate the time the passenger stays in the area adjacent to the driver's seat.

[0110] Step 102: If the time exceeds a preset first threshold, video data and first voice data are collected from the passenger.

[0111] Under normal circumstances, the area adjacent to the driving seat is where passengers get on the bus. Generally, there are coin boxes, time clocks, temperature detectors and other equipment. After users insert coins, swipe bus cards and take their temperatures, they will walk into the interior of the bus to prevent congestion.

[0112] Therefore, if the bus compartment is not crowded, and the time the passenger stays in the area adjacent to the driver's seat exceeds a preset first threshold (such as 5 seconds), it means that the passenger stays in the area adjacent to the driver's seat for a long time, which is not normal. At this time, the camera can be called to collect video data of the passenger and the microphone can be called to collect first voice data of the passenger.

[0113] This embodiment uses the residence time in the area adjacent to the driver's seat as a condition for screening detection objects (passengers). While maintaining detection accuracy, it can greatly reduce the subsequent amount of calculations and reduce resource consumption. In addition, the passenger's video data and first voice data, in addition to being used for detection, can also be used as evidence to restore the event after determining that a conflict exists between the passenger and the driver.

[0114] Step 103: collect second voice data from the driver in the driving seat.

[0115] If a driver in the driving seat has a conflict with a passenger, the driver will usually respond to the passenger's words. Therefore, the microphone can be called to collect the second voice data of the driver.

[0116] In addition to being used for detection, the driver's second voice data can also be used as evidence to restore the event after it is determined that there is a conflict between the passenger and the driver.

[0117] Step 104: Detect emotions expressed by the passenger in the video data and / or the first voice data.

[0118] The emotions generated by bus passengers on the bus can be expressed through multiple dimensions. Therefore, in this embodiment, the emotions generated by bus passengers on the bus can be detected in real time through different dimensions in the video data and / or the first voice data.

[0119] In video data, classification algorithms such as CNN and PCNN can be used to identify the facial expressions and body movements of passengers.

[0120] In the first voice data, the voice information uttered by the passenger can be recognized by a voice recognition algorithm.

[0121] If the passenger's face shows one or more expressions of anger, sadness, fear, surprise, and disgust, and his or her limbs produce one or more movements of pulling, throwing, hitting, and kicking, and he or she makes aggressive or insulting voice messages, then it is determined that the passenger is expressing negative emotions on the bus.

[0122] In this embodiment, passengers expressing negative emotions on the bus are detected through facial expressions, body movements, and voice, thereby realizing multimodal feature extraction, thereby improving the accuracy of detecting passengers expressing emotions on the bus. In addition, the conditions of the detection objects (passengers) are further screened through the detection of emotions, and a smaller model can be used, with a faster detection speed. While maintaining the detection accuracy, the subsequent amount of calculation can be further reduced, thereby reducing resource consumption.

[0123] Step 105: If the emotion is negative, detect the passenger's sight direction in the video data.

[0124] There is less difference between a passenger's general actions and aggressive actions, and relying on actions to detect whether a passenger is aggressive towards the driver will result in large errors. In addition, there is less difference between a passenger's general language and aggressive language, and relying on language to detect whether a passenger is aggressive towards the driver will also result in large errors.

[0125] However, when a passenger has a conflict with the driver, his or her line of sight is mostly pointed toward the driver, which can be used as evidence that the passenger is aggressive toward the driver. Therefore, when a passenger stays in the area adjacent to the driver's seat for a long time and expresses negative emotions, there is a greater probability that a conflict will occur with the driver. At this time, the video data collected from the passenger can detect the direction of the passenger's line of sight.

[0126] In one embodiment of the present invention, step 105 may include the following steps:

[0127] Step 1051: extract the image data of the area where the passenger's eyeball is located from the image data.

[0128] In this embodiment, the passengers on the bus may be taken as the target and tracked in the video data by a target tracking algorithm, wherein the target tracking algorithm may include the following types:

[0129] 1. Classic tracking algorithm

[0130] Classical tracking algorithms are mainly based on target modeling or tracking target features.

[0131] 1.1. Modeling method based on target model

[0132] By modeling the target appearance model, the target is found in the subsequent frames. For example, region matching, feature point tracking, active contour-based tracking algorithm, optical flow method, etc. Among them, for the feature matching method, the target features are first extracted, and then the most similar features are found in the subsequent frames to locate the target. The target features include SIFT features, SURF features, Harris corner points, etc.

[0133] 1.2. Search-based methods

[0134] The method based on target modeling processes the entire image, which has poor real-time performance. Therefore, the prediction algorithm is added to the tracking, and the target search is performed near the predicted value, which reduces the search range. One type of prediction algorithm includes Kalman filtering and particle filtering methods. Another method to reduce the search range is the kernel method: using the principle of the steepest descent method, the target template is gradually iterated in the direction of gradient descent until it is iterated to the optimal position, such as Meanshift and Camshift algorithms.

[0135] 2. Tracking algorithm based on kernel correlation filtering

[0136] The correlation filtering (measurement of the similarity between two signals) in the communication field is introduced into target tracking, resulting in tracking algorithms based on correlation filtering, such as MOSSE, CSK, KCF, BACF, SAMF, etc., which can reach a speed of hundreds of frames per second and can be widely used in real-time tracking systems.

[0137] 3. Tracking algorithm based on deep learning

[0138] In the context of big data, the convolution feature output obtained by training the network model with deep learning has stronger expressive power. In target tracking, the initial application method is to directly apply the features learned by the network to the correlation filter or Struck tracking framework to obtain better tracking results, such as the DeepSRDCF method. In essence, the feature expression obtained by the convolution output is better than HOG or CN features.

[0139] There are multiple frames of image data in the video data. In the process of tracking passengers, face detection can be performed on these image data to extract the image data of the area where the passenger's eyeballs are located.

[0140] Among them, face detection is also called face key point detection, positioning or face alignment. It means locating the key areas of the face, including eyebrows, eyes, nose, mouth, facial contours, etc., given face data.

[0141] In this embodiment, face detection can be performed on the original image data in at least one of the following ways:

[0142] 1. Use artificially extracted features, such as Haar features, use the features to train classifiers, and use the classifiers for face detection.

[0143] 2. Inherit face detection from general object detection algorithms, for example, using Faster R-CNN to detect faces.

[0144] 3. Use cascaded convolutional neural networks, for example, Cascade CNN (Cascaded Convolutional Neural Networks), MTCNN (Multi-task Cascaded Convolutional Networks).

[0145] In a specific implementation, these methods for implementing face detection can be integrated into a module of an application, and the application can directly call the module to detect face data in the original image data. These methods for implementing face detection can also be integrated into an SDK (Software Development Kit). The SDK serves as assembly data for an application, and the application can request the SDK to perform face detection on image data. The SDK detects face data in the image data and returns the face data.

[0146] It should be noted that the number of facial key points in the facial data can be set according to actual conditions. For static image processing, the real-time requirement is relatively low, and relatively dense facial key points, such as 1000, can be detected. In addition to locating important feature points of the face, the contours of the facial features can also be accurately described. For multimedia data, etc., the real-time requirement is relatively high, and relatively sparse facial key points, such as 68, 81, and 106, can be detected to locate more obvious and important feature points on the face (such as eye key points, eyebrow key points, nose key points, mouth key points, contour key points, etc.) to reduce the processing volume, reduce processing time, etc. The embodiments of the present invention are not limited to this.

[0147] Step 1052: extract iris image data representing the iris and eyeball image data representing the eyeball from the regional image data.

[0148] The relationship between the iris and the eyeball can represent the user's line of sight direction. In this embodiment, iris image data representing the iris and eyeball image data representing the eyeball can be extracted from the regional image data.

[0149] In one extraction method, the regional image data can be first encoded (down-sampled) and then decoded (upsampling) to regress to a classification of pixels of the same size as the original image data (i.e., the regional image data), thereby combining the bottom-level and high-level information to extract iris image data representing the iris and eye image data representing the eyeball from the regional image data.

[0150] like Figure 2 As shown, during the encoding process:

[0151] The region image data is encoded in the first encoder 210 to obtain first feature image data.

[0152] The first feature image data is encoded in the second encoder 220 to obtain second feature image data.

[0153] Both the first encoder 210 and the second encoder 220 can be implemented as a multi-layer convolutional layer (Convolutional layer) 211, such as a three-layer convolutional layer 211, and the size of the output feature of each convolutional layer 211 is continuously reduced, so as to obtain the bottom layer (deep layer) information.

[0154] The underlying (deep) information can provide semantic information about the context of the segmented target in the entire region image data, which can be understood as a feature that reflects the relationship between the target and its environment. This feature helps to judge the category of the eyeball and iris.

[0155] like Figure 2 As shown, during the decoding process:

[0156] Decoding the second feature image data in the first decoder 230 to obtain third feature image data;

[0157] The third characteristic image data and the first characteristic image data are decoded in the second decoder 240 to obtain fourth characteristic image data.

[0158] Both the first decoder 230 and the second decoder 240 can be implemented as a multi-layer convolutional layer (Convolutional layer) 241, such as a 3-layer convolutional layer 241. The size of the output features of each convolutional layer 241 continues to increase, so as to obtain high-level (shallow) information, and the high-resolution information directly transmitted from the Encoder to the Decoder at the same height after the concatenate operation (i.e., adding the first feature image data for decoding) can provide more refined features for segmentation, such as gradients.

[0159] Afterwards, if Figure 2 As shown, a residual module 250 is added. The residual module 250 may be sequentially distributed with one BN (batch normalization) layer and multiple convolution layers (such as one 3×3 convolution layer and two 1×1 convolution layers). After passing through the last convolution layer, the features after the previous BN layer operation are added element by element as output. The residual module 250 takes into account that when extracting image features, some useful feature information may be lost in the subsequent output response value. Therefore, the original input information in each residual module is added with the information after convolution processing, so that the subsequent network can further extract high-order effective features and keep some useful information of the original input.

[0160] The fourth feature image data is processed in the residual network 250 and mapped into outputs of two channels, the output of one channel is iris image data 260 representing the iris, and the output of the other channel is eye image data 270 representing the eyeball, thereby obtaining iris image data 260 representing the iris and eye image data 270 representing the eyeball.

[0161] For the iris and eyeball, the imaging is of a fixed structure, not the surrounding environment, and usually has the following two characteristics:

[0162] 1. Because the boundaries of regional image data are fuzzy and the gradients are complex, more high-resolution information is required. High-resolution information is used to accurately segment the iris and eyeball.

[0163] 2. The exterior of the iris and eyeball is relatively fixed. The distribution of the segmented iris and eyeball in the regional image data is very regular, with simple and clear semantics. Low-resolution information can provide this information for the recognition of the iris and eyeball.

[0164] The above structure combines low-resolution information (providing a basis for classifying the iris and eyeballs) and high-resolution information (providing a basis for accurate segmentation and positioning), which can ensure accurate segmentation of the iris and eyeballs.

[0165] It should be noted that the iris image data representing the iris is a set of vectors (features) that multi-dimensionally represent the iris, and does not refer to the iris itself in the video data; the eye image data representing the eyeball is a set of vectors (features) that multi-dimensionally represent the eyeball, and does not refer to the eyeball itself in the video data.

[0166] Step 1053: Calculate the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data.

[0167] Obtaining the sight direction of the passenger from the iris image data and the eyeball image data is a regression problem. Therefore, the posture of the passenger's iris relative to the eyeball can be calculated based on the iris image data and the eyeball image data by regression.

[0168] In a regressive manner, such as Figure 3 As shown, the iris image data and the eyeball image data are processed in the first dense block 310 to obtain the first reference image data.

[0169] A first batch of normalization (BN) 311 and a first pooling operation (pooling) 312 are performed on the first reference image data.

[0170] The first reference image data is processed in the second dense block Dense Block 320 to obtain the first reference image data.

[0171] A second batch normalization BN 321 and a second pooling operation pooling 322 are performed on the second reference image data.

[0172] The second reference image data is processed in the third dense block Dense Block 330 to obtain the second reference image data.

[0173] Among them, the first dense block Dense Block 310, the second dense block Dense Block 320, and the third dense block Dense Block 330 all belong to dense blocks Dense Block. The dense block Dense Block has multiple convolution layers, such as 4 3×3 convolution layers. Each convolution specifies the output of a feature matrix of multiple (such as 8) channels for extracting features of iris image data and eyeball image data.

[0174] The input of each convolutional layer is a new feature matrix formed by channel-wise concatenation of the feature matrices output by all previous convolutional layers. This channel-wise concatenation method can directly weight the features extracted by each previous convolutional layer according to their effectiveness, thereby obtaining more efficient line of sight direction features.

[0175] The first pooling operation pooling 312 and the second pooling operation pooling 322 may be average pooling operations, which are used to reduce the matrix size of the output features to reduce the amount of calculation and to combine the extracted features.

[0176] The second reference image data is processed in the fully connected layers (FC) 340 to obtain the yaw angle and the pitch angle (two-dimensional vector) to represent the posture of the passenger's iris relative to the eyeball in the regional image data 350 (such as the direction of the arrow).

[0177] Step 1054: Convert the posture into the passenger's sight direction.

[0178] The image data of the area where the passenger's eyeball is located cannot actually show the outline of the eyeball. Only parts (sclera, iris, pupil, cornea, etc.) are displayed outside. The actual eyeball size may even be larger than the height size of the area image data. If the area image data is projected onto a two-dimensional plane according to the real size to obtain a projection image, the actual size of the eyeball will often exceed the edge of the area image data. Therefore, the pixel size of the projected image can be set to m×n, and the pixel size of the eyeball can be set to r. Each time the projection is performed, the height n of the projected image and the radius r of the eyeball have the following limiting relationship:

[0179] 2r=1.2n

[0180] When calculating the passenger's sight direction based on the regional image data, the set projection image takes into account the relative size and position of the eyeball and the iris, without caring about the size of the area where the actual eyeball is located in the regional image data. Therefore, the influence of the camera's translation and rotation deviation on the sight direction calculation is eliminated. No matter what changes occur in the appearance of the human eye in the regional image data, the position area of ​​the eyeball in the projection image remains unchanged, and the area of ​​the iris projection changes. Therefore, by measuring the iris, the change in the passenger's sight direction can be characterized.

[0181] In one embodiment, if the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinate of the passenger's iris. Specifically:

[0182] The rotation of the passenger's head posture can be represented by three quantities: pitch, yaw, and roll, which correspond to the pitch angle generated by rotating around the x-axis, the yaw angle generated by rotating around the y-axis, and the roll angle generated by rotating around the z-axis. Considering that the regional image data is two-dimensional image data and the center of the eyeball is fixed in the projected image, the pitch angle and yaw angle can be used to represent the three-dimensional posture of the passenger's line of sight relative to the camera imaging plane.

[0183] Assuming the pitch angle pitch = α and the yaw angle yaw = β, the coordinates O(u, v) of the center of the iris on the mapping plane are:

[0184]

[0185]

[0186] Among them, Figure 4 As shown, j represents the distance from the center of the eyeball to the center of the plane circle formed by the iris boundary inside the eyeball. Assuming that the radius of the iris is half of the radius r of the eyeball, the surface circle of the iris covering the surface of the eyeball in the projection image is ( Figure 4 The diameter of the eyeball center (in bold) is d = r, and the opening angle γ = d / r = 1 of the iris area can be obtained. At this time, the corresponding relationship between j and r is:

[0187] j = rcos (sin -1 (0.5))

[0188] The two-dimensional vector yaw angle β and pitch angle α can be used to represent the three-dimensional posture of the passenger's line of sight relative to the camera imaging plane. Considering that the line of sight is marked from the center of the pupil (iris), in order to facilitate the calculation of whether the line of sight is pointing to the driver, the present embodiment can combine the yaw angle, pitch angle and center coordinates to form the passenger's line of sight, and use the four-dimensional vector g = (u, v, β, α) to characterize the passenger's line of sight. It is simpler to construct the eyeball model and convenient for calculation. In addition, the four-dimensional vector g is more intuitive and contains more information, which is convenient for calculating whether the line of sight is pointing to the driver.

[0189] Step 106: Perform natural language matching on the first voice data and the second voice data.

[0190] The passenger's first voice data and the driver's second voice data can be matched in the dimension of natural language to determine whether the two constitute a dialogue in natural language.

[0191] In a specific implementation, voice recognition may be performed on the first voice data to obtain the first text information, and voice recognition may be performed on the second voice data to obtain the second text information.

[0192] Assuming there is a contradiction, the conversation between the passenger and the driver is relatively scattered, not in the form of question and answer, and direct comparison will have difficulties in aligning the time sequence.

[0193] Therefore, this embodiment can pre-train a chat robot, which is used to communicate with users when conflicts arise with users. To reduce the workload of training, a chat robot pre-trained by a third party can be used. On this basis, conversations between users when conflicts arise, especially conflicts between drivers and passengers, are added as training corpus to fine-tuning the chat robot.

[0194] Of course, in addition to fine-tuning, the chatbot can also be retrained directly using the complete corpus (including situations where conflicts occur between users, especially situations where conflicts occur between drivers and passengers, conversations between users, and normal conversations between users), and this embodiment does not impose any restrictions on this.

[0195] The first text information is used as the first sentence, that is, the first text information is divided into one or more complete sentences according to the semantic relationship, and the first sentence is input into the chat robot to obtain a second sentence that conducts a dialogue on the sentences through natural language.

[0196] The similarity between the second sentence and the second text information is identified through keyword matching (such as N-gram, Jaccard, etc.), vector space (such as Word2vec, TF-IDF, LSA, Euclidean distance, cosine distance, etc.), deep learning (such as DSSM, ConvNet, Tree-LSTM, etc.), etc.

[0197] If the similarity is greater than a preset second threshold, it means that the driver has a greater probability of having a conversation with the passenger in a conflicting situation, and it can be determined that the first voice data and the second voice data are successfully matched in natural language.

[0198] Step 107: If the line of sight is pointing to the driver and the first voice data matches the second voice data successfully, an alarm operation is performed on the passenger and a driving control operation is performed on the bus.

[0199] Since the driving seat is fixed, the driver can move within a limited range, which can be ignored within the normal range of vision. Therefore, the area where the driver is located can be marked in the camera's coordinate system in advance, without the need to detect the driver's area in real time in the video data, thereby reducing the amount of calculation.

[0200] At this time, it is possible to detect whether the passenger's line of sight is directed toward the driver's area. If the line of sight direction is represented by a four-dimensional vector g = (u, v, β, α), the center of the iris (u, v) can be used as a reference, and a ray can be extended along the posture of the iris relative to the eyeball (β, α) to determine whether the line of sight passes through the driver's area. If so, it is determined that the passenger's line of sight is directed toward the driver. Otherwise, it is determined that the passenger's line of sight is not directed toward the driver.

[0201] Under the two conditions that the passenger's line of sight is pointing to the driver and the first voice data successfully matches the second voice data, it can be considered that a conflict has occurred between the passenger and the driver. Of course, in order to ensure the accuracy of the detection, an audit task can be generated, and the first voice data (optionally with the first text information), the second voice data (optionally with the second text information), and the video data can be written into the audit task as evidence. The audit task is distributed to a client for manual review, and the back-end technicians log in to the client, retrieve the first voice data (optionally with the first text information), the second voice data (optionally with the second text information), and the video data for review to determine whether a conflict has occurred between the passenger and the driver.

[0202] When it is clear that there is a conflict between a passenger and the driver, in order to ensure the safety of the bus, on the one hand, an alarm operation can be performed on the passenger, for example, an alarm message can be issued through the bus's loudspeaker, etc. On the other hand, when necessary, a driving control operation can be performed on the bus, that is, the driving of the bus can be controlled.

[0203] In a specific implementation, a bus route is generally a sequence formed by multiple stops arranged in order. For example, if the bus route is stop A → stop B → stop C..., it means that the bus first arrives at stop A, then stops B, and finally stops C. It can be seen that the bus route is not the actual route of the bus.

[0204] Therefore, multiple trajectories recorded when the bus travels along the route can be obtained from the historical record database as original trajectories, wherein each original trajectory can be represented by multiple discrete positions, or can also be represented by a polynomial function, which is not limited in this embodiment.

[0205] The average value of multiple original trajectories is calculated as the target trajectory, which is the trajectory with higher confidence for the bus to travel on this route, to facilitate subsequent calculations.

[0206] The satellite positioning module is called to perform positioning operations on the bus to record the current driving trajectory.

[0207] The degree to which the driving trajectory deviates from the target trajectory can be calculated in real time based on shape distances (such as Hausdorff Distance, Frechet Distance, etc.), segmented distances (such as One Way Distance (OWD), Locality In-between Polylines (LIP)), etc.).

[0208] The bus routes are fixed, and the road conditions on these routes are relatively stable. If the driver drives the bus normally, the current driving trajectory will follow the target trajectory in principle.

[0209] If the degree is greater than the preset third threshold, it means that the bus has deviated from the normal track, and the following driving control operations can be performed:

[0210] 1. Start the bus's double flashing lights

[0211] Activating the bus's double flashing lights can alert surrounding vehicles and pedestrians that an emergency has occurred on the bus, allowing them to make timely emergency avoidance maneuvers.

[0212] 2. Reduce the speed of the bus until it stops.

[0213] By reducing the speed of the bus until it stops, that is, the speed is 0, abnormal driving behavior can be eliminated when passengers affect the driver's driving of the bus, major traffic accidents can be avoided, and the safety of surrounding vehicles and pedestrians and the safety of people on the bus can be guaranteed.

[0214] In this embodiment, when the bus travels along a designated route, the time a passenger stays in an area adjacent to the driver's seat is detected. If the time exceeds a preset first threshold, video data and first voice data are collected from the passenger, and second voice data is collected from the driver in the driver's seat. The emotion expressed by the passenger is detected in the video data and / or the first voice data. If the emotion is negative, the direction of the passenger's gaze is detected in the video data, and the first voice data is matched with the second voice data in natural language. If the gaze direction points to the driver and the first voice data matches the second voice data successfully, an alarm operation is performed on the passenger and a driving control operation is performed on the bus. The passengers are screened by the time they stay around the driver and the emotions they express. This can reduce the amount of calculation while ensuring the accuracy of detection. The direction of the passenger's gaze and the conversation between the passenger and the driver are used as the basis for strong correlation between the conflict, which can improve the accuracy of detecting whether a conflict occurs between the passenger and the driver, thereby reducing the number of manual reviews and the cost of manual reviews.

[0215] It should be noted that, for the sake of simplicity, the method embodiments are described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0216] Embodiment 2

[0217] Figure 5 The structural block diagram of a bus driving control device provided in the second embodiment of the present invention may specifically include the following modules:

[0218] The stay time detection module 501 is used to detect the time that the passenger stays in the area adjacent to the driver's seat when the bus travels along the designated route;

[0219] A passenger data collection module 502, configured to collect video data and first voice data of the passenger if the time exceeds a preset first threshold;

[0220] A driver data collection module 503 is used to collect second voice data from the driver in the driving position;

[0221] A passenger emotion detection module 504, configured to detect the emotion expressed by the passenger in the video data and / or the first voice data;

[0222] A sight line direction detection module 505, configured to detect the sight line direction of the passenger in the video data if the emotion is a negative emotion;

[0223] A voice matching module 506, configured to perform natural language matching on the first voice data and the second voice data;

[0224] The control operation execution module 507 is used to execute an alarm operation on the passenger and a driving control operation on the bus if the line of sight is directed toward the driver and the first voice data matches the second voice data successfully.

[0225] In one embodiment of the present invention, the video data has multiple frames of image data, and the sight direction detection module 505 is further used to:

[0226] Extracting the image data of the area where the eyeball of the passenger is located from the image data;

[0227] extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data;

[0228] Calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data;

[0229] The posture is converted into the viewing direction of the passenger.

[0230] In one embodiment of the present invention, the sight direction detection module 505 is further used for:

[0231] Encoding the regional image data in a first encoder to obtain first feature image data;

[0232] encoding the first feature image data in a second encoder to obtain second feature image data;

[0233] Decoding the second feature image data in a first decoder to obtain third feature image data;

[0234] Decoding the third characteristic image data and the first characteristic image data in a second decoder to obtain fourth characteristic image data;

[0235] The fourth feature image data is processed in a residual network to obtain iris image data representing the iris and eyeball image data representing the eyeball.

[0236] In one embodiment of the present invention, the sight direction detection module 505 is further used for:

[0237] Processing the iris image data and the eyeball image data in a first dense block to obtain first reference image data;

[0238] performing a first batch of normalization and a first pooling operation on the first reference image data;

[0239] Processing the first reference image data in a second dense block to obtain first reference image data;

[0240] performing a second batch of normalization and a second pooling operation on the second reference image data;

[0241] Processing the second reference image data in a third dense block to obtain second reference image data;

[0242] The second reference image data is processed in a fully connected layer to obtain a yaw angle and a pitch angle to represent the posture of the passenger's iris relative to the eyeball.

[0243] In one embodiment of the present invention, the sight direction detection module 505 is further used for:

[0244] If the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinates of the passenger's iris;

[0245] The yaw angle, the pitch angle and the center coordinates are combined to form the sight direction of the passenger.

[0246] In one embodiment of the present invention, the voice matching module 506 is further used for:

[0247] Performing speech recognition on the first speech data to obtain first text information;

[0248] Performing voice recognition on the second voice data to obtain second text information;

[0249] Using the first text information as a first sentence, a second sentence of a dialogue with the sentence in natural language;

[0250] identifying a similarity between the second sentence and the second text information;

[0251] If the similarity is greater than a preset second threshold, it is determined that the first voice data and the second voice data are matched successfully.

[0252] In one embodiment of the present invention, the control operation execution module 507 is further used to:

[0253] Acquire multiple trajectories recorded when the bus travels along the route as original trajectories;

[0254] Calculate an average value of the multiple original trajectories as a target trajectory;

[0255] Performing a positioning operation on the bus to record the current driving trajectory;

[0256] Calculating the degree to which the driving trajectory deviates from the target trajectory;

[0257] If the degree is greater than a preset third threshold, the double flashing lights of the bus are activated and the speed of the bus is reduced until it stops.

[0258] The bus driving control device provided in the embodiment of the present invention can execute the bus driving control method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0259] Embodiment 3

[0260] Figure 6 A schematic diagram of the structure of a computer device provided in Embodiment 3 of the present invention. Figure 6 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Figure 6The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0261] like Figure 6 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0262] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0263] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0264] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Figure 6 not shown, usually called a "hard drive"). Although Figure 6 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0265] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in the memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0266] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), may communicate with one or more devices that enable a user to interact with the computer device 12, and / or may communicate with any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0267] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, such as implementing the bus driving control method provided in the embodiment of the present invention.

[0268] Embodiment 4

[0269] Embodiment 4 of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned bus driving control method are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0270] Among them, computer-readable storage media may include, for example, but are not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0271] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for controlling the travel of a public transport, characterized in that: include: When the bus is traveling along a designated route, the time that the passenger stays in the area adjacent to the driver's seat is detected; If the time exceeds a preset first threshold, collecting video data and first voice data of the passenger; collecting second voice data from the driver in the driving seat; detecting an emotion expressed by the passenger in the video data and / or the first voice data; If the emotion is a negative emotion, detecting the sight direction of the passenger in the video data; Performing natural language matching on the first voice data and the second voice data; If the sight line is pointing to the driver and the first voice data matches the second voice data successfully, an alarm operation is performed on the passenger and a driving control operation is performed on the bus; The video data has multiple frames of image data, and detecting the sight direction of the passenger in the video data includes: Extracting the image data of the area where the eyeball of the passenger is located from the image data; extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data; Calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data; Converting the posture into the sight direction of the passenger; The converting the posture into the sight direction of the passenger comprises: If the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinates of the passenger's iris; The yaw angle, the pitch angle and the center coordinates are combined to form the sight direction of the passenger.

2. The method according to claim 1, characterized in that The step of extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data comprises: Encoding the regional image data in a first encoder to obtain first feature image data; encoding the first feature image data in a second encoder to obtain second feature image data; Decoding the second feature image data in a first decoder to obtain third feature image data; Decoding the third characteristic image data and the first characteristic image data in a second decoder to obtain fourth characteristic image data; The fourth feature image data is processed in a residual network to obtain iris image data representing the iris and eyeball image data representing the eyeball.

3. The method according to claim 1, characterized in that: The step of calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data comprises: Processing the iris image data and the eyeball image data in a first dense block to obtain first reference image data; performing a first batch of normalization and a first pooling operation on the first reference image data; Processing the first reference image data in a second dense block to obtain second reference image data; performing a second batch of normalization and a second pooling operation on the second reference image data; Processing the second reference image data in a third dense block to obtain second reference image data; The second reference image data is processed in a fully connected layer to obtain a yaw angle and a pitch angle to represent the posture of the passenger's iris relative to the eyeball.

4. The method according to any one of claims 1 to 3, characterized in that The matching of the first voice data with the second voice data in natural language includes: Performing speech recognition on the first speech data to obtain first text information; Performing voice recognition on the second voice data to obtain second text information; Using the first text information as a first sentence, a second sentence of a dialogue with the sentence in natural language; identifying a similarity between the second sentence and the second text information; If the similarity is greater than a preset second threshold, it is determined that the first voice data and the second voice data are matched successfully.

5. The method according to any one of claims 1 to 3, characterized in that: The performing driving control operation on the bus includes: Acquire multiple trajectories recorded when the bus travels along the route as original trajectories; Calculate an average value of the multiple original trajectories as a target trajectory; Performing a positioning operation on the bus to record the current driving trajectory; Calculating the degree to which the driving trajectory deviates from the target trajectory; If the degree is greater than a preset third threshold, the double flashing lights of the bus are activated and the speed of the bus is reduced until it stops.

6. A bus driving control device, characterized in that: include: A stay time detection module, used to detect the time a passenger stays in an area adjacent to the driver's seat when the bus travels along a designated route; A passenger data collection module, configured to collect video data and first voice data of the passenger if the time exceeds a preset first threshold; A driver data collection module, used for collecting second voice data of the driver in the driving position; a passenger emotion detection module, configured to detect the emotion expressed by the passenger in the video data and / or the first voice data; a sight line direction detection module, configured to detect the sight line direction of the passenger in the video data if the emotion is a negative emotion; A voice matching module, used for matching the first voice data with the second voice data in natural language; a control operation execution module, configured to execute an alarm operation on the passenger and a driving control operation on the bus if the line of sight points to the driver and the first voice data matches the second voice data successfully; The video data has multiple frames of image data, and the sight direction detection module is further used for: Extracting the image data of the area where the eyeball of the passenger is located from the image data; extracting iris image data representing the iris and eyeball image data representing the eyeball from the regional image data; Calculating the posture of the passenger's iris relative to the eyeball according to the iris image data and the eyeball image data; Converting the posture into the sight direction of the passenger; The sight direction detection module is also used for: If the posture includes a yaw angle and a pitch angle, the yaw angle and the pitch angle are used to calculate the center coordinates of the passenger's iris; The yaw angle, the pitch angle and the center coordinates are combined to form the sight direction of the passenger.

7. A computer device, characterized in that: The computer device comprises: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the bus driving control method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for controlling the travel of a public bus as described in any one of claims 1 to 5 is implemented.

Images