An interactive auxiliary safety driving system

Abstract

An interactive assisted safe driving system relates to the field of safe driving technology. It aims to solve the problem that the existing technology cannot detect the speed of vehicles in front and behind in real time and perceive dangerous situations in time, thus leading to the occurrence of traffic accidents. The present application provides a pedestrian detection algorithm (AFCNN) for new complex scenarios, which effectively improves the detection accuracy of pedestrians in congested and blocked scenarios, thereby further reducing the risk of pedestrian collisions; the present application proposes a road surface disease information detection algorithm (SI‑SSD), which reduces the probability of traffic accidents caused by poor road conditions; in view of the fact that vehicle collision detection algorithms in common products are difficult to cope with problems such as icy and snowy weather, a function of real-time detection of the driving speed of vehicles in front and behind the driver is introduced, which is convenient for the driver to adjust the speed in time, further reducing the risk of traffic accidents.

Description

Technical Field

[0001] The present invention relates to the technical field of safe driving, and in particular to an interactive auxiliary safe driving system. Background Art

[0002] With the advancement of science and technology and the development of society, people own more and more cars. As the number of cars continues to increase, the traffic environment has become more congested, and the risk of traffic accidents for drivers while driving has also increased. The main reason is that the scene outside the car cannot be effectively perceived. This problem seriously affects the driving safety of drivers and the life safety of pedestrians. Therefore, how to more effectively protect the lives of drivers and pedestrians has become an urgent problem to be solved.

[0003] At present, common safe driving systems are mainly based on AI intelligent systems to process data, which effectively improves driving safety by making drivers aware of possible dangers in advance. Generally, they include the following functions: vehicle-to-vehicle close-distance alarm and pedestrian collision alarm, etc.

[0004] The design idea of ​​the pedestrian collision alarm module is to give a corresponding voice or visual display alarm prompt when it is detected that the distance between the driving vehicle and the pedestrian is compared with the safe distance and there is a potential risk. However, in order to ensure the safety of pedestrians in a complex and congested environment (i.e. high-precision pedestrian collision perception), there are extremely high requirements for the accuracy of pedestrian detection. In actual scenarios, pedestrians often block each other or are blocked by stationary objects when crossing the road. Traditional pedestrian detection algorithms alone do not have high reliability and still pose potential dangers.

[0005] In addition, accurate detection of road surface information ahead of driving is also an important part. Since it is difficult for drivers to ensure smooth driving when the road surface is poorly flat, the risk of traffic accidents will increase. At present, common methods for detecting road surface flatness mainly include detection by special vehicles equipped with professional detection equipment and detection based on deep learning. Most of the common products on the market today do not have road condition detection modules, and the high detection cost, low detection frequency and low real-time performance of traditional road surface detection instruments have led to limitations in practical applications of this method. The idea of ​​road surface flatness detection based on deep learning is: first, collect vibration data information during driving, set effective acceleration indicators, and then use a clustering framework plus data mining methods to more accurately detect road surface flatness while reducing costs.

[0006] For the vehicle-to-vehicle distance alarm module, the common market products currently mainly detect and calculate the distance between the driving vehicle and the vehicle in front, and compare it with the preset safety distance value. When the distance is lower than the preset safety distance threshold, the alarm system will trigger an alarm prompt. However, in actual driving, if the speed difference between the driving vehicle and the vehicle in front is large or in icy and snowy weather, even if the distance to the vehicle in front is large, it may not be possible to brake in time, resulting in a traffic accident.

[0007] Due to the poor comprehensiveness of existing technologies, they cannot effectively protect the life safety of drivers and reduce the probability of traffic accidents. Summary of the invention

[0008] The purpose of the present invention is to propose an interactive auxiliary safe driving system to address the problem that the prior art is poorly comprehensive, difficult to effectively protect the driver's life safety, and reduce the probability of traffic accidents.

[0009] The technical solution adopted by the present invention to solve the above technical problems is:

[0010] An interactive auxiliary safe driving system, including a pedestrian detection module in complex congestion scenes, a road surface disease information detection module, a driving surrounding traffic sign detection and recognition module, and a driving front and rear vehicle speed detection and front vehicle collision alarm module;

[0011] The pedestrian detection module in the complex congestion scenario is used to detect pedestrians in front of the vehicle, and when the distance between the pedestrian and the vehicle is less than the safety threshold, it will provide voice feedback to the driver;

[0012] The road surface damage information detection module is used to detect road surface damage information and feed back the detected information to the driver through voice;

[0013] The driving surrounding traffic sign detection and recognition module is used to detect and recognize the traffic signs in front of the vehicle, and feed back the detection and recognition results to the driver through voice;

[0014] The front and rear vehicle speed detection and front vehicle collision warning module is used to detect the distance between the front and rear vehicles and the driving vehicle and the speed of the front and rear vehicles. If the distance between the front and rear vehicles and the driving vehicle is less than the safety distance threshold or the speed of the front and rear vehicles is greater than the speed threshold, the driver is fed back through voice;

[0015] The pedestrian detection module in the complex congestion scene is implemented by a pedestrian detection algorithm, and the pedestrian detection algorithm specifically includes the following steps:

[0016] Step 1: Obtain the video stream file, and then extract the video stream file frame by frame. For each frame of the extracted image, use the non-maximum suppression algorithm to identify whether there is occlusion and occluded objects, and define the occluded object as active and the occluded object as inactive;

[0017] Step 2: Before regressing the inactive and active state trajectories, assume that the pedestrian moves at a constant speed, and use the MM model to predict the positions of the occluded and occluded object trajectories in the next frame, and obtain the speeds of the occluded and occluded objects by calculating the average displacement of the centers of the first n bounding boxes, specifically:

[0018]

[0019] Among them, S i represents the displacement of the center of the i-th bounding box, t l Indicates the time interval between two adjacent frames of images;

[0020] Step 3: Use the CMC model to get the camera motion speed, let P D is the center position of the blocked pedestrian, and its coordinates are (x D ,y D ), let the estimated position P of the inactive state trajectory of the blocked pedestrian be L The coordinates are (x L ,y L ), when the difference between the two coordinates is lower than the set threshold, the occluded object will activate the inactive state, that is, enter the active state, specifically:

[0021] ||P D -P L ||≤D max

[0022]

[0023] Among them, D max Follow inactive tracks The velocity and the time t that the trajectory has been inactive inactive increases with the increase of γ, which indicates the control D max Numerical parameters;

[0024] Step 4: Obtain the camera motion speed based on the CMC model and then obtain the detection marker frame. Then, extract any two tracklets from the motion trajectory of the active object in step 3. If the two tracklets completely overlap, that is, the IoU value in each frame where both tracklets have detection marker frames is higher than the threshold, then the two belong to the same target. If the two tracklets partially overlap, that is, there are frames without overlap, and the IoU value is lower than the threshold, then only the detection marker frame corresponding to the longest tracklet in each frame is retained.

[0025] Step 5: When the overlap between the detection marker box and the object entering the active state decreases, the regression score S will decrease accordingly. When the regression score S is less than the threshold S min When the boundary of the detection mark box of the object entering the active state coincides with the boundary of the image, the two-dimensional velocity vector of the motion trajectory of the detection mark box is calculated if If the direction points to the image boundary, the detection mark box is deleted from the motion trajectory, otherwise, the detection mark box is retained;

[0026] Step 6: Extract 17 key points of the tracked pedestrian through the posture estimation network. The tracked pedestrian is the object entering the active state, and construct a response map based on the 17 key points as data input, and calculate the attention weight of each key point of the tracked pedestrian. k , specifically:

[0027] attention k =average_pool(relu(conv(K)))

[0028]

[0029] Among them, average_pool represents the global average pool, which is used to encode the features of each channel into the corresponding global features, attention k Represents the attention weight of the key points of the tracked pedestrian, attention k ∈R C×1×1 , conv represents a 3×3 convolution kernel, relu represents an activation function, and f represents the extracted pedestrian key point features. Represents the feature value after NMS suppression of the occluded part;

[0030] Step 7: First, a mask is constructed for the visible part of the tracked pedestrian, that is, the visible part is marked as 1 and the invisible part is marked as 0. Finally, the feature map of the visible part of the tracked pedestrian is constructed. It is expressed as:

[0031]

[0032] Among them, ⊙ represents the multiplication operation according to the corresponding position;

[0033] Step 8: Let the tracked pedestrian be T, let the set of all candidate marker boxes corresponding to the tracked pedestrian be C, and after parameter adjustment, calculate the value M of each element in the set C by the following formula:

[0034] M=3*Err cls +Err reg

[0035] According to the value M of each element in the set C and combined with the following formula, the smallest element T is selected end :

[0036]

[0037] Among them, Err cls With Err reg They refer to the error and regression error between the predicted label box and the true label box respectively;

[0038] Step 9: Take the smallest element T end The detection mark boxes outside the corresponding detection mark boxes are suppressed, that is, normalized, and expressed as:

[0039]

[0040]

[0041] grade=α*Dis cls +βDis reg

[0042] Dis cls =y i *log(σ(p(y i )))+(1-y i )*log(1-σ(p(y i )))

[0043]

[0044] Among them, Dis cls Indicates the gap between the candidate detection box sample and the real box category, y i ∈{0, 1}, σ represents the Sigmoid function, which is used to map values ​​to (0, 1), Dis reg Represents the gap between the candidate detection box sample and the real box regression, Z x and Z yRespectively represent the center point coordinates of the candidate detection box, V x and V y Respectively represent the center point coordinates of the real box, W Z and H Z Respectively represent the width and height of the candidate detection box;

[0045] Step 10: Set the loss function L;

[0046] Step 11: Train and test the entire model according to steps 1 to 10, and calculate various model evaluation indicators.

[0047] Furthermore, the loss function L includes a center point loss, a pedestrian loss, an offset loss, a regional pixel learning loss, and a regional fine-grained segmentation learning loss;

[0048] L=λ1L center +λ2L scale +λ3L offset +λ4L pixel +λ5L f

[0049] Among them, L center Indicates the loss at the center point, L scale represents pedestrian loss, L offset Indicates the offset loss, L pixel represents the region pixel learning loss, L f represents the regional fine-grained segmentation learning loss, and the control coefficient parameters λ1~λ5 are set to 0.02, 0.005, 0.1, 0.0002, and 0.0001 respectively.

[0050] Furthermore, the center point loss is defined as follows:

[0051] Center point L center Including the center point of the pedestrian's body, the center point of the pedestrian's head and the center point of the visible part of the pedestrian. There are 3 real image output channels corresponding to these three center points. The value of the center point in channel 1 is set to 1, and the center point map values ​​of the other positions are generated according to the 2D Gaussian function. The value of the pedestrian center point in channel 2 is set to 1, and the values ​​of the other positions are set to 0; the value of the pedestrian detection box area in channel 3 is assigned to 1, and the values ​​of the other areas are set to 0. When the real image of the center point already has a value, the larger value between the existing value and the current value is taken. The center point loss is expressed as:

[0052]

[0053]

[0054]

[0055]

[0056] Among them, N represents the number of people in each picture, (x a ,y a , σ wa , σ ha ) represents the coordinates of the pedestrian. If there are multiple Gaussian values ​​in a certain coordinate value, the maximum value max is taken, σ w represents the Gaussian value of the pedestrian's corresponding width, σ h represents the Gaussian value of the pedestrian's height, r ij represents the treatment confidence value of (i, j) in the prediction graph, Indicates different corresponding y ij The predicted value under In actual execution, α is set to 4, β is set to 2, and y ij represents the value at the pedestrian center point, β ij Represents the coefficient parameters under different ij.

[0057] Furthermore, the pedestrian loss is defined as follows:

[0058] For pedestrian loss, based on the L1Loss loss function, it includes the loss of width and height respectively.

[0059] L scale =L height +L width

[0060]

[0061]

[0062] Among them, s a represents the logarithm of the true value of the pedestrian width, t a Indicates the logarithm of the predicted value of pedestrian width, s b Indicates the logarithm of the true value of the pedestrian height, t b It represents the logarithm of the predicted value of the pedestrian height, a=1...N, b=1...N.

[0063] Furthermore, the offset loss is defined as follows:

[0064] The offset loss function is based on the SmoothL1 loss function and is expressed as:

[0065]

[0066] Among them, k and k1 Represent the true value and predicted value of the offset respectively.

[0067] Furthermore, the regional pixel learning loss is defined as follows:

[0068] Let A be the number of all pixels in the pedestrian area, m t , m p They represent the pixel values ​​of the real area where the pedestrian is located and the pixel values ​​of the area where the pedestrian is located predicted by the model. The regional pixel learning loss is expressed as:

[0069]

[0070] Furthermore, the regional fine-grained segmentation learning loss is defined as follows:

[0071]

[0072] Among them, n t , n p They respectively represent the fine-grained segmentation image of the actual area where the pedestrian is located and the fine-grained segmentation image of the area where the pedestrian is located predicted by the model.

[0073] Furthermore, the pavement damage information detection module is implemented by a pavement damage information detection algorithm, and the pavement damage information detection algorithm specifically includes the following steps:

[0074] Step 1: Obtain a road surface image and perform feature extraction on the road surface image. Then, perform Sobel edge detection based on the extracted features. Specifically:

[0075]

[0076]

[0077]

[0078]

[0079] Among them, g x Represents the horizontal gradient, g y Represents the vertical gradient, g x ′ and g y ′ represents the diagonal gradient, z i (i∈[1,9]) represents the value displayed at each coordinate in the 3×3 Sobel operator template;

[0080] Step 2: Replace VGG16 in the SSD model with the Inception network structure, and introduce the SE-Net attention mechanism between Conv6-Conv12;

[0081] Step 3: Introduce the loss function, which includes the position loss αLloc (x, l, g) and classification loss L conf (x, c), the total loss is:

[0082]

[0083] Among them, x and c represent the indication amount and the set confidence respectively, l and g represent the prediction box and the background box respectively;

[0084] Step 3.1: Calculate the position loss. The position loss refers to the loss between the predicted detection frame and the real detection frame, expressed as:

[0085]

[0086]

[0087]

[0088]

[0089]

[0090] Among them, L loc Represents the sub-variable in the position loss, Pos represents the area covered by the detection box, represents the prediction box represented by the i-th sample in the detection box, represents the intermediate variable, Represents the background box corresponding to the jth sample in the central horizontal coordinate, Represents the background box corresponding to the jth sample in the central ordinate, Indicates the network's own predetermined binding box corresponding to the i-th sample in the central horizontal coordinate, Indicates the network's own predetermined binding box corresponding to the i-th sample in the central ordinate, represents the bounding box of the network itself with width w in the i-th sample, represents the network's own predetermined binding box that highly represents w in the i-th sample, represents the background box with width w in the jth sample, and h represents the corresponding height of the pedestrian;

[0091] Step 3.2: Calculate the classification loss, which is the Softmax loss, specifically:

[0092]

[0093]

[0094] Among them, N represents the number of samples, x and c represent the indication amount and the confidence level of the setting respectively, l and g represent the prediction box and the background box respectively, α represents the weight of the position box loss value, and C x , C y , w, h represent the center coordinates (x, y) of the marker box and the height and width of the marker box respectively, d represents the marker box predicted by the network itself, represents the jth indicator under the i-th sample, represents the intermediate variable, Neg represents the regression area of ​​the sample, represents the confidence of the i-th sample, and p represents the calculation coefficient;

[0095] Step 4: Set the batch size to 20, the image size to 400×400, the learning rate to 0.005, the number of iterations to 5000, use the L2 regularization standard to train the model, and use the test set to test the model performance.

[0096] Furthermore, the front and rear vehicle speed detection and front vehicle collision warning module specifically performs the following steps:

[0097] Step 1: Define car A as the vehicle driven by the driver. Car A has an external camera O2 in front of it, f is the focal length of the camera, car B is the vehicle in front during driving, M is the rear of car B, P is the horizontal position of the center point of car B, H is the height of the camera on the car, d1 is the distance between car A and car B, O3M is the distance between the point of car B's parking space vertically projected onto the ground and the center line of the image in the image, and the distance O3P between the center points of the two cars can be calculated based on the geometric relationship, specifically:

[0098]

[0099]

[0100] Among them, α1 represents the angle between O2 and the center point of car B and the horizontal line, and γ1 represents the angle between O2 and M and the horizontal line;

[0101] According to the above formula, the coordinate O2P in the vertical direction Y and the coordinate PQ in the vertical direction X are expressed as:

[0102]

[0103]

[0104]

[0105] Among them, Q represents the mapping point of the intersection of O2 with the horizontal plane at an angle of α1 on the x-axis, β1 represents the angle between O2 and P and the horizontal plane, P1 represents the intersection of the extended line of O2P and the y-axis, O1 represents the intersection of the extended line of the line connecting O2 and M and the y-axis, and x pix represents the test time, v represents the speed of vehicle A, and v center represents the speed of vehicle B;

[0106] Step 2: Assume that the height of the vehicle to be tested on the imaging plane at different times is u i (i=0,1,2...), let the relative distances between the two vehicles at two different times be d r1 and d r2 , the relative distance between two different moments can be calculated according to the following formula:

[0107]

[0108] Among them, KH represents the actual height of the car, α0 represents the control angle. According to the above formula, the relative distance between the two cars changes as follows:

[0109]

[0110] According to the above formula, when l′ is less than 0, it means that the two cars are gradually approaching each other; when l′ is greater than 0, the two cars are gradually moving away from each other; when the value of |l′| approaches 1 infinitely, the change in the relative distance between the two cars increases infinitely; when the value of |l′| approaches 0 infinitely, the change in the relative distance between the two cars decreases infinitely;

[0111] Step 3: The relative speed between vehicles is expressed as:

[0112]

[0113] Wherein, ΔT represents the time interval between two adjacent frames of detected images, and i=1...n.

[0114] Furthermore, the system also includes: an off-duty inspection and identity authentication module, a fatigue state detection and emotional state detection module, and a dangerous driving behavior detection module;

[0115] The off-post inspection and identity authentication module includes an off-post inspection unit and an identity authentication unit. The off-post inspection unit is used to detect a face through an in-vehicle camera and perform liveness detection. When no face is detected, the face is blocked or fails the liveness detection, an alarm is triggered. The identity authentication unit is used to match the detected face information with the vehicle owner information set in the data storage. If the match is successful, driving is allowed. If the match fails, an alarm is triggered.

[0116] The fatigue state detection and emotional state detection module is used to detect whether the driver is driving fatigued through the driver's facial, eye and body features after detecting that driving is allowed, and if so, trigger an alarm, otherwise, do not trigger an alarm;

[0117] The dangerous driving behavior detection module is used to detect the real-time posture of the driver during driving through a camera. If smoking, drinking, making phone calls, playing with the mobile phone, and looking for things are detected, an alarm is triggered.

[0118] The beneficial effects of the present invention are:

[0119] This application provides a new pedestrian detection algorithm (AFCNN) for complex scenarios, which effectively improves the detection accuracy of pedestrians in congested and obstructed scenarios, thereby further reducing the risk of pedestrian collisions; this application proposes a road surface disease information detection algorithm (SI-SSD), which reduces the probability of traffic accidents caused by poor road conditions; in view of the fact that vehicle collision detection algorithms in common products are difficult to cope with problems such as icy and snowy weather, a function of real-time detection of the driving speed of vehicles in front and behind the driver is introduced, which facilitates the driver to adjust the speed in time, further reducing the risk of traffic accidents. This application reduces the probability of traffic accidents caused by poor road conditions through the detection function of road disease information (potholes, surface cracks, road damage, etc.); this application proposes an improved road disease information detection algorithm (SI-SSD) based on the SSD model, which is verified on the RDD data set: compared with the detection performance of the traditional SSD model, the SI-SSD model improves the detection accuracy of transverse cracks by about 10%, reaching 94.5%, and the detection accuracy of longitudinal cracks by about 7.5%, reaching 91.2%; this application also proposes an occluded pedestrian detection algorithm in complex congestion scenes (pedestrian detection algorithm AFCNN based on visible part feature extraction of attention mechanism), which improves the accuracy of pedestrian detection in complex scenes, and can achieve 40.6% on the Heavy subset of the CityPersons dataset, and 10.3% on the Reasonable subset (the evaluation standard is MR -2 ), which is significantly improved compared with common algorithms and reduces the probability of pedestrian collision. In short, compared with existing products, this application can more effectively protect the driver's life safety and reduce the probability of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 It is a flow chart of on-the-job / off-the-job detection and identity verification in the in-vehicle scenario;

[0121] Figure 2 It is a flow chart of driver driving status detection and analysis (emotion detection, fatigue detection) in the in-car scene;

[0122] Figure 3 It is a flow chart for the detection and analysis of dangerous driving behaviors in in-car scenarios;

[0123] Figure 4 It is a flow chart of voice interaction and gesture interaction in the in-car scene;

[0124] Figure 5 It is a flow chart of road surface roughness detection in the scene outside the vehicle;

[0125] Figure 6 It is a flow chart of pedestrian detection in complex congested scenarios outside the vehicle;

[0126] Figure 7 It is a flow chart of traffic sign detection in the out-of-car scene;

[0127] Figure 8 It is a flow chart of vehicle speed detection in front and behind the vehicle in the off-vehicle scene;

[0128] Fig. 9 It is a geometric relationship image based on visual measurement of vehicle distance;

[0129] Fig.10 This is a comparative analysis of the detector performance after adding different additional features to the AFCNN algorithm;

[0130] Fig.11 This is a test chart comparing the performance of the AFCNN algorithm with other algorithms;

[0131] Fig.12 This is a performance comparison test chart between the SI-SSD model and the original SSD model. DETAILED DESCRIPTION

[0132] It should be particularly noted that, in the absence of conflict, the various embodiments disclosed in this application can be combined with each other.

[0133] Specific implementation method 1: An interactive auxiliary safe driving system described in this implementation method includes a pedestrian detection module in complex congestion scenes, a road disease information detection module, a driving surrounding traffic sign detection and recognition module, and a driving front and rear vehicle speed detection and front vehicle collision alarm module;

[0134] The pedestrian detection module in the complex congestion scenario is used to detect pedestrians in front of the vehicle, and when the distance between the pedestrian and the vehicle is less than the safety threshold, it will provide voice feedback to the driver;

[0135] The road surface damage information detection module is used to detect road surface damage information and feed back the detected information to the driver through voice;

[0136] The driving surrounding traffic sign detection and recognition module is used to detect and recognize the traffic signs in front of the vehicle, and feed back the detection and recognition results to the driver through voice;

[0137] The front and rear vehicle speed detection and front vehicle collision warning module is used to detect the distance between the front and rear vehicles and the driving vehicle and the speed of the front and rear vehicles. If the distance between the front and rear vehicles and the driving vehicle is less than the safety distance threshold or the speed of the front and rear vehicles is greater than the speed threshold, the driver is fed back through voice;

[0138] The pedestrian detection module in the complex congestion scene is implemented by a pedestrian detection algorithm, and the pedestrian detection algorithm specifically includes the following steps:

[0139] Step 1: Obtain the video stream file, and then extract the video stream file frame by frame. For each frame of the extracted image, use the non-maximum suppression algorithm to identify whether there is occlusion and occluded objects, and define the occluded object as active and the occluded object as inactive;

[0140] Step 2: Before regressing the inactive and active state trajectories, assume that the pedestrian moves at a constant speed, and use the MM model to predict the positions of the occluded and occluded object trajectories in the next frame, and obtain the speeds of the occluded and occluded objects by calculating the average displacement of the centers of the first n bounding boxes, specifically:

[0141]

[0142] Among them, S i represents the displacement of the center of the i-th bounding box, t l Indicates the time interval between two adjacent frames of images;

[0143] Step 3: Use the CMC model to get the camera motion speed, let P D is the center position of the blocked pedestrian, and its coordinates are (x D ,y D ), let the estimated position P of the inactive state trajectory of the blocked pedestrian be L The coordinates are (x L ,y L ), when the difference between the two coordinates is lower than the set threshold, the occluded object will activate the inactive state, that is, enter the active state, specifically:

[0144] ||P D -P L ||≤D max

[0145]

[0146] Among them, D maxFollow inactive tracks The velocity and the time t that the trajectory has been inactive inactive increases with the increase of γ, which indicates the control D max Numerical parameters;

[0147] Step 4: Obtain the camera motion speed based on the CMC model and then obtain the detection marker frame. Then, extract any two tracklets from the motion trajectory of the active object in step 3. If the two tracklets completely overlap, that is, the IoU value in each frame where both tracklets have detection marker frames is higher than the threshold, then the two belong to the same target. If the two tracklets partially overlap, that is, there are frames without overlap, and the IoU value is lower than the threshold, then only the detection marker frame corresponding to the longest tracklet in each frame is retained.

[0148] Step 5: When the overlap between the detection marker box and the object entering the active state decreases, the regression score S will decrease accordingly. When the regression score S is less than the threshold S min When the boundary of the detection mark box of the object entering the active state coincides with the boundary of the image, the two-dimensional velocity vector of the motion trajectory of the detection mark box is calculated if If the direction points to the image boundary, the detection mark box is deleted from the motion trajectory, otherwise, the detection mark box is retained;

[0149] Step 6: Extract 17 key points of the tracked pedestrian through the posture estimation network. The tracked pedestrian is the object entering the active state, and construct a response map based on the 17 key points as data input, and calculate the attention weight of each key point of the tracked pedestrian. k , specifically:

[0150] attention k =average_pool(relu(conv(K)))

[0151]

[0152] Among them, average_pool represents the global average pool, which is used to encode the features of each channel into the corresponding global features, attention k Represents the attention weight of the key points of the tracked pedestrian, attention k ∈R C×1×1 , conv represents a 3×3 convolution kernel, relu represents an activation function, and f represents the extracted pedestrian key point features. Represents the feature value after NMS suppression of the occluded part;

[0153] Step 7: First, a mask is constructed for the visible part of the tracked pedestrian, that is, the visible part is marked as 1 and the invisible part is marked as 0. Finally, the feature map of the visible part of the tracked pedestrian is constructed. It is expressed as:

[0154]

[0155] Among them, ⊙ represents the multiplication operation according to the corresponding position;

[0156] Step 8: Let the tracked pedestrian be T, let the set of all candidate marker boxes corresponding to the tracked pedestrian be C, and after parameter adjustment, calculate the value M of each element in the set C by the following formula:

[0157] M=3*Err cls +Err reg

[0158] According to the value M of each element in the set C and combined with the following formula, the smallest element T is selected end :

[0159]

[0160] Among them, Err cls With Err reg They refer to the error and regression error between the predicted label box and the true label box respectively;

[0161] Step 9: Take the smallest element T end The detection mark boxes outside the corresponding detection mark boxes are suppressed, that is, normalized, and expressed as:

[0162]

[0163]

[0164] grade=α*Dis cls +βDis reg

[0165] Dis cls =y i *log(σ(p(y i )))+(1-y i )*log(1-σ(p(y i )))

[0166]

[0167] Among them, Dis cls Indicates the gap between the candidate detection box sample and the real box category, y i∈{0, 1}, σ represents the Sigmoid function, which is used to map values ​​to (0, 1), Dis reg Represents the gap between the candidate detection box sample and the real box regression, Z x and Z y Respectively represent the center point coordinates of the candidate detection box, V x and V y Respectively represent the center point coordinates of the real box, W Z and H Z Respectively represent the width and height of the candidate detection box;

[0168] Step 10: Set the loss function L;

[0169] Step 11: Train and test the entire model according to steps 1 to 10, and calculate various model evaluation indicators.

[0170] As an implementation of this application, the following modules may be included:

[0171] Module 1. Off-duty inspection and identity verification in the in-car scene. The in-car camera is turned on to detect whether the face is on the display screen. When the driver is off-duty or the driver's face is blocked, the alarm system will be triggered; identity verification refers to matching the detected face information with the owner information set in the data storage. In order to avoid false "photo" detection, a liveness detection link is set. If the identity verification is successful after the liveness detection, driving is allowed; if the verification fails, the alarm system will be triggered.

[0172] Module 2. Fatigue state detection and emotional state detection in the in-car scene. When the terminal device detects that the driver is on duty, it intelligently analyzes the driver's facial, eye, body and other detailed features to detect whether the driver is driving fatigued. If so, an alarm is triggered to remind the driver to take a rest. If the driver is in a non-neutral state for a long time, an alarm is triggered to remind the driver to focus on driving.

[0173] Module 3. Dangerous driving behavior detection in the car scene. The camera will detect the driver's behaviors such as smoking, drinking, making phone calls, playing with mobile phones, and looking for things while driving. At the same time, it will detect whether the number of passengers in the vehicle is overloaded. Overloading of passengers is also a dangerous driving behavior. If dangerous driving behavior is detected, an alarm will be triggered to remind the driver to drive carefully.

[0174] Module 4. Voice recognition and interaction and gesture recognition and interaction in the in-car scene. The built-in microphone of the terminal device extracts the driver's speech content, and performs semantic analysis through the algorithm model in the program. Different functions can be awakened by the driver speaking specific content, including: waking up the music player, increasing the volume, lowering the volume, broadcasting the weather conditions, displaying the current time, waking up the navigation system, etc.; while the gesture recognition and interaction part perceives the hand gesture through the built-in camera of the terminal device, and can make different responses to different gestures. The functions that can be awakened are consistent with voice awakening. In short, the recognition and interaction of hand gestures further expands the applicable population of the entire safety system, especially providing convenience for disabled people to drive cars.

[0175] Module 5. Pedestrian detection in complex congested scenes outside the vehicle. Since there is a lot of overlap in the pedestrian targets detected in complex congested scenes, the features actually extracted do not have high reliability. Therefore, in actual driving scenarios, if the extracted features are not sufficient to be identified as pedestrians or the detection accuracy is very low, it is often easy to cause pedestrian collisions. To address this problem, this application makes improvements based on the Fast R-CNN framework and proposes a pedestrian detection algorithm (AFCNN) based on visible part feature extraction based on the attention mechanism. To a certain extent, it can reduce missed detections due to occlusion, improve the efficiency of pedestrian detection in complex scenes, and reduce the risk of pedestrian collisions.

[0176] Module 6. Detection of road surface smoothness and disease information in scenes outside the vehicle. Since the number of neural network layers of the traditional recurrent neural network algorithm will continue to increase when receiving complex road surface information data input, the number of activation functions will also increase accordingly, and the gradient will decrease in the back propagation link, affecting the final detection accuracy. To address this problem, this application introduces an algorithm based on the LSTM network. The input data is road surface information data. The road surface roughness index (IRI) is calculated through the LSTM network; after obtaining the IRI value, the final road surface smoothness prediction result is obtained according to the defined evaluation index, and the abnormal IRI condition is fed back to the driver through the voice model, so that the driver can make adjustments according to the actual situation; for the road surface disease detection link, the traditional SSD network is difficult to efficiently handle the problem that the road surface image has a single color and the image features are not obvious. To address this problem, this application adds a Sobel edge detection operator to the convolutional layer, and proposes an improved road surface disease information detection algorithm (SI-SSD) based on the SSD model, which speeds up the execution of the model while also improving the efficiency of extracting road surface disease image features. Finally, the real-time detected road surface roughness and road damage information is fed back to the driver through the voice module, which helps the driver make judgments based on the real-time environment.

[0177] Module 7. Detection and recognition of traffic signs around the driver in the scene outside the car. Detecting traffic signs is essentially a candidate area detection and classification problem. At the same time, the detection of traffic signs during driving requires extremely high real-time performance. Since the YOLOv3 model has a fast feature extraction network DarkNet-53, and the K-means model has a better classification effect. Therefore, this application uses the YOLOv3+K-means combination method to detect and recognize traffic signs, and feeds back the real-time detection results to the driver through the voice module, which is beneficial for the driver to make adjustments based on the real-time situation.

[0178] Module 8. Vehicle speed detection in front and behind the driver and front vehicle collision alarm in the scene outside the vehicle. This module must first monitor and identify the vehicles in front and behind the driver. Because the car has many distinct features, such as color, edges, shadows, lights and textures. These features can be used to further obtain the area where the car is located in the image, and then the located vehicle state is detected based on the motion model. In the relative speed detection link, this application uses a method based on multi-frame image superposition to calculate the change in vehicle distance based on a geometric relationship model. When the distance between the two vehicles is lower than the safety threshold distance or the relative speed is higher than the safety threshold speed, the voice module reminds the driver to make adjustments based on the real-time situation.

[0179] 1. Road surface smoothness detection

[0180] Step 1: Dataset preprocessing

[0181] This step is mainly to fill the missing values ​​in the data set. In this application, the K-means algorithm is used to fill the missing values. The normalization formula is shown in formula (1):

[0182]

[0183] Where D is the distance value of the road data in the data set, and max and min represent the maximum and minimum values ​​in the data respectively.

[0184] Step 2: Pooling Layer

[0185] The pooling layer is mainly added after the convolution layer. The purpose of adding the pooling layer is to prevent overfitting and improve the efficiency of the neural network. The pooling layer uses 3×3 average pooling, which can better preserve the collected distance data information.

[0186] Step 3: Calculate the output value of the forget gate

[0187] This step is mainly to store and manage all input information, and at the same time screen and filter all information to remove useless information. The specific calculation formula is shown in formula (2):

[0188] ft =σ(W f ·[h t-1 , x t ]+b f ) (2)

[0189] Among them, W f represents the weight of the input gate, and σ is the sigmoid activation function. f is the bias direction of the forget gate.

[0190] Step 4: Calculate the value of the input gate and update the network unit status information. The calculation formula is shown in equations (3)-(4)

[0191] i t =σ(W i ·[h t-1 , x t ]+b i ) (3)

[0192]

[0193] W i represents the weight of the input gate, W c represents the weight of the unit state, b i represents the bias term of the input gate, b c A bias term that represents the cell state. Represents the intermediate vector for calculating the update information required at the next moment.

[0194] Step 5: Update the unit status. This step consists of three steps. First, delete the old information, then use To add new information, finally calculate the value of the output gate and update the unit state. The calculation formula is shown in equations (5)-(7)

[0195]

[0196] o t =σ(W o ·[h t-1 , x t ]+b o ) (6)

[0197] h t =o t *tanh(C t ) (7)

[0198] Among them t Indicates the value after activation.

[0199] Step 6: Calculate the IRI value, continuously update the unit state, perform back propagation operations to update the model parameters, and use the prediction model to make predictions. The specific calculation formula is shown in formula (8):

[0200]

[0201] 2. Road surface disease information detection (potholes, surface cracks, road surface damage, etc.)

[0202] Step 1: Sobel edge detection and extraction operator

[0203] This step is mainly aimed at the problem that the features of pavement disease images are weak and difficult to process. The specific calculation is shown in equations (9)-(12):

[0204]

[0205]

[0206]

[0207]

[0208] where g x Represents the horizontal gradient, g y Represents the vertical gradient, g x ′ and g y ′ represents the diagonal gradient. i (i∈[1,9]) represents the value displayed at each coordinate in the 3×3 Sobel operator template.

[0209] Step 2: Constructing the pavement damage detection model (SI-SSD)

[0210] The SSD model extracts features based on the VGG16 network. Although it can extract rich information, it also has some shortcomings. Too many convolution kernel operations lead to too many extracted features and too many parameters, which has a negative impact on the detection efficiency. At the same time, the VGG16 network needs to undergo a large number of pooling operations, which is easy to lose important feature information. In response to these problems, this application proposes a SI-SSD model, which is improved based on the SSD model. The overall structure is shown in Table 1. First, the VGG16 structure is changed to the Inception network structure, because there is a bottleneck layer (1×1 convolution) in the Inception module, which can reduce the number of parameters and reduce the impact on the detection efficiency. The Inception Block is set to 5 layers, and the Transition Layer is used to perform feature dimensionality reduction. Then, a depth-separable convolution with a step of 2 is used to replace the maximum pooling layer, and an attention mechanism (SE-Net) is introduced between Conv6-Conv12 to detect the feature map to obtain the default box, and finally perform non-maximum suppression screening. In the process of training the data set, the introduction of the attention mechanism can effectively increase the weight coefficient of the crack location, improve the efficiency of extracting important features of the crack, and also improve the accuracy of the loss function calculation.

[0211] Table 1. SI-SSD model structure

[0212]

[0213]

[0214] Step 3: Calculate the loss function. This step mainly includes two parts: position loss and classification loss. The specific calculation formula is shown in formula (13):

[0215]

[0216] Step 3.1: Calculate the position loss. The position loss refers to the loss between the predicted box and the real box. The calculation formula is shown in equations (14)-(18)

[0217]

[0218]

[0219]

[0220]

[0221]

[0222] Step 3.2: Calculate the classification loss. The classification loss refers to the Softmax loss. The calculation formula is shown in equations (19)-(20):

[0223]

[0224]

[0225] Where N is the number of samples, x and c are the indicator and the confidence level respectively, l and g are the predicted box and the background box respectively. α is the weight of the position box loss value, C x ,C y ,w,h represent the center coordinates (x,y) of the marker box and the height and width of the marker box respectively, and d represents the marker box predicted by the network itself.

[0226] Step 4: Training and validating the model. To validate the performance of the SI-SSD model, this application uses the RDD dataset for validation and compares the performance of the SI-SSD model with the original SSD model. The dataset contains images of different pavement damage in seven cities, and the pavement damage is divided into eight categories and labeled with their category names, as shown in Table 2.

[0227] Table 2. Various pavement damage types and their definitions in the RDD dataset

[0228]

[0229] For the model training parameters, the batch number is 20, the image size is set to 400×400, the learning rate is 0.005, the number of iterations is 5000, and the regularization standard uses L2 regularization.

[0230] 3. Vehicle distance and relative speed calculation

[0231] Step 1: Calculate the positional relationship between the vehicle to be tested and the imaging plane. Since the basic principles of detecting the distance and speed of the front and rear vehicles are the same, the only difference is in the reference system, so here only the joint analysis of the driving vehicle and the vehicle in front is listed. Fig. 9 In the equation, car A represents the vehicle driven by the driver. Car A has an external camera O2 in front of it, and f represents the focal length of the camera. Car B represents the vehicle in front during driving, H represents the height of the camera on the car, and d1 represents the distance between car A and car B. O3M represents the distance from the point on the ground where the parking space of car B is vertically projected to the image to the center line of the image. The length of O3P can be calculated based on the geometric relationship, as shown in equations (21)-(22)

[0232]

[0233]

[0234] According to the above formula, the coordinate O2P in the vertical direction Y and the coordinate PQ in the vertical direction X are as shown in formulas (23)-(25):

[0235]

[0236]

[0237]

[0238] Step 2: Calculate the relative distance between vehicles using the frame difference method. Suppose the height of the vehicle to be tested on the imaging plane at different times is u i (i=0,1,2…), let the relative distances between the two vehicles at two different times be d r1 ,d r2 , according to formula (26), the relative distance between the two vehicles can be calculated.

[0239]

[0240] Where K represents the actual height of the car. α0 approaches 0 infinitely. According to the above formula, the change of the relative distance between the two cars can be obtained as shown in formula (27).

[0241]

[0242] According to the above formula, when l' is less than 0, it means that the two cars are gradually approaching each other; when l' is greater than 0, the two cars are gradually moving away from each other; when the value of |l'| approaches 1 infinitely, the change in the relative distance between the two cars increases infinitely; when the value of |l'| approaches 0 infinitely, the change in the relative distance between the two cars decreases infinitely;

[0243] Step 3: Calculate the relative speed between vehicles. The specific calculation formula is as follows:

[0244]

[0245] Wherein ΔT represents the time interval between two adjacent frames of images detected.

[0246] 4. AFCNN Algorithm

[0247] Step 1: Occlusion definition and detection, when the motion trajectories of two pedestrians intersect each other, the overlap of the detected pedestrian bounding boxes will increase frame by frame until the non-maximum suppression (NMS) filters out the accurate regression box. If pedestrian A is in an active state and pedestrian B is in an inactive state, pedestrian A is defined as occluding pedestrian B.

[0248] Step 2: MM (motion model) + CMC (camera motion compensation model), before the regression of inactive and active tracks, it is assumed that the pedestrian moves at a constant speed. The MM model is used before the CMC model works. To predict the position of the track in the next frame, its speed is calculated by averaging the displacement of the centers of the first n bounding boxes. The specific calculation is as follows

[0249]

[0250] Where S i represents the displacement of the center of each bounding box, t l Indicates the time interval between two adjacent image frames.

[0251] Step 3: Activate the inactive state of the blocked pedestrian. First, process the inactive state with the MM model and the CMC model. Let P D is the center position of the blocked pedestrian, and its coordinates are (x D ,x D ); Assume that P L The estimated position coordinates of the inactive track of the occluded pedestrian are (x L ,y L ). When the difference between the two coordinates is lower than the set threshold, the newly detected pedestrian will activate the inactive state and enter the active state. The specific calculation is as follows

[0252] ||P D -P L ||≤D max (30)

[0253]

[0254] Where D max With inactive track V L The speed of the trajectory increases with the increase of the time that the trajectory has been inactive, and γ is the control D max Numerical parameters.

[0255] Step 4: Merge multiple tracklets. If two tracklets (small fragments of tracked trajectories) completely overlap, that is, the IoU in each frame where both tracklets have boxes is higher than the threshold, then it can be determined that the two belong to the same target. If the two tracks only partially overlap, that is, there are frames without overlap (IoU is lower than the threshold), then delete the box of the shorter track in each frame and determine that the longer track is more reliable and accurate than the shorter track.

[0256] Step 5: Pedestrian trajectory identification. When the pedestrian to be detected is about to leave the detection field of view, the regression score will decrease. When the regression score S is less than the threshold S minWhen its bounding box is at the edge of the image, calculate the two-dimensional velocity vector of its motion trajectory if If the direction points to the image boundary, it will be deleted from the motion trajectory.

[0257] Step 6: Detect and track the occluded pedestrian. In this step, the key points of the human body are first extracted. The 17 key points of the tracked pedestrian are extracted through the posture estimation network. According to these 17 key points (the human body parts corresponding to the 17 key points of COCO include: 0: nose, 1: left eye, 2: right eye, 3: left ear, 4: right ear, 5: left shoulder, 6: right shoulder, 7: left elbow, 8: right elbow, 9: left wrist, 10: right wrist, 11: left hip, 12: right hip, 13: left knee, 14: right knee, 15: left ankle, 16: right ankle), a response map is constructed as the data input. The attention weights of the key points of the tracked person can be obtained through calculations of formulas (32)-(33), which are represented by the symbol attention. k .

[0258] attention k =average_pool(relu(conv(K))) (32)

[0259]

[0260] Among them, average_pool represents the global average pool, which is used to encode the features of each channel into the corresponding global features. k Represents the attention weight of the key points of the tracked pedestrian, attention k ∈R C×1×1 , conv represents a 3×3 convolution kernel, relu represents an activation function, and f represents the extracted pedestrian key point features. Represents the feature value after NMS suppression of the occluded part.

[0261] Step 7: Visible parts detection and tracking based on attention mechanism. Visible human body parts refer to the parts that are not blocked during the detection and tracking process, such as the head, arms, legs, visible center, etc. First, a mask should be constructed for the visible parts, that is, the visible parts are marked as 1 and the invisible parts are marked as 0. Finally, the visible parts feature map of the tracked pedestrian From formula (34), we can get

[0262]

[0263] Where ⊙ represents the multiplication operation according to the corresponding position. After the improvement, the influence of the features of the occluded part on the pedestrian detection under overall occlusion is further reduced, effectively ensuring the accuracy of pedestrian detection and tracking.

[0264] Step 8: Filter high-quality markers. Let the pedestrian target be T, and let the set of all corresponding candidate markers be C. After parameter adjustment, the value M of each element in the set C is calculated by formula (35), and the element T of the smallest candidate marker is filtered out according to formula (36): end

[0265] M=3*Err cls +Err reg (35)

[0266]

[0267] Among them, Err cls With Err reg They refer to the error between the predicted box and the true box and the regression error respectively.

[0268] Step 9: Suppress other marked frames. After selecting high-quality marked frames, other frames should be suppressed to prevent them from interfering with the selected marked frames. The specific method is to normalize other frames, as shown in equations (37)-(38)

[0269]

[0270]

[0271] This method suppresses other non-optimal marked frames, which can ensure that the frame used for each detection is the optimal frame, thereby reducing the interference of low-quality marked frames on the overall detection.

[0272] Step 10: Set the loss function. The loss function includes center point loss, height loss, offset loss, regional pixel learning loss, and regional fine-grained segmentation learning loss.

[0273] Step 10.1: Center point loss. The center points include the center point of the pedestrian's body, the center point of the pedestrian's head, and the center point of the visible part of the pedestrian. The number of real image output channels corresponding to these three center points is 3. The value of the center point in channel 1 is set to 1, and the center point map values ​​of the other positions are generated according to the 2D Gaussian function. The value of the pedestrian center point in channel 2 is set to 1, and the values ​​of the other positions are set to 0; the value of the pedestrian detection box area in channel 3 is assigned to 1, and the values ​​of the other areas are set to 0. When the real image of the center point already has a value, the larger value between the existing value and the current value is taken.

[0274]

[0275]

[0276] N represents the number of people in each picture, (xk ,y k , σ wk , σ hk ) represents the coordinates of the pedestrian. If there are multiple Gaussian values ​​in a certain coordinate value, the maximum value max is taken.

[0277]

[0278]

[0279] Q ij α =G(i, j; x k ,y k , σ wk , σ hk ) (43)

[0280] where r ij Represents the treatment confidence value of (i, j) in the prediction graph, and the prediction graph size is In actual execution, α is set to 4 and β is set to 2.

[0281] Step 10.2: Pedestrian loss. For pedestrian loss, based on the L1Loss loss function, it includes width and height losses respectively.

[0282]

[0283] L scale =L height +L width (45)

[0284] Step 10.3: Offset loss, the offset loss function is based on the SmoothL1 loss function.

[0285]

[0286] Among them k and k1 Represent the true value and predicted value of the offset respectively.

[0287] Step 10.4: Regional pixel learning loss, let A be the number of all pixels in the pedestrian area, m t , m p They respectively represent the pixel value of the real area where the pedestrian is located and the pixel value of the area where the pedestrian is located predicted by the model.

[0288]

[0289] Step 10.5: Region fine-grained segmentation learning loss, n t , n pThey respectively represent the fine-grained segmentation image of the actual area where the pedestrian is located and the fine-grained segmentation image of the area where the pedestrian is located predicted by the model.

[0290]

[0291] Step 10.6: Calculate the overall model loss function

[0292] L=λ i L center +λ2L scale +λ3L offset +λ4L pixel +λ5L f (49)

[0293] The control coefficient parameters λ1~λ5 are set to 0.02, 0.005, 0.1, 0.0002, and 0.0001 respectively.

[0294] Step 11: According to the operations in steps 1 to 10, the entire algorithm model is trained and tested, and various model evaluation indicators are calculated.

[0295] Compared with existing products, this application has the following features:

[0296] 1. In the pedestrian detection link, compared with the existing pedestrian detection algorithm, the AFCNN algorithm in this application adds the extraction and fusion of additional features, which is more suitable for scenes with more severe occlusion. The detection performance is improved and the missed detection rate is reduced to a certain extent.

[0297] 2. In the process of pavement disease information detection, compared with the existing SSD model detection algorithm, the SI-SSD algorithm model in this application does not need to perform too many convolution kernel pooling operations. By using a lighter network structure, the execution speed of the entire model is accelerated, while the detection confidence is improved, and the detection confidence of multiple targets is also improved.

[0298] 3. Compared with common products on the market, this application has added a road surface flatness detection function, which alleviates the problem of unstable driving caused by poor road quality.

[0299] 4. Compared with common products on the market, this application refines and improves the concept of safe vehicle distance, and adds a real-time monitoring function of the speed of vehicles around the driver, which can further reduce the probability of car collision during driving.

[0300] The effectiveness of this application can be illustrated by the following simulation:

[0301] 1. AFCNN Algorithm

[0302] 1. Analyze the impact of adding different additional features on the detection performance of the detector. In order to further verify the effect of AFCNN algorithm on pedestrian detection under occlusion conditions, this application designed an ablation experiment. The performance impact of pedestrian detection area, head features, visible center, and predicted pedestrian height on the pedestrian detector was tested respectively. The overall backbone network uses ResNeSt-200. In order to maintain stability during model training, a moving sliding average strategy is used. The learning rate is set to 0.0001 and the number of training times is set to 1000. Since the data type in the CityPersons dataset is closer to the actual application scenario of this application, the CityPersons dataset is selected as the verification dataset.

[0303] 2. Comparison and analysis of the performance of AFCNN algorithm and other model algorithms. In order to further verify the performance of AFCNN algorithm, the detection performance of common models on CityPersons dataset was tested respectively. This application mainly tested the effect of MGAN, AP2M, CSP, ALFNET, TLL and AEVB models on the Heavy subset and Reasonable subset in the dataset. The evaluation criteria were MR -2 The learning rate and the number of training times remain unchanged.

[0304] 3. Simulation results and analysis. For the analysis experiment on the effect of adding different additional features on the detection performance of the detector, the specific results are shown in Table 3. Fig.10 It reflects the detection effect of the detector after adding head features, visible center and predicted pedestrian height features when the pedestrian area is used as the detection area. Fig.10 From the information, we can see that the performance of pedestrian detectors continues to improve with the addition of additional features, and the missed detection rate in severe occlusion conditions can be significantly reduced.

[0305] Table 3. Analysis of ablation experiment results

[0306]

[0307]

[0308] According to the data in Table 3, it can be seen that using the pedestrian area as the detection area is better than using the entire image as the detection area. On the Heavy subset, the effect of reducing 1.4 percentage points was achieved. Therefore, the pedestrian area was used as the detection area in the subsequent experiments. After introducing the head information feature and the visible center module, a result of 41.3% can be achieved on the Heavy subset, and the effect of pedestrian detection has been significantly improved. After introducing the pedestrian height prediction module, a result of 40.6% can be achieved on the Heavy subset, and an effect of 10.3% can be achieved on the Reasonable subset. Therefore, the introduction of head features, visible center detection, and pedestrian height prediction modules have a positive effect on the final detection effect.

[0309] Table 4 shows the comparison of the indicators of the AFCNN algorithm evaluated on the CityPersons dataset with other common network models. Fig.11 This is a test chart that reflects the performance comparison between the AFCNN algorithm proposed in this application and other models. Fig.11 From the information, it can be seen that the network model proposed in this application can more efficiently concentrate the features on pedestrians, thereby effectively reducing the missed detection rate in severe occlusion conditions.

[0310] Table 4. Comparison of evaluation indicators of various models in the CityPersons dataset

[0311]

[0312] From the data in Table 4, it can be seen that the pedestrian detection module based on the AFCNN model has better effect in congested scenes, better processing effect on missed detection, and lower missed detection rate. Therefore, the experimental results show that the AFCNN algorithm proposed in this application is more suitable for actual driving scenes than other pedestrian detection algorithms, and can effectively reduce the probability of pedestrian collision.

[0313] 2. SI-SSD Algorithm

[0314] 1. Compare the loss values ​​of the SI-SSD model algorithm and the SSD model during 5000 iterations. By comparing the loss values ​​of the SI-SSD model and the SSD model during the iteration process, the performance of the two can be indirectly compared. Table 5 lists the loss values ​​of the SI-SSD model and the SSD model at each stage during 5000 iterations.

[0315] Table 5. Statistics of loss values ​​in 5000 iterations

[0316]

[0317] According to Table 5, the SI-SSD model has a lower loss value and its effect is better than that of the original SSD model. Then, 2000 images were extracted from the data set as the test set to verify the accuracy of the SSD model and SI-SSD model respectively. The specific results are shown in Table 6.

[0318] Table 6. Accuracy statistics and comparison of SSD model and SI-SSD model under test set

[0319]

[0320] It can be seen that the loss value of the improved SI-SSD model is lower than that of the original SSD model, and the accuracy on the test set has been significantly improved.

[0321] 2. Compare the confidence of the SI-SSD algorithm model and the SSD model in detecting disease information. In order to more intuitively compare the performance of the two model detectors, some images were selected for detection and comparative analysis. The detection results are shown in the figure below. Fig.12 As shown in the figure, the detection result graph includes the location of the disease information, the category it belongs to, and the confidence level. Fig.12 From the information, we can see that the SI-SSD model proposed in this application has improved the detection confidence and also improved the detection confidence of multiple targets. Fig.12 It can be seen that the traditional SSD model is difficult to effectively detect smaller road surface damage information, while the SI-SSD model proposed in this application can detect smaller targets more accurately. Therefore, the experimental results show that the SI-SSD algorithm model proposed in this application is more suitable for actual driving scenarios and has better performance in detecting road surface damage information.

[0322] It should be noted that the specific implementation is only an explanation and description of the technical solution of the present invention, and cannot be used to limit the scope of protection of the rights. Any partial changes made according to the claims and description of the present invention should still fall within the scope of protection of the present invention.

Claims

1. An interactive auxiliary safety driving system, characterized in that It includes a pedestrian detection module in complex congestion scenarios, a road surface disease information detection module, a driving surrounding traffic sign detection and recognition module, and a driving front and rear vehicle speed detection and front vehicle collision alarm module; The pedestrian detection module in the complex congestion scenario is used to detect pedestrians in front of the vehicle, and when the distance between the pedestrian and the vehicle is less than the safety threshold, it will provide voice feedback to the driver; The road surface damage information detection module is used to detect road surface damage information and feed back the detected information to the driver through voice; The driving surrounding traffic sign detection and recognition module is used to detect and recognize the traffic signs in front of the vehicle, and feed back the detection and recognition results to the driver through voice; The front and rear vehicle speed detection and front vehicle collision warning module is used to detect the distance between the front and rear vehicles and the driving vehicle and the speed of the front and rear vehicles. If the distance between the front and rear vehicles and the driving vehicle is less than the safety distance threshold or the speed of the front and rear vehicles is greater than the speed threshold, the driver is fed back through voice; The pedestrian detection module in the complex congestion scene is implemented by a pedestrian detection algorithm, and the pedestrian detection algorithm specifically includes the following steps: Step 1: Obtain the video stream file, and then extract the video stream file frame by frame. For each frame of the extracted image, use the non-maximum suppression algorithm to identify whether there is occlusion and occluded objects, and define the occluded object as active and the occluded object as inactive; Step 2: Before regressing the inactive and active state trajectories, assume that the pedestrian moves at a constant speed, and use the MM model to predict the positions of the occluded and occluded object trajectories in the next frame, and obtain the speeds of the occluded and occluded objects by calculating the average displacement of the centers of the first n bounding boxes, specifically: Among them, S i represents the displacement of the center of the i-th bounding box, t l Indicates the time interval between two adjacent frames of images; Step 3: Use the CMC model to get the camera motion speed, let P D is the center position of the blocked pedestrian, and its coordinates are (x D ,y D ), let the estimated position P of the inactive state trajectory of the blocked pedestrian be L The coordinates are (x L ,y L ), when the difference between the two coordinates is lower than the set threshold, the occluded object will activate the inactive state, that is, enter the active state, specifically: ||P D -P L ||≤D max Among them, D max Follow inactive tracks The velocity and the time t that the trajectory has been inactive inactive increases with the increase of γ, which indicates the control D max Numerical parameters; Step 4: Obtain the camera motion speed based on the CMC model and then obtain the detection marker frame. Then, extract any two tracklets from the motion trajectory of the active object in step 3. If the two tracklets completely overlap, that is, the IoU value in each frame where both tracklets have detection marker frames is higher than the threshold, then the two belong to the same target. If the two tracklets partially overlap, that is, there are frames without overlap, and the IoU value is lower than the threshold, then only the detection marker frame corresponding to the longest tracklet in each frame is retained. Step 5: When the overlap between the detection marker box and the object entering the active state decreases, the regression score S will decrease accordingly. When the regression score S is less than the threshold S min When the boundary of the detection mark box of the object entering the active state coincides with the boundary of the image, the two-dimensional velocity vector of the motion trajectory of the detection mark box is calculated if If the direction points to the image boundary, the detection mark box is deleted from the motion trajectory, otherwise, the detection mark box is retained; Step 6: Extract 17 key points of the tracked pedestrian through the posture estimation network. The tracked pedestrian is the object entering the active state, and construct a response map based on the 17 key points as data input, and calculate the attention weight of each key point of the tracked pedestrian. k , specifically: attention k average_pool(relu(conv(K))) Among them, average_pool represents the global average pool, which is used to encode the features of each channel into the corresponding global features, attention k Represents the attention weight of the key points of the tracked pedestrian, attention k ∈R C×1×1 , conv represents a 3×3 convolution kernel, relu represents an activation function, and f represents the extracted pedestrian key point features. Represents the feature value after NMS suppression of the occluded part; Step 7: First, a mask is constructed for the visible part of the tracked pedestrian, that is, the visible part is marked as 1 and the invisible part is marked as 0. Finally, the feature map of the visible part of the tracked pedestrian is constructed. It is expressed as: Among them, ⊙ represents the multiplication operation according to the corresponding position; Step 8: Let the tracked pedestrian be T, let the set of all candidate marker boxes corresponding to the tracked pedestrian be C, and after parameter adjustment, calculate the value M of each element in the set C by the following formula: M=3*Err cls +Err reg According to the value M of each element in the set C and combined with the following formula, the smallest element T is selected end : Among them, Err cls With Err reg They refer to the error and regression error between the predicted label box and the true label box respectively; Step 9: Take the smallest element T end The detection mark boxes outside the corresponding detection mark boxes are suppressed, that is, normalized, and expressed as: grade=α*Dis cls +βDis reg Dis cls =y i *log(σ(p(y i )))+(1-y i )*log(1-σ(p(y i ))) Among them, Dis cls Indicates the gap between the candidate detection box sample and the real box category, y i ∈{0,1}, σ represents the Sigmoid function, which is used to map the value to between (0,1), Dis reg Represents the gap between the candidate detection box sample and the real box regression, Z x and Z y Respectively represent the center point coordinates of the candidate detection box, V x and V y Respectively represent the center point coordinates of the real box, W Z and H Z Respectively represent the width and height of the candidate detection box; Step 10: Set the loss function L; Step 11: Train and test the entire model according to steps 1 to 10, and calculate various model evaluation indicators.

2. The interactive auxiliary safety driving system according to claim 1, characterized in that The loss function L includes center point loss, pedestrian loss, offset loss, regional pixel learning loss and regional fine-grained segmentation learning loss; L=λ1L center +λ2L scale +λ3L offset +λ4L pixel +λ5L F Among them, L center Represents the loss at the center point, L scale represents pedestrian loss, L offset Indicates the offset loss, L pixel represents the region pixel learning loss, L f represents the regional fine-grained segmentation learning loss, and the control coefficient parameters λ1~λ5 are set to 0.02, 0.005, 0.1, 0.0002, and 0.0001 respectively.

3. The interactive auxiliary safety driving system according to claim 2 is characterized in that The center point loss is defined as follows: Center point L center Including the center point of the pedestrian's body, the center point of the pedestrian's head and the center point of the visible part of the pedestrian. There are 3 real image output channels corresponding to these three center points. The value of the center point in channel 1 is set to 1, and the center point map values ​​of the other positions are generated according to the 2D Gaussian function. The value of the pedestrian center point in channel 2 is set to 1, and the values ​​of the other positions are set to 0; the value of the pedestrian detection box area in channel 3 is assigned to 1, and the values ​​of the other areas are set to 0. When the real image of the center point already has a value, the larger value between the existing value and the current value is taken. The center point loss is expressed as: Among them, N represents the number of people in each picture, (x a ,y a ,σ wa ,σ ha ) represents the coordinates of the pedestrian. If there are multiple Gaussian values ​​in a certain coordinate value, the maximum value max is taken, σ w represents the Gaussian value of the pedestrian's corresponding width, σ h represents the Gaussian value of the pedestrian's height, r ij represents the treatment confidence value of (i,j) in the prediction graph, Indicates different corresponding y ij The predicted value under the prediction graph size indicates In actual execution, α is set to 4, β is set to 2, and y ij represents the value at the pedestrian center point, β ij Represents the coefficient parameters under different ij.

4. The interactive auxiliary safety driving system according to claim 3 is characterized in that The pedestrian loss is defined as follows: For pedestrian loss, based on the L1Loss loss function, it includes the loss of width and height respectively. L scale =L height +L width Among them, s a represents the logarithm of the true value of the pedestrian width, t a Indicates the logarithm of the predicted value of pedestrian width, s b Indicates the logarithm of the true value of the pedestrian height, t b It represents the logarithm of the predicted value of the pedestrian height, a=1...N, b=1...N.

5. The interactive auxiliary safety driving system according to claim 4, characterized in that The offset loss is defined as follows: The offset loss function is based on the SmoothL1 loss function and is expressed as: Among them, k and k1 Represent the true value and predicted value of the offset respectively.

6. The interactive auxiliary safe driving system according to claim 5, characterized in that The region pixel learning loss is defined as follows: Let A be the number of all pixels in the pedestrian area, m t ,m p They represent the pixel values ​​of the real area where the pedestrian is located and the pixel values ​​of the area where the pedestrian is located predicted by the model. The regional pixel learning loss is expressed as:

7. The interactive auxiliary safe driving system according to claim 6, characterized in that The regional fine-grained segmentation learning loss is defined as follows: Among them, n t ,n p They respectively represent the fine-grained segmentation image of the actual area where the pedestrian is located and the fine-grained segmentation image of the area where the pedestrian is located predicted by the model.

8. The interactive auxiliary safe driving system according to claim 7, characterized in that The pavement damage information detection module is implemented by a pavement damage information detection algorithm, and the pavement damage information detection algorithm specifically includes the following steps: Step 1: Obtain a road surface image and perform feature extraction on the road surface image. Then, perform Sobel edge detection based on the extracted features. Specifically: Among them, g x Represents the horizontal gradient, g y Represents the vertical gradient, g x ' and g y ' represents the diagonal gradient, z i (i∈[1,9]) represents the value displayed at each coordinate in the 3×3 Sobel operator template; Step 2: Replace VGG16 in the SSD model with the Inception network structure, and introduce the SE-Net attention mechanism between Conv6-Conv12; Step 3: Introduce the loss function, which includes the position loss αL loc (x,l,g) and classification loss L conf (x,c), the total loss is: Among them, x and c represent the indication amount and the set confidence respectively, l and g represent the prediction box and the background box respectively; Step 3.1: Calculate the position loss. The position loss refers to the loss between the predicted detection frame and the real detection frame, expressed as: Among them, L loc Represents the sub-variable in the position loss, Pos represents the area covered by the detection box, represents the prediction box represented by the i-th sample in the detection box, represents the intermediate variable, Represents the background box corresponding to the jth sample in the central horizontal coordinate, Represents the background box corresponding to the jth sample in the central ordinate, Indicates the network's own predetermined binding box corresponding to the i-th sample in the central horizontal coordinate, Indicates the network's own predetermined binding box corresponding to the i-th sample in the central ordinate, represents the bounding box of the network itself with width w in the i-th sample, represents the network's own predetermined binding box that highly represents w in the i-th sample, represents the background box with width w in the jth sample, and h represents the corresponding height of the pedestrian; Step 3.2: Calculate the classification loss, which is the Softmax loss, specifically: Among them, N represents the number of samples, x and c represent the indication amount and the confidence level of the setting respectively, l and g represent the prediction box and the background box respectively, α represents the weight of the position box loss value, and C x ,C y ,w,h represent the center coordinates (x,y) of the marker box and the height and width of the marker box respectively, and d represents the marker box predicted by the network itself. represents the jth indicator under the i-th sample, represents the intermediate variable, Neg represents the regression area of ​​the sample, represents the confidence of the i-th sample, and p represents the calculation coefficient; Step 4: Set the batch size to 20, the image size to 400×400, the learning rate to 0.005, the number of iterations to 5000, use the L2 regularization standard to train the model, and use the test set to test the model performance.

9. The interactive auxiliary safe driving system according to claim 8, characterized in that The driving front and rear vehicle speed detection and front vehicle collision warning module specifically performs the following steps: Step 1: Define car A as the vehicle driven by the driver. Car A has an external camera O2 in front of it, f is the focal length of the camera, car B is the vehicle in front during driving, M is the rear of car B, P is the horizontal position of the center point of car B, H is the height of the camera on the car, d1 is the distance between car A and car B, O3M is the distance between the point of car B's parking space vertically projected onto the ground and the center line of the image in the image, and the distance O3P between the center points of the two cars can be calculated based on the geometric relationship, specifically: Among them, α1 represents the angle between O2 and the center point of car B and the horizontal line, and γ1 represents the angle between O2 and M and the horizontal line; According to the above formula, the coordinate O2P in the vertical direction Y and the coordinate PQ in the vertical direction X are expressed as: Among them, Q represents the mapping point of the intersection of O2 with the horizontal plane at an angle of α1 on the x-axis, β1 represents the angle between O2 and P and the horizontal plane, P1 represents the intersection of the extended line of O2P and the y-axis, O1 represents the intersection of the extended line of the line connecting O2 and M and the y-axis, and x pix represents the test time, v represents the speed of vehicle A, and v center represents the speed of vehicle B; Step 2: Assume that the height of the vehicle to be tested on the imaging plane at different times is u i (i=0,1,2…), let the relative distances between the two vehicles at two different times be d r1 and d r2 , the relative distance between two different moments can be calculated according to the following formula: Among them, KH represents the actual height of the car, α0 represents the control angle. According to the above formula, the relative distance between the two cars changes as follows: According to the above formula, when l' is less than 0, it means that the two cars are gradually approaching each other; when l' is greater than 0, the two cars are gradually moving away from each other; when the value of |l'| approaches 1 infinitely, the change in the relative distance between the two cars increases infinitely; when the value of |l'| approaches 0 infinitely, the change in the relative distance between the two cars decreases infinitely; Step 3: The relative speed between vehicles is expressed as: Wherein, ΔT represents the time interval between two adjacent frames of detected images, and i=1...n.

10. The interactive auxiliary safe driving system according to claim 9, characterized in that The system also includes: an off-duty inspection and identity authentication module, a fatigue state detection and emotional state detection module, and a dangerous driving behavior detection module; The off-post inspection and identity authentication module includes an off-post inspection unit and an identity authentication unit. The off-post inspection unit is used to detect a face through an in-vehicle camera and perform liveness detection. When no face is detected, the face is blocked or fails the liveness detection, an alarm is triggered. The identity authentication unit is used to match the detected face information with the vehicle owner information set in the data storage. If the match is successful, driving is allowed. If the match fails, an alarm is triggered. The fatigue state detection and emotional state detection module is used to detect whether the driver is driving fatigued through the driver's facial, eye and body features after detecting that driving is allowed, and if so, trigger an alarm, otherwise, do not trigger an alarm; The dangerous driving behavior detection module is used to detect the real-time posture of the driver during driving through a camera. If smoking, drinking, making phone calls, playing with mobile phones, or looking for things are detected, an alarm will be triggered.

Images