Shunting safety intelligent analysis method and system

Abstract

The invention discloses a shunting safety intelligent analysis method and system, and the method comprises the steps: firstly, obtaining driver and passenger video data based on 5G millimeter wave communication; secondly, constructing a neural network algorithm based on a graph space transformation module, an attention module and a bidirectional feature fusion module; then, based on the constructed neural network algorithm, a training model is established, iterative training is carried out, and a discrimination model is established; and finally, identifying the obtained driver and passenger video data by using a discrimination model, and analyzing the fatigue state and behavior state of the driver. Intelligent analysis of multi-dimensional data is implemented through an artificial intelligence deep learning algorithm model, omnibearing diplopia in the shunting operation process is achieved, and professional and intelligent support is provided for safe operation of the shunting locomotive.

Description

Technical Field

[0001] The present invention belongs to the technical field of rail transportation, and in particular relates to a shunting safety intelligent analysis method and system. Background Art

[0002] With the rapid development of my country's economy, the importance of railway transportation in the national economy has become increasingly prominent. Driven by the development of locomotive informationization, the safety of locomotive operation has been significantly improved. However, safety accidents still occur frequently in the shunting operation. In locomotive shunting operations, crossing blocking signals, colliding with earth barriers, running over derailers, and speeding often occur, which may lead to safety risks such as "rushing", "squeezing", and "derailment", becoming a high-incidence area for locomotive accidents.

[0003] At present, the railway safety management department mainly relies on analyzing the LKJ operation record data files generated by the LKJ train operation control recording device to ensure the safety of locomotive shunting. By analyzing the retrieval points of the operation record, the full operation record of the locomotive, and the various equipment index curves of the locomotive operation, the control of the safe operation of the locomotive and the safe operation process of the crew can be achieved. By analyzing the LKJ operation record data files, the analysts can understand the operation status of the locomotive and count the index data of various locomotive on-board equipment to achieve the purpose of operation safety inspection. The locomotive on-board safety protection system device (referred to as the 6A system) realizes the collection of the crew's operation screen and the front, back, left and right road conditions during the driving process by installing a video monitoring system on the locomotive, and also records the voice interaction between the crew and the system. This series of measures effectively ensures that the audio, video and road condition information during the entire operation process are fully recorded. However, the 6A system generates a large amount of multimedia data. The work of locomotive data transfer and management is not only time-consuming and laborious, but also has security issues such as data loss and leakage. The manual copying is inefficient and has poor security. The traditional wireless access technology has low bandwidth and other shortcomings. The safety personnel of the control center manually analyze the data of each device, so this method also faces serious challenges. Therefore, how to more effectively improve the information level of shunting operations has become an increasingly urgent technical problem to be solved. Summary of the invention

[0004] In view of the above problems, the present invention provides a shunting safety intelligent analysis method and system, which are highly intelligent and improve the safety and reliability of shunting.

[0005] The object of the present invention is to provide a shunting safety intelligent analysis method, comprising: Based on 5G millimeter wave communication, obtain video data of drivers and passengers; Construct a neural network algorithm based on the graphic space transformation module, attention module and bidirectional feature fusion module; Based on the constructed neural network algorithm, a training model is established, and iterative training is performed to establish a discriminant model; The discriminant model is used to identify the acquired video data of drivers and passengers, and the driver's fatigue state and behavioral state are analyzed.

[0006] Furthermore, the constructed neural network algorithm includes Backbone module, Neck module and Head module, among which, Introduce multiple graphics space transformation modules into the Backbone module; Multiple attention modules, multiple bidirectional feature fusion modules, and the C2f module in the graphic space transformation module are introduced into the Neck module.

[0007] Furthermore, based on the constructed neural network algorithm, a training model is established, and iterative training is performed to form a discriminant model including: Collect facial video data of the driver including blinking and yawning pictures, and photos and video data of gestures made according to LKJ instructions; The collected photos and video data are screened and framed, and the driver's action posture photo samples and face image samples are selected through web crawlers; Perform data enhancement processing and labeling on the selected driver action posture photo samples and face image samples; The labeled images are formed into a data set, and the data set is divided into a first training set and a first test set according to a certain ratio; The training model is iteratively trained using the first training set, and after the training is completed, the training model is tested using the first test set until the detection accuracy, detection speed, model parameter quantity and model weight size of the training model meet the preset requirements, and the trained training model is used as the discriminant model.

[0008] Further, iteratively training the training model using the first training set includes: In the Backbone module, the graphic space transformation module enhances the driver's face image and behavior image in the first training set to form new training data and sends it to the Neck module; In the Neck module, based on the new training data, the bidirectional feature fusion module extracts feature maps from different layers of the basic network to obtain multi-scale fused feature maps; The multi-scale fusion feature map is input into the C2f module, the multi-scale fusion feature map is enhanced and input into the attention module; The enhanced multi-scale fusion feature map passes through the channel attention module and the spatial attention module of the attention module, and the output features of the channel attention module and the spatial attention module are multiplied element by element to obtain the final attention enhanced feature map, which is sent to the Head module; In the Head module, the attention-enhanced feature map is converted into a prediction result, where the prediction result includes the eye state, mouth state, and behavior state, as well as the PER parameters of the eye state and mouth state, and whether the behavior state is in violation.

[0009] Furthermore, the discriminant model is used to identify the acquired video data of the driver and passenger, and the fatigue state of the driver is analyzed, including: Obtaining eye area images and mouth area images from the driver and passenger video data, and inputting them into the discrimination model; Based on the discriminant model, the eye state and the mouth state are obtained, and the PER parameters of the eye state and the mouth state are calculated; It is determined whether the driver is in a fatigue state based on the calculated PER parameters of the eye state and the mouth state.

[0010] Furthermore, the PER parameter of the eye state satisfies:

[0011] In the formula, f BF Indicates the proportion of time that the eyes are closed within a certain period of time, f close Indicates the total number of frames in the eye-closed state, f total Indicates the total number of time frames within a certain period of time; The PER parameters of the mouth state satisfy:

[0012] In the formula, f yawn The proportion of time the mouth is open over a period of time, t represents the time the mouth is open, T total Indicates the total time over a period of time.

[0013] Further, judging whether the driver is in a fatigue state based on the calculated PER parameters of the eye state and the mouth state includes, When f BF When it is greater than or equal to the first threshold value or f yawn When it is less than the second threshold, the driver is in a fatigue state.

[0014] Furthermore, the discriminant model is used to identify the acquired video data of the driver and passenger, and the driver's behavior status is analyzed, including: Obtaining behavioral images from the driver and passenger video data and inputting them into the discrimination model; Based on the discriminant model, the behavior status is obtained, and based on the LKJ data and voice instructions, it is judged and identified whether the driver's operation is standardized. If the obtained behavior status is consistent with the LKJ data and voice instructions, the driver's operation is standardized, otherwise it is not standardized.

[0015] Furthermore, it also includes establishing a behavior discrimination model, including, Based on the behavior detection algorithm, a behavior training model is established and iterative training is performed; Collect photos and video data of the gestures made by drivers and passengers according to LKJ instructions; The collected photos and video data are screened and framed, and the driver's action and posture photo samples are selected through web crawlers; Perform data enhancement processing on the selected driver action posture photo samples and label them; The labeled images are formed into a data set, and the data set is divided into a second training set and a second test set according to a predetermined ratio; The behavior training model is iteratively trained using the second training set, and after the training is completed, the training model is tested using the second test set until the detection accuracy, detection speed, model parameter quantity and model weight size of the training model meet the preset requirements, and the trained behavior training model is used as the behavior discrimination model.

[0016] Further, iteratively training the behavior training model using the second training set includes: The backbone network VGG19 is used to extract the features of the image and obtain the correlation between key points; The features of the extracted images are sequentially input into multiple serial stage modules. Each stage module is divided into two branches to process the features of the images respectively. The first branch pairs and groups the joint points according to the number of human bodies based on the confidence set of the key points and the confidence of each part. The second branch estimates the position of each joint part of the human body through the affinity vector set of the critical points of the human skeleton, and solves the results obtained by the two branches through the loss function to obtain the solution result. The solution results of the loss functions in all stage modules are summed up, and the driver's behavior status is output whether it is illegal.

[0017] Further, the key points include: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear and left ear; Each coordinate point in the skeleton represents a key point, and the part connecting two key points is called a limb.

[0018] Another object of the present invention is to provide a shunting safety intelligent analysis system, including a 5G ground station, a 5G vehicle-mounted station, a video server, a construction module, a building module and an intelligent analysis device, wherein: The 5G ground station is used to receive the driver and passenger video data sent by the 5G vehicle station and store it in the video server; A construction module is used to construct a neural network algorithm based on a graph space transformation module, an attention module, and a bidirectional feature fusion module; Establish a module for establishing a training model based on the constructed neural network algorithm, and perform iterative training to establish a discriminant model; Intelligent analysis equipment is used to identify the acquired video data of drivers and passengers using the connected discriminant model, and analyze the driver's fatigue state and behavioral state.

[0019] Furthermore, the 5G ground station is an integrated millimeter wave and Sub-6G base station, including a main control unit, a power module, a duplexer and an antenna array, among which: The main control unit includes a 5G NR protocol stack, a millimeter wave baseband unit, a millimeter wave radio frequency, a Sub6G baseband unit, and a Sub6G radio frequency. The 5G NR protocol stack is connected to the millimeter wave baseband unit and the Sub6G baseband unit respectively. The millimeter wave baseband unit is also connected to the millimeter wave radio frequency, and the Sub6G baseband unit is also connected to the Sub6G radio frequency. The power supply module is respectively connected to the main control unit, the duplexer and the antenna array; The duplexer is connected to the Sub6G radio frequency and the antenna array respectively; The antenna array is connected to the millimeter wave radio frequency.

[0020] Further, the 5G millimeter wave baseband unit is used to handle signal processing in the millimeter wave digital domain; The Sub-6G baseband unit is used to process signals in the Sub-6G digital domain; The millimeter wave radio frequency is used to transmit the millimeter wave radio frequency signal to the antenna array; Sub-6G radio frequency is used to convert intermediate frequency signals into radio frequency signals.

[0021] Furthermore, the 5G ground station also includes an OS / BSP operating system.

[0022] The intelligent analysis method of the present invention automatically transfers a large amount of multimedia data through 5G millimeter wave communication technology. The background data center implements intelligent analysis of multi-dimensional data through artificial intelligence deep learning algorithm models, which can more comprehensively identify risk points in the shunting operation process and identify and track the specific operation of the crew, thereby comprehensively evaluating the entire shunting operation. This method helps to further eliminate safety hazards in shunting operations, realize all-round double vision of the shunting operation process, and provide professional and intelligent support for the safe operation of shunting locomotives.

[0023] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 A method for intelligent analysis of shunting safety in an embodiment of the present invention is shown; Figure 2 A diagram showing a spatial transformation network structure in an embodiment of the present invention is shown; Figure 3 A structural diagram of an attention module in an embodiment of the present invention is shown; Figure 4 A block diagram of a neural network algorithm constructed in an embodiment of the present invention is shown; Figure 5 A training module and a discriminant model establishment flow chart in an embodiment of the present invention are shown; Figure 6 A flow chart of a method for analyzing a driver's fatigue state and behavior state in an embodiment of the present invention is shown; Figure 7 A flow chart of a face recognition method in an embodiment of the present invention is shown; Figure 8 A behavior detection algorithm block diagram in an embodiment of the present invention is shown; Fig. 9 A diagram showing the effect of applying a discriminant model in an embodiment of the present invention is shown; Fig.10A structural diagram of a shunting safety intelligent analysis system in an embodiment of the present invention is shown; Fig.11 A block diagram of a 5G machine adjustment security intelligent analysis system in an embodiment of the present invention is shown; Fig.12 A block diagram of a 5G ground base station in an embodiment of the present invention is shown; Fig.13 A 5G ground base station software architecture diagram in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] like Figure 1 As shown, an embodiment of the present invention introduces a shunting safety intelligent analysis method, which is applied to the shunting operation process. The method includes, first, obtaining the video data of the driver and passengers based on 5G millimeter wave communication; then, constructing a neural network algorithm based on the graphic space transformation module, the attention module and the bidirectional feature fusion module; then, establishing a training model based on the constructed neural network algorithm, and performing iterative training to establish a discriminant model; finally, using the discriminant model to identify the obtained video data of the driver and passengers, and analyzing the fatigue state and behavior state of the driver. Through 5G millimeter wave communication technology, a large amount of video data is automatically transferred and stored, and the background data center implements intelligent analysis of multi-dimensional data through an artificial intelligence deep learning algorithm model, which can more comprehensively identify the risk points in the shunting operation process, and identify and track the specific operation of the crew, so as to comprehensively evaluate the entire shunting operation. The above method helps to further eliminate the safety hazards in the shunting operation, realize the all-round double vision of the shunting operation process, and provide professional and intelligent support for the safe operation of the shunting locomotive.

[0028] Specifically, the 5G ground station used in 5G millimeter wave communication obtains video data of drivers and passengers. The 5G ground station is an integrated millimeter wave and Sub-6G base station. The use of an integrated millimeter wave and Sub-6G base station increases the reliability of the communication system, reduces the intensity of manual on-board copying operations, and improves work efficiency and data security.

[0029] Furthermore, based on the graphic space transformation module, the attention module and the bidirectional feature fusion module, a neural network algorithm is constructed. Figure 4 As shown, the constructed neural network algorithm includes a Backbone module, a Neck module and a Head module, wherein the Backbone module includes a first ConvModule module (convolution module), a second ConvModule module, a first C2f_STN module (i.e., a graphic space transformation module), a third ConvModule module, a second C2f_STN module, a fourth ConvModule module, a third C2f_STN module, and an SPPF module (SpatialPyramid Pooling – Fast: pyramid pooling technology) connected in sequence. Multiple graphic space transformation modules are introduced between the ConvModule modules, namely the first C2f_STN module to the third C2f_STN module. Among them, the first C2f_STN module, the second C2f_STN module and the SPPF module are also output to the Neck module. It should be noted that: for the convenience of description Figure 4 The same modules in the Backbone module, Neck module and Head module are distinguished by first, second, etc. according to the execution and / or arrangement order.

[0030] Furthermore, the first C2f_STN module to the third C2f_STN module perform spatial transformation on the facial images and behavioral images of the driver and passengers, thereby enhancing the generalization ability of the algorithm model and helping to improve the extraction of driving micro-motion features. Figure 2As shown in the figure, the spatial transformer network (STN: Spatial Transformer Networks) includes parameter prediction Localisation net, coordinate mapping Grid generator, pixel collection Sampler. STN allows the network to explicitly perform spatial transformations, such as translation, rotation and scaling. By introducing STN in the Backbone module to learn how to perform spatial transformations on the input data, the network can automatically adjust its geometric structure according to the content of the input data, and can handle changes in the position and perspective of objects in the image, and dynamically adjust the position, size and direction of the input data, thereby improving the robustness of the neural network algorithm to input changes. Furthermore, the C2f (CSP Bottleneck with 2 convolutions: CSP bottleneck structure with two convolutions) module improves the feature representation ability and computational efficiency of the network by dividing the feature map into multiple parts, processing these parts separately, and then merging them together. The C2f module is used for feature fusion and information transmission in the Backbone module. Combining the STN algorithm with the C2f module to form the C2f_STN module can reduce the amount of training data and data enhancement processing of the CNN (Convolutional Neural Network), allowing the CNN to learn the shape transformation of the data. The C2f_STN module first performs Mosaic Data Augmentation on the input data set, specifically including randomly selecting four pictures from the input data set, randomly cropping these pictures and splicing them into a new synthetic image as training data. The above data enhancement method is particularly suitable for data sets containing small targets, because it can enrich the image background and expand the scale of the data set, thereby improving the model's detection accuracy and anti-interference ability for small targets.

[0031] The Neck module includes a first Upsample (up-sampling) module, a first Concat BiFPN module (bidirectional feature fusion module), a first C2f module, a second Upsample module, a second Concat BiFPN module, a second C2f module, a first Attention module (attention module), a fifth ConvModule module, a third Concat BiFPN module, a third C2f module, a second Attention module, a sixth ConvModule module, a fourth Concat BiFPN module, a fourth C2f module and a third Attention module, which are connected in sequence. Among them, the first C2f module is also connected to the third Concat BiFPN module, and the second C2f module, the third C2f module and the third Attention module are also output to the head module. The head module includes multiple groups of ConvModule modules, conv2d (convolutional neural network), and bboxloss (bounding box regression loss) modules connected in sequence.

[0032] Specifically, Figure 3As shown in the figure, the attention module includes a channel attention module and a spatial attention module. The channel attention module and the spatial attention module can improve the perception ability of the model, thereby improving performance without increasing the complexity of the network. The channel attention module is used to calculate the importance of each channel in order to better distinguish the features between different channels. The input feature map is subjected to maximum pooling and average pooling in the channel dimension, and then the two pooling results are input into a fully connected layer, and finally a channel attention weight vector is output. The attention weight vector is used to weight each channel in the input feature map, so as to better distinguish the features of different channels. The spatial attention module is used to calculate the importance of each pixel in space, so as to better capture the spatial structure in the image. The input feature map is subjected to average pooling and maximum pooling in the channel dimension, and then the two pooling results are input into a fully connected layer, and finally a spatial attention weight tensor is output. The attention weight tensor is used to weight each pixel in space, so as to better capture the spatial structure in the image. Therefore, the attention module multiplies the output features of the channel attention module and the spatial attention module element by element to obtain the final attention-enhanced feature map, namely the refined feature. The feature maps extracted from different levels are input into multiple attention modules to obtain attention-enhanced features, which further enhances the effect of the feature pyramid network in multi-scale feature extraction and fusion, thereby improving the performance of the model in detecting objects of different sizes. The main function is to identify the driver's micro-movements, such as standing far away and small pictures, and to identify micro-expressions and movements.

[0033] The bidirectional feature fusion module fuses features at different levels through a bidirectional path to improve the model's ability to detect multi-scale targets. The bidirectional feature fusion module is a feature pyramid network structure used for target detection and semantic segmentation tasks. It improves the model's detection performance for targets of different sizes by efficiently fusing feature maps at different levels. Each time the extracted feature map passes through the bidirectional feature fusion BiFPN module, it is an iteration, and each iteration can further enhance the expressiveness of the feature map. Compared with the traditional feature pyramid network, it fuses features through a bidirectional path from top to bottom and from bottom to top, thereby better preserving the semantic information of high-level features and the spatial information of low-level features.

[0034] Specifically, Figure 5 As shown, based on the constructed neural network algorithm, a training model is established, and iterative training is performed to form a discriminant model including: First, a training data set is created, which includes collecting facial video data of drivers including blinking and yawning pictures, and photos and video data of gestures made according to LKJ (train control and monitoring system) instructions; the collected photos and video data are screened and framed, and the photos and facial image samples of the drivers' gestures are selected through web crawlers; Secondly, the selected driver action posture photo samples and face image samples are subjected to data enhancement processing and annotated; the images are annotated using the X-AnyLabeling tool, specifically annotating the face, eyes, mouth, blinking, yawning or specific gestures in the image. The annotated images are formed into a data set, and the data set is divided into a first training set and a first test set according to a certain ratio; the certain ratio may be 4:1, but is not limited thereto, and 5:1 is also applicable to the present invention.

[0035] Then, the training model is iteratively trained using the first training set, and after the training is completed, the training model is tested using the first test set until the detection accuracy, detection speed, model parameter quantity and model weight of the training model meet the preset requirements, and the trained training model is used as the discriminant model. Figure 4 The training of the neural network constructed in includes, First, in the Backbone module, the graphic space transformation module enhances the driver's face image and behavior image in the first training set to form new training data, and sends it to the Neck module; Secondly, in the Neck module, based on the new training data, the bidirectional feature fusion module extracts feature maps from different levels of the basic network to obtain a multi-scale fusion feature map; then, the multi-scale fusion feature map is input into the C2f module, the multi-scale fusion feature map is enhanced and input into the attention module; finally, the enhanced multi-scale fusion feature map passes through the channel attention module and the spatial attention module of the attention module, and the output features of the channel attention module and the spatial attention module are multiplied element by element to obtain the final attention-enhanced feature map, which is sent to the Head module.

[0036] Then, in the Head module, the attention-enhanced feature map is converted into a prediction result, where the prediction result includes the eye state, mouth state and behavior state, as well as the PER parameters of the eye state and mouth state, and whether the behavior state is in violation.

[0037] Furthermore, the method also includes evaluating the training model using the average precision mean mAP, the number of images detected per second FPS by the model, the parameters of the network model and the size of the network model weight evaluation indicators. Among them, mAP (0.5) is mainly used to reflect the recognition ability of the model, where (0.5) means that the intersection-over-union ratio IOU is equal to 0.5, mAP (0.5∶0.95) means the average mAP calculated under multiple IOU thresholds (from 0.5 to 0.95, with a step size of 0.05), mAP ≥ 88.6%, FPS ≥ 50 frames per second, mainly used to reflect the positioning effect and boundary regression ability; FPS is a measure of the running speed, the larger the value, the higher the real-time performance; the number of model parameters and the size of the model weight are important indicators for edge deployment, and the smaller the better without losing accuracy. For example, the number of original data sets is 1395, the image formats of the data sets are jpg and png, and the input image size is 640×640. The SGD (Stochastic Gradient Descent) optimizer is used for training. SGD uses a small batch of training samples to estimate the gradient at each iteration instead of using the entire data set. This can speed up training and avoid falling into local minima by introducing randomness. The initial learning rate is set to 0.02, the total number of training times is set to 200, and the batch size is set to 16. The final discriminant model is connected to the shunting safety intelligent analysis system for testing and verification. The effect is as follows: Fig. 9 shown.

[0038] Finally, the analysis method further includes connecting the discrimination system to the intelligent analysis system to analyze the fatigue state and behavior state of the driver. Specifically, further, the discrimination model is connected to the intelligent analysis system, that is, the intelligent analysis device of the following system. Figure 6 As shown, after the discriminant model in the analysis system reads the video, face recognition and behavior detection are performed on the current frame image at the same time. If fatigue or irregular behavior is found in the detection, an alarm is given and a log is written. If fatigue or irregular behavior does not exist, face recognition and behavior detection are performed on the next frame of the video.

[0039] Further, if Figure 7 As shown, the discriminant model is used to identify the acquired driver and passenger video data, and the driver's fatigue state is analyzed, including: Obtain the eye area image and mouth area image in the video data of the driver and passenger, and input them into the discrimination model, that is, use the constructed neural network to complete the recognition of fatigue status; Based on the discriminant model, the eye state and mouth state are obtained, and the PER parameters of the eye state and mouth state are calculated. Among them, the PER parameters of the eye state satisfy:

[0040] In the formula, f BF Indicates the proportion of time that the eyes are closed within a certain period of time, f close Indicates the total number of frames in the eye-closed state, f total Indicates the total number of time frames within a certain period of time; The PER parameters of the mouth state satisfy:

[0041] In the formula, f yawn The proportion of time the mouth is open over a period of time, t represents the time the mouth is open, T total Indicates the total time within a period of time. The above period of time can be 5 seconds, but is not limited to this, and can be updated according to the application scenario.

[0042] The PER parameters based on the calculated eye state and mouth state are used to determine whether the driver is in a fatigue state, including when f BF When it is greater than or equal to the first threshold value or f yawn When the first threshold is less than the second threshold, the driver is in a fatigued state, otherwise, the driver is in a normal state. Further, the first threshold may be, but not limited to, 0.35, and the second threshold may be, but not limited to, 0.25. Furthermore, when the driver is in a fatigued state, an alarm is given and a log is written.

[0043] In an embodiment of the present invention, a discriminant model is used to identify the acquired video data of the driver and passenger, and the driver's behavior status is analyzed, including, first, acquiring the behavior image in the video data of the driver and passenger, and inputting it into the discriminant model; then, based on the discriminant model, acquiring the behavior status, and judging and identifying whether the driver and passenger's operation is standardized based on the LKJ data and voice instructions, wherein, if the acquired behavior status is consistent with the LKJ data and the voice instructions, the driver's operation is standardized, otherwise it is not standardized.

[0044] In an embodiment of the present invention, the analysis method further includes establishing a behavior discrimination model, including: Based on the behavior detection algorithm, a behavior training model is established and iterative training is performed; Collect photos and video data of the gestures made by drivers and passengers according to LKJ instructions; The collected photos and video data are screened and framed, and the driver's action and posture photo samples are selected through web crawlers; The selected images are subjected to data enhancement and labeling; the images are labeled using the X-AnyLabeling tool, specifically labeling specific gestures in the images. The predetermined ratio may be 4:1, but is not limited thereto, and 5:1 is also applicable to the present invention. The labeled images are formed into a data set, and the data set is divided into a second training set and a second test set according to a predetermined ratio; The behavior training model is iteratively trained using the second training set, and after the training is completed, the training model is tested using the second test set until the detection accuracy, detection speed, model parameter quantity and model weight size of the training model meet the preset requirements, and the trained behavior training model is used as the behavior discrimination model.

[0045] Behavior detection algorithms such as Figure 8 As shown in the figure, it includes VGG19 and multiple serial stage modules, wherein VGG19 (Visual Geometry Group with 19 layers: deep convolutional neural network module) is used to extract the features of the image and obtain the correlation between the key points; then, the features of the extracted images are sequentially input into multiple serial stage modules, and the stage module includes parallel partial affinity fields paf and confidence maps pcm, and loss_pcm and loss_paf represent the corresponding loss functions. The paf part is used to estimate the position of each joint of the human body, which solves the problem of only encoding the position and too many support areas. The pcm confidence map part is used to pair and group the joint points according to the number of human bodies according to the confidence of each part. The confidence map module improves the accuracy through continuous iterative prediction. In order to strengthen the data results of the next stage, the partial affinity fields trained in the current stage are refined. The affinity module converts the pixels detected in the image into a flow field with direction, so that the pixels have directionality. In the case of a single person, it is only necessary to connect each part in turn; in the case of multiple people, it is necessary to vectorize and encode each part and pair it to different people before connecting. Through the joint action of these two modules, the recognition of behavioral states is completed. The network predicts the PCM and PAF of body parts through continuous iteration, and in order to estimate the relationship between the detection results and the true labels, the Loss loss function is added to the end of each stage of each branch network. In addition, a partial affinity field is introduced to represent the correlation between key points, making the clustering process more efficient and accurate.

[0046] Furthermore, each stage module is divided into two branches to process the features of the image and perform convs convolution operations respectively. Each branch is composed of multiple convolution kernel layers and loss function layers. The iterative training is carried out in two stages. The first branch pairs and groups the joints based on the confidence set of the key points according to the number of human bodies; the second branch estimates the position of each joint of the human body through the affinity vector set of the critical points of the human skeleton; one branch generates PCM and one branch generates PAF. The results obtained by the two branches are summed, that is, the PCM and PAF of each stage will perform Loss solution and output the summation result. Finally, all Losses (loss functions) are summed to determine whether the behavior of the driver and passengers is in violation of the rules. The key points include: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear and left ear; each coordinate point in the skeleton represents a key point, and the part connecting the two key points is called a limb. Furthermore, the behavioral states of drivers and passengers include smoking, making phone calls, playing with mobile phones, leaving their posts, not patrolling, standing guard, not making gestures, and not looking out.

[0047] In an embodiment of the present invention, a behavior discrimination model is used to identify the acquired video data of the driver and passenger, and the driver's behavior status is analyzed, including, first, acquiring the behavior image in the video data of the driver and passenger, and inputting it into the behavior discrimination model; then, based on the behavior discrimination model, acquiring the behavior status, and judging and identifying whether the driver and passenger's operation is standardized based on the LKJ data and voice instructions, wherein, if the acquired behavior status is consistent with the LKJ data and the voice instructions, the driver's operation is standardized, otherwise it is not standardized.

[0048] Through 5G millimeter wave technology, a large amount of vehicle-mounted video and operation and maintenance data (i.e., LKJ data and voice commands) needs to be transmitted back to the ground control center in a timely, safe, and reliable manner, and then the data is uniformly and intelligently analyzed and processed through the artificial intelligence deep learning algorithm model, realizing the automatic association of recording files, video data, and LKJ data. A comprehensive, continuous, complete, accurate, and timely data chain between people and vehicles has been constructed, and a shunting safety intelligent analysis system covering all-round information of locomotives has been established. The system has the characteristics of ultra-large capacity, fully automatic dumping, super strong anti-interference, strong environmental suitability, simple operation and maintenance, and safety and reliability, which fully meets the needs of intelligent analysis of vehicle-mounted data transmission in scenarios such as smart railways, smart urban rail, and smart ports.

[0049] like Fig.10As shown, the embodiment of the present invention also introduces a shunting safety intelligent analysis system capable of executing the above method, the intelligent analysis system includes a 5G ground station, a 5G vehicle-mounted station, a video server, a construction module, a building module and an intelligent analysis device, wherein the 5G ground station is used to receive the video information of the driver and passenger sent by the 5G vehicle-mounted station and store it in the video server; the construction module is used to construct a neural network algorithm based on the graphic space transformation module, the attention module and the bidirectional feature fusion module, and the building module is used to establish a training model based on the constructed neural network algorithm, and iterative training is performed to establish a discriminant model; the intelligent analysis device is used to use the accessed discriminant model to identify the acquired driver and passenger video data and analyze the driver's fatigue state and behavior state. Based on the integrated base station equipment of millimeter wave and Sub-6G, the on-board video data is wirelessly transmitted, which reduces the intensity of manual on-board copying operations and improves work efficiency and data security. In addition, through the analysis of the intelligent analysis device, a comprehensive, continuous, complete, accurate and timely human and vehicle data chain is constructed, and a shunting safety intelligent analysis system covering all-round information of locomotives is established. The system has the characteristics of ultra-large capacity, fully automatic dump, super anti-interference, strong environmental adaptability, simple operation and maintenance, and safety and reliability, fully meeting the needs of on-board data transmission and intelligent analysis in scenarios such as smart railways and smart urban rail.

[0050] Specifically, Fig.11 As shown, the 5G ground station, video server and intelligent analysis equipment are set up at the station, and the 5G vehicle-mounted station is set up on the train. Further, the system also includes a vehicle-mounted gateway. Further, as Fig.12As shown, the 5G ground station is a millimeter wave and Sub-6G integrated base station, including a main control unit, a power module, a duplexer and an antenna array, wherein the main control unit includes a 5G NR protocol stack, a millimeter wave baseband unit, a millimeter wave radio frequency, a Sub6G baseband unit and a Sub6G radio frequency, and the 5G NR (New Radio, wireless access technology of 5G network) protocol stack is respectively connected to the millimeter wave baseband unit and the Sub6G baseband unit, the millimeter wave baseband unit is also connected to the millimeter wave radio frequency, and the Sub6G baseband unit is also connected to the Sub6G radio frequency; the power module is respectively connected to the main control unit, the duplexer and the antenna array; the duplexer is respectively connected to the Sub6G radio frequency and the antenna array; the antenna array is connected to the millimeter wave radio frequency. Among them, the 5G millimeter wave baseband unit is used to process the signal processing in the millimeter wave digital domain; the Sub-6G baseband unit is used to process the signal processing in the Sub-6G digital domain; the millimeter wave radio frequency is used to transmit the millimeter wave radio frequency signal to the antenna array; the Sub-6G radio frequency is used to convert the intermediate frequency signal into the radio frequency signal. The power module uses an AC (alternating current) input power module of 100~220V, 50~60Hz. Through the switch and linear power module, it converts and outputs DC (direct current) 12V, 48V, 5V and other DC inputs required by the internal mainboards and modules. The power supply meets the power consumption requirements of the base station. The integrated base station is connected to the core network using optical fiber, and uses Ethernet to interconnect with the local LMT as a local maintenance port. It supports operation and maintenance through the network management (OMCB / LMT): configuration management, fault management, performance management, version management, front-end and back-end communication management, and diagnostic management.

[0051] 5G millimeter wave AIP (Antenna in Package) antenna module indicators: frequency range 26.5Ghz~27.5Ghz (gigahertz), carrier bandwidth 200Mhz / 400Mhz (megahertz), MIMO (Multiple-Input Multiple-Output) method, antenna array scale , the beam scanning range is ±60° horizontally and ±45° vertically. The specific parameter indicators are shown in the following table: Table 1 AIP antenna module parameter indicators

[0052] In the embodiment of the present invention, Fig.13As shown, the 5G ground station also includes an OS / BSP operating system, which is used to support the operation of the entire software and the provision of system services. After the platform layer encapsulates the OS (Operating System) service, it provides the required service interface to all applications upward. Specifically, the NRL1 physical layer protocol stack communicates with Layer 2 through the FAPI interface; the complete NR Layer 2 protocol stack (L2) includes the scheduler control plane and the user plane, and communicates with Layer 1 through the FAPI interface; the complete NR Layer 3 protocol stack (L3) includes RRC, RRM and Ng / Xn interface management; OAM (Operations, Administration and Maintenance) also includes adjustable and testable functional modules of the base station, such as including but not limited to Log, Trace, WebLmt and other operation and maintenance test modules.

[0053] Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent substitutions for some of the technical features therein; and these modifications or substitutions do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent analysis of shunting safety, characterized in that: include, Based on 5G millimeter wave communication, obtain video data of drivers and passengers; Construct a neural network algorithm based on the graphic space transformation module, attention module and bidirectional feature fusion module; Based on the constructed neural network algorithm, a training model is established, and iterative training is performed to establish a discriminant model; The discriminant model is used to identify the acquired video data of drivers and passengers, and the driver's fatigue state and behavioral state are analyzed.

2. The intelligent analysis method for shunting safety according to claim 1 is characterized in that: The constructed neural network algorithm includes Backbone module, Neck module and Head module, among which, Introduce multiple graphics space transformation modules into the Backbone module; Multiple attention modules, multiple bidirectional feature fusion modules, and the C2f module in the graphic space transformation module are introduced into the Neck module.

3. The intelligent analysis method for shunting safety according to claim 2 is characterized in that: Based on the constructed neural network algorithm, a training model is established and iterative training is performed to form a discriminant model including: Collect facial video data of the driver including blinking and yawning pictures, and photos and video data of gestures made according to LKJ instructions; The collected photos and video data are screened and framed, and the driver's action posture photo samples and face image samples are selected through web crawlers; Perform data enhancement processing and labeling on the selected driver action posture photo samples and face image samples; The labeled images are formed into a data set, and the data set is divided into a first training set and a first test set according to a certain ratio; The training model is iteratively trained using the first training set, and after the training is completed, the training model is tested using the first test set until the detection accuracy, detection speed, model parameter quantity and model weight size of the training model meet the preset requirements, and the trained training model is used as the discriminant model.

4. The intelligent analysis method for shunting safety according to claim 3 is characterized in that: Iteratively training the training model using the first training set includes: In the Backbone module, the graphic space transformation module enhances the driver's face image and behavior image in the first training set to form new training data and sends it to the Neck module; In the Neck module, based on the new training data, the bidirectional feature fusion module extracts feature maps from different layers of the basic network to obtain multi-scale fused feature maps; The multi-scale fusion feature map is input into the C2f module, the multi-scale fusion feature map is enhanced and input into the attention module; The enhanced multi-scale fusion feature map passes through the channel attention module and the spatial attention module of the attention module, and the output features of the channel attention module and the spatial attention module are multiplied element by element to obtain the final attention enhanced feature map, which is sent to the Head module; In the Head module, the attention-enhanced feature map is converted into a prediction result, where the prediction result includes the eye state, mouth state, and behavior state, as well as the PER parameters of the eye state and mouth state, and whether the behavior state is in violation.

5. The intelligent analysis method for shunting safety according to claim 4 is characterized in that: The discriminant model is used to identify the acquired video data of drivers and passengers, and the driver's fatigue state is analyzed, including: Obtaining eye area images and mouth area images from the driver and passenger video data, and inputting them into the discrimination model; Based on the discriminant model, the eye state and the mouth state are obtained, and the PER parameters of the eye state and the mouth state are calculated; It is determined whether the driver is in a fatigue state based on the calculated PER parameters of the eye state and the mouth state.

6. The intelligent analysis method for shunting safety according to claim 5 is characterized in that: The PER parameters of the eye state satisfy: In the formula, f BF Indicates the proportion of time that the eyes are closed within a certain period of time, f close Indicates the total number of frames in the eye-closed state, f total Indicates the total number of time frames within a certain period of time; The PER parameters of the mouth state satisfy: In the formula, f yawn The proportion of time the mouth is open over a period of time, t represents the time the mouth is open, T total Represents the total time over a period of time.

7. The intelligent analysis method for shunting safety according to claim 6 is characterized in that: Judging whether the driver is in a fatigue state based on the calculated PER parameters of the eye state and mouth state includes, When f BF When it is greater than or equal to the first threshold value or f yawn When it is less than the second threshold, the driver is in a fatigue state.

8. The intelligent analysis method for shunting safety according to claim 4 is characterized in that: The discriminant model is used to identify the acquired video data of the driver and passenger, and the driver's behavior status is analyzed, including: Obtaining behavioral images from the driver and passenger video data and inputting them into the discrimination model; Based on the discriminant model, the behavior status is obtained, and based on the LKJ data and voice instructions, it is judged and identified whether the driver's operation is standardized. If the obtained behavior status is consistent with the LKJ data and voice instructions, the driver's operation is standardized, otherwise it is not standardized.

9. The intelligent analysis method for shunting safety according to claim 1 is characterized in that: It also includes building a behavioral discrimination model, including, Based on the behavior detection algorithm, a behavior training model is established and iterative training is performed; Collect photos and video data of the gestures made by drivers and passengers according to LKJ instructions; The collected photos and video data are screened and framed, and the driver's action and posture photo samples are selected through web crawlers; Perform data enhancement processing on the selected driver action posture photo samples and label them; The labeled images are formed into a data set, and the data set is divided into a second training set and a second test set according to a predetermined ratio; The behavior training model is iteratively trained using the second training set, and after the training is completed, the training model is tested using the second test set until the detection accuracy, detection speed, model parameter quantity and model weight size of the training model meet the preset requirements, and the trained behavior training model is used as the behavior discrimination model.

10. The intelligent analysis method for shunting safety according to claim 9 is characterized in that: Iteratively training the behavior training model using the second training set includes: The backbone network VGG19 is used to extract the features of the image and obtain the correlation between key points; The features of the extracted images are sequentially input into multiple serial stage modules. Each stage module is divided into two branches to process the features of the images respectively. The first branch pairs and groups the joint points according to the number of human bodies based on the confidence set of the key points and the confidence of each part. The second branch estimates the position of each joint part of the human body through the affinity vector set of the critical points of the human skeleton, and solves the results obtained by the two branches through the loss function to obtain the solution result. The solution results of the loss functions in all stage modules are summed up, and the driver's behavior status is output whether it is illegal.

11. The intelligent analysis method for shunting safety according to claim 10 is characterized in that: The key points include: nose, neck, right shoulder, right elbow, right wrist, left shoulder, left elbow, left wrist, right hip, right knee, right ankle, left hip, left knee, left ankle, right eye, left eye, right ear and left ear; Each coordinate point in the skeleton represents a key point, and the part connecting two key points is called a limb.

12. A shunting safety intelligent analysis system, characterized in that: It includes 5G ground station, 5G vehicle station, video server, construction module, establishment module and intelligent analysis equipment, among which: The 5G ground station is used to receive the driver and passenger video data sent by the 5G vehicle station and store it in the video server; A construction module is used to construct a neural network algorithm based on a graph space transformation module, an attention module, and a bidirectional feature fusion module; Establish a module for establishing a training model based on the constructed neural network algorithm, and perform iterative training to establish a discriminant model; Intelligent analysis equipment is used to identify the acquired video data of drivers and passengers using the connected discriminant model, and analyze the driver's fatigue state and behavioral state.

13. The intelligent analysis system for shunting safety according to claim 12 is characterized in that: The 5G ground station is an integrated millimeter wave and Sub-6G base station, including a main control unit, a power module, a duplexer, and an antenna array. The main control unit includes a 5G NR protocol stack, a millimeter wave baseband unit, a millimeter wave radio frequency, a Sub6G baseband unit, and a Sub6G radio frequency. The 5G NR protocol stack is connected to the millimeter wave baseband unit and the Sub6G baseband unit respectively. The millimeter wave baseband unit is also connected to the millimeter wave radio frequency, and the Sub6G baseband unit is also connected to the Sub6G radio frequency. The power supply module is respectively connected to the main control unit, the duplexer and the antenna array; The duplexer is connected to the Sub6G radio frequency and the antenna array respectively; The antenna array is connected to the millimeter wave radio frequency.

14. The intelligent analysis system for shunting safety according to claim 13 is characterized in that: 5G millimeter wave baseband unit, which handles signal processing in the millimeter wave digital domain; The Sub-6G baseband unit is used to process signals in the Sub-6G digital domain; The millimeter wave radio frequency is used to transmit the millimeter wave radio frequency signal to the antenna array; Sub-6G radio frequency is used to convert intermediate frequency signals into radio frequency signals.

15. The intelligent analysis system for shunting safety according to claim 13, characterized in that: The 5G ground station also includes an OS / BSP operating system.

Images