Locomotive crew behavior detection method, system and equipment based on computer vision

Abstract

The invention provides a locomotive steward behavior detection method, system and equipment based on computer vision. The method comprises the following steps: constructing a steward key skeleton point training set and an object target training set; respectively training an object target detection model and a steward attitude skeleton point detection model based on a deep neural network; video images of a locomotive cab are obtained in real time in the locomotive driving process and input into the detection model, and object target detection information and steward posture skeleton point information are deduced; and analyzing the LKJ monitoring data in real time and searching whether an LKJ event exists or not, and if the LKJ event exists and is within the required time or mileage range at present, judging whether a first type of violation behaviors exist or not according to the object target detection information which is inferred at present and the attitude skeleton point information of the steward. The method has the advantages of being simple in implementation method, low in cost, high in efficiency and precision, stable, reliable and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of rail transit intelligent detection, and in particular to a method, system and device for detecting locomotive crew behavior based on computer vision. Background Art

[0002] In order to ensure the safe and reliable operation of the locomotive, it is necessary to conduct intelligent detection and analysis of the behavior of the crew members in the cab of the locomotive, including the driver, to monitor in real time whether the crew members' behavior complies with the regulations and whether there are any violations. For intelligent detection of human behavior, the prior art usually extracts features by collecting human images, such as grayscale information of the image background, joint coordinate data, etc., and judges the type of human movement based on the extracted features. However, there are many detection targets and action types for locomotive crew members, and the action postures of different types are also varied. The above-mentioned human behavior detection in the prior art is only a simple judgment of human behavior based on human action detection, and is not suitable for the detection of complex locomotive crew behavior.

[0003] The behavior of locomotive crew members is often associated with related objects in the cab or carriage environment. Simple human motion detection cannot directly determine whether the crew members' behavior is illegal. For example, to detect whether the driver's hands have left the console, it is necessary to monitor the state of the driver's hands and the distance between the driver and the console at the same time. It is not possible to determine whether the driver has left the console directly based on the state of the driver's hands. At the same time, there are multiple events involved in the locomotive driving process, such as entering the section, exiting the section, entering shunting, shunting parking, etc. Different events may involve a variety of different detection points, and there are many types of detection targets and actions. Each detection target and action is often associated with various events in the locomotive driving process, such as gesture action detection in driving operations, probe action detection in shunting operations, etc. Traditional human behavior detection based on human motion detection is difficult to accurately detect and identify the behavior of locomotive crew members in different events in a timely manner. Summary of the invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a locomotive crew behavior detection method, system and equipment based on computer vision, which has a simple implementation method, low cost, high efficiency and accuracy, and is stable and reliable.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for detecting locomotive crew behavior based on computer vision, comprising:

[0007] Construct a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants;

[0008] Based on a deep neural network, the crew key skeleton point training set and the object target training set are used to train a crew posture skeleton point detection model and an object target detection model respectively;

[0009] Acquire a video image of the locomotive cab in real time during the locomotive driving process, input the image into the object target detection model and the crew posture skeleton point detection model, respectively, and infer the object target detection information and the crew posture skeleton point information;

[0010] Real-time analysis of LKJ (train operation monitoring and recording device) monitoring data and retrieval of whether there is an LKJ recording event corresponding to the required monitoring violation item. If so and the current time or mileage is within the required time or mileage range of the LKJ recording event, determine whether there is a first type of violation based on the currently inferred object target detection information and the crew posture skeleton point information. The first type of violation is a violation related to the LKJ recording event.

[0011] Furthermore, the violation items include any one or more of the main driver's failure to make hand signals, failure to look behind, failure to raise the pantograph for confirmation, failure to wear a vest when getting off the vehicle, and the main driver's hands leaving the operating console; the LKJ recorded events include any one or more of entering the section, exiting the section, entering shunting, shunting parking, shunting starting, exiting shunting, parking in the station, starting in the station, LKJ siding input, and over-phase events.

[0012] Furthermore, it also includes constructing corresponding item point detection models for each violation item that needs to be monitored, and each item point detection model determines whether there is a corresponding violation based on the inferred object target detection information and the crew member's posture skeleton point information. The item point detection models include any one or more of the main driver's no hand ratio detection model, no rear lookout detection model, no bow raising confirmation detection model, no vest wearing detection model when getting off the vehicle, and the main driver's hands off the console detection model.

[0013] Furthermore, the main driver's hand gesture detection model calculates the angle between the driver's upper arm vector and the trunk vector, the angle between the upper arm vector and the forearm vector, and the angle between the forearm vector and the vertical direction according to the crew member's posture skeleton point information, and determines the execution of the main driver's hand gesture action; the rear lookout detection model calculates the horizontal angle between the crew member's left shoulder and right shoulder vector according to the position area of ​​the doors and windows in the object target detection information, and determines the execution of the crew member's bow raising confirmation action according to the calculated horizontal angle and the crew member's posture skeleton point information; the crew member getting off the vehicle without wearing a vest detection model determines the position area of ​​the doors and windows and whether the crew member is wearing a vest according to the object target detection information, and determines the action state of the crew member getting off the vehicle according to the crew member's posture skeleton point information, and confirms the execution of the crew member getting off the vehicle action; the main driver's hands leaving the control panel detection model determines the edge position of the locomotive control panel according to the body target detection information, and determines the state of the crew member's hands according to the crew member's posture skeleton point information, and confirms the execution of the main driver's hands leaving the control panel action.

[0014] Furthermore, it also includes judging whether there is a second type of violation based on the inferred object target detection information and the crew member's posture skeleton point information, wherein the first type of violation is a violation that requires full-time monitoring.

[0015] Furthermore, the construction of the training set of key skeleton points of the flight attendants and the training set of object targets required for analyzing various behaviors of the flight attendants includes:

[0016] Obtain historical video image data of the locomotive cab;

[0017] Perform frame extraction and image deduplication processing on the acquired video image data;

[0018] Annotate the key skeleton points of the object target and the crew member in the processed data frame;

[0019] Based on the labeled image data, the flight attendant key skeleton point training set and the object target training set are constructed respectively.

[0020] Furthermore, the YOLOv model was used to train the object target detection model and the flight attendant posture skeleton point detection model. During the model training process, additional auxiliary heads were set in the middle layer of the deep neural network, weights for guiding auxiliary losses were set for the shallow network, and the batch normalization operation was implemented by integrating the mean and variance of the batch normalization into the bias and weight of the convolutional layer to assist in completing the training.

[0021] Furthermore, in the process of processing the video image, the video image data is read and analyzed based on the streaming media analysis and processing plug-in under the DeepStream framework, the hardware decoder is called to accelerate the decoding of the video image data, and the decoded image data is preprocessed.

[0022] Furthermore, in the training of the object target detection model and the crew posture skeleton point detection model, it also includes calling the inference acceleration optimizer to optimize any one or more of the weight parameters, fusion network layers and optimization of video memory allocation in the object target detection model and the crew posture skeleton point detection model.

[0023] Furthermore, it also includes using two parallel pipelines for constructing object target detection model and crew member posture skeleton point detection model inference, and inputting the stream data to be inferred into the parallel pipelines of the target detection model and the crew member posture skeleton point detection model for model inference.

[0024] A locomotive crew behavior detection system based on computer vision, comprising:

[0025] A training set construction module is used to construct a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants;

[0026] A model building module, for training a flight attendant posture skeleton point detection model and an object target detection model using the flight attendant key skeleton point training set and the object target training set respectively based on a deep neural network;

[0027] A detection module, used to obtain a video image of the locomotive cab in real time during the locomotive driving process, input it into the object target detection model and the crew posture skeleton point detection model respectively, and infer the object target detection information and the crew posture skeleton point information;

[0028] The behavior detection module is used to parse the LKJ monitoring data in real time and retrieve whether there is an LKJ recorded event corresponding to the required monitoring violation item. If so and the current time is within the time or mileage range required by the LKJ recorded event, it is determined whether there is a first type of violation based on the currently inferred object target detection information and the crew posture skeleton point information. The first type of violation is a violation related to the LKJ recorded event.

[0029] Furthermore, each module is constructed in the DeepStream framework, and the streaming media analysis and processing plug-in is used to read and parse the video image data, and the hardware decoder is called to accelerate the decoding of the video image data.

[0030] Furthermore, DeepStream also includes two parallel pipelines for performing reasoning using the object target detection model and the crew posture skeleton point detection model, respectively, and / or DeepStream also includes an inference acceleration optimizer for performing acceleration optimization.

[0031] An electronic device comprises a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0032] A computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0033] Compared with the prior art, the advantages of the present invention are as follows: the present invention first constructs a crew key skeleton point training set and an object target training set, and then trains to form an object target detection model and a crew posture skeleton point detection model. During the locomotive driving process, the video image of the locomotive cab is used to detect the crew posture skeleton points and related object targets in real time by using the two detection models. At the same time, LKJ record events are retrieved by real-time analysis of LKJ monitoring data. When the LKJ record events are retrieved, the crew behavior is identified and judged based on the model reasoning results. The crew behavior detection can be associated with the LKJ data, and various point detections can be started in time. The LKJ data and the inferred object target detection information and crew posture skeleton point information are fully integrated to improve the detection efficiency and accuracy, and effectively reduce the complexity of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of the implementation flow of the locomotive crew behavior detection method based on computer vision in this embodiment.

[0035] Figure 2 It is a schematic diagram of the implementation process of locomotive image data annotation and training set construction in this embodiment.

[0036] Figure 3 It is a schematic diagram of the implementation process of model training, conversion and optimization on a server platform equipped with multiple GPU graphics cards in a specific application embodiment.

[0037] Figure 4 It is a schematic diagram of the implementation process of constructing a two-item detection model in a specific application embodiment.

[0038] Figure 5 It is a schematic diagram of the implementation process of realizing the detection of locomotive crew behavior at various points in a specific application embodiment.

[0039] Figure 6It is a schematic diagram of the structural principle of a locomotive crew behavior detection system built under the DeepStream framework in a specific application embodiment.

[0040] Figure 7 It is a schematic diagram of the connection relationship principle of the locomotive crew behavior detection system in a specific application embodiment. DETAILED DESCRIPTION

[0041] The present invention is further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the protection scope of the present invention is not limited thereby.

[0042] As shown in the disclosure of the present invention, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not specifically refer to the singular, but may also include the plural. The words "first", "second" and similar words used in the disclosure of the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, the words "include" or "comprise" and the like mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

[0043] like Figure 1 As shown, the steps of the locomotive crew behavior detection method based on computer vision in this embodiment include:

[0044] Step S1. Constructing a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants;

[0045] Step S2. Based on a deep neural network, the crew key skeleton point training set and the object target training set are used to train a crew posture skeleton point detection model and an object target detection model respectively;

[0046] Step S3. Acquire the video image of the locomotive cab in real time during the locomotive driving process, input it into the object target detection model and the crew posture skeleton point detection model respectively, and infer the object target detection information and the crew posture skeleton point information;

[0047] Step S4. Real-time analysis of LKJ monitoring data and retrieval of whether there is an LKJ recorded event corresponding to the required monitoring violation point. If yes and the current time or mileage range required by the LKJ recorded event is within, based on the currently inferred object target detection information and crew posture skeleton point information, determine whether there is a first type of violation, which is a violation related to the LKJ recorded event. The above LKJ recorded event is the event recorded in the LKJ monitoring data.

[0048] In this embodiment, by first constructing a training set of crew key skeleton points and an object target training set, and then training to form two detection models (object target detection model, crew posture skeleton point detection model), the video image of the locomotive cab is used in the process of locomotive driving, and the two detection models are used to detect the crew posture skeleton points and related object targets in real time. At the same time, LKJ recorded events are retrieved by real-time analysis of LKJ monitoring data. When the LKJ recorded events are retrieved, the inferred object target detection information and the crew posture skeleton point information are fully integrated according to the model reasoning results to perform behavior recognition and judgment, so as to ensure the recognition accuracy. By associating the crew behavior detection with the LKJ data, it is possible to start the detection of each point in a timely and accurate manner, effectively reducing the complexity of detection and improving detection efficiency and accuracy.

[0049] In this embodiment, the construction of the crew key skeleton point training set and the object target training set required for analyzing various behaviors of the crew includes:

[0050] Step S101. Acquire historical video image data of the locomotive cab;

[0051] Step S102: performing frame extraction and image deduplication processing on the acquired video image data;

[0052] Step S103: marking key skeleton points of the object target and the crew member in the processed data frame;

[0053] Step S104: construct a flight attendant key skeleton point training set and an object target training set based on the labeled image data.

[0054] In a specific application embodiment, Figure 2 As shown in the figure, the locomotive 6A system (locomotive onboard safety protection system) driver's cab video files are collected from the data storage server of the locomotive depot data center. After frame extraction and image deduplication processing, the key skeleton points of the object targets such as mobile phones in the locomotive cab and the crew are annotated by Labelme software, and the object target training set and the training set of 17 key skeleton points of the crew are respectively produced. The training data set specifically includes two files: image and annotation. The object target annotation file includes target category labels, normalized horizontal and vertical coordinates of the center point, and normalized width and height, and the human body posture skeleton point annotation file includes skeleton labels, normalized horizontal and vertical coordinates of the center point, and other information.

[0055] This embodiment specifically processes the video image under the DeepStream framework, reads and analyzes the video image data based on the GStreamer streaming media analysis and processing plug-in in the DeepStream framework, and accelerates the decoding of the video image data by calling the NVDEC hardware decoder to achieve fast video parsing and decoding, thereby solving the problem of low real-time performance of traditional methods of reading, parsing and encoding and decoding large-scale locomotive 6A driver's cab video data, and then preprocessing operations are performed on the decoded image data.

[0056] In this embodiment, the YOLOv7 model is used as the deep neural network target detection framework of the training model, and the object target detection model and the crew posture skeleton point model are uniformly trained through the video analysis server equipped with NVIDIA multi-GPU graphics cards. The YOLOv7 detector has very high detection accuracy and detection speed. It is understandable that other types of deep neural network models (such as YOLOv4) of the YOLOv series can also be used, and even other types of deep neural network models other than the YOLOv series can be used. An auxiliary training module is added during the training process of the model. During the training process of the object target detection model and the crew posture skeleton point detection model, an additional auxiliary head is set in the middle layer of the deep neural network, and the weight for guiding the auxiliary loss is set for the shallow network, and the batch normalization operation is implemented by integrating the mean and variance of the batch normalization into the bias and weight of the convolution layer to assist in completing the training and improve the detection accuracy of the model. After training, two detection models (object target detection model and crew posture skeleton point detection model) are generated, which are the object detection model object.pt file and the crew skeleton point detection model pose.pt file.

[0057] The use of CUDA's parallel computing platform and model compression method can improve the reasoning speed and computing efficiency of deep neural network models and optimize embedded hardware deployment. This embodiment specifically defines a set of standard formats in ONNX that are independent of the environment and platform to enhance the interactivity of various AI models, so that for training models under various training frameworks (such as TensorFlow, Pytorch, and Paddle, etc.), the models of the above frameworks can be uniformly converted to the ONNX unified format for storage after training. Then use the model conversion tool to convert the object.pt and pose.pt model files under the pytorch deep learning framework into open neural network pose.onnx and object.onnx model files, such as Figure 3 shown.

[0058] In the training of the object target detection model and the crew posture skeleton point detection model, this embodiment also includes calling the TensorRT reasoning acceleration optimizer to optimize the weight parameters, network layer fusion, and video memory allocation in the object target detection model and the crew posture skeleton point detection model, that is, to achieve optimization through weight parameter optimization, network layer fusion optimization, and GPU kernel call optimization. TensorRT is a high-performance reasoning framework that runs on an NVIDIA image processing unit (GPU). This embodiment optimizes the model using the TensorRT reasoning acceleration optimizer, which can improve the model operation and reasoning speed, achieve low latency and high throughput of model reasoning, and facilitate deployment on embedded hardware platforms.

[0059] In a specific application embodiment, Figure 3 As shown, the TensorRT inference acceleration engine tool is used to optimize the training model in ONNX format to form a TensorRT Engine file, and an Engine file with fewer parameters and faster inference speed can be obtained. The optimization specifically includes: reducing the flow of data and the use of video memory by fusing or deleting redundant operators. Although the object.engine and pose.engine with INT8 precision have fewer network parameters and occupy less memory, the inference speed of the model is faster, but the model detection accuracy will be sacrificed. In this embodiment, the trained model is quantized to replace its data type with INT8 or FP16 with lower precision, that is, the object.engine and pose.engine with FP32 precision are quantized to FP16. It is also possible to generate an engine file with INT8 precision instead of FP16 during the optimization process of the training model, so that the inference speed of the deep neural network model is faster. Optimize the original layer of the network vertically, and merge the CONV, BN, and RELU layers into a CBR layer. Optimize the network layer horizontally, for example, merge all 1x1 CBRs into a large CBR, or directly delete the concat layer of the network, and directly input the input of the contact layer into the network layer below to reduce the transmission throughput. According to different graphics card architectures, the number of stream processors, and core frequencies, select the optimization strategy and calculation method that best suits the current platform.

[0060] In this embodiment, DeepStream is also used to build two parallel pipelines for object target detection model and crew member posture skeleton point detection model reasoning, and the stream data to be inferred are respectively input into the parallel pipelines of the target detection model and the crew member posture skeleton point detection model for model reasoning, thereby realizing parallel processing of models in a unified operating configuration environment, so that object target and crew member posture skeleton point reasoning analysis can be performed in parallel.

[0061] In a specific application embodiment, Figure 4 As shown in the figure, the GStreamer streaming media analysis and processing framework is used to build the reading, parsing, decoding and preprocessing plug-ins for the locomotive 6A driver's cab video stream, including: first, the rtsp video stream of the locomotive 6A system is read and parsed to obtain the H264 format code stream, and then the NVDEC hardware decoder is used to accelerate its decoding, and the decoded image data is preprocessed (color space conversion and image correction, etc.). Through DeepStream, two parallel pipelines for inference of the target detection model and the crew posture skeleton point detection model are built, and the preprocessed image data of a batch batch is aggregated through nvstreammux to form the stream data to be queued for inference, and then the stream data to be inferred is input into the two parallel pipelines of the target detection model and the human posture skeleton point detection model through nvstreamdemux respectively for model inference. The nvinfer plug-in of DeepStream is used to infer the stream data of a batch of 6A driver's cab images waiting in line. The nvinfer plug-in converts the format and resizes the stream data according to the configuration requirements of the deep neural network model. The converted data is then passed to the underlying library of DeepStream, which preprocesses the converted data (subtracting the mean, dividing the variance, and normalizing operations, etc.). The nvinfer plug-in calls the TensorRT inference acceleration optimizer, and performs GPU accelerated inference on a batch of queued streaming data through the object.engine and pose.engine engine files. The first time the engine file of the model is called, it is serialized and saved, and then reasoning is performed. When it is called again, the serialized engine file can be directly deserialized without having to start over. The above-mentioned use of the DeepStream framework integrates complex tasks such as the reading, parsing, and decoding tasks of the locomotive 6A video, the preprocessing operations of the video image, and the parallel reasoning process of the image into an overall pipeline.

[0062] It is understandable that other streaming media plug-ins besides GStreamer, such as FFmpeg, etc., may also be used for the tasks of reading, parsing and decoding the video.

[0063] In this embodiment, the violation points specifically include the main driver's failure to compare hands, failure to look behind, failure to confirm by raising the pantograph, failure to wear a vest when getting off the train, and the main driver's hands leaving the control panel, etc. The LKJ recorded events specifically include entry into the section, exit from the section, entry into shunting, shunting parking, shunting start, exit from shunting, parking in the station, starting in the station, LKJ siding input, and over-phase events, etc. The violation points to be monitored and the LKJ recorded events can be specifically configured according to actual needs.

[0064] In this embodiment, it also includes constructing corresponding item point detection models for each violation item that needs to be monitored. Each item point detection model determines whether there is a corresponding violation based on the inferred object target detection information and the crew member's posture skeleton point information. The item point detection model includes a detection model for the main driver not to use his hands to compare, a detection model for not looking behind, a detection model for not raising the bow to confirm, a detection model for not wearing a vest when getting off the vehicle, and a detection model for the main driver taking both hands off the console.

[0065] In this embodiment, it is also included to determine whether there is a second type of violation based on the inferred object target detection information and the crew member posture skeleton point information, and the first type of violation is a violation that needs to be monitored throughout the process, such as smoking, playing with mobile phones, etc. That is, for the violation that needs to be monitored throughout the process, the object target detection information and the crew member posture skeleton point information inferred by the two detection models are used in real time for identification and judgment, without the need to associate with the LKJ data.

[0066] In a specific application embodiment, the GPU inference results output by the two detection models are transmitted to the CPU, and according to the required item point business logic, it is determined whether it is necessary to perform real-time correlation analysis on the inference results of the two detection models and the LKJ data. For items that are not related to LKJ recorded events, such as playing with mobile phones and smoking, which belong to the first type of violations that require full monitoring, no correlation analysis is required; for items related to LKJ, such as the main driver not making hand comparisons, not looking behind, not raising the bow for confirmation, not wearing a vest when getting off the car, and the main driver leaving the console with both hands, it is necessary to perform correlation analysis with the LKJ data, that is, based on the LKJ data, when it is monitored that the time or mileage range required by the corresponding LKJ event is reached, the crew behavior judgment of the corresponding item is initiated, and the LKJ event is combined with the information inferred by the two detection models to determine whether there is a violation of the corresponding item. This can effectively integrate the target detection information in the cab, the crew skeleton point information and the LKJ data information to quickly and accurately realize crew behavior detection and identification.

[0067] For the violation points in driving and shunting operations, this embodiment specifically sets five points: the main driver does not compare with his hands, does not look behind, does not raise the bow to confirm, does not wear a vest when getting off the car, and the main driver leaves the control panel with both hands. The LKJ data is analyzed in real time and the LKJ events corresponding to the points are retrieved, including "entering the section", "exiting the section", "entering shunting", "shunting parking", "shunting driving", "exiting shunting", "station parking", "station driving", "LKJ siding input", and "over-phase" key events. The LKJ data is integrated with the reasoning results of the detection model. Within the time or mileage range corresponding to the specific events of the above-mentioned points, whether the locomotive crew gesture or probe action occurs is detected; if no action occurs, the violation point is output in real time, otherwise a null value is output. For the violation points of playing with mobile phones and smoking in the whole operation, there is no need to associate LKJ data. The reasoning results of the model are analyzed and processed by the detection model of the playing with mobile phones and smoking points, and the violation points or null values ​​are output. Considering that the environment inside the cab is complex and easily affected by environmental factors such as lighting, video image pixel information can be further integrated during the detection process to improve detection accuracy and robustness.

[0068] This embodiment specifically combines the relevant angle, module length, shape, position, timing, color tone and other information of the item point behavior in the physical 3D space of the driver's cab as the important parameters of the item point detection model, and establishes 7 item point detection models respectively. Among them, in the main driver's hand comparison detection model, the angle between the driver's upper arm vector and the trunk vector, the angle between the upper arm vector and the forearm vector, and the angle between the forearm vector and the vertical direction are calculated according to the crew member's posture skeleton point information to judge the execution of the main driver's hand comparison action; the rear lookout detection model calculates the horizontal direction of the crew member's left shoulder to right shoulder vector according to the position area of ​​the door and window in the object target detection information. Angle, according to the calculated horizontal angle and the crew member's posture skeleton point information, determine the execution of the crew member's bow raising confirmation action; the crew member getting off the vehicle without a vest detection model determines the position area of ​​the door and window and whether the crew member is wearing a vest according to the object target detection information, and determines the action state of the crew member getting off the vehicle according to the crew member's posture skeleton point information, and confirms the execution of the crew member getting off the vehicle action; the main driver's hands leaving the control panel detection model determines the edge position of the locomotive control panel according to the body target detection information, and determines the state of the crew member's hands according to the crew member's posture skeleton point information, and confirms the execution of the main driver's hands leaving the control panel action.

[0069] In a specific application embodiment, the detailed steps of configuring the above-mentioned point detection model are as follows:

[0070] In the mobile phone playing item point detection model, the inference results of the object target detection model and the flight attendant posture skeleton point detection model are obtained. If the two models output the center point coordinates of the mobile phone detection frame and the human wrist point coordinates, and the two-norm distance between the two coordinates meets the condition of being less than a certain threshold, it is confirmed that the mobile phone playing violation behavior exists.

[0071] In the smoking point detection model, the inference results of the object target detection model and the flight attendant posture skeleton point detection model are obtained. If the model outputs the center point coordinates of the cigarette detection frame and the human wrist point coordinates, and the two-norm distance between the two coordinates satisfies the condition of being less than a certain threshold, it is confirmed that there is a smoking violation.

[0072] In the main driver's non-hand gesture item point detection model, the coordinates of the main driver's right shoulder, right hip, right elbow and right wrist output by the crew member posture skeleton point detection model are obtained, and the angle between the upper arm vector and the trunk vector, the angle between the upper arm vector and the forearm vector, and the angle between the forearm vector and the vertical direction are calculated. If the three angles simultaneously meet a certain angle range, the execution of the main driver's hand gesture action is determined.

[0073] In the non-raised bow confirmation item point detection model, the inference results of the object target detection model and the crew member posture skeleton point detection model are obtained. The object target detection model infers and outputs the detection frames of the two crew members standing on the doors and windows, and the crew member posture skeleton point detection model outputs the shoulder coordinates and hip coordinates of the two crew members. The image edge line detection algorithm is used to process the door and window interest areas of the image, and the position area of ​​the doors and windows in the locomotive video image is determined according to the camera's viewing angle, the door and window edge angle and the empirical threshold of the length. The horizontal angle from the crew member's left shoulder to the right shoulder vector is calculated. If it is less than a certain threshold, it can be confirmed that the crew member is facing the door. It is calculated whether the shoulder coordinates and hip coordinates are within the area of ​​the doors and windows. If the above conditions are met, it is confirmed that the crew member's raised bow confirmation action is executed.

[0074] In the driver's rear-looking point detection model, the inference results of the object target detection model and the crew member posture skeleton point detection model are obtained. The object target detection model infers and outputs the detection frame of a single crew member standing on the door, and the crew member posture skeleton point detection model infers and outputs the shoulder coordinates and hip coordinates of the crew member. The horizontal angle from the crew member's left shoulder to the right shoulder vector is calculated. If it is less than a certain threshold, it can be confirmed that the crew member is facing the door. The shoulder coordinates and hip coordinates are calculated whether they are within the area of ​​the door. If the above conditions are met, it is confirmed that the crew member's rear-looking action is executed.

[0075] In the point detection model of the attendant getting off the train without wearing a vest, the image edge line detection algorithm is used to process the door and window interest areas of the image, and the position area of ​​the door and window in the locomotive video image is determined according to the camera's viewing angle, the door and window edge angle and the empirical threshold of the length; the human body key skeleton point model is used to infer the image of the locomotive video, and the shoulder coordinates, hip coordinates and elbow coordinates of the attendant are output; the horizontal angle from the attendant's right shoulder to the left shoulder vector is calculated, and if it is less than a certain threshold, it can be confirmed that the attendant is facing the inside of the cab; the angle between the upper arm vector and the torso vector is calculated, and if it is greater than a certain threshold, it can be confirmed that the attendant's upper arm is in a raised state; within a certain image frame sequence, the coordinates of the torso centroid are calculated to determine whether the ordinate is in an increasing state in the image coordinates; if the above conditions are met, it is confirmed that the action of the attendant getting off the train is executed; finally, the area proportion of the vest yellow color in the attendant's torso area is detected, and if it is greater than a certain threshold, it is confirmed that the attendant is wearing a vest, and the detection of the point of the attendant getting off the train without wearing a vest is completed.

[0076] In the point detection model for the main driver's hands leaving the console, the image edge line detection algorithm is used to process the console area of ​​interest in the image; the edge position of the locomotive console is determined based on the camera's viewing angle, the console edge angle and the empirical threshold of the length; the human body key skeleton point model is used to infer the image of the locomotive video and output the key skeleton point coordinates of the crew; the right wrist coordinates of the main driver's crew are screened out based on the empirical threshold of the main driver's key skeleton point coordinates; based on the positional relationship between the main driver's right wrist coordinates and the console edge vector, it is determined whether the right wrist is on the left side of the edge; the perpendicular distance between the right wrist coordinates and the console edge vector is calculated. If it is greater than a certain threshold, it is determined that the main driver has left the console.

[0077] This embodiment performs decision-level multimodal information fusion by fusing object target detection information, human skeleton point detection information, image pixel information and LKJ data to establish an item point detection model, which can accurately identify the behavior of flight attendants at various types of item points. It not only has high recognition accuracy and stability, but also has better robustness.

[0078] In a specific application embodiment, Figure 5As shown, when analyzing each set of data of the locomotive 6A video stream, two detection models are used for reasoning to obtain object target detection information and crew posture skeleton point detection information, and determine whether to output a label. If a label is output, it indicates that there is a target related to the required monitoring behavior, and then it is further determined whether it needs to be associated with the LKJ data, that is, by real-time analysis of the LKJ data to retrieve whether there is an LKJ event. If so, the LKJ data is combined for analysis to determine whether it is currently within the time or mileage required by the LKJ event to confirm whether it meets the various point detection conditions. If so, the object target detection information and crew posture skeleton point detection information obtained by the model reasoning of the various point detection models are used to determine whether there is a violation. If it is determined that there is a violation, an alarm is issued.

[0079] To implement the above method, take the use of DeepStream framework to build a detection system in a specific application embodiment as an example, Figure 6 As shown, it includes a data layer, an algorithm layer, and an application layer, wherein the data layer includes 6A image data, 6A image training set, and LKJ data; the algorithm layer includes an image processing model / two detection models (object target detection model and crew posture skeleton point detection model) built based on a deep network model, an LKJ event analysis model, and an item point detection model. The item point detection model uses image pixel information, target detection information, crew key skeleton point information, and LKJ data to perform decision-level multimodal information fusion to establish and form, and performs item point behavior recognition and judgment by fusing the reasoning results of the two detection models and the analysis results of the LKJ event; the application layer reads, decodes, preprocesses, batches, performs model reasoning, and item point detection on the video in sequence under the DeepStream framework, uses the GStreamer streaming media analysis and processing framework, builds the reading, parsing, and decoding plug-ins of the locomotive 6A driver's cab video stream, as well as the image preprocessing plug-in, and accelerates and optimizes through TensorRT and the like. TensorRT is used to accelerate the reasoning of the preprocessed image frame data, and DeepStream is used to build two parallel target detection models and human posture skeleton point detection model pipelines. The above system is specifically deployed on the NVIDIA Jetson AGX Xavier embedded platform, which can provide a higher level of computing density, energy efficiency and AI reasoning capabilities, while running at a lower power consumption, and can quickly deploy deep neural network models, facilitating later upgrade optimization and operation and maintenance. The NVIDIA Jetson AGX Xavier embedded platform can use the lower-performance Jetson AGX Xavier 8GB to achieve full software and hardware compatibility with the existing Jetson AGX Xavier.

[0080] The locomotive crew behavior detection system based on computer vision in this embodiment includes:

[0081] A training set construction module is used to construct a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants;

[0082] A model building module, for training a flight attendant posture skeleton point detection model and an object target detection model using the flight attendant key skeleton point training set and the object target training set respectively based on a deep neural network;

[0083] A detection module, used to obtain a video image of the locomotive cab in real time during the locomotive driving process, input it into the object target detection model and the crew posture skeleton point detection model respectively, and infer the object target detection information and the crew posture skeleton point information;

[0084] The behavior detection module is used to parse the LKJ monitoring data in real time and retrieve whether there is an LKJ recorded event corresponding to the required monitoring violation item. If so and the current time is within the time or mileage range required by the LKJ recorded event, it is determined whether there is a first type of violation based on the currently inferred object target detection information and the crew posture skeleton point information. The first type of violation is a violation related to the LKJ recorded event.

[0085] In this embodiment, if Figure 6 As shown, each module is constructed in the DeepStream framework, and the GStreamer streaming media analysis and processing plug-in is used in the DeepStream framework to read and parse the video image data, and the NVDEC hardware decoder is called to accelerate the decoding of the video image data. DeepStream also includes two parallel pipelines, which are used to perform reasoning using the object target detection model and the crew posture skeleton point detection model respectively, and DeepStream also includes a TensorRT reasoning acceleration optimizer for acceleration optimization.

[0086] like Figure 7 As shown, the detection system of this embodiment is connected to the locomotive 6A system, and obtains data from the locomotive 6A system in real time to detect the crew behavior. The output end of the detection system is also connected to an alarm. When the detection system detects a violation, an alarm signal is output through the alarm.

[0087] This embodiment further provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0088] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices in cooperation with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing related programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other applications. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0089] This embodiment further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the above method is implemented.

[0090] Those skilled in the art will appreciate that the above-mentioned embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0091] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for detecting locomotive crew behavior based on computer vision, characterized in that: include: Construct a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants; Based on a deep neural network, the crew key skeleton point training set and the object target training set are used to train a crew posture skeleton point detection model and an object target detection model respectively; Acquire a video image of the locomotive cab in real time during the locomotive driving process, input the image into the object target detection model and the crew posture skeleton point detection model, respectively, and infer the object target detection information and the crew posture skeleton point information; Parse the LKJ monitoring data in real time and search whether there is an LKJ recorded event corresponding to the required monitoring violation item. If so and the current time or mileage is within the required time or mileage range of the LKJ recorded event, determine whether there is a first type of violation based on the currently inferred object target detection information and the crew posture skeleton point information. The first type of violation is a violation related to the LKJ recorded event.

2. The method for detecting locomotive crew behavior based on computer vision according to claim 1, characterized in that: The violation items include any one or more of the following: the main driver failed to make hand signals, failed to look behind, failed to raise the pantograph for confirmation, failed to wear a vest when getting off the vehicle, and the main driver took both hands off the control panel; the LKJ recorded events include any one or more of the following: entering the section, exiting the section, entering shunting, shunting parking, shunting starting, exiting shunting, parking in the station, starting in the station, LKJ siding input, and over-phase events.

3. The method for detecting locomotive crew behavior based on computer vision according to claim 1, characterized in that: It also includes constructing corresponding item point detection models for each violation item that needs to be monitored, and each item point detection model determines whether there is a corresponding violation based on the inferred object target detection information and the crew member's posture skeleton point information. The item point detection models include any one or more of the main driver's no hand comparison detection model, no rear lookout detection model, no bow raising confirmation detection model, no vest wearing detection model when getting off the car, and the main driver's hands off the console detection model.

4. The method for detecting locomotive crew behavior based on computer vision according to claim 3, characterized in that: The main driver's hand gesture detection model calculates the angle between the driver's upper arm vector and the trunk vector, the angle between the upper arm vector and the forearm vector, and the angle between the forearm vector and the vertical direction according to the crew member's posture skeleton point information, and determines the execution of the main driver's hand gesture action; the rear lookout detection model calculates the horizontal angle between the crew member's left shoulder and right shoulder vector according to the position area of ​​the door and window in the object target detection information, and determines the execution of the crew member's bow raising confirmation action according to the calculated horizontal angle and the crew member's posture skeleton point information; the crew member getting off the vehicle without wearing a vest detection model determines the position area of ​​the door and window and whether the crew member is wearing a vest according to the object target detection information, and determines the action state of the crew member getting off the vehicle according to the crew member's posture skeleton point information, and confirms the execution of the crew member getting off the vehicle action; the main driver's hands leaving the control panel detection model determines the edge position of the locomotive control panel according to the body target detection information, and determines the state of the crew member's hands according to the crew member's posture skeleton point information, and confirms the execution of the main driver's hands leaving the control panel action.

5. The method for detecting locomotive crew behavior based on computer vision according to claim 1, characterized in that: It also includes judging whether there is a second type of violation based on the inferred object target detection information and the crew member's posture skeleton point information, wherein the first type of violation is a violation that requires full-time monitoring.

6. The method for detecting locomotive crew behavior based on computer vision according to any one of claims 1 to 5, characterized in that: The training set of key skeleton points of the flight attendants and the training set of object targets required for analyzing various behaviors of the flight attendants include: Obtain historical video image data of the locomotive cab; Perform frame extraction and image deduplication processing on the acquired video image data; Annotate the key skeleton points of the object target and the crew member in the processed data frame; Based on the labeled image data, the flight attendant key skeleton point training set and the object target training set are constructed respectively.

7. The method for detecting locomotive crew behavior based on computer vision according to any one of claims 1 to 5, characterized in that: The YOLOv model is used to train the object target detection model and the flight attendant posture skeleton point detection model. During the model training process, additional auxiliary heads are set in the middle layer of the deep neural network, weights for guiding auxiliary losses are set for the shallow network, and the mean and variance of batch normalization are integrated into the bias and weight of the convolutional layer to implement batch normalization operations to assist in completing the training.

8. The method for detecting locomotive crew behavior based on computer vision according to any one of claims 1 to 5, characterized in that: In the process of processing video images, the video image data is read and analyzed based on the streaming media analysis and processing plug-in under the DeepStream framework, the hardware decoder is called to accelerate the decoding of the video image data, and the decoded image data is preprocessed.

9. The method for detecting locomotive crew behavior based on computer vision according to claim 8, characterized in that: In the training of the object target detection model and the crew posture skeleton point detection model, it also includes calling the inference acceleration optimizer to optimize any one or more of the weight parameters, fusion network layers and optimization of video memory allocation in the object target detection model and the crew posture skeleton point detection model.

10. The method for detecting locomotive crew behavior based on computer vision according to claim 8, characterized in that: It also includes using two parallel pipelines for building object target detection models and crew member posture skeleton point detection model reasoning, and inputting the stream data to be reasoned into the parallel pipelines of the target detection model and the crew member posture skeleton point detection model for model reasoning.

11. A locomotive crew behavior detection system based on computer vision, characterized in that: include: A training set construction module is used to construct a training set of key skeleton points of flight attendants and a training set of object targets required for analyzing various behaviors of flight attendants; A model building module, for training a flight attendant posture skeleton point detection model and an object target detection model using the flight attendant key skeleton point training set and the object target training set respectively based on a deep neural network; A detection module, used to obtain a video image of the locomotive cab in real time during the locomotive driving process, input it into the object target detection model and the crew posture skeleton point detection model respectively, and infer the object target detection information and the crew posture skeleton point information; The behavior detection module is used to parse the LKJ monitoring data in real time and retrieve whether there is an LKJ recorded event corresponding to the required monitoring violation item. If so and the current time is within the time or mileage range required by the LKJ recorded event, it is determined whether there is a first type of violation based on the currently inferred object target detection information and the crew posture skeleton point information. The first type of violation is a violation related to the LKJ recorded event.

12. The locomotive crew behavior detection system based on computer vision according to claim 11, characterized in that: Each module is constructed in the DeepStream framework, and the streaming media analysis and processing plug-in is used to read and parse the video image data, and the hardware decoder is called to accelerate the decoding of the video image data.

13. The locomotive crew behavior detection system based on computer vision according to claim 12, characterized in that: DeepStream also includes two parallel pipelines for performing inference using the object target detection model and the crew posture skeleton point detection model respectively, and / or DeepStream also includes an inference acceleration optimizer for performing acceleration optimization.

14. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 10.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 10 is implemented.

Images