Maintenance personnel operation behavior monitoring method based on artificial intelligence

Abstract

The invention discloses a maintenance personnel operation behavior monitoring method based on artificial intelligence, and relates to the technical field of artificial intelligence. In order to solve the problems that the current locomotive depot personnel behavior monitoring is mainly performed by manual monitoring and cannot give an alarm at the first time when an accident occurs, so that hidden dangers are caused to safety production, and video monitoring personnel often face relatively high working pressure. Through the face recognition and human body posture estimation technology, the system can accurately recognize the identity and the position of the maintenance personnel and monitor the behaviors of the maintenance personnel in real time, the workload of the monitoring personnel is relieved, the behaviors of the maintenance personnel are automatically recognized through the OpenPose algorithm, manual monitoring is not needed, the monitoring efficiency and the automation degree are improved, powerful support is provided for maintenance safety management, and the system is suitable for popularization and application. The system can effectively improve the monitoring efficiency, reduce the safety risk, improve the safety awareness, and promote the scientific and intelligent development of the maintenance safety management, thereby effectively preventing and controlling the safety accidents, and improving the maintenance safety management level.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an artificial intelligence-based method and system for monitoring the operating behavior of maintenance personnel. Background Art

[0002] At present, the China Railway Corporation and all stations and sections are paying more and more attention to ground video and increasing their monitoring efforts. Many potential safety hazards have been discovered and prevented in a timely manner through ground video. Due to the large number of video monitoring points in the locomotive system and the various types of violations by employees, there is currently no effective technical means for auxiliary monitoring. The current monitoring of the behavior of locomotive section personnel is mainly based on manual monitoring, which is inefficient due to manual real-time verification and inspection. It cannot achieve all-weather, all-round monitoring, and it is impossible to issue an alarm in the first time of an incident, which poses a hidden danger to production safety. Due to the large number of scenes that need to be monitored and the need to maintain a high degree of concentration in real time, video monitoring personnel often face great work pressure. Summary of the invention

[0003] The purpose of the present invention is to provide a method and system for monitoring the operating behavior of maintenance personnel based on artificial intelligence, which can realize real-time monitoring and early warning of the operating behavior of maintenance depot personnel through automated and intelligent means, help the maintenance depot management personnel to deal with possible safety hazards in a timely manner, ensure the safety of operations, and reduce the operating pressure of video monitoring personnel, so as to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence includes the following steps:

[0006] Step 1: Video acquisition: Use video surveillance equipment to capture the real-time video stream of the on-site operation scene of the locomotive depot, pre-process the collected video stream to obtain video frames, and extract target features;

[0007] Step 2: Behavior recognition: Identify the position features of the maintenance personnel in the video frame, and select the human body area based on the position features of the maintenance personnel. In the detected human body area, obtain the key point positions of the human body based on the key point features in the posture estimation library;

[0008] Step 3: Dangerous behavior judgment: Identify the behavior of the maintenance personnel based on the detected key points of the human body, classify the identified behavior, and determine whether the behavior is illegal or dangerous;

[0009] Step 4: Intelligent warning: When it is determined that the maintenance personnel's behavior is illegal or dangerous, a warning signal is immediately issued to remind the management personnel to take timely measures. At the same time, the warning information is sent to the remote management terminal where the management personnel is located, and the corresponding video stream data segment is displayed on the remote management terminal;

[0010] Step 5: Data analysis and evaluation: Integrate the collected data on violations and dangerous behaviors to generate a violation behavior dataset of maintenance personnel. Determine the frequency and time period of violations based on the maintenance personnel's historical violations, and evaluate the maintenance personnel's work behavior.

[0011] Furthermore, the collected video stream is preprocessed in step 1, including:

[0012] Decoding the obtained compressed video stream based on the video decoder to obtain the original image data of each frame of the decoded video stream;

[0013] According to the frame rate of the video stream, the continuous raw image data is divided into a corresponding number of independent video frames, and brightness compensation and denoising are performed;

[0014] Extract feature points from each frame of the processed image, compare the feature points of each frame of the image based on the extraction results, and determine the target environment feature points and target human feature points that appear in each frame of the image;

[0015] The number of target persons in the target monitoring area in the video stream is determined according to the target environment feature points and the target human feature points.

[0016] Furthermore, the continuous original image data is divided into a corresponding number of independent video frames according to the frame rate of the video stream, and brightness compensation is performed, including:

[0017] Extracting each frame of image corresponding to the video frame;

[0018] Traverse all the pixels contained in each frame of the image and obtain the grayscale value corresponding to each pixel;

[0019] Obtaining a gray balance index parameter corresponding to the image grayscale of each frame image according to the grayscale value corresponding to each pixel point;

[0020] The gray balance index parameter corresponding to the image grayscale of each frame image is obtained by the following formula:

[0021]

[0022] Where S represents the gray balance index parameter corresponding to the image grayscale of each frame image; H brepresents the standard deviation of the gray values ​​of n pixels contained in each frame image; n represents the total number of pixels contained in the frame image; H i represents the pixel value corresponding to the i-th pixel; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image; H max and H min They respectively represent the maximum grayscale value and the minimum grayscale value of the n pixels contained in the frame image;

[0023] Compare the gray balance index parameter corresponding to the image grayscale of each frame image with a preset gray balance index parameter threshold to obtain a comparison result;

[0024] Brightness compensation is performed on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold.

[0025] Furthermore, brightness compensation is performed on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold, including:

[0026] When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is greater than the preset gray balance index parameter threshold, calling the first compensation coefficient model;

[0027] The brightness compensation coefficient is obtained by using the first compensation coefficient model; wherein the structure of the first compensation coefficient model is as follows:

[0028]

[0029] Among them, B 01 represents the compensation coefficient of brightness obtained by the first compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image;

[0030] When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is not greater than the preset gray balance index parameter threshold, calling the second compensation coefficient model;

[0031] The brightness compensation coefficient is obtained by using the second compensation coefficient model; wherein the structure of the second compensation coefficient model is as follows:

[0032]

[0033] Among them, B 02 represents the compensation coefficient of brightness obtained by the second compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image;

[0034] Using the brightness compensation coefficient to compensate and adjust the original brightness of each frame of image to obtain an adjusted image of each frame;

[0035] The brightness after compensation adjustment is obtained by the following formula:

[0036]

[0037] Wherein, L represents the value of the brightness after compensation adjustment; L0 represents the value of the original brightness of each frame image; B 01 Indicates that the first compensation coefficient model obtains the compensation coefficient of brightness; B 02 It indicates that the first compensation coefficient model obtains the compensation coefficient of brightness.

[0038] Furthermore, the key point positions of the human body are extracted in step 2, specifically:

[0039] Extracting key human point features from the target human feature points based on a posture estimation library, and identifying position information of the key human points;

[0040] Determine the pixel coordinates of each human key point based on the human key point position information, perform smoothing processing on the key point coordinates, and determine the position information of each human key point in each frame image based on the processing result;

[0041] And the posture type of the human body is determined based on the position information of each human key point in each frame image.

[0042] Furthermore, the behavior identification in step 2 further includes:

[0043] Obtaining basic data of maintenance personnel actively uploaded by the management personnel, and extracting personnel data from the basic data of maintenance personnel;

[0044] Obtaining the number of maintenance personnel, and establishing corresponding personnel storage space based on the number of personnel;

[0045] In each personnel storage space, a time axis is established according to the data length of the video stream, and face recognition is performed on each frame of the video stream based on the facial feature points of the personnel data;

[0046] According to the identification results, the maintenance personnel are identified, and the identity information and the time point of their appearance are recorded on the time axis of the personnel storage space to generate dynamic data;

[0047] The position information of each maintenance personnel in each frame of image is extracted from the dynamic data, the initial position of each maintenance personnel is determined, and the position characteristics of the maintenance personnel are analyzed.

[0048] Furthermore, the position characteristics of the maintenance personnel are analyzed, including:

[0049] Dividing the dynamic data into a plurality of unit data segments, and obtaining the movement position corresponding to each maintenance personnel in each unit data segment;

[0050] The initial position of the same maintenance personnel and the moving position corresponding to each unit data segment are obtained, and the personnel movement trajectory is established. The total distance and average movement speed of the maintenance personnel in the unit data segment are calculated according to the movement trajectory, and the movement characteristics corresponding to each maintenance personnel are generated;

[0051] At the same time, in each unit data segment, it is determined whether the maintenance personnel stay at the same position for more than a preset pause time threshold, and the position and the stay time are recorded to generate the pause characteristics of the maintenance personnel;

[0052] Obtain the target unit data segment where the pause feature corresponding to each maintenance personnel is located, and at the same time, based on the time series of the target unit data segment in the dynamic data, determine whether there are multiple maintenance personnel pausing in the same time period, and determine the communication features between the maintenance personnel;

[0053] A location feature data set of the maintenance personnel is established according to the movement features and communication features corresponding to each maintenance personnel, and the maintenance personnel behavior patterns contained in the location feature data set are analyzed.

[0054] Furthermore, the risk behavior determination in step 3 further includes:

[0055] The extracted key point position information and posture type of the human body are matched with the preset dangerous behavior pattern library, and the similarity between the current behavior of the computer personnel and the behavior pattern in the dangerous behavior pattern library is calculated to identify possible dangerous behaviors;

[0056] At the same time, the identified behavior is contextually analyzed based on the extracted target environment feature points and target human feature points;

[0057] Combined with the analysis results, a risk assessment is conducted on the identified possible dangerous behaviors to evaluate the severity and probability of their possible consequences.

[0058] Furthermore, the frequency of historical violations of maintenance personnel in step 5 is determined, specifically including:

[0059] Collect historical behavior data of maintenance personnel, including all monitored violations, classify violations into different types, and assign a unique code to each violation;

[0060] For each maintenance personnel, the number of occurrences of various violations is counted, the frequency of each violation is calculated, the time is divided into different segments, and the frequency of violations in each time segment is determined;

[0061] Draw a trend chart of violations based on historical violation data, determine the changing trend of violations, establish a violation frequency prediction model, and predict the frequency of violations in the future based on the violation frequency prediction model;

[0062] Combined with the predicted results of the frequency of violations, the operating level of each maintenance personnel is evaluated, a maintenance personnel evaluation report is generated, and the corresponding maintenance personnel handling plan is matched according to the operating level evaluation results.

[0063] The present invention provides another technical solution, an artificial intelligence-based maintenance personnel operation behavior monitoring system, comprising:

[0064] The data acquisition module is used to capture real-time video streams through video acquisition devices installed at various key locations of the locomotive depot. At the same time, thermal imaging devices are installed at various key locations to collect temperature data at various key locations.

[0065] A data processing module is used to pre-process and extract features of the video streams and temperature data collected at various key locations based on the server, and store the received video streams and temperature data;

[0066] The artificial intelligence analysis module is used to identify and classify the behaviors of maintenance personnel based on feature extraction results, and determine whether there are any violations or dangerous behaviors. At the same time, it combines temperature data to analyze whether there are any safety hazards within the monitoring range, and determines the severity and urgency of the risks.

[0067] The early warning module is used to immediately trigger the early warning mechanism when violations and dangerous behaviors of maintenance personnel are detected, or abnormal temperature data is monitored, and to match corresponding early warning methods and measures according to the severity and urgency of the risk;

[0068] The remote control module is used to display the video images and temperature data of the locomotive depot in real time, and to remotely adjust the parameter settings of the video acquisition equipment and thermal imaging equipment based on the remote control terminal, as well as call historical data for analysis and review.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] Through face recognition and human posture estimation technology, the system can accurately identify the identity and location of maintenance personnel and monitor their behavior in real time, reducing the workload of monitoring personnel. Through the OpenPose algorithm, the behavior of maintenance personnel is automatically identified without manual monitoring, which improves the monitoring efficiency and degree of automation, and provides strong support for maintenance safety management. It can effectively improve monitoring efficiency, reduce safety risks, enhance safety awareness, and promote the scientific and intelligent development of maintenance safety management, thereby effectively preventing and controlling safety accidents and improving the level of maintenance safety management. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 This is a flow chart of the maintenance personnel operation behavior monitoring method based on artificial intelligence of the present invention;

[0072] Figure 2 This is a module diagram of the maintenance personnel operation behavior monitoring system based on artificial intelligence of the present invention. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0074] In order to solve the technical problem that the monitoring of the behavior of the maintenance personnel is mainly based on manual monitoring, which cannot give an alarm in the first time, causing hidden dangers to production safety, and video monitoring personnel often face great work pressure, please refer to Figure 1 , this embodiment provides the following technical solutions:

[0075] The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence includes the following steps:

[0076] Step 1: Video acquisition: Use video monitoring equipment to capture real-time video streams of on-site operation scenes in the locomotive depot, pre-process the acquired video streams to obtain video frames, and extract target features, including environmental features, human body features, etc.

[0077] In this embodiment, the preprocessing of the collected video stream in step 1 includes:

[0078] Decoding the obtained compressed video stream based on the video decoder to obtain the original image data of each frame of the decoded video stream;

[0079] According to the frame rate of the video stream, the continuous raw image data is divided into a corresponding number of independent video frames, and brightness compensation and denoising are performed;

[0080] Extract feature points from each processed image frame, compare the feature points of each image frame based on the extraction results, determine the target environment feature points and target human feature points that appear in each image frame, such as the interior of the maintenance depot, the oiling area, the hot work area, etc., and judge whether the maintenance personnel are using the correct tools to operate and whether they comply with the safety operating procedures;

[0081] Determine the number of target persons in the target monitoring area in the video stream according to the target environment feature points and the target human feature points, such as the number of maintenance personnel who are oiling, the number of maintenance personnel who are performing hot work, etc.;

[0082] In this embodiment, identifying the scene type and the equipment in the video based on the target environment feature points helps the system to identify and judge behaviors according to specific safety rules of different scenes. Identifying the color and style of the clothing of the maintenance personnel based on the target human feature points helps to distinguish the human features of maintenance personnel and non-maintenance personnel, and judge whether the maintenance personnel are wearing clothing that meets safety requirements.

[0083] Step 2: Behavior recognition: Identify the position features of the maintenance personnel in the video frame, and select the human body area based on the position features of the maintenance personnel. In the detected human body area, obtain the key point positions of the human body based on the key point features in the posture estimation library, such as the joints of the head, hands, legs, etc. Specifically:

[0084] Extracting key human point features from the target human feature points based on a posture estimation library, and identifying position information of the key human points;

[0085] Determine the pixel coordinates of each human key point based on the human key point position information, perform smoothing processing on the key point coordinates, and determine the position information of each human key point in each frame image based on the processing result;

[0086] And based on the position information of each human key point in each frame of the image, the human body posture type is determined, such as standing, walking, bending, etc.;

[0087] In this embodiment, the key features of the human body include the head, shoulders, elbows, wrists, waist, knees, ankles and other features, which are helpful for identifying the behavior of maintenance personnel, such as whether they are wearing a helmet, whether they are using tools correctly, whether they are in a dangerous area, etc.;

[0088] Step 3: Dangerous behavior judgment: Identify the behavior of the maintenance personnel based on the detected key points of the human body, classify the identified behavior, and determine whether the behavior is illegal or dangerous;

[0089] Step 4: Intelligent warning: When it is determined that the maintenance personnel's behavior is illegal or dangerous, a warning signal is immediately issued to remind the management personnel to take timely measures. At the same time, the warning information is sent to the remote management terminal where the management personnel is located, and the corresponding video stream data segment is displayed on the remote management terminal to facilitate the management personnel to investigate and deal with it;

[0090] Step 5: Data analysis and evaluation: Integrate the collected data on violations and dangerous behaviors to generate a violation behavior data set for maintenance personnel. Determine the frequency and time period of violations based on the maintenance personnel's historical violations, and evaluate the maintenance personnel's work behavior; in order to better understand safety risks, continuously optimize the behavior recognition model, and improve recognition accuracy and real-time performance.

[0091] In this embodiment, the behavior identification in step 2 further includes:

[0092] Obtaining basic data of maintenance personnel actively uploaded by management personnel, including maintenance personnel certificate information, certificate photos and historical work behavior monitoring data, and extracting personnel data from the maintenance personnel basic data, including each person's unique identifier, name, certificate photo, etc.;

[0093] Obtain the number of maintenance personnel, and establish corresponding personnel storage spaces based on the number of personnel, each space being used to store relevant data of a specific maintenance personnel, including their identity information, recognition results in video frames, etc.;

[0094] In each personnel storage space, a time axis is established according to the data length of the video stream to record the time point and behavior data of the maintenance personnel appearing in the video stream, and face recognition is performed on each frame of the video stream based on the facial feature points of the personnel data;

[0095] According to the identification results, the maintenance personnel are identified, and the identity information and the time point of their appearance are recorded on the time axis of the personnel storage space to generate dynamic data;

[0096] Extracting the position information of each maintenance personnel in each frame of image from the dynamic data, determining the initial position of each maintenance personnel, and analyzing the position characteristics of the maintenance personnel;

[0097] Dividing the dynamic data into a plurality of unit data segments, and obtaining the movement position corresponding to each maintenance personnel in each unit data segment;

[0098] The initial position of the same maintenance personnel and the moving position corresponding to each unit data segment are obtained, and the personnel movement trajectory is established. The total distance and average movement speed of the maintenance personnel in the unit data segment are calculated according to the movement trajectory, and the movement characteristics corresponding to each maintenance personnel are generated, including the movement direction and movement mode, such as forward, backward, left, right, etc., linear movement, curved movement, and stepping on the spot, etc.;

[0099] At the same time, in each unit data segment, it is determined whether the maintenance personnel stay at the same position for more than a preset pause time threshold, and the position and the stay time are recorded to generate the pause characteristics of the maintenance personnel;

[0100] Obtain the target unit data segment where the pause feature corresponding to each maintenance personnel is located, and at the same time, based on the time series of the target unit data segment in the dynamic data, determine whether there are multiple maintenance personnel pausing in the same time period, and determine the communication features between the maintenance personnel;

[0101] A location feature data set of the maintenance personnel is established according to the movement features and communication features corresponding to each maintenance personnel, and the maintenance personnel behavior patterns contained in the location feature data set are analyzed, such as work flow, activity range, rest time, etc.

[0102] In this embodiment, the identity of the maintenance personnel is identified based on the feature point information of the face extracted based on the feature points of the target body, and individual behavior analysis is performed and their behavior is monitored in real time, which reduces the workload of the monitoring personnel, conducts personalized management, monitors the behavior of the maintenance personnel in real time, and identifies abnormal behavior, which helps managers to discover problems in time and deal with them, prevent the occurrence of safety accidents, and promote the scientific and intelligent development of maintenance safety management.

[0103] In this embodiment, the continuous original image data is divided into a corresponding number of independent video frames according to the frame rate of the video stream, and brightness compensation is performed, including:

[0104] Extracting each frame of image corresponding to the video frame;

[0105] Traverse all the pixels contained in each frame of the image and obtain the grayscale value corresponding to each pixel;

[0106] Obtaining a gray balance index parameter corresponding to the image grayscale of each frame image according to the grayscale value corresponding to each pixel point;

[0107] The gray balance index parameter corresponding to the image grayscale of each frame image is obtained by the following formula:

[0108]

[0109] Where S represents the gray balance index parameter corresponding to the image grayscale of each frame image; H b represents the standard deviation of the gray values ​​of n pixels contained in each frame image; n represents the total number of pixels contained in the frame image; H i represents the pixel value corresponding to the i-th pixel; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image; H max and H min They respectively represent the maximum grayscale value and the minimum grayscale value of the n pixels contained in the frame image;

[0110] Compare the gray balance index parameter corresponding to the image grayscale of each frame image with a preset gray balance index parameter threshold to obtain a comparison result;

[0111] Brightness compensation is performed on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold.

[0112] The technical effect of the above technical solution is: by dividing the continuous original image data into a corresponding number of independent video frames according to the frame rate of the video stream, it is ensured that each video frame is an accurate representation of the original video data, which is conducive to subsequent image processing and analysis. The solution traverses all the pixels of each frame image and obtains the gray value of each pixel, and then calculates the gray balance index parameter (S) of each frame image. This step helps to evaluate the uniformity of the gray distribution of the image, that is, the gray balance state. Gray balance is one of the important indicators for measuring image quality. Good gray balance can make the image look more natural and clear. By comparing the gray balance index parameter (S) of each frame image with the preset gray balance index parameter threshold, it can be determined whether the image needs brightness compensation. If the S value exceeds the threshold range, it means that the gray distribution of the image is not uniform enough, and brightness adjustment may be required. This adaptive brightness compensation method can be accurately adjusted according to the actual situation of the image, thereby improving the visual effect of the image. By performing brightness compensation on each frame image, the image quality of the entire video sequence can be improved. Brightness compensation helps to reduce the overly bright or dark areas in the image, making the details of the image clearer and the colors more saturated. This is of great significance for improving the visualization effect and user experience of the video. By calculating the gray balance index parameters and performing brightness compensation, the solution can enhance the robustness of the algorithm to a certain extent. Even in the face of different lighting conditions or video streams with poor image quality, the solution can maintain the stability and clarity of the image through adaptive adjustment.

[0113] In summary, this technical solution effectively improves the quality and visualization effect of video images and enhances the robustness of the algorithm by accurately segmenting video frames, optimizing grayscale balance and adaptive brightness compensation, which is of great significance to the field of video processing and analysis.

[0114] Specifically, performing brightness compensation on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold includes:

[0115] When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is greater than the preset gray balance index parameter threshold, calling the first compensation coefficient model;

[0116] The brightness compensation coefficient is obtained by using the first compensation coefficient model; wherein the structure of the first compensation coefficient model is as follows:

[0117]

[0118] Among them, B 01 represents the compensation coefficient of brightness obtained by the first compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image;

[0119] When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is not greater than the preset gray balance index parameter threshold, calling the second compensation coefficient model;

[0120] The brightness compensation coefficient is obtained by using the second compensation coefficient model; wherein the structure of the second compensation coefficient model is as follows:

[0121]

[0122] Among them, B 02 represents the compensation coefficient of brightness obtained by the second compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H xIndicates the grayscale intermediate value corresponding to the frame image;

[0123] Using the brightness compensation coefficient to compensate and adjust the original brightness of each frame of image to obtain an adjusted image of each frame;

[0124] The brightness after compensation adjustment is obtained by the following formula:

[0125]

[0126] Wherein, L represents the value of the brightness after compensation adjustment; L0 represents the value of the original brightness of each frame image; B 01 Indicates that the first compensation coefficient model obtains the compensation coefficient of brightness; B 02 It indicates that the first compensation coefficient model obtains the compensation coefficient of brightness.

[0127] The technical effect of the above technical solution is: the solution can dynamically determine whether the image needs brightness compensation by comparing the gray balance index parameter of each frame image with the preset threshold. This adjustment method based on the gray balance index can more accurately reflect the gray distribution of the image, so as to make a more reasonable brightness adjustment decision. According to the comparison result of the gray balance index parameter and the threshold, the solution can flexibly select the first compensation coefficient model or the second compensation coefficient model to obtain the brightness compensation coefficient. This design makes the brightness compensation more targeted and adaptable, and can be personalized according to the gray characteristics of different images. The first compensation coefficient model and the second compensation coefficient model both consider multiple factors, including gray balance index parameters, preset thresholds, middle gray average, center gray value and gray intermediate value. The comprehensive consideration of these factors makes the calculation of the brightness compensation coefficient more accurate, so that the brightness of the image can be adjusted more accurately. By using the brightness compensation coefficient to compensate and adjust the original brightness of each frame image, the solution can improve the brightness distribution of the image and make the image look more natural and clear. This helps to improve the overall quality of the image and enhance the visual effect of the image. This solution uses a clear mathematical model and formula to calculate the brightness compensation coefficient, avoiding the ambiguity and uncertainty that may exist in traditional methods. This design makes the algorithm more stable and reliable, and can maintain good performance under different lighting conditions and image quality.

[0128] In this embodiment, the risk behavior determination in step 3 further includes:

[0129] The extracted key point position information and posture type of the human body are matched with the preset dangerous behavior pattern library. The dangerous behavior pattern library contains various known violations and dangerous behaviors of maintenance personnel, such as failure to wear protective equipment correctly, illegal operation of equipment, entering dangerous areas, incorrect use of tools, etc. The similarity between the current behavior of the maintenance personnel and the behavior pattern in the dangerous behavior pattern library is calculated to identify possible dangerous behaviors;

[0130] At the same time, based on the extracted target environment feature points and target human feature points, the identified behavior is analyzed in context. For example, when it is identified that the maintenance personnel are not wearing protective glasses in the hot work area, this behavior will be regarded as a dangerous behavior in a specific context.

[0131] Combined with the analysis results, a risk assessment is conducted on the identified possible dangerous behaviors to evaluate the severity and probability of their possible consequences.

[0132] In this embodiment, based on the detected key points, OpenPose performs posture estimation, determines the connection relationship and posture information between each key point, generates output data containing the detected human key points and posture information, and matches it with the dangerous behavior pattern library. Combined with context analysis for risk assessment, the system can effectively identify and warn of dangerous behaviors of maintenance personnel, provide strong support for maintenance safety management, help improve monitoring efficiency, reduce safety risks, and enhance safety awareness.

[0133] In this embodiment, the frequency of historical violations of flight attendants in step 5 is determined, specifically including:

[0134] Collect historical behavior data of maintenance personnel, including all detected violations, classify violations into different types, such as failure to wear protective equipment, illegal operation of equipment, entering dangerous areas, etc., and assign a unique code to each violation;

[0135] For each maintenance personnel, count the number of violations of each type, calculate the frequency of each violation, divide the time into different periods, such as month, quarter, year, etc., determine the frequency of violations in each period, so as to understand the seasonal or cyclical characteristics of violations;

[0136] Draw a trend chart of violations based on historical violation data, determine the changing trend of violations, establish a violation frequency prediction model, and predict the frequency of violations in the future based on the violation frequency prediction model;

[0137] Combined with the predicted results of the frequency of violations, the operating level of each maintenance personnel is evaluated and a maintenance personnel evaluation report is generated. The maintenance personnel evaluation report includes the basic data of the maintenance personnel, the time when the violation occurred, the corresponding violation code and the evaluation score. The corresponding maintenance personnel processing plan is matched according to the operating level evaluation results to provide targeted decision-making support for managers, such as strengthening training, adjusting work arrangements, optimizing monitoring strategies, etc.

[0138] In this embodiment, by analyzing the frequency of violations, it is possible to predict possible safety risks in the future, and take preventive measures to reduce the incidence of safety accidents. The evaluation and early warning functions can improve the safety awareness of maintenance personnel, prompt maintenance personnel to comply with safety operating procedures, reduce the occurrence of violations, and help managers understand the safety behavior status of maintenance personnel, evaluate their operating level, and take targeted management measures, thereby effectively preventing and controlling safety accidents and improving the level of maintenance safety management.

[0139] In order to better implement the AI-based maintenance personnel operation behavior monitoring method, please refer to Figure 2 The present invention proposes an artificial intelligence-based maintenance personnel operation behavior monitoring system, comprising:

[0140] The data acquisition module is used to capture real-time video streams through video acquisition devices installed at various key locations of the locomotive depot, such as the oiling area, the hot work area, etc. At the same time, thermal imaging devices are installed at various key locations to collect temperature data at various key locations;

[0141] The data processing module is used to pre-process and extract features of the video streams and temperature data collected at various key locations based on the server, store the received video streams and temperature data, and store and transmit the video streams based on the network video recorder;

[0142] The artificial intelligence analysis module is used to identify and classify the behaviors of maintenance personnel based on feature extraction results, and determine whether there are any violations or dangerous behaviors. At the same time, it combines temperature data to analyze whether there are safety hazards such as overheating and fire within the monitoring range, and determines the severity and urgency of the risks.

[0143] The early warning module is used to immediately trigger the early warning mechanism when violations and dangerous behaviors of maintenance personnel are detected, or abnormal temperature data is monitored. The early warning information is sent to management personnel and relevant departments in a timely manner through sound and light alarms, SMS notifications, email alerts, etc., and the corresponding early warning methods and measures are matched according to the severity and urgency of the risk;

[0144] The remote control module is used to display the video images and temperature data of the locomotive depot in real time, and remotely adjust the parameter settings of the video acquisition equipment and thermal imaging equipment based on the remote control terminal, as well as call historical data for analysis and review, ensuring that managers can monitor and manage anytime and anywhere.

[0145] In this embodiment, real-time acquisition and diagnosis of thermal imaging data are carried out to warn of dangerous conditions such as oiling areas and hot work. Through video equipment, network video recorders and servers, efficient, stable and reliable connections are established to achieve precise control of the model. First, the video equipment captures real-time video streams and converts them into digital signals. The network video recorder (NRV) receives the video streams, stores and transmits the videos. The server processes the stored data and analyzes and processes the videos. The equipment is connected through a high-speed and stable network to form a complete networking structure. The video equipment and NRV are connected to the server to ensure smooth transmission of the video stream. The server is connected to the PC. The user remotely controls and monitors the model to identify human behaviors in the video, including simple actions, single-person behaviors, interactive behaviors and group behaviors. Furthermore, according to the OpenPose algorithm, the joints in the video are first detected, and then matched and connected to achieve real-time detection of multiple people, generate images containing human posture information, achieve real-time monitoring of employee behaviors, and warn of dangerous conditions through thermal imaging technology, thereby effectively preventing and controlling safety hazards.

[0146] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A method for monitoring the operation behavior of maintenance personnel based on artificial intelligence, characterized in that: The following steps are involved: Step 1: Video acquisition: Use video surveillance equipment to capture the real-time video stream of the on-site operation scene of the locomotive depot, pre-process the collected video stream to obtain video frames, and extract target features; Step 2: Behavior recognition: Identify the position features of the maintenance personnel in the video frame, and select the human body area based on the position features of the maintenance personnel. In the detected human body area, obtain the key point positions of the human body based on the key point features in the posture estimation library; Step 3: Dangerous behavior judgment: Identify the behavior of the maintenance personnel based on the detected key points of the human body, classify the identified behavior, and determine whether the behavior is illegal or dangerous; Step 4: Intelligent warning: When it is determined that the maintenance personnel's behavior is illegal or dangerous, a warning signal is immediately issued to remind the management personnel to take timely measures. At the same time, the warning information is sent to the remote management terminal where the management personnel is located, and the corresponding video stream data segment is displayed on the remote management terminal; Step 5: Data analysis and evaluation: Integrate the collected data on violations and dangerous behaviors to generate a violation behavior dataset of maintenance personnel. Determine the frequency and time period of violations based on the maintenance personnel's historical violations, and evaluate the maintenance personnel's work behavior.

2. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 1, characterized in that: The step 1 pre-processes the collected video stream, including: Decoding the obtained compressed video stream based on the video decoder to obtain the original image data of each frame of the decoded video stream; According to the frame rate of the video stream, the continuous raw image data is divided into a corresponding number of independent video frames, and brightness compensation and denoising are performed; Extract feature points from each frame of the processed image, compare the feature points of each frame of the image based on the extraction results, and determine the target environment feature points and target human feature points that appear in each frame of the image; The number of target persons in the target monitoring area in the video stream is determined according to the target environment feature points and the target human feature points.

3. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 2, characterized in that: According to the frame rate of the video stream, the continuous raw image data is divided into a corresponding number of independent video frames, and brightness compensation is performed, including: Extracting each frame of image corresponding to the video frame; Traverse all the pixels contained in each frame of the image and obtain the grayscale value corresponding to each pixel; Obtaining a gray balance index parameter corresponding to the image grayscale of each frame image according to the grayscale value corresponding to each pixel point; The gray balance index parameter corresponding to the image grayscale of each frame image is obtained by the following formula: Where S represents the gray balance index parameter corresponding to the image grayscale of each frame image; H b represents the standard deviation of the gray values ​​of n pixels contained in each frame image; n represents the total number of pixels contained in the frame image; H i represents the pixel value corresponding to the i-th pixel; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image; H max and H min They respectively represent the maximum grayscale value and the minimum grayscale value of the n pixels contained in the frame image; Compare the gray balance index parameter corresponding to the image grayscale of each frame image with a preset gray balance index parameter threshold to obtain a comparison result; Brightness compensation is performed on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold.

4. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 3, characterized in that: Performing brightness compensation on each frame of the image according to a comparison result between a gray balance index parameter corresponding to the image grayscale of each frame of the image and a preset gray balance index parameter threshold value, including: When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is greater than the preset gray balance index parameter threshold, calling the first compensation coefficient model; The brightness compensation coefficient is obtained by using the first compensation coefficient model; wherein the structure of the first compensation coefficient model is as follows: Among them, B 01 represents the compensation coefficient of brightness obtained by the first compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image; When the comparison result between the gray balance index parameter corresponding to the image grayscale of each frame image and the preset gray balance index parameter threshold indicates that the gray balance index parameter is not greater than the preset gray balance index parameter threshold, calling the second compensation coefficient model; The brightness compensation coefficient is obtained by using the second compensation coefficient model; wherein the structure of the second compensation coefficient model is as follows: Among them, B 02 represents the compensation coefficient of brightness obtained by the second compensation coefficient model; S represents the gray balance index parameter corresponding to the image grayscale of each frame image; S y Indicates the preset gray balance index parameter threshold; H p Indicates the average grayscale value corresponding to the frame image; H z Indicates the central gray value corresponding to the frame image; H x Indicates the grayscale intermediate value corresponding to the frame image; Using the brightness compensation coefficient to compensate and adjust the original brightness of each frame of image to obtain an adjusted image of each frame; The brightness after compensation adjustment is obtained by the following formula: L=(1+B 01 )·L0,or, Wherein, L represents the value of the brightness after compensation adjustment; L0 represents the value of the original brightness of each frame image; B 01 Indicates that the first compensation coefficient model obtains the compensation coefficient of brightness; B 02 It indicates that the first compensation coefficient model obtains the compensation coefficient of brightness.

5. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 2, characterized in that: The key point positions of the human body are extracted in step 2, specifically: Extracting key human point features from the target human feature points based on a posture estimation library, and identifying position information of the key human points; Determine the pixel coordinates of each human key point based on the human key point position information, perform smoothing processing on the key point coordinates, and determine the position information of each human key point in each frame image based on the processing result; And the posture type of the human body is determined based on the position information of each human key point in each frame image.

6. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 5, characterized in that: The behavior identification in step 2 further includes: Obtaining basic data of maintenance personnel actively uploaded by the management personnel, and extracting personnel data from the basic data of maintenance personnel; Obtaining the number of maintenance personnel, and establishing corresponding personnel storage space based on the number of personnel; In each personnel storage space, a time axis is established according to the data length of the video stream, and face recognition is performed on each frame of the video stream based on the facial feature points of the personnel data; According to the identification results, the maintenance personnel are identified, and the identity information and the time point of their appearance are recorded on the time axis of the personnel storage space to generate dynamic data; The position information of each maintenance personnel in each frame of image is extracted from the dynamic data, the initial position of each maintenance personnel is determined, and the position characteristics of the maintenance personnel are analyzed.

7. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 6, characterized in that: Analyze the location characteristics of maintenance personnel, including: Dividing the dynamic data into a plurality of unit data segments, and obtaining the movement position corresponding to each maintenance personnel in each unit data segment; The initial position of the same maintenance personnel and the moving position corresponding to each unit data segment are obtained, and the personnel movement trajectory is established. The total distance and average movement speed of the maintenance personnel in the unit data segment are calculated according to the movement trajectory, and the movement characteristics corresponding to each maintenance personnel are generated; At the same time, in each unit data segment, it is determined whether the maintenance personnel stay at the same position for more than a preset pause time threshold, and the position and the stay time are recorded to generate the pause characteristics of the maintenance personnel; Obtain the target unit data segment where the pause feature corresponding to each maintenance personnel is located, and at the same time, based on the time series of the target unit data segment in the dynamic data, determine whether there are multiple maintenance personnel pausing in the same time period, and determine the communication features between the maintenance personnel; A location feature data set of the maintenance personnel is established according to the movement features and communication features corresponding to each maintenance personnel, and the maintenance personnel behavior patterns contained in the location feature data set are analyzed.

8. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 7, characterized in that: The risk behavior determination in step 3 further includes: The extracted key point position information and posture type of the human body are matched with the preset dangerous behavior pattern library, and the similarity between the current behavior of the computer personnel and the behavior pattern in the dangerous behavior pattern library is calculated to identify possible dangerous behaviors; At the same time, the identified behavior is contextually analyzed based on the extracted target environment feature points and target human feature points; Combined with the analysis results, a risk assessment is conducted on the identified possible dangerous behaviors to evaluate the severity and probability of their possible consequences.

9. The method for monitoring the operation behavior of maintenance personnel based on artificial intelligence according to claim 8, characterized in that: The frequency of historical violations of aircraft maintenance personnel in step 5 is determined, specifically including: Collect historical behavior data of maintenance personnel, including all monitored violations, classify violations into different types, and assign a unique code to each violation; For each maintenance personnel, count the number of violations of each type, calculate the frequency of each violation, divide the time into different segments, and determine the frequency of violations in each time segment; Draw a trend chart of violations based on historical violation data, determine the changing trend of violations, establish a violation frequency prediction model, and predict the frequency of violations in the future based on the violation frequency prediction model; Combined with the predicted results of the frequency of violations, the operating level of each maintenance personnel is evaluated, a maintenance personnel evaluation report is generated, and the corresponding maintenance personnel handling plan is matched according to the operating level evaluation results.

Images