A transportation monitoring system based on artificial intelligence video analysis

Abstract

The present invention relates to the field of transportation monitoring technology, and in particular to a transportation monitoring system based on artificial intelligence video analysis. It includes a data acquisition unit, an end data storage unit, a monitoring platform, a video preprocessing unit, a video analysis unit, an early warning unit, and a display unit; the video preprocessing unit is used to determine whether it is a normal situation, and the video analysis unit performs enhanced processing on the video data and determines whether there is a fault and the type of fault; the early warning unit is used to analyze the probability of future faults in the absence of faults; the present invention processes the collected video data step by step, and enhances the video data, so that the characteristics of the video data are more obvious, which is conducive to the training of the preset model and the output of the results; the present invention adopts an artificial intelligence algorithm to analyze the collected video data and output the conclusion, reducing human interference; the output conclusion can be directly reflected in the video, which is more intuitive.

Description

Technical Field

[0001] The present invention relates to the field of transportation monitoring technology, and in particular to a transportation monitoring system based on artificial intelligence video analysis. Background Art

[0002] Production and transportation underground in mines are typical high-risk and arduous industries. With the continuous popularization of the industrial Internet, a large number of electronic equipment are also used in production and transportation underground in mines, especially video surveillance for the transportation link, which is of great significance to the safety of transportation.

[0003] However, the mine environment is complex and dark in most cases, which brings many problems to video surveillance. Although the video display can meet the monitoring needs, it cannot carry out timely intelligent analysis and requires a lot of manpower to manage. Therefore, how to improve the quality of video collected in the mine so that it can meet the requirements of intelligent analysis and determine whether there is a transportation fault through artificial intelligence is an urgent problem that needs to be solved.

[0004] In view of the above technical defects, a solution is now proposed. Summary of the invention

[0005] The purpose of the present invention is to provide a transportation monitoring system based on artificial intelligence video analysis to solve the above-mentioned technical defects. The present invention processes the collected video data step by step and enhances the video data to make the characteristics of the video data more obvious, which is conducive to the training of the preset model and the output of the results; the present invention adopts an artificial intelligence algorithm to analyze the collected video data and output conclusions to reduce human interference; the output conclusions can be directly reflected in the video, which is more intuitive.

[0006] The object of the present invention can be achieved by the following technical solutions: a transportation monitoring system based on artificial intelligence video analysis, specifically comprising a data acquisition unit, an end data storage unit, a monitoring platform, a video preprocessing unit, a video analysis unit, an early warning unit, and a display unit;

[0007] The data acquisition unit is connected to the end data storage unit in a one-way communication manner; the end data storage unit is connected to the video preprocessing unit in a two-way communication manner; the end data storage unit is connected to the video analysis unit in a two-way communication manner; the end data storage unit is connected to the early warning unit in a two-way communication manner; the video preprocessing unit is connected to the video analysis unit and the monitoring platform in a one-way communication manner respectively; the video analysis unit is connected to the early warning unit and the monitoring platform in a one-way communication manner respectively; the early warning unit is connected to the monitoring platform in a one-way communication manner; the display unit is used to display the video data stored in the monitoring platform.

[0008] The data acquisition unit is used to collect video data of the belt and its carrier, and send the collected original video data to the end data storage unit through a wired or wireless network for temporary storage.

[0009] The end data storage unit immediately sends the received original video data to the video preprocessing unit. The video preprocessing unit identifies the original video data and decomposes the original video data into continuous image frames. If it is less than a preset threshold value Q after calculation, it is determined to be a normal situation. The video preprocessing unit sends the original video data to the monitoring platform and outputs it through the display unit. At the same time, the video preprocessing unit generates a normal video signal and sends the normal video signal to the end data storage unit. After receiving the normal video signal, the end data storage unit deletes the original video data determined to be normal; otherwise, it is determined to be an abnormal situation. The video preprocessing unit deletes the original video data. At the same time, the video preprocessing unit outputs a processing signal and sends the processing signal to the end data storage unit. After receiving the processing signal, the end data storage unit sends the original video data determined to be abnormal to the video analysis unit.

[0010] Normal videos in historical video data are manually screened and the features of normal videos are used to train unsupervised learning models as judgment criteria;

[0011] The video preprocessing unit receives the original video data and decomposes the original video data into continuous image frames Ft, extracts the image frames and performs grayscale and normalization processing to obtain standardized grayscale image frames. , extract the features of each standardized grayscale frame, including spatial features and temporal features; input the extracted features of the original video data into the unsupervised learning model, calculate the anomaly score and determine whether it is abnormal.

[0012] Among them, the spatial features include the brightness mean feature , brightness standard deviation characteristics , edge density feature Et, texture feature Ct; the time feature includes frame difference feature , the features of each standardized grayscale image frame are represented by a vector as Zt, ;

[0013] The unsupervised learning model uses a Gaussian mixture model GMM to calculate the probability distribution of GMM: ;

[0014] Where K is the number of Gaussian distributions;

[0015] is the weight of the kth Gaussian distribution, satisfying: ;

[0016] represents the probability density of the kth Gaussian distribution;

[0017] represents the mean of the kth Gaussian distribution; represents the covariance matrix of the kth Gaussian distribution;

[0018] Calculate the negative logarithmic probability of GMM: GS=-log(p(Zt)). If the anomaly score GS is greater than or equal to the threshold Q, it is judged as abnormal, otherwise it is normal.

[0019] Specifically, the brightness mean feature , ;

[0020] Brightness standard deviation characteristics ;

[0021] The edge density feature Et is the ratio of the number of pixels identified as edges in the grayscale image frame to the total number of pixels in the grayscale image frame;

[0022] Texture features , where i,j are grayscale values, is the gray-level co-occurrence matrix, which indicates the number of times gray-level values ​​i and j appear in a given direction;

[0023] Temporal features include frame difference features , The larger the value, the more drastic the change. is the grayscale value of the tth frame, The gray value of the t+1th frame, Represents the absolute value of the difference in grayscale value between the tth frame and the t+1th frame;

[0024] Where N is the total number of pixels.

[0025] When the video preprocessing unit determines that the original video data is abnormal:

[0026] The video analysis unit receives the original video data, performs image enhancement processing on the original video data to obtain video enhanced image data, and analyzes the video enhanced image data through a preset model to identify whether there is a fault and the type of fault; if there is a fault, the fault is marked, and then the marked video data is sent to the monitoring platform for storage and output through the display unit. At the same time, the video analysis unit generates a fault signal and sends it to the end data storage unit. After receiving the fault signal, the end data storage unit deletes the original video data; if there is no fault, the video analysis unit generates a prediction signal and sends it to the early warning unit, and sends the feature vector data to the end data storage unit for storage.

[0027] The specific operation is as follows: the video analysis unit receives the original video data and decomposes the original video data into continuous image frames, extracts the image frames for noise reduction and defogging, and performs grayscale processing to obtain a pixel-adjusted image; dynamically adjusts the brightness of the pixel-adjusted image, and then performs contrast enhancement, local contrast enhancement, and edge enhancement to obtain video enhanced image data; in the brightness enhancement process, adaptive brightness enhancement is adopted, and the dynamic adjustment brightness gain α(x, y) and brightness offset β(x, y) are set; the video enhanced image data is analyzed by a preset convolutional neural network model to identify whether there is a fault and the type of fault, which includes coal flow breakage, belt deviation, coal overflow, belt tearing, and abnormal objects; if there is a fault, it is marked at the fault location, and then the marked video data is sent to the monitoring platform and output through the display unit, and the video analysis unit deletes the video data identified as a fault; if there is no fault, the video analysis unit sends the feature vector data to the end data storage unit for storage, and generates a prediction signal to send to the early warning unit.

[0028] The video analysis unit performs two operations: the first step is to decompose the original video data into image frames through the OpenCV tool, and then perform image processing on the obtained image frames; the second step is to perform fault judgment and classification based on the obtained video enhanced image data;

[0029] The first step of image processing is as follows:

[0030] After the extracted image frame is subjected to denoising and defogging and grayscale processing, the specific processing steps include:

[0031] S1. Perform brightness adjustment on the pixel-adjusted image obtained, and obtain the adaptive brightness adjustment formula by applying adaptive parameters α(x, y) and β(x, y) to the original grayscale value I(x, y) obtained by grayscale processing: I'(x, y) = α(x, y) * I(x, y) + β(x, y); α(x, y) and β(x,y) are local adaptive parameters based on pixels; then further enhance the image details and calculate I"(x, y) = (I'(x, y) - I'_min) *CE, where I'_min represents the minimum grayscale value of the image;

[0032] CE represents the contrast enhancement amplitude;

[0033] I'(x, y) represents the original grayscale value after brightness processing;

[0034] I"(x, y) represents the enhanced gray value;

[0035] S2. Divide the image after brightness enhancement into several regions and calculate the grayscale value after equalization. The formula is:

[0036] H(v) = round((C(v) / MN) * (L - 1)), where

[0037] H(v) represents the gray value after equalization;

[0038] C(v) represents the cumulative histogram function (CDF);

[0039] M, N are the width and height of the local area;

[0040] L is the gray level;

[0041] Assign the grayscale value obtained by H(v) to the grayscale value after adjusting the contrast, and we get .

[0042] S3, edge enhancement processing: edge enhancement is performed through convolution operation, the second-order derivative of the image is calculated using the Laplacian operator, and then it is superimposed on the image with adjusted contrast;

[0043] The formula is: = -λ * ∇² ;

[0044] Represents the original grayscale value, where I(x, y)=H(v);

[0045] Represents the enhanced gray value;

[0046] Represents the second-order derivative of the pixel, calculated using the Laplace operator;

[0047] λ represents the sharpening factor, which controls the strength of edge enhancement.

[0048] The second step is to perform fault determination and classification based on the obtained video enhanced image data, and to perform identification and classification through a convolutional neural network model, including the following:

[0049] After obtaining the video enhanced image data, it is analyzed through the preset convolutional neural network model to identify whether there is a fault and the fault type. The training steps of the convolutional neural network model are as follows:

[0050] (1) Data acquisition: Extract frames from historical video data at a fixed frequency (e.g., 5 frames per second) as image data; enhance the image data of each frame to obtain video enhanced image data to ensure that the fault features are clear and obtain a training data set;

[0051] (2) Data labeling: The obtained training data set is divided into a training set and a test set. Fault labels are performed on the images in the training set so that each image has a label and is saved in a csv format file.

[0052] (3) Feature extraction: The ResNet50 model is used to automatically extract image features and generate feature vector data. The convolution layer is divided into three layers. The first convolution layer extracts edge features; the middle convolution layer extracts texture and shape information; belt deviation is manifested as the offset of the belt boundary; the high-level convolution layer captures the global pattern of the fault;

[0053] (4) Use the ResNet50 model for training: The image data of the training set is used to train the ResNet50 model; the ResNet50 model consists of three layers of convolutional layers, pooling layers, and fully connected layers; the ResNet50 model abstracts features from the input image layer by layer, and the extracted features are mapped to specific category labels through the fully connected layer.

[0054] After receiving the prediction signal, the early warning unit immediately retrieves the feature vector data and original video data stored in the end data storage unit, and analyzes them through a preset model; when the early warning unit completes data reception, the end data storage unit deletes the original video data and feature vector data; if there is a risk after analysis, the early warning unit will mark the original video data and send it to the monitoring platform for storage and output it through the display unit, and then delete the marked original video data and feature vector data; if there is no risk, the early warning unit will send the original video data to the monitoring platform for storage and output it through the display unit, and then delete the original video data and feature vector data.

[0055] The early warning unit uses historical normal data features to predict the probability of a failure at a certain moment in the future. If the predicted failure probability exceeds the set threshold, it is considered that there is a greater risk of failure in the future and a prompt is given. The early warning unit generates a fixed-length feature vector from the feature vector data. Then, the continuous image features over a period of time are used as input, and the time series model LSTM is used to analyze the changes of these features over time. Then, the fully connected layer and Sigmoid activation function are used to map the output of the LSTM to a value in the range of [0, 1], which represents the probability of failure in the future.

[0056] Furthermore, the steps for training the time series model LSTM are as follows:

[0057] Step 1: Data acquisition: Get a vector of fixed length through feature vector data; then extract the feature vector of the time series, the shape is [batch_size, seq_length, feature_dim]; batch_size represents the number of time series samples input into the LSTM network for processing each time;

[0058] seq_length indicates the length of the time series, for example, the most recent 10 frames;

[0059] feature_dim represents the feature dimension of each frame;

[0060] The second step is to calibrate the data: add labels to the acquired data: whether a failure will occur in the future, 0 means normal, 1 means failure;

[0061] Step 3: Data training:

[0062] (1) Data loading and preparation: Divide the time series data into small batches; each batch is input into the model, and the label is used to calculate the loss;

[0063] (2) Forward propagation: Input time series data, LSTM processes it step by step according to the time step; extract the hidden state of the last time step, pass it to the fully connected layer, and output the failure probability;

[0064] (3) Calculate loss: Use the cross entropy loss function to compare the predicted value and the true label and calculate the error;

[0065] (4) Back propagation: Error back propagation, gradient calculation; optimizer updates model parameters to minimize loss;

[0066] (5) Verification and debugging: Evaluate model performance on the validation set to avoid overfitting.

[0067] The beneficial effects of the present invention are as follows:

[0068] (1) The present invention processes the collected video, and then analyzes the video through a trained artificial intelligence model to assist the staff in monitoring the transportation system; the present invention has a video preprocessing unit to preprocess and analyze the collected video data, quickly filter out the video data of normal conditions, reduce complex image processing steps, and reduce the computing pressure of the video analysis unit; the present invention has a video analysis unit that can further analyze the collected video data to determine whether there is a fault; the present invention has an early warning unit that can analyze the probability of a fault; the present invention performs hierarchical processing according to the transportation situation, reduces resource occupation, and ensures the stable operation of the operation monitoring system;

[0069] (2) The video analysis unit of the present invention performs image processing on the collected video data, including noise reduction, defogging, brightness enhancement, contrast enhancement, edge enhancement and other processing methods; it avoids problems such as brightness overexposure caused by simply performing overall enhancement processing, local distortion caused by contrast enhancement, and loss of details; the processed data is more conducive to artificial intelligence model analysis, reducing the occurrence of underreporting and false reporting. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The present invention will be further described below in conjunction with the accompanying drawings;

[0071] Figure 1 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION

[0072] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Embodiment 1:

[0074] See also Figure 1 As shown, the present invention is a transportation monitoring system based on artificial intelligence video analysis, which specifically includes a data acquisition unit, an end data storage unit, a monitoring platform, a video preprocessing unit, a video analysis unit, an early warning unit, and a display unit;

[0075] The data acquisition unit is connected to the end data storage unit in a one-way communication manner; the end data storage unit is connected to the video preprocessing unit in a two-way communication manner; the end data storage unit is connected to the video analysis unit in a two-way communication manner; the end data storage unit is connected to the early warning unit in a two-way communication manner; the video preprocessing unit is connected to the video analysis unit and the monitoring platform in a one-way communication manner respectively; the video analysis unit is connected to the early warning unit and the monitoring platform in a one-way communication manner respectively; the early warning unit is connected to the monitoring platform in a one-way communication manner; the display unit is used to display the video data stored in the monitoring platform.

[0076] The data acquisition unit includes an industrial camera for acquiring video data of the belt and its carrier, and sending the acquired original video data to the end data storage unit through a wired or wireless network for temporary storage.

[0077] The end data storage unit immediately sends the received original video data to the video preprocessing unit. The video preprocessing unit identifies the original video data and decomposes the original video data into continuous image frames. If it is less than a preset threshold value Q after calculation, it is determined to be a normal situation. The video preprocessing unit sends the original video data to the monitoring platform and outputs it through the display unit. At the same time, the video preprocessing unit generates a normal video signal and sends the normal video signal to the end data storage unit. After receiving the normal video signal, the end data storage unit deletes the original video data determined to be normal; otherwise, it is determined to be an abnormal situation. The video preprocessing unit deletes the original video data. At the same time, the video preprocessing unit outputs a processing signal and sends the processing signal to the end data storage unit. After receiving the processing signal, the end data storage unit sends the original video data determined to be abnormal to the video analysis unit.

[0078] The video preprocessing unit operates as follows:

[0079] Retrieve historical video data as the data source for model training;

[0080] Normal videos in historical video data are screened out through manual judgment, and the features of normal videos are used to train unsupervised learning models as judgment criteria;

[0081] The video preprocessing unit receives the original video data and decomposes the original video data into continuous image frames Ft, where Ft represents the t-th frame image, and the continuous image frames Ft are set as V. , where T is the total number of frames in the video;

[0082] Extract image frames and perform grayscale and normalization processing to obtain standardized grayscale frames; extract each frame of the image from the video file, usually at a fixed frame rate (such as 30 frames per second). If the video frame rate is too high (such as 60fps), frame sampling can be performed, extracting once every k frames (such as k=2).

[0083] Convert the image frame Ft to grayscale ,calculate =0.299R+0.587G+0.114B, is the grayscale value, R, G, B are the pixel values ​​of the RGB channels; then the calculated Perform normalization: ; We get a set of standardized grayscale frames: .

[0084] Extract each standardized grayscale frame The characteristics of the system include spatial characteristics and temporal characteristics;

[0085] Spatial features include brightness mean features , brightness standard deviation characteristics , edge density feature Et, texture feature Ct;

[0086] Brightness mean feature ; Brightness standard deviation characteristics ;N is the total number of pixels.

[0087] The edge is extracted by the edge detection algorithm Canny, and the edge pixel ratio is calculated to obtain the edge density feature Et as a grayscale frame. The number of pixels identified as edges in the grayscale image frame The ratio of the total number of pixels;

[0088] The gradient amplitude (edge ​​strength) is calculated by the edge detection algorithm Canny, and the gradient amplitude (edge ​​strength) of each pixel is calculated:

[0089] ; Then the gradient amplitude is set according to the threshold Convert to binary edge image ; Then calculate the edge density feature Et ,in, Represents the total number of pixels in the image. Indicates the number of edge pixels.

[0090] The texture features of the image are calculated using the gray-level co-occurrence matrix GLCM: , where i,j are grayscale values, is the gray-level co-occurrence matrix, which indicates the number of times gray-level values ​​i and j appear in a given direction; Reflects the degree of local change between grayscale values. The larger the value, the stronger the image texture contrast.

[0091] For example, if the matrix size is 4×4, then count the number of times gray values ​​i and j appear in the horizontal direction and construct the matrix , normalized , then calculate

[0092] =(0.125*0 2 )+(0.125*1 2 )+(0.0625*1 2 )+(0.0625*2 2 )+(0.1875*0 2 )=0.4375.

[0093] Temporal features include frame difference features , The larger the value, the more drastic the change. is the grayscale value of the tth frame, The gray value of the t+1th frame, Represents the absolute value of the difference in grayscale values ​​between the tth frame and the t+1th frame.

[0094] The features of each standardized grayscale image frame are represented by a vector as Zt, .

[0095] The features of the extracted raw video data are input into the unsupervised learning model, the anomaly score is calculated and whether it is abnormal is determined.

[0096] Among them, the unsupervised learning model uses the Gaussian mixture model GMM to calculate the probability distribution of GMM: ;

[0097] Where K is the number of Gaussian distributions;

[0098] is the weight of the kth Gaussian distribution, satisfying: ;

[0099] represents the probability density of the kth Gaussian distribution;

[0100] represents the mean of the kth Gaussian distribution; represents the covariance matrix of the kth Gaussian distribution;

[0101] in, ,d represents the dimension of the feature vector Zt, is the determinant of the covariance matrix, which represents the volume of the Gaussian distribution; It is the inverse of the covariance matrix, which is used to measure how to scale the data differences in different directions.

[0102] Calculate the negative logarithmic probability of GMM: GS=-log(p(Zt)). If the anomaly score GS is greater than or equal to the threshold Q, it is judged as abnormal, otherwise it is normal.

[0103] If the calculated GS value is less than the threshold value Q, it indicates that the current operation is normal. The video preprocessing unit sends the original video data to the monitoring platform and outputs it through the display unit. At the same time, the video preprocessing unit generates a normal video signal and sends the normal video signal to the end data storage unit. The end data storage unit deletes the original video data that is judged to be normal; the end data storage unit matches the timestamp of the normal video signal with the timestamp of the original video data to identify the original video data.

[0104] By training an unsupervised learning model to make preliminary judgments on the collected videos, normal videos can be quickly screened out, and in-depth analysis can be performed on uncertain situations. This reduces the complexity of calculations, does not require complex image processing steps, and is suitable for most running times.

[0105] Embodiment 2:

[0106] The difference between this embodiment and the first embodiment is that after the video preprocessing unit determines that the original video data is abnormal, the video preprocessing unit deletes the original video data stored in the video preprocessing unit, leaving storage space for the video preprocessing unit to store and analyze other original video data;

[0107] At the same time, the video preprocessing unit outputs a processing signal and sends the processing signal to the end data storage unit. After the end data storage unit receives the processing signal, the end data storage unit sends the original video data determined to be abnormal to the video analysis unit; the end data storage unit matches the timestamp of the processing signal with the timestamp of the original video data to identify the original video data.

[0108] When the video preprocessing unit determines that the original video data is abnormal:

[0109] The video analysis unit receives the original video data, performs image enhancement processing on the original video data to obtain video enhanced image data, and analyzes the video enhanced image data through a preset model to identify whether there is a fault and the type of fault; if there is a fault, the fault is marked, and then the marked video data is sent to the monitoring platform for storage and output through the display unit. At the same time, the video analysis unit generates a fault signal and sends it to the end data storage unit. After receiving the fault signal, the end data storage unit deletes the original video data; if there is no fault, the video analysis unit generates a prediction signal and sends it to the early warning unit, and sends the feature vector data to the end data storage unit for storage.

[0110] The specific operation is as follows: the video analysis unit receives the original video data and decomposes the original video data into continuous image frames. The operation here is the same as that of the video preprocessing unit and will not be repeated here;

[0111] Then, the image frame is extracted for noise reduction and dehazing, and grayscale processing is performed to obtain a pixel-adjusted image; the brightness of the pixel-adjusted image is dynamically adjusted, and then contrast enhancement, local contrast enhancement, and edge enhancement are performed respectively to obtain video enhanced image data; in the brightness enhancement process, adaptive brightness enhancement is used, and the dynamically adjusted brightness gain α(x, y) and brightness offset β(x, y) are set.

[0112] Since the light in the mine is weak and uneven, the original video data needs to be processed before video data analysis, including noise reduction, defogging, brightness improvement, contrast enhancement, edge enhancement and other processing methods; if the overall enhancement is simply performed, the output video will not meet the subsequent analysis requirements, such as overexposure caused by brightness enhancement, local distortion caused by contrast enhancement, and loss of details.

[0113] The original grayscale value I(x, y) is obtained by grayscale processing. If the linear brightness enhancement formula I'(x, y) = α* I(x, y) + β is directly applied to the entire image, when the pixel value with higher brightness in the image is close to 255 (maximum value), the enhancement may cause pixel overflow and thus lose details. Therefore, by analyzing the brightness distribution of the image, the amplitude of brightness enhancement is adaptively adjusted to avoid distortion in areas with too high brightness. Adaptive parameters α(x, y) and β(x, y) are introduced to obtain the adaptive brightness adjustment formula: I'(x, y) = α(x, y) * I(x, y) + β(x, y); α(x, y) and β(x, y) are local adaptive parameters based on pixels: for pixel values ​​with lower brightness, a larger gain α(x, y) and offset β(x, y) are used; for pixel values ​​with higher brightness, α(x, y) and β(x, y) are reduced to avoid overexposure.

[0114] After the image frame is processed, video enhanced image data is obtained, and the video enhanced image data is analyzed by a preset convolutional neural network model to identify whether there is a fault and the type of fault, which includes coal flow breakage, belt deviation, coal overflow, belt tearing, and abnormal objects. If a fault exists, the fault is marked, and then the marked video data is sent to the monitoring platform and output through the display unit. The video analysis unit deletes the video data identified as a fault. If no fault exists, the video analysis unit sends the feature vector data to the end data storage unit for storage, and generates a prediction signal to send to the early warning unit.

[0115] The video analysis unit is divided into two steps: the first step is to decompose the original video data into image frames through the OpenCV tool, and then perform image processing on the obtained image frames; the second step is to perform fault judgment and classification based on the obtained video enhanced image data;

[0116] The first step of image processing is as follows:

[0117] After the extracted image frame is subjected to denoising and defogging and grayscale processing, the specific processing steps include:

[0118] S1. Perform brightness adjustment on the pixel-adjusted image obtained, and obtain the adaptive brightness adjustment formula by applying adaptive parameters α(x, y) and β(x, y) to the original grayscale value I(x, y) obtained by grayscale processing: I'(x, y) = α(x, y) * I(x, y) + β(x, y); α(x, y) and β(x,y) are local adaptive parameters based on pixels; then further enhance the image details and calculate I"(x, y) = (I'(x, y) - I'_min) *CE, where I'_min represents the minimum grayscale value of the image;

[0119] CE represents the contrast enhancement amplitude;

[0120] I'(x, y) represents the original grayscale value after brightness processing;

[0121] I"(x, y) represents the enhanced gray value;

[0122] S2. Divide the image after brightness enhancement into several regions and calculate the grayscale value after equalization. The formula is:

[0123] H(v) = round((C(v) / MN) * (L - 1)), where

[0124] H(v) represents the gray value after equalization;

[0125] C(v) represents the cumulative histogram function (CDF);

[0126] M, N are the width and height of the local area;

[0127] L is the number of gray levels, usually 256;

[0128] Assign the grayscale value obtained by H(v) to the grayscale value after adjusting the contrast, and we get .

[0129] S3, edge enhancement processing: edge enhancement is performed through convolution operation, the second-order derivative of the image is calculated using the Laplacian operator, and then it is superimposed on the image with adjusted contrast;

[0130] The formula is: = -λ * ∇² ;

[0131] Represents the original grayscale value, where I(x, y)=H(v);

[0132] Represents the enhanced gray value;

[0133] Represents the second-order derivative of the pixel, calculated using the Laplace operator;

[0134] λ represents the sharpening factor, which controls the strength of edge enhancement.

[0135] During the image processing process, it is necessary to dynamically adjust the corresponding parameters according to the different characteristics of the image data, and output the corresponding parameters through the AI ​​model;

[0136] The training steps of the AI ​​model are as follows:

[0137] (1) Dataset preparation

[0138] Video data collection:

[0139] Multiple video clips are collected from different scenarios (e.g. different lighting conditions, belt surface states, coal flow height, etc.) to cover the diversity of brightness and contrast distribution.

[0140] Frames are uniformly sampled from the video (eg, 1 frame per second) to form an image dataset.

[0141] Data annotation and distribution analysis:

[0142] Brightness and contrast characteristics (such as average brightness, contrast, dynamic range) are extracted from the acquired image dataset to analyze the distribution of image samples.

[0143] Ensure that the brightness distribution is balanced to avoid data bias. For example, the number of image samples with low brightness (<50 grayscale value), medium brightness (50~200 grayscale value), and high brightness (>200 grayscale value) are roughly evenly distributed.

[0144] Perform feature extraction: extract multiple statistical characteristics from the image as input features of the model to ensure that the model can accurately reflect the brightness and contrast distribution of the image;

[0145] Features include: Brightness mean feature , which is used to measure the overall brightness of the image;

[0146] Brightness standard deviation characteristics , used to measure the discrete degree of image brightness;

[0147] Global contrast GC is used to measure the distribution range of the overall gray level of the image.

[0148] GC = (I_max - I_min) / (I_max + I_min);

[0149] Dynamic range DR, used to represent the span of gray levels, DR = I_max - I_min;

[0150] (3) Generate target parameters based on rule optimization:

[0151] Generate the optimal parameters for each image through rule-based algorithms:

[0152] The dynamic adjustment rules for calculating α(x, y) and β(x, y) are as follows:

[0153] α(x, y) = 1 + k * (1 - / 255);

[0154] β(x, y) = c * (128 - ), where k represents the brightness gain adjustment coefficient; c represents the offset adjustment coefficient.

[0155] Calculate the contrast enhancement amplitude CE,

[0156] CE= min(1 + g * (TC - GC), C_max),

[0157] Where g represents the adjustment coefficient of contrast enhancement;

[0158] TC is the target contrast;

[0159] C_max is the maximum enhancement limit.

[0160] Local contrast enhancement sets a limiting threshold to avoid over-enhancement of local areas; a limiting threshold CL is generated based on the dynamic range and local contrast of the image;

[0161] CL = min(Max_Limit, a * DR); where,

[0162] a is the dynamic range adjustment coefficient;

[0163] Max_Limit is the maximum threshold limit set.

[0164] (4) Model construction and training:

[0165] The model includes: input layer, hidden layer and output layer;

[0166] The input layer inputs the input features obtained based on the image dataset;

[0167] The hidden layer includes a fully connected layer and a convolutional layer. The fully connected layer h = ReLU(Wx + b) is used to extract global features, and the convolutional layer h = ReLU(Conv(x) + b) is used to capture local features. The global features and local features are fused as high-order input features.

[0168] The output layer is used to output the target parameters [α(x, y), β(x, y), CE, CL];

[0169] The mean squared error (MSE) is used as the loss function to ensure that the predicted values ​​are close to the target values.

[0170] After obtaining the trained AI model, the target parameters [α(x, y), β(x, y), CE, CL] are predicted in real time, and the image enhancement algorithm is dynamically adjusted according to the predicted parameters.

[0171] The second step is to perform fault determination and classification based on the obtained video enhanced image data, and to perform identification and classification through a convolutional neural network model, including the following:

[0172] After obtaining the video enhanced image data, it is analyzed through the preset convolutional neural network model to identify whether there is a fault and the fault type. The training steps of the convolutional neural network model are as follows:

[0173] (1) Data acquisition: Extract frames from historical video data at a fixed frequency, for example, 5 frames per second as image data; enhance the image data of each frame to obtain video enhanced image data to ensure that the fault features are clear and obtain a training data set.

[0174] (2) Data labeling: The obtained training data set is divided into a training set and a test set. Fault labels are performed on the images in the training set so that each image has a label and is saved in a csv format file.

[0175] Coal flow break is marked as type 1; belt deviation is marked as type 2; coal spillage is marked as type 3; belt tear is marked as type 4; abnormal objects are marked as type 5; normal state is marked as type 0;

[0176] like:

[0177] frame_0.jpg: Coal flow fracture (label 1)

[0178] frame_1.jpg: normal (label 0)

[0179] frame_2.jpg: coal spill (label 3);

[0180] The saved csv file will display the following format:

[0181] image_name,label

[0182] frame_0.jpg,1;

[0183] frame_1.jpg,0;

[0184] frame_2.jpg,3;

[0185] In order to detect the specific location of the fault, it is necessary to mark the fault area and obtain the extreme values ​​of the horizontal and vertical coordinates of the fault area, such as:

[0186] frame_0.jpg: The coal flow fracture area is at [xmin=50, ymin=100, xmax=200, ymax=300].

[0187] (3) Feature extraction: The ResNet50 model is used to automatically extract image features and generate feature vector data. The convolution layer is divided into three layers. The first convolution layer extracts edge features including belt edges, coal particles, etc. The middle convolution layer extracts texture and shape information: for example, coal flow breakage is manifested as texture interruption in the coal area; belt deviation is manifested as the offset of the belt boundary; the high-level convolution layer captures the global pattern of the fault: such as abnormal coal distribution (coal overflow) or the appearance of objects that do not conform to the characteristics of coal (abnormal objects).

[0188] (4) Use the ResNet50 model for training: The image data of the training set is used to train the ResNet50 model; the ResNet50 model consists of three convolutional layers, a pooling layer, and a fully connected layer; the ResNet50 model abstracts features from the input image layer by layer, from pixels to edges, from edges to shapes, and then to global patterns. The extracted features are mapped to specific category labels through the fully connected layer.

[0189] The video analysis unit decomposes the newly acquired video data into image frames using the OpenCV tool, and then processes the image frames to obtain video enhanced image data. The image features are extracted through the trained model to form feature vector data, and the fault category is obtained after model analysis.

[0190] If there is a fault, the fault location is marked, and then the marked video data is sent to the monitoring platform and output through the display unit. The video analysis unit deletes the video data identified as a fault;

[0191] The video analysis unit analyzes the collected video data through the convolutional neural network model to obtain the fault type, superimposes the text on the image by using OpenCV's cv2.putText, then writes each frame to a new video file by using cv2.VideoWriter, and uses cv2.imshow to display the processed frames.

[0192] Embodiment three:

[0193] The difference between this embodiment and the second embodiment is that the video analysis unit analyzes the collected video data through the convolutional neural network model and if no fault type is output, it is determined to be a normal situation; at this time, it is necessary to further judge the probability of future failures and take measures as soon as possible.

[0194] After receiving the prediction signal, the early warning unit immediately retrieves the feature vector data and original video data stored in the end data storage unit, and analyzes them through a preset model; when the early warning unit completes data reception, the end data storage unit deletes the original video data and feature vector data; if there is a risk after analysis, the early warning unit will mark the original video data and send it to the monitoring platform for storage and output it through the display unit, and then delete the marked original video data and feature vector data; if there is no risk, the early warning unit will send the original video data to the monitoring platform for storage and output it through the display unit, and then delete the original video data and feature vector data.

[0195] The early warning unit uses historical normal data features to predict the probability of a failure at a certain moment in the future. If the predicted failure probability exceeds the set threshold, it is considered that there is a greater risk of failure in the future and a prompt is given. The early warning unit generates a fixed-length feature vector from the feature vector data. Then, the continuous image features over a period of time are used as input, and the time series model LSTM is used to analyze the changes of these features over time. Then, the fully connected layer and Sigmoid activation function are used to map the output of LSTM to a value in the range of [0, 1], indicating the probability of failure in the future. The failure probability text is superimposed on the image by using OpenCV's cv2.putText, and then each frame is written to a new video file by using cv2.VideoWriter, and the processed frames are displayed using cv2.imshow.

[0196] Furthermore, the steps for training the time series model LSTM are as follows:

[0197] Step 1: Data acquisition: Get a fixed-length vector through feature vector data, such as 2048-dimensional through the output of the ResNet50 model; then extract the feature vector of the time series, with a shape of [batch_size, seq_length, feature_dim]; batch_size represents the number of time series samples input into the LSTM network for processing each time;

[0198] seq_length indicates the length of the time series, for example, the most recent 10 frames;

[0199] feature_dim represents the feature dimension of each frame of the image, which is 2048 in this embodiment;

[0200] For example, if the input tensor shape data is [4, 10, 2048], it means that 4 time series are input at a time, each time series contains 10 time steps, and each time step has 2048 features;

[0201] The second step is to calibrate the data: add labels to the acquired data: whether a failure will occur in the future, 0 means normal, 1 means failure;

[0202] Step 3: Data training:

[0203] (1) Data loading and preparation: Divide the time series data into small batches; each batch is input into the model, and the label is used to calculate the loss;

[0204] (2) Forward propagation: Input time series data, LSTM processes it step by step according to the time step; extract the hidden state of the last time step, pass it to the fully connected layer, and output the failure probability;

[0205] (3) Calculate loss: Use the cross entropy loss function to compare the predicted value and the true label and calculate the error;

[0206] (4) Back propagation: Error back propagation, gradient calculation; optimizer updates model parameters to minimize loss;

[0207] (5) Verification and debugging: Evaluate model performance on the validation set to avoid overfitting.

[0208] The specific training steps are as follows:

[0209] S21 initializes training parameters and sets training hyperparameters:

[0210] num_epochs: the total number of training rounds;

[0211] Batch_size: the number of samples input into the model each time;

[0212] learning_rate: The learning rate of the optimizer.

[0213] S22 model training steps:

[0214] Step 1: In PyTorch, use model.train() to set the model to training mode.

[0215] Step 2: Data loading and batch processing:

[0216] Use DataLoader to divide the time series data into small batches and train one batch of data at a time;

[0217] The input shape of each batch is [batch_size, seq_length, feature_dim], and the label is [batch_size, output_dim]. output_dim means that the output of each time series is a scalar, indicating the probability of failure; batch_size means the number of prediction results output at a time;

[0218] Step 3: Batch For each epoch (training round), the model will perform the following steps on each batch of data:

[0219] 1. Input data: Get the input data batch_data and corresponding labels batch_labels of the current batch;

[0220] 2. Forward propagation: pass the input data into the model to obtain the predicted output predictions;

[0221] 3. Calculate loss: Use the loss function to calculate the error between the predicted value and the true label. Specifically, the binary cross entropy loss function is suitable for the binary classification task of predicting the probability of failure.

[0222] 4. Back propagation: calculate the gradient through the error and update the model parameters;

[0223] After each epoch, the model performance can be tested on the validation set to assess whether it is overfitting.

[0224] This example requires the use of LSTM to process time series:

[0225] LSTM receives a time sequence (length seq_length, feature dimension feature_dim) and gradually processes the features of each time step;

[0226] For each time series, LSTM generates a hidden state h_t, which represents the comprehensive representation of the features;

[0227] The hidden state h_t of the last time step (expressed as [batch_size, hidden_dim]) is used as a summary of the entire time series;

[0228] The fully connected layer then maps to a scalar: the hidden state h_t is mapped to a scalar, denoted as [batch_size, 1]; this scalar represents the model’s prediction score for future failures.

[0229] Finally, the probability is obtained through the Sigmoid activation function:

[0230] The score is mapped to [0, 1] through the Sigmoid activation function to represent the failure probability:

[0231] ;

[0232] Where p is the failure probability and y is the output score of the fully connected layer.

[0233] Where y=W*h_t +b; W is the weight matrix of the fully connected layer (expressed as [hidden_dim, 1]), and b is the bias term.

[0234] The threshold value in the present invention is set for the convenience of comparison. The size of the threshold value depends on the amount of sample data and the number of bases set by technical personnel in this field for each group of sample data; as long as it does not affect the proportional relationship between the parameter and the quantized value.

[0235] The above formulas are obtained by collecting a large amount of data for software simulation and selecting a formula that is close to the actual value. The coefficients in the formula are set by technical personnel in this field according to actual conditions. The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited to this. Any technical personnel familiar with the technical field within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, make equivalent replacement or change, which should be covered within the protection scope of the present invention.

Claims

1. A transportation monitoring system based on artificial intelligence video analysis, characterized in that: It includes data acquisition unit, terminal data storage unit, monitoring platform, video preprocessing unit, video analysis unit, early warning unit and display unit; The data acquisition unit is used to collect video data of the belt and its carrier, and send the collected original video data to the end data storage unit through a wired or wireless network for temporary storage; The end data storage unit immediately sends the received original video data to the video preprocessing unit. The video preprocessing unit identifies the original video data and decomposes the original video data into continuous image frames. If it is less than a preset threshold value Q after calculation, it is determined to be a normal situation. The video preprocessing unit sends the original video data to the monitoring platform and outputs it through the display unit. At the same time, the video preprocessing unit generates a normal video signal and sends the normal video signal to the end data storage unit. After receiving the normal video signal, the end data storage unit deletes the original video data determined to be normal; otherwise, it is determined to be an abnormal situation. The video preprocessing unit deletes the original video data. At the same time, the video preprocessing unit outputs a processing signal and sends the processing signal to the end data storage unit. After receiving the processing signal, the end data storage unit sends the original video data determined to be abnormal to the video analysis unit. The video analysis unit receives original video data, performs image enhancement processing on the original video data to obtain video enhanced image data, analyzes the video enhanced image data through a preset model to obtain feature vector data and identifies whether there is a fault and the type of fault; if there is a fault, the fault is marked, and then the marked video data is sent to the monitoring platform for storage and output through the display unit. At the same time, the video analysis unit generates a fault signal and sends it to the end data storage unit. After receiving the fault signal, the end data storage unit deletes the original video data; if there is no fault, the video analysis unit generates a prediction signal and sends it to the early warning unit, and sends the feature vector data to the end data storage unit for storage; After receiving the prediction signal, the early warning unit immediately retrieves the feature vector data and the original video data stored in the end data storage unit, and analyzes them through a preset model; when the early warning unit completes data reception, the end data storage unit deletes the original video data and the feature vector data; After analysis, if there is a risk, the early warning unit will mark the original video data and send it to the monitoring platform for storage and output it through the display unit, and then delete the marked original video data and feature vector data; if there is no risk, the early warning unit will send the original video data to the monitoring platform for storage and output it through the display unit, and then delete the original video data and feature vector data.

2. A transportation monitoring system based on artificial intelligence video analysis according to claim 1, characterized in that: The video preprocessing unit operates as follows: Normal videos in historical video data are manually screened and the features of normal videos are used to train unsupervised learning models as judgment criteria; The video preprocessing unit receives the original video data and decomposes the original video data into continuous image frames Ft, extracts the image frames and performs grayscale and normalization processing to obtain standardized grayscale image frames. , extract the features of each standardized grayscale image frame, including spatial features and temporal features; The features of the extracted raw video data are input into the unsupervised learning model, the anomaly score is calculated and whether it is abnormal is determined.

3. A transportation monitoring system based on artificial intelligence video analysis according to claim 2, characterized in that: The spatial feature includes a brightness mean feature , brightness standard deviation characteristics , edge density feature Et, texture feature Ct; the time feature includes frame difference feature , the features of each standardized grayscale image frame are represented by a vector as Zt, ; The unsupervised learning model uses a Gaussian mixture model GMM to calculate the probability distribution of GMM: ; Where K is the number of Gaussian distributions; is the weight of the kth Gaussian distribution, satisfying: ; represents the probability density of the kth Gaussian distribution; represents the mean of the kth Gaussian distribution; represents the covariance matrix of the kth Gaussian distribution; Calculate the negative logarithmic probability of GMM: GS=-log(p(Zt)). If the anomaly score GS is greater than or equal to the threshold Q, it is judged as abnormal, otherwise it is normal.

4. A transportation monitoring system based on artificial intelligence video analysis according to claim 3, characterized in that: The brightness mean feature , ; The brightness standard deviation characteristic ; The edge density feature Et is the ratio of the number of pixels identified as edges in the grayscale image frame to the total number of pixels in the grayscale image frame; Texture features , where i,j are grayscale values, is the gray-level co-occurrence matrix, which indicates the number of times gray-level values ​​i and j appear in a given direction; Temporal features include frame difference features , The larger the value, the more drastic the change. is the grayscale value of the tth frame, The gray value of the t+1th frame, Represents the absolute value of the difference in grayscale value between the tth frame and the t+1th frame; Where N is the total number of pixels.

5. The transportation monitoring system based on artificial intelligence video analysis according to claim 1 is characterized in that: The video analysis unit operates as follows: The video analysis unit receives the original video data and decomposes the original video data into continuous image frames, extracts the image frames for noise reduction and defogging, and performs grayscale processing to obtain a pixel-adjusted image; Dynamically adjust the brightness of the pixel-adjusted image, and then perform contrast enhancement, local contrast enhancement, and edge enhancement to obtain video enhanced image data; Adaptive brightness enhancement is used in the brightness enhancement process, and the brightness gain α(x, y) and brightness offset β(x, y) are set to be dynamically adjusted; The video enhanced image data is analyzed through a preset convolutional neural network model to identify whether there is a fault and the type of fault, which includes coal flow breakage, belt deviation, coal overflow, belt tearing, and abnormal objects; if there is a fault, it is marked at the fault location, and then the marked video data is sent to the monitoring platform and output through the display unit, and the video analysis unit deletes the video data identified as a fault; if there is no fault, the video analysis unit sends the feature vector data to the end data storage unit for storage, and generates a prediction signal to send to the early warning unit.

6. A transportation monitoring system based on artificial intelligence video analysis according to claim 5, characterized in that: After the extracted image frame is subjected to denoising and defogging and grayscale processing, the specific processing steps include: S1. Perform brightness adjustment on the pixel-adjusted image obtained: The original grayscale value I(x, y) obtained by grayscale processing is adjusted through adaptive parameters α(x, y) and β(x, y) to obtain the adaptive brightness adjustment formula: I'(x, y) = α(x, y) * I(x,y) + β(x, y); α(x, y) and β(x, y) are local adaptive parameters based on pixels; then further enhance the image details and calculate I"(x, y) = (I'(x, y) - I'_min) *CE, where I'_min represents the minimum grayscale value of the image; CE represents the contrast enhancement amplitude; I'(x, y) represents the original grayscale value after brightness processing; I"(x, y) represents the enhanced gray value; S2. Divide the image after brightness enhancement into several regions and calculate the grayscale value after equalization. The formula is: H(v) = round((C(v) / MN) * (L - 1)), where H(v) represents the gray value after equalization; C(v) represents the cumulative histogram function (CDF); M, N are the width and height of the local area; L is the gray level; Assign the grayscale value obtained by H(v) to the grayscale value after adjusting the contrast, and we get ; S3, edge enhancement processing: edge enhancement is performed through convolution operation, the second-order derivative of the image is calculated using the Laplacian operator, and then it is superimposed on the image with adjusted contrast; The formula is: = -λ * ∇² ; Represents the original grayscale value, where I(x, y)=H(v); Represents the enhanced gray value; Represents the second-order derivative of the pixel, calculated using the Laplace operator; λ represents the sharpening factor, which controls the strength of edge enhancement.

7. A transportation monitoring system based on artificial intelligence video analysis according to claim 6, characterized in that: After obtaining the video enhanced image data, it is analyzed through the preset convolutional neural network model to identify whether there is a fault and the fault type. The training steps of the convolutional neural network model are as follows: Data acquisition: extract frames from historical video data at a fixed frequency as image data; enhance the image data of each frame to obtain video enhanced image data to ensure that the fault features are clear and obtain a training data set; Data labeling: The obtained training data set is divided into a training set and a test set, and fault labels are performed on the images in the training set so that each image has a label and is saved in a csv format file; Feature extraction: The ResNet50 model is used to automatically extract image features and generate feature vector data. The convolution layer is divided into three layers. The first convolution layer extracts edge features; the middle convolution layer extracts texture and shape information; belt deviation is manifested as the offset of the belt boundary; the high-level convolution layer captures the global pattern of the fault. Use the ResNet50 model for training: Use the image data of the training set to train the ResNet50 model; the ResNet50 model includes three convolutional layers, a pooling layer, and a fully connected layer; the ResNet50 model abstracts features from the input image layer by layer, and the extracted features are mapped to specific category labels through the fully connected layer.

8. The transportation monitoring system based on artificial intelligence video analysis according to claim 1 is characterized in that: The early warning unit uses historical normal data features to predict the probability of a failure at a certain moment in the future. If the predicted failure probability exceeds the set threshold, it is considered that there is a greater risk of failure in the future and a prompt is given; the early warning unit generates a feature vector of fixed length from the feature vector data; then the continuous image features over a period of time are used as input, and the changes of these features over time are analyzed through the time series model LSTM; then the fully connected layer and the Sigmoid activation function are used to map the output of the LSTM to a value in the range of [0, 1], indicating the probability of a failure in the future.

9. A transportation monitoring system based on artificial intelligence video analysis according to claim 8, characterized in that: The steps for training the time series model LSTM are as follows: Step 1: Data acquisition: Get a vector of fixed length through feature vector data; then extract the feature vector of the time series, the shape is [batch_size, seq_length, feature_dim]; batch_size represents the number of time series samples input into the LSTM network for processing each time; seq_length indicates the length of the time series; feature_dim represents the feature dimension of each frame; The second step is to calibrate the data: add labels to the acquired data to indicate whether a failure will occur in the future, 0 indicates normal and 1 indicates failure; Step 3: Data training: (1) Data loading and preparation: Divide the time series data into small batches; each batch is input into the model, and the label is used to calculate the loss; (2) Forward propagation: Input time series data, LSTM processes it step by step, extracts the hidden state of the last time step, passes it to the fully connected layer, and outputs the failure probability; (3) Calculate loss: Use the cross entropy loss function to compare the predicted value and the true label and calculate the error; (4) Back propagation: Error back propagation, gradient calculation; optimizer updates model parameters to minimize loss; (5) Verification and debugging: Evaluate model performance on the validation set to avoid overfitting.

Images