Truck loading and unloading anomaly detection method and system based on image processing technology
By processing the GPS trajectory data of trucks and combining it with image and acoustic sensors, the problem of response delay in traditional monitoring systems has been solved, enabling real-time anomaly detection and alarm during the loading and unloading process of trucks.
Patent Information
- Application Number
- CN202411227313.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-03
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-09-03
AI Technical Summary
Traditional video surveillance systems cannot achieve efficient real-time data processing during truck loading and unloading, are easily affected by subjective factors, resulting in response delays and low monitoring efficiency, especially prone to misjudgment in complex environments.
By acquiring and preprocessing the GPS trajectory data of the trucks, and combining it with loading and unloading identification strategies, anomaly detection is performed using camera equipment and acoustic sensors to trigger an alarm mechanism.
It enables real-time monitoring of the loading and unloading status of trucks, accurately identifies abnormal situations and promptly issues alarms, thus improving monitoring efficiency and accuracy.
Smart Images

Figure CN118736511B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cargo transportation, and in particular to a method and system for detecting abnormal loading and unloading of trucks based on image processing technology. Background Art
[0002] With the rapid development of the logistics industry, trucks are often loaded and unloaded during transportation. However, when there are abnormalities during the loading and unloading process, it may cause problems such as cargo damage and transportation delays, which in turn affects logistics efficiency and customer satisfaction. Therefore, it is of great significance to detect abnormalities during the loading and unloading process of trucks in a timely and accurate manner. Traditional video surveillance systems rely on manual observation, cannot effectively process large amounts of video data in real time, and are easily affected by subjective factors.
[0003] For example, the patent with publication number CN112163818A provides a cargo transportation control method and device, storage medium and electronic device. Among them, the cargo transportation control method includes: the server receives the loading image sent by the carrier terminal; the server receives the unloading image sent by the carrier terminal; the server uses the image recognition algorithm based on the loading image and the unloading image to determine whether there is any abnormality in the transportation of the cargo; the server sends an abnormal prompt information to the consignor terminal and / or the carrier terminal when there is an abnormality in the cargo. In this method, the server first uses the image recognition algorithm to automatically detect the abnormality of the cargo, and then automatically notifies the relevant personnel. It is efficient in handling abnormalities of the cargo and is conducive to reducing labor costs. In addition, this method only needs to use the cargo images collected during loading and unloading to detect abnormalities of the cargo, and there is no need to monitor the cargo throughout the process. Therefore, its deployment cost is low and its practical value is high.
[0004] The above patents all have the problems raised by this background technology: traditional video surveillance systems usually require manual observation and analysis, and cannot achieve efficient real-time data processing, resulting in delayed response and low monitoring efficiency. Especially in complex loading and unloading environments, relying only on GPS data and truck location to judge the loading and unloading status is prone to misjudgment. To solve the above problems, this application designs a truck loading and unloading anomaly detection method and system based on image processing technology. Summary of the invention
[0005] The technical problem to be solved by the present invention is to address the deficiencies of the prior art and provide a method and system for detecting abnormalities in truck loading and unloading based on image processing technology. First, the GPS trajectory data is acquired and processed to determine whether the truck is at the loading and unloading point, and then the loading and unloading identification strategy is used to determine whether the truck is in the loading and unloading operation. Once it is confirmed that the vehicle is in the loading and unloading state, the system will use an abnormality detection strategy to further analyze whether there are any abnormal conditions. If an abnormal event is detected, an alarm mechanism will be triggered.
[0006] To achieve the above object, the present invention provides the following technical solutions:
[0007] The method for detecting abnormalities in truck loading and unloading based on image processing technology includes the following steps:
[0008] S1: Obtain GPS track data of the truck, and determine whether the truck is at a loading and unloading point by processing the GPS track data;
[0009] S2: If the truck is at the loading and unloading point, the loading and unloading identification strategy is used to determine whether the truck is in the loading and unloading state;
[0010] S3: If the truck is in the loading and unloading state, determine whether there is any abnormality in the loading and unloading of the truck through the anomaly detection strategy;
[0011] S4: When abnormal loading and unloading events are detected, the alarm mechanism is triggered.
[0012] The GPS track data includes the truck GPS longitude, the truck GPS latitude, the truck GPS instantaneous speed, the truck GPS direction angle and the truck GPS timestamp. The specific steps of processing the GPS track data are as follows:
[0013] S1.1: preprocessing the GPS trajectory data to obtain smoothed data, wherein the preprocessing includes trajectory data cleaning, trajectory data noise reduction and trajectory data smoothing;
[0014] S1.2: Calculate the average speed of the truck according to the GPS longitude of the truck, the GPS latitude of the truck, the instantaneous GPS speed of the truck and the GPS timestamp of the truck;
[0015] S1.3: Calculate the average yaw rate of the truck according to the instantaneous GPS speed of the truck, the GPS direction angle of the truck and the GPS timestamp of the truck;
[0016] S1.4: establishing a truck trajectory feature set according to the GPS trajectory data, the average speed of the truck and the average deflection rate of the truck, calculating the weight of the truck trajectory feature by using the hierarchical analysis method, and obtaining the truck motion matrix according to the weight and the truck trajectory feature;
[0017] S1.5: Construct a location recognition network, use the truck motion matrix as the input parameter of the location recognition network, train the input parameter through the location recognition network, and output the location recognition result of the truck GPS trajectory data, wherein the location recognition result includes driving points, stop points and loading and unloading points.
[0018] The step of constructing a location identification network includes:
[0019] The input layer is used to perform cluster analysis on the truck motion matrix through a clustering algorithm, set the truck trajectory features belonging to the driving points in the truck motion matrix to zero according to the clustering results, obtain the stay feature matrix, and mark the corresponding GPS trajectory data as driving points;
[0020] The convolution pooling layer is used to extract the abstract features of the dwell feature matrix and reshape the abstract features into a one-dimensional long vector through the convolution kernel. The one-dimensional long vector is downsampled according to the maximum pooling to obtain the posture features of the GPS trajectory data.
[0021] The output layer is used to perform weighted splicing on the posture features, calculate the location confidence probability of the GPS trajectory data through the sigmoid activation function, mark the GPS trajectory data with a location confidence probability greater than the confidence threshold as a loading and unloading point, mark the GPS trajectory data with a location confidence probability less than or equal to the confidence threshold as a stay point, and output the location recognition result of the GPS trajectory data. The calculation formula of the location confidence probability is:
[0022] ,
[0023] in, represents the location confidence probability of GPS trajectory data, represents the sigmoid activation function, Represents the probability weight value of the change of the truck direction feature, represents the change probability weight value of the truck location feature, represents the change probability weight value of the truck stop feature, represents vector multiplication, represents the maximum pooling aggregation function, C(•) represents the convolution kernel operation, represents the abstract feature, and b represents the bias constant.
[0024] The specific steps of the loading and unloading identification strategy are as follows:
[0025] S2.1: Shooting a video sequence of a truck body through a camera device, obtaining inter-frame images of the truck body frame by frame, and performing differential denoising on the inter-frame images of the truck body;
[0026] Perform background subtraction on the box-frame inter-frame image after differential denoising, and calculate the contour features of the box-frame inter-frame image through morphological opening operation;
[0027] S2.2: Process the contour features through the elliptical Fourier descriptor, extract the local feature map, perform regional cutting on the local feature map, and obtain the regional temporal features and regional spatial features;
[0028] S2.3: Fuse the regional temporal features and the regional spatial features to obtain the bilinear regional features, and use the pooling function to integrate the bilinear regional features of all regions to obtain the global bilinear feature descriptor;
[0029] S2.4: Activate the global bilinear feature descriptor through the Relu function, obtain the door area features, detect the door area features through the detection frame, calculate the door state value, compare the door state value with the opening threshold, if it is greater than or equal to the opening threshold, mark the box frame image of the corresponding frame as 1, if it is less than the opening threshold, mark the box frame image of the corresponding frame as 0;
[0030] S2.5: Traverse all the box frame images and obtain the mark values of the box frame images. If the proportion of the box frame images with a mark value of 1 is greater than the loading and unloading threshold, it means that the truck is in the loading and unloading state.
[0031] The calculation formula of the door status value is:
[0032] ,
[0033] in, represents the door state value, k represents a single detection frame, N represents the total number of detection frames, O represents the plane dimension of the door area feature, o represents the single plane coordinate of the door area feature, represents the door closure discriminant function, The horizontal coordinate of the plane representing the characteristics of the hatch area, The average value of the horizontal coordinate of the plane representing the characteristics of the hatch area, The plane ordinate representing the characteristics of the hatch area, represents the average plane ordinate of the hatch area feature, V represents the three-dimensional dimension of the hatch area feature, p represents the single three-dimensional coordinate of the hatch area feature, The vertical coordinates of the hatch area feature, The average vertical coordinate of the plane representing the characteristics of the hatch area.
[0034] The anomaly detection strategy includes a location anomaly detection sub-strategy and a handling anomaly detection sub-strategy. The location anomaly detection sub-strategy is used to identify whether the truck is located in a predetermined loading and unloading area through fence detection technology. The handling anomaly detection sub-strategy is used to obtain sound signals emitted by abnormal noises of goods in the carriage during the loading and unloading process through acoustic sensors, process the sound signals, and identify whether there are abnormal conditions during the loading and unloading process, wherein the abnormal conditions include cargo vibration, cargo collision, and cargo scratches.
[0035] The specific steps of the location anomaly detection sub-strategy are as follows:
[0036] S3.1: Obtain the geographical boundary of the loading and unloading area, and convert the geographical boundary into a fence model through a geographic information system, wherein the fence model includes a circular model and a polygonal model;
[0037] S3.2: Matching the truck GPS longitude and the truck GPS latitude of the truck loading and unloading location with the fence model;
[0038] S3.3: Determine whether the truck is located in the predetermined loading and unloading area based on the matching result. If the truck is not in the loading and unloading area, it indicates that an abnormal location event exists.
[0039] The specific steps of the handling anomaly detection sub-strategy are as follows:
[0040] S3.4: Acquire sound signals generated by abnormal noises of cargo in the carriage during cargo loading and unloading through an acoustic sensor, and pre-process the sound signals;
[0041] S3.5: extracting features from the preprocessed sound signal to obtain a sound feature signal, performing short-time Fourier transform on the sound feature signal, and calculating a time domain spectrum feature of the sound feature signal;
[0042] S3.6: Perform logarithmic operation and dimension reconstruction on the time domain spectrum features to obtain input parameters of the cargo status recognition network, perform convolution on the input parameters, and calculate input features of the input parameters;
[0043] S3.7: According to the input features of the input parameters, the cargo status is identified using the cargo status recognition network, the cargo status category parameters are calculated through the attention mechanism, and the abnormal situation classification results are output according to the cargo status category parameters.
[0044] A truck loading and unloading anomaly detection system based on image processing technology, the system includes a GPS trajectory data processing module, a loading and unloading identification module, an anomaly detection module and an alarm mechanism module;
[0045] The GPS track data processing module is used to acquire and pre-process the truck GPS data in real time, calculate the track features, and identify the truck position status;
[0046] The loading and unloading identification module is used to identify whether the truck is in a loading and unloading state through camera equipment and image processing technology;
[0047] The anomaly detection module is used to determine abnormal behavior during the loading and unloading process of the truck based on the location and handling anomaly detection strategy;
[0048] The alarm mechanism module is used to trigger an alarm and notify relevant personnel to take measures when an abnormal loading and unloading event is detected;
[0049] The GPS trajectory data processing module includes:
[0050] GPS data acquisition unit: responsible for obtaining the real-time GPS track data of the truck, including longitude, latitude, instantaneous speed, direction angle and timestamp;
[0051] Data preprocessing unit: preprocess the collected GPS data, including data cleaning, noise reduction and smoothing, to obtain smooth data;
[0052] Trajectory feature calculation unit: calculates the average speed and average deflection rate of the truck based on the preprocessed GPS data, and establishes a truck trajectory feature set;
[0053] Motion matrix generation unit: calculates the weight of trajectory features through the hierarchical analysis method and generates the truck motion matrix;
[0054] Location recognition unit: calculates the location confidence probability of GPS trajectory data through the sigmoid activation function and outputs the location recognition result;
[0055] The loading and unloading identification module comprises:
[0056] Video acquisition unit: uses a camera to capture a video sequence of the truck body and obtains inter-frame images of the body frame by frame;
[0057] Image processing unit: perform differential denoising, background subtraction and morphological opening operations on the inter-frame images of the box to extract contour features;
[0058] Feature extraction unit: Process contour features through elliptical Fourier descriptors, extract local feature maps, perform regional cutting, and obtain regional temporal and spatial features;
[0059] Feature fusion unit: fuses regional temporal features and spatial features to obtain bilinear regional features, and obtains global bilinear feature descriptors through pooling functions;
[0060] Door status detection unit: activates the global bilinear feature descriptor through the ReLU function, detects the door area features, calculates the door status value, and marks the box inter-frame image;
[0061] The anomaly detection module comprises:
[0062] Location anomaly detection unit: Uses fence detection technology to determine whether the truck is within the designated loading and unloading area and whether there is a location anomaly event;
[0063] Abnormal handling detection unit: The acoustic sensor is used to obtain the sound signals emitted by abnormal noises in the carriage during loading and unloading of goods, process the sound signals, and output the abnormal situation classification results.
[0064] Compared with the prior art, the present invention has the following beneficial effects:
[0065] The present invention obtains the GPS trajectory data of the truck, pre-processes and analyzes the data, and can accurately determine whether the truck is at the loading and unloading point. Further combined with the loading and unloading identification strategy, it can effectively identify whether the truck is performing loading and unloading operations, thereby providing real-time loading and unloading status monitoring. BRIEF DESCRIPTION OF THE DRAWINGS
[0066] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0067] Figure 1 This is a flow chart of a method for detecting abnormalities in loading and unloading of trucks based on image processing technology according to Embodiment 1 of the present invention;
[0068] Figure 2 This is a schematic diagram of the principle structure of a location identification network according to Embodiment 1 of the present invention;
[0069] Figure 3 This is a flow chart of the clustering algorithm of Example 1 of the present invention;
[0070] Figure 4 This is a flow chart of a loading and unloading identification algorithm according to Embodiment 1 of the present invention;
[0071] Figure 5 This is a flowchart of the box inter-frame image differential denoising process according to Embodiment 1 of the present invention;
[0072] Figure 6 It is a structural schematic diagram of a truck loading and unloading anomaly detection system based on image processing technology of the present invention. DETAILED DESCRIPTION
[0073] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0074] Example 1
[0075] See also Figure 1 , an embodiment of the present invention provides: a method for detecting abnormal loading and unloading of trucks based on image processing technology, comprising the following steps:
[0076] S1: Obtain GPS track data of the truck, and determine whether the truck is at a loading and unloading point by processing the GPS track data;
[0077] S2: If the truck is at the loading and unloading point, the loading and unloading identification strategy is used to determine whether the truck is in the loading and unloading state;
[0078] S3: If the truck is in the loading and unloading state, determine whether there is any abnormality in the loading and unloading of the truck through the anomaly detection strategy;
[0079] S4: When abnormal loading and unloading events are detected, an alarm mechanism is triggered;
[0080] The GPS trajectory data includes the truck GPS longitude, the truck GPS latitude, the truck GPS instantaneous speed, the truck GPS direction angle and the truck GPS timestamp. The truck GPS longitude and the truck GPS latitude are longitude and latitude coordinates based on the GPS positioning system, which are used to describe the spatial information of the truck. The truck GPS instantaneous speed and the truck GPS direction angle are the vehicle driving speed and angle recorded by the on-board sensor, which are used to describe the movement trend of the truck. The truck GPS timestamp refers to the sampling time point of the GPS trajectory data.
[0081] The specific steps of processing the GPS trajectory data are as follows:
[0082] Before S1.1: preprocessing the GPS trajectory data to obtain smooth data, wherein the preprocessing includes trajectory data cleaning, trajectory data noise reduction and trajectory data smoothing;
[0083] The trajectory data cleaning includes the following steps:
[0084] A1: Remove obvious error points in the GPS trajectory data, including points where the speed exceeds the reasonable range and the position jump is too large;
[0085] B1: Completing the missing trajectory data through interpolation algorithm;
[0086] C1: Filter out redundant trajectory points in a stationary state to reduce the amount of data and improve computational efficiency;
[0087] S1.1: Calculate the average speed of the truck according to the GPS longitude, GPS latitude, instantaneous GPS speed and GPS timestamp of the truck. The calculation formula of the average speed of the truck is:
[0088] ,
[0089] in, represents the average speed of the truck corresponding to the i-th GPS trajectory data of the truck, represents the distance correction factor, Indicates the GPS timestamp of the truck corresponding to the i+1th GPS track data, represents the GPS timestamp of the i-th truck, Har(•) represents the Harvey function, represents the GPS longitude of the i-th truck, represents the GPS longitude of the i+1th truck, represents the GPS latitude of the i-th truck, represents the GPS latitude of the i+1th truck, represents the instantaneous speed correction coefficient, represents the GPS instantaneous speed of the i+1th truck, represents the GPS instantaneous speed of the i-th truck;
[0090] S1.2: Calculate the average deflection rate of the truck according to the instantaneous GPS speed of the truck, the GPS direction angle of the truck and the GPS timestamp of the truck. The calculation formula of the average deflection rate of the truck is:
[0091] ,
[0092] in, represents the average deflection rate of the truck corresponding to the i-th GPS trajectory data of the truck, represents the acceleration correction factor, represents the GPS timestamp of the i-1th truck, represents the GPS instantaneous speed of the i-1th truck, represents the direction angle correction coefficient, represents the GPS direction angle of the i+1th truck, represents the GPS direction angle of the i-th truck;
[0093] S1.3: Using the GPS trajectory data, the average speed of the truck and the average deflection rate of the truck as the truck trajectory features, a truck trajectory feature set is established, and the weight of each truck trajectory feature in the truck trajectory feature set is calculated by the hierarchical analysis method. According to the weight and the truck trajectory features, a truck motion matrix is obtained;
[0094] The method of calculating the weight of each truck trajectory feature in the truck trajectory feature set by the hierarchical analysis method includes establishing an analytical hierarchical structure model, constructing a judgment matrix, hierarchical sorting and information entropy determination. The method of establishing an analytical hierarchical structure model divides the decision-making goals, factors to be considered and decision-making objects into a target layer, a criterion layer and a scheme layer according to the correlation. The method of constructing a judgment matrix determines the proportional scale table of the truck motion data by the expert experience method, determines the appropriate scale, and obtains the judgment matrix by consistency test. The hierarchical sorting is used to perform normalization calculation on the judgment matrix, and the criterion layer in the hierarchical structure model is updated and sorted by calculating the maximum feature of the judgment matrix. The information entropy determination is used to calculate the weight of each truck trajectory feature by the entropy weight method.
[0095] S1.4: constructing a location recognition network, taking the truck motion matrix as an input parameter of the location recognition network, training the input parameter through the location recognition network, and outputting a location recognition result of the truck GPS trajectory data, wherein the location recognition result includes a driving point, a stop point, and a loading and unloading point;
[0096] The step of constructing a location identification network includes:
[0097] The input layer is used to perform cluster analysis on the truck motion matrix through a clustering algorithm, set the truck trajectory features belonging to the driving points in the truck motion matrix to zero according to the clustering results, obtain the stay feature matrix, and mark the corresponding GPS trajectory data as driving points;
[0098] The convolution pooling layer is used to extract the abstract features of the dwell feature matrix and reshape the abstract features into a one-dimensional long vector through the convolution kernel. The one-dimensional long vector is downsampled according to the maximum pooling to obtain the posture features of the GPS trajectory data.
[0099] The output layer is used to perform weighted splicing on the posture features, calculate the location confidence probability of the GPS trajectory data through the sigmoid activation function, mark the GPS trajectory data with a location confidence probability greater than the confidence threshold as a loading and unloading point, mark the GPS trajectory data with a location confidence probability less than or equal to the confidence threshold as a stay point, and output the location recognition result of the GPS trajectory data. The calculation formula of the location confidence probability is:
[0100] ,
[0101] in, represents the location confidence probability of GPS trajectory data, represents the sigmoid activation function, Represents the probability weight value of the change of the truck direction feature, represents the change probability weight value of the truck location feature, represents the change probability weight value of the truck stop feature, represents vector multiplication, represents the maximum pooling aggregation function, C(•) represents the convolution kernel operation, represents the abstract feature, b represents the bias constant;
[0102] See also Figure 2 , a structural diagram of a location recognition network according to an embodiment of the present invention, specifically, the location recognition network is provided with an input layer, a convolutional pooling layer and an output layer;
[0103] The input layer is used to perform cluster analysis on the truck motion matrix using DBSCAN and K-means clustering algorithms, and also includes obtaining truck standard driving data. Figure 3 , a flow chart of a clustering algorithm according to an embodiment of the present invention, performing preliminary clustering on standard driving data of trucks according to the DBSCAN clustering algorithm, obtaining driving core points of trucks, and determining the number of clustering center clusters according to the number of driving core points of trucks, using the driving core points of trucks after DBSCAN clustering as the initial center clusters of the K-means clustering algorithm, calculating the Euclidean distance from the truck motion matrix to each initial center cluster, performing clustering division according to the Euclidean distance, and calculating the clustering average error of the initial center cluster, updating the center cluster according to the clustering average error, defining a convergence condition, judging whether the updated center cluster meets the convergence condition, if not meeting the convergence condition, continuing to update the center cluster, if meeting the convergence condition, outputting the clustering result, setting the truck trajectory features belonging to the driving point in the truck motion matrix to zero according to the clustering result, obtaining the stay feature matrix, and marking the corresponding GPS trajectory data as the driving point, wherein the clustering result includes the driving point and the original location;
[0104] The convolution pooling layer is used to extract the abstract features of the dwell feature matrix through bilinear difference, and reshape the abstract features into a one-dimensional long vector through the convolution kernel. In this process, the convolution pooling layer includes a convolution kernel of 3x3 size, a pooling kernel of 2x2 size and a RELU activation function. The convolution pooling layer first performs a convolution operation on the dwell feature matrix, and uses multiple convolution kernels to scan the matrix. Each convolution kernel can extract a specific feature. These features are combined together to form a feature map. The one-dimensional long vector is downsampled according to the maximum pooling operation to obtain the posture features of the GPS trajectory data, and the feature map is subjected to the maximum pooling operation. Maximum pooling is a downsampling method that can reduce the dimension of the feature map while retaining the most important feature information. In the maximum pooling process, the convolution pooling layer divides the feature map into multiple small areas, and selects the largest eigenvalue in each area as the representative of the area. In this way, the size of the feature map will be reduced, but the key feature information is still retained. Through convolution and maximum pooling operations, the convolution pooling layer can effectively extract the abstract features in the stay feature matrix and reshape it into a one-dimensional long vector through the RELU activation function. This one-dimensional long vector contains the posture features of the GPS trajectory data, which can provide important information for subsequent location recognition. The posture features include truck orientation features, truck location features, and truck stay features;
[0105] The output layer is used to perform weighted splicing on the posture features, calculate the location confidence probability of the GPS trajectory data through the sigmoid activation function, mark the GPS trajectory data with a location confidence probability greater than the confidence threshold as a loading and unloading point, mark the GPS trajectory data with a location confidence probability less than or equal to the confidence threshold as a stay point, and output the location recognition result of the GPS trajectory data. Specifically, the output layer plays the role of the final decision in the location recognition network. It is responsible for further integration and analysis of the posture features extracted and processed by the convolution pooling layer to determine the specific category of the GPS trajectory data point, that is, whether it is a loading and unloading point or a stay point;
[0106] See also Figure 4 , a flow chart of the loading and unloading identification algorithm according to an embodiment of the present invention, wherein the specific steps of the loading and unloading identification strategy are as follows:
[0107] S2.1: Shooting a video sequence of a truck body through a camera device, obtaining inter-frame images of the truck body frame by frame, and performing differential denoising on the inter-frame images of the truck body;
[0108] See also Figure 5 , a flowchart of differential denoising of box-frame images according to an embodiment of the present invention, wherein the specific steps of performing differential denoising on the box-frame images are as follows:
[0109] S2.1.1: extract feature information from the box inter-frame images of the current frame and the next frame according to Gaussian difference filtering, wherein the Gaussian difference filtering includes Gaussian filter parameters 1, Gaussian filter parameters 2, Gaussian filter parameters 3 and Gaussian filter parameters 4, and obtain Gaussian filter image 1, Gaussian filter image 2, Gaussian filter image 3 and Gaussian filter image 4;
[0110] S2.1.2: performing differential processing on the four Gaussian filter images to extract four layers of two-dimensional Gaussian difference filter box frame images, wherein the four layers of two-dimensional Gaussian difference filter box frame images include a first layer of Gaussian difference images, a second layer of Gaussian difference images, a third layer of Gaussian difference images and a fourth layer of Gaussian difference images;
[0111] S2.1.3: Segment the four-layer two-dimensional Gaussian difference filtered box frame image into four directions, including the 0° direction, the 45° direction, the 90° direction and the 135° direction, filter out high-frequency noise from the segmented four-layer two-dimensional Gaussian difference filtered box frame image by filtering, fuse the four-layer two-dimensional Gaussian difference filtered box frame image after denoising in four directions, and obtain the four-layer two-dimensional Gaussian difference filtered box frame image after high-frequency noise removal;
[0112] S2.1.4: Fusing the two-dimensional Gaussian difference filtered box inter-frame images after denoising at each layer to obtain a box denoised inter-frame image;
[0113] Background subtraction is performed on the box-frame inter-frame image after differential denoising. Background subtraction is a technology used to separate foreground images from image sequences. Moving objects are used as foregrounds and stationary objects are used as backgrounds. The background subtraction method of the mixed Gaussian model can effectively distinguish foreground objects and backgrounds, thereby improving the accuracy of foreground separation. The contour features of the box-frame inter-frame image are calculated through morphological opening operations. Morphological operations are used to process the shape and structure of images. Common operations include corrosion and expansion, which effectively remove small noise points in the image and smooth the boundaries of foreground objects to make the contour features clearer. The calculation formula for the contour features of the box-frame inter-frame image is:
[0114] ,
[0115] in, represents the contour features of the box-frame image, j represents a single Gaussian distribution of the box-frame image in the mixed Gaussian model, M represents the total number of Gaussian distributions of the box-frame image in the mixed Gaussian model, represents the weight value of the j-th Gaussian distribution, represents the probability density function of Gaussian distribution, X represents the box frame image, represents the mean of the Gaussian distribution of the images between the boxes, represents the Gaussian distribution variance of the box frame image, Represents the pixel mean of the box frame image, represents the corrosion operation, represents the expansion operation;
[0116] The background subtraction of the box frame image after differential denoising includes initializing the model, updating the model, foreground separation and background subtraction, the initialization model includes random initialization parameters, the parameters include the number, mean, variance and weight of Gaussian distribution, and the model update includes probability density function and update parameters;
[0117] S2.2: Describe the contour features through the elliptical Fourier descriptor, extract the local feature map, perform regional cutting on the local feature map, and obtain the regional temporal features and regional spatial features;
[0118] Specifically, by decomposing the contour features into elliptical Fourier coefficients, the local and overall features of the shape can be effectively captured. Different shape features can be represented by a set of digital descriptors, which is convenient for subsequent feature extraction and comparison. Next, based on the extracted elliptical Fourier descriptors, the corresponding local feature map is generated. The local feature map is an image that represents the shape features in the image space, which can intuitively reflect the contour characteristics of different parts. By analyzing the local feature map, more meaningful shape information can be further refined and extracted. Then, the generated local feature map is subjected to regional cutting. The purpose of regional cutting is to divide the local feature map into multiple small regions, each region containing pixels with similar features. In this way, the features of each local region can be extracted more accurately, and it is convenient for subsequent Feature fusion and analysis: The specific region cutting method can use edge detection technology. Finally, the temporal features and spatial features of each region are extracted respectively. The regional temporal features refer to the changes of a specific region in the time dimension, including the trend of the shape, position and size of the region over time. The regional spatial features refer to the shape, position and boundary change characteristics of a specific region in the spatial dimension. The regional temporal features are calculated by the temporal feature extraction function, and the regional image is reduced in dimension according to the time gradient. The regional temporal features are extracted by updating the learning rate and convolution operations. The regional spatial features are calculated by the attention mechanism, and the attention mechanism includes compression operation, excitation operation and calibration operation. By extracting regional temporal features and regional spatial features, the regional temporal features can reflect the changes of objects between different time frames. For example, the changes of the opening and closing actions of the hatch on the time axis can be captured and identified by temporal features, and spatial features can describe the position, shape and boundary of the object in the image. Through spatial features, hatches, cargo and other related objects can be accurately located. Spatial features help distinguish different objects and regions, such as distinguishing hatch areas from cargo areas.
[0119] S2.3: The outer product of two feature vectors is calculated by bilinear pooling, and then a Fourier transform operation is performed to obtain the fused bilinear regional features. The regional time features and regional spatial features are fused. The fusion of regional time features and regional spatial features can provide more comprehensive and rich information, so that the system can better understand and identify the loading and unloading status. By fusing features, the dynamic changes and static features of the object can be combined to improve the accuracy and robustness of recognition. The bilinear regional features of all regions are integrated through the pooling function to obtain the global bilinear feature descriptor. The calculation formula of the global bilinear feature descriptor is:
[0120] ,
[0121] in, represents the global bilinear feature descriptor, a represents the cut region of a single local feature map, A represents the total number of cut regions of the local feature map, b{·} represents the pooling function, represents the time feature extraction function, represents the local feature map, represents the fusion operation, Ex[•] represents the excitation function in the attention mechanism, W represents the horizontal length of the half-body feature image, H represents the vertical length of the local feature map, n represents the unit horizontal length of the local feature map, m represents the unit vertical length of the local feature map, Represents the convolution kernel corresponding to the cut area;
[0122] S2.4: Activate the global bilinear feature descriptor through the Relu function, obtain the door area feature, detect the door area feature through the detection frame, calculate the door state value, compare the door state value with the opening threshold, if it is greater than or equal to the opening threshold, mark the box frame image of the frame as 1, if it is less than the opening threshold, mark the box frame image of the frame as 0, wherein the activation includes plane dimension activation mapping and stereo dimension activation mapping;
[0123] The door region feature is a feature related to the state of the truck door extracted from the global feature, which is used to reflect the opening and closing state and position information of the door. The activated global feature is mapped to the door region to obtain the door region feature. The detection frame is a bounding box used to locate and identify a specific target area in the image. The opening and closing state of the door is determined by the detection frame, so as to calculate the door state value. The calculation formula of the door state value is:
[0124] ,
[0125] in, represents the door state value, k represents a single detection frame, N represents the total number of detection frames, O represents the plane dimension of the door area feature, o represents the single plane coordinate of the door area feature, represents the door closure discriminant function, The horizontal coordinate of the plane representing the characteristics of the hatch area, The average value of the horizontal coordinate of the plane representing the characteristics of the hatch area, The plane ordinate representing the characteristics of the hatch area, represents the average plane ordinate of the hatch area feature, V represents the three-dimensional dimension of the hatch area feature, p represents the single three-dimensional coordinate of the hatch area feature, The vertical coordinates of the hatch area feature, The average vertical coordinate of the plane representing the characteristics of the hatch area;
[0126] S2.5: traverse all the box frame images, obtain the mark values of all the box frame images, if the box frame images with a mark value of 1 have a proportion greater than the loading and unloading threshold, it means that the truck is in the loading and unloading state;
[0127] The anomaly detection strategy includes a location anomaly detection sub-strategy and a handling anomaly detection sub-strategy. The location anomaly detection sub-strategy is used to identify whether the truck is located in the predetermined loading and unloading area through fence detection technology. The handling anomaly detection sub-strategy is used to obtain the sound signal emitted by the abnormal sound of the cargo in the compartment during the cargo loading and unloading process through the acoustic sensor, process the sound signal, and identify whether there is an abnormal situation during the cargo loading and unloading process, wherein the abnormal situation includes cargo vibration, cargo collision and cargo scratch;
[0128] The specific steps of the location anomaly detection sub-strategy are as follows:
[0129] S3.1: Obtain the geographical boundary of the loading and unloading area, and convert the geographical boundary into a fence model through a geographic information system, wherein the fence model includes a circular model and a polygonal model;
[0130] S3.2: Matching the truck GPS longitude and the truck GPS latitude of the truck loading and unloading location with the fence model;
[0131] S3.3: judging whether the truck is located in the predetermined loading and unloading area according to the matching result; if the truck is not located in the loading and unloading area, it indicates that there is an abnormal location event;
[0132] The matching includes polygon fence matching and circular fence matching. The polygon fence matching is performed by drawing a ray from the truck positioning point to any direction and counting the number of intersections between the ray and the polygon boundary. If the number of intersections is an odd number, the point is inside the polygon; if the number of intersections is an even number, the point is outside the polygon. The circular fence matching is performed by calculating the distance between the truck positioning point and the center of the circle to determine whether the point is within the circle.
[0133] The specific steps of the handling anomaly detection sub-strategy are as follows:
[0134] S3.4: Acquire the sound signal of abnormal noise of cargo in the carriage during cargo loading and unloading through the acoustic sensor, and pre-process the sound signal. The pre-processing includes normalization, pre-emphasis and framing. The normalization is used to reduce the distortion caused by the long distance between the acoustic sensor and the special equipment. The pre-emphasis is used to weaken the high-frequency radiation, make the sound signal tend to be flat, and enhance the energy of the high-frequency part of the sound signal. The framing is used to cut the sound signal according to the window function to generate a plurality of permutation and combination sound signals.
[0135] S3.5: extracting features from the preprocessed sound signal to obtain a sound feature signal, performing short-time Fourier transform on the sound feature signal, and calculating a time domain spectrum feature of the sound feature signal;
[0136] S3.6: Perform logarithmic operation and dimension reconstruction on the time domain spectrum features to obtain the input parameters of the cargo status recognition network, perform convolution on the input parameters, and calculate the input features of the input parameters. The calculation formula of the input features is:
[0137] ,
[0138] Among them, F represents the input characteristics of the input parameters, Represents a 3x3 convolution kernel convolution operation, represents the historical input parameters of the training phase, represents the standard deviation of historical input parameters, represents the variance of historical input parameters, represents the error in the training phase, Represents a vector dot product operation, Represents input parameters;
[0139] S3.7: Based on the input features of the input parameters, the cargo status recognition network is used to recognize the status of the special equipment, the cargo status category parameters are calculated through the attention mechanism, and the abnormal situation classification results are output based on the cargo status category parameters;
[0140] The cargo status recognition network includes an input layer, a hidden layer and an output layer. The input layer is used to input time domain spectrum features, perform logarithmic operations and size reconstruction on the time domain spectrum features to obtain input parameters. The hidden layer is used to train the input parameters. The output layer is used to output classification results according to cargo status category parameters.
[0141] The input layer includes logarithmic operation, size reconstruction and convolution;
[0142] The hidden layer includes a processing module AM and a processing module BM, the processing module AM includes a block A1 and a block A2, the blocks A1 and A2 include two convolutional layers, a normalization layer and an average pooling layer, the processing module BM includes a block B1 and a block B2, the block B1 includes a channel attention module and a spatial attention module, and the block B2 includes three convolutional layers and a maximum pooling layer;
[0143] The output layer includes error reconstruction and activation function. The error reconstruction is used to perform reverse error analysis on the special equipment status category parameters to determine whether the error threshold is met. If not, retraining is performed. If so, the classification category is output according to the activation function.
[0144] Example 2
[0145] See also Figure 6 , the present invention provides an embodiment: a truck loading and unloading anomaly detection system based on image processing technology, the system includes a GPS trajectory data processing module, a loading and unloading identification module, an anomaly detection module and an alarm mechanism module;
[0146] The GPS track data processing module is used to acquire and pre-process the truck GPS data in real time, calculate the track features, and identify the truck position status;
[0147] The loading and unloading identification module is used to identify whether the truck is in a loading and unloading state through camera equipment and image processing technology;
[0148] The anomaly detection module is used to determine abnormal behavior during the loading and unloading process of the truck based on the location and handling anomaly detection strategy;
[0149] The alarm mechanism module is used to trigger an alarm and notify relevant personnel to take measures when an abnormal loading and unloading event is detected;
[0150] The GPS trajectory data processing module includes:
[0151] GPS data acquisition unit: responsible for obtaining the real-time GPS track data of the truck, including longitude, latitude, instantaneous speed, direction angle and timestamp;
[0152] Data preprocessing unit: preprocess the collected GPS data, including data cleaning, noise reduction and smoothing, to obtain smooth data;
[0153] Trajectory feature calculation unit: calculates the average speed and average deflection rate of the truck based on the preprocessed GPS data, and establishes a truck trajectory feature set;
[0154] Motion matrix generation unit: calculates the weight of trajectory features through the hierarchical analysis method and generates the truck motion matrix;
[0155] Location recognition unit: calculates the location confidence probability of GPS trajectory data through the sigmoid activation function and outputs the location recognition result;
[0156] The loading and unloading identification module comprises:
[0157] Video acquisition unit: uses a camera to capture a video sequence of the truck body and obtains inter-frame images of the body frame by frame;
[0158] Image processing unit: perform differential denoising, background subtraction and morphological opening operations on the inter-frame images of the box to extract contour features;
[0159] Feature extraction unit: Process contour features through elliptical Fourier descriptors, extract local feature maps, perform regional cutting, and obtain regional temporal and spatial features;
[0160] Feature fusion unit: fuses regional temporal features and spatial features to obtain bilinear regional features, and obtains global bilinear feature descriptors through pooling functions;
[0161] Door status detection unit: activates the global bilinear feature descriptor through the ReLU function, detects the door area features, calculates the door status value, and marks the box inter-frame image;
[0162] The anomaly detection module comprises:
[0163] Location anomaly detection unit: Uses fence detection technology to determine whether the truck is within the designated loading and unloading area and whether there is a location anomaly event;
[0164] Abnormal handling detection unit: The acoustic sensor is used to obtain the sound signals emitted by abnormal noises in the carriage during loading and unloading of goods, process the sound signals, and output the abnormal situation classification results.
[0165] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A truck loading and unloading anomaly detection method based on image processing technology, characterized in that: The following steps are involved: S1: Obtain GPS track data of the truck, and determine whether the truck is at a loading and unloading point by processing the GPS track data; S2: If the truck is at the loading and unloading point, the loading and unloading identification strategy is used to determine whether the truck is in the loading and unloading state; S3: If the truck is in the loading and unloading state, determine whether there is any abnormality in the loading and unloading of the truck through the anomaly detection strategy; S4: When abnormal loading and unloading events are detected, an alarm mechanism is triggered; The specific steps of the loading and unloading identification strategy are as follows: S2.1: Shoot a video sequence of the truck body through a camera device, obtain inter-frame images of the box body frame by frame, perform background subtraction on the inter-frame images of the box body, and calculate the contour features of the inter-frame images of the box body through morphological opening operations; S2.2: Process the contour features through the elliptical Fourier descriptor, extract the local feature map, perform regional cutting on the local feature map, and obtain the regional temporal features and regional spatial features; S2.3: Fuse the regional temporal features and the regional spatial features to obtain the bilinear regional features, and use the pooling function to integrate the bilinear regional features of all regions to obtain the global bilinear feature descriptor; S2.4: Activate the global bilinear feature descriptor through the Relu function, obtain the door area features, detect the door area features through the detection frame, calculate the door state value, compare the door state value with the opening threshold, if it is greater than or equal to the opening threshold, mark the box frame image of the corresponding frame as 1, if it is less than the opening threshold, mark the box frame image of the corresponding frame as 0. S2.5: traverse all the box frame images and obtain the label values of the box frame images. If the proportion of the box frame images with a label value of 1 is greater than the loading and unloading threshold, it means that the truck is in the loading and unloading state; The calculation formula of the door status value is: , in, represents the door state value, k represents a single detection frame, N represents the total number of detection frames, O represents the plane dimension of the door area feature, o represents the single plane coordinate of the door area feature, represents the door closure discriminant function, The horizontal coordinate of the plane representing the characteristics of the hatch area, The average value of the horizontal coordinate of the plane representing the characteristics of the hatch area, The plane ordinate representing the characteristics of the hatch area, represents the average plane ordinate of the hatch area feature, V represents the three-dimensional dimension of the hatch area feature, p represents the single three-dimensional coordinate of the hatch area feature, The vertical coordinates of the hatch area feature, The average vertical coordinate of the plane representing the characteristics of the hatch area.
2. According to the method for detecting abnormalities in truck loading and unloading based on image processing technology in claim 1, it is characterized in that: The GPS track data includes the truck GPS longitude, the truck GPS latitude, the truck GPS instantaneous speed, the truck GPS direction angle and the truck GPS timestamp. The specific steps of processing the GPS track data are as follows: S1.1: Calculate the average speed of the truck according to the GPS longitude of the truck, the GPS latitude of the truck, the GPS instantaneous speed of the truck and the GPS timestamp of the truck; S1.2: Calculate the average deflection rate of the truck according to the truck GPS instantaneous speed, the truck GPS direction angle and the truck GPS timestamp; S1.3: establishing a truck trajectory feature set according to the GPS trajectory data, the average speed of the truck and the average deflection rate of the truck, calculating the weight of the truck trajectory feature by using the hierarchical analysis method, and obtaining the truck motion matrix according to the weight and the truck trajectory feature; S1.4: Construct a location recognition network, use the truck motion matrix as the input parameter of the location recognition network, train the input parameter through the location recognition network, and output the location recognition result of the truck GPS trajectory data, wherein the location recognition result includes driving points, stop points and loading and unloading points.
3. The method for detecting abnormalities in truck loading and unloading based on image processing technology according to claim 2 is characterized in that: Build a place recognition network, including: The input layer is used to perform cluster analysis on the truck motion matrix through a clustering algorithm, set the truck trajectory features belonging to the driving points in the truck motion matrix to zero according to the clustering results, obtain the stay feature matrix, and mark the corresponding GPS trajectory data as driving points; The convolution pooling layer is used to extract the abstract features of the dwell feature matrix and reshape the abstract features into a one-dimensional long vector through the convolution kernel. The one-dimensional long vector is downsampled according to the maximum pooling to obtain the posture features of the GPS trajectory data. The output layer is used to perform weighted splicing on the posture features, calculate the location confidence probability of the GPS trajectory data through the sigmoid activation function, mark the GPS trajectory data with a location confidence probability greater than the confidence threshold as a loading and unloading point, mark the GPS trajectory data with a location confidence probability less than or equal to the confidence threshold as a stay point, and output the location recognition result of the GPS trajectory data. The calculation formula of the location confidence probability is: , in, represents the location confidence probability of GPS trajectory data, represents the sigmoid activation function, Represents the probability weight value of the change of the truck direction feature, represents the change probability weight value of the truck location feature, represents the change probability weight value of the truck stop feature, represents vector multiplication, represents the maximum pooling aggregation function, C(•) represents the convolution kernel operation, represents the abstract feature, and b represents the bias constant.
4. The method for detecting abnormalities in truck loading and unloading based on image processing technology according to claim 1 is characterized in that: The anomaly detection strategy includes a location anomaly detection sub-strategy and a handling anomaly detection sub-strategy. The location anomaly detection sub-strategy is used to identify whether the truck is located in a predetermined loading and unloading area through fence detection technology. The handling anomaly detection sub-strategy is used to obtain sound signals emitted by abnormal noises of goods in the carriage during the loading and unloading process through acoustic sensors, process the sound signals, and identify whether there are abnormal conditions during the loading and unloading process, wherein the abnormal conditions include cargo vibration, cargo collision, and cargo scratches.
5. The method for detecting abnormalities in truck loading and unloading based on image processing technology according to claim 4 is characterized in that: The specific steps of the location anomaly detection sub-strategy are as follows: S3.1: Obtain the geographical boundary of the loading and unloading area, and convert the geographical boundary into a fence model through a geographic information system, wherein the fence model includes a circular model and a polygonal model; S3.2: Matching the truck GPS longitude and the truck GPS latitude of the truck loading and unloading location with the fence model; S3.3: Determine whether the truck is located in the predetermined loading and unloading area based on the matching result. If the truck is not in the loading and unloading area, it indicates that an abnormal location event exists.
6. The method for detecting abnormalities in truck loading and unloading based on image processing technology according to claim 5 is characterized in that: The specific steps of the handling anomaly detection sub-strategy are as follows: S3.4: Acquire sound signals generated by abnormal noises of cargo in the carriage during cargo loading and unloading through an acoustic sensor, and pre-process the sound signals; S3.5: extracting features from the preprocessed sound signal to obtain a sound feature signal, performing short-time Fourier transform on the sound feature signal, and calculating a time domain spectrum feature of the sound feature signal; S3.6: Perform logarithmic operation and dimension reconstruction on the time domain spectrum features to obtain input parameters of the cargo status recognition network, perform convolution on the input parameters, and calculate input features of the input parameters; S3.7: According to the input features of the input parameters, the cargo status is identified using the cargo status recognition network, the cargo status category parameters are calculated through the attention mechanism, and the abnormal situation classification results are output according to the cargo status category parameters.
7. A truck loading and unloading anomaly detection system based on image processing technology, used to implement a truck loading and unloading anomaly detection method based on image processing technology as claimed in any one of claims 1 to 6, characterized in that: The system includes a GPS trajectory data processing module, a loading and unloading identification module, an anomaly detection module and an alarm mechanism module; The GPS track data processing module is used to acquire and pre-process the truck GPS data in real time, calculate the track features, and identify the truck position status; The loading and unloading identification module is used to identify whether the truck is in a loading and unloading state through camera equipment and image processing technology; The anomaly detection module is used to determine abnormal behavior during the loading and unloading process of the truck based on the location and handling anomaly detection strategy; The alarm mechanism module is used to trigger an alarm and notify relevant personnel to take measures when an abnormal loading and unloading event is detected.
