A method for recognizing loading and unloading pallet status patterns

Abstract

This invention discloses a method for recognizing loading and unloading pallet status patterns. First, a pallet status dataset is constructed. Based on camera positions, the pallet status is divided into five categories and labeled. Next, a ResNet50 network model is constructed, adding ReLU, dropout, and logSoftmax regression layers to adapt to the dataset for sample training. After model convergence, a pallet status pattern recognition system is built. The video stream is input into the recognition system using the RTSP protocol to complete the initial recognition of the pallet status. Median filtering is used to denoise and improve the recognition accuracy, ultimately achieving an accuracy of 99%. This solves the problem of mismatch between work time and vehicle number caused by misjudgments in traditional algorithms. The license plate area is located using the YOLOv5 algorithm, and the target text is extracted using a text detection algorithm, combined with a convolutional recurrent neural network to recognize the license plate information. Finally, the recorded pallet vehicle docking and exit times and work duration data provide a basis for pallet management. This invention realizes the status recognition of each pallet, records the pallet operation status, and can effectively provide feedback on abnormal operations, facilitating management and review.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and specifically relates to a method for recognizing loading and unloading pallet status patterns. Background Technology

[0002] The postal district center bureau is a postal production unit responsible for sorting, sealing, transferring, and shipping mail along land routes. It handles the sorting and transportation of mail entering and leaving the district. Improving the operational efficiency of the center bureau and monitoring abnormal production conditions have been major research directions for cost reduction and efficiency improvement in the postal service in recent years. Postal district center bureaus experience frequent vehicle traffic and numerous loading and unloading points. Relying solely on random video checks from the dispatch center or manual patrols at the processing center has limitations and cannot achieve comprehensive, round-the-clock coverage. Therefore, automating the assessment of loading point operations and reducing manual patrols to achieve 24-hour comprehensive coverage is of profound significance.

[0003] Image state recognition techniques based on feature extraction or image classification based on deep learning models, when applied, suffer from limitations such as changes in the barricade environment, lighting conditions, and the camera's field of view, resulting in classification accuracy that cannot reach 100%. Their limitations are as follows:

[0004] (1) When the sampling interval is short and the data flow is large, misjudgment will cause a single operation within the operation time to be split into two or more operations or the operation time to be discontinuous, resulting in the inability to match the train number and operation time.

[0005] (2) During vehicle operation, the same vehicle is pushed out and then pushed back in for operation.

[0006] (3) When the loading and unloading platform is idle, vehicles pass by and vehicles temporarily stop at the platform in the picture.

[0007] (4) When the roller shutter door of the loading and unloading stack is pulled up, the camera's field of view is blocked and the status cannot be determined. Summary of the Invention

[0008] Purpose of the invention: To address the problems existing in the prior art, this invention provides a method for recognizing the status patterns of loading and unloading pallets. It can improve the neural network model by using median filtering based on the characteristics of continuous sequences, so that the accuracy of pallet status recognition reaches more than 99%, and can obtain the precise operation time of the work vehicle, as well as the time when the work vehicle approaches and exits the platform.

[0009] Technical Solution: To solve the above technical problems, the present invention provides a method for recognizing the status pattern of loading and unloading pallets, comprising the following steps:

[0010] Step 1: Set the video capture interval to 1 second / frame, capture video data, create a dataset from the captured video data, and preprocess the dataset;

[0011] Step 2: Divide the data into training set, test set, and cross-validation set, construct a ResNet50 pattern recognition network model, and train the constructed ResNet50 pattern recognition network model.

[0012] Step 3: Use the NVIDIA edge computing box as the model deployment container. Utilize the decoding and encoding chip of the NVIDIA edge computing box to realize video stream parsing through hardware decoding technology. At the same time, the core computing power of the NVIDIA edge computing box meets the system's computing power requirements.

[0013] Step 4: Using the RTSP protocol, edge computing boxes are used for decoding and encoding, and the video stream is transmitted in real time to the trained ResNet50 pattern recognition network model. The current time, pallet ID, and recognition result are recorded. The license plate areas of vehicles at the pallet and the positions of workers at the pallet are marked. A YOLOv5 algorithm model is built to locate the license plate areas in the images and count the number of workers. A text detection algorithm is built to locate the license plate areas, and a convolutional recurrent neural network is used to recognize the text sequence, extract the license plate information, and input it into the database.

[0014] Step 5: Improve the ResNet50 pattern recognition network model based on median filtering to correct the state of the stack.

[0015] Step 6: Based on the stability of the time series, construct a state machine filtering method to further correct the stack state after median filtering and denoising;

[0016] Step 7: Calculate the corrected results, record the arrival and departure times and durations of vehicles at the pallet, complete the pattern recognition of the pallet status, and generate a time-series-based work report.

[0017] Furthermore, the preprocessing steps for the dataset in step one are as follows:

[0018] The dataset is divided into two main categories based on the camera angle, orientation, and operating scenario: idle state and stacking gate operating state. The idle state includes the stacking gate idle state when the camera is low, the stacking gate idle state, and the stacking gate roller shutter state. The stacking gate operating state includes the stacking gate operating state and the stacking gate unloaded state.

[0019] Furthermore, the specific steps for constructing the ResNet50 pattern recognition network model in step two, and training the constructed ResNet50 pattern recognition network model, are as follows:

[0020] Step 2.1: Construction of ResNet50 Pattern Recognition Network Model

[0021] The initial ResNet50 network classifies 2048 categories, with an output dimension of 2048. The network structure is adjusted to accommodate the stacked state: a new Dropout layer and logSoftmax are added, and a fully connected layer with dimension 5 is added to the end of the original ResNet50 network structure. Assuming a total of k = 5 output nodes, the logSoftmax for the i-th node is:

[0022] L i =-logo(z i )

[0023]

[0024] Where L i Output for the i-th node;

[0025] Step 2.2: Training the ResNet50 pattern recognition network model

[0026] The dataset is input into the network for training. In the initial stage, the model is trained multiple times to quickly achieve residual convergence through transfer learning, resulting in weights with generalizability. Then, the model is tested using a test set.

[0027] Furthermore, the specific steps of improving the ResNet50 pattern recognition network model based on median filtering in step five are as follows:

[0028] Step 5.1: Let the sequence of the video stream be f1, f2, ..., f n Take the window length s, where s is an odd number; apply median filtering to the recognition performance of this video stream:

[0029] Y i =Med{f i-t ,...,f i ,...f i+t}

[0030] Where Med is the formula for calculating the median of a sequence. s numbers are extracted from the video recognition sequence, and i is the median index of the sequence. The sequence is sorted in descending order, and the index is taken as the median value as the output of the entire sequence.

[0031] Step 5.2: Model optimization based on median filtering algorithm:

[0032] The video feeds from each channel are transmitted in real time to the ResNet50 pattern recognition network model to complete pattern recognition and output a time-series recognition sequence of the stack status. The time-series recognition sequence is as follows: 0, stack idle state when the camera is low; 1, stack idle state; 2, stack roller shutter state; 3, stack working state; 4, stack unloaded state.

[0033] The sequence 0, 1, 2 is marked as 0 to indicate idle time, and 3, 4 is marked as 1 to indicate work. The final result is a square wave of the 0, 1 sequence.

[0034] A nonlinear estimation equation for the step frequency and the filter window length is constructed, and kernel density estimation is performed on both:

[0035]

[0036] Where x (1) Indicates the step frequency, x (2) Let η represent the filter window length and η be the smoothing parameter bandwidth. A window length selection model based on a flexible Bayesian classifier is constructed to establish a median filter with an adaptive window length.

[0037] Furthermore, the specific steps for constructing the window length selection model based on the flexible Bayesian classifier in step 5.2 are as follows:

[0038] Step 5.2.1: Segment the dataset according to the window length. Dataset features include step frequency and sequence length.

[0039] Step 5.2.2: Construct a generative model for each dataset using kernel density estimation, and then calculate the likelihood estimate P(x|y), where x represents the feature and y represents the window length;

[0040] Step 5.2.3: Find the bandwidth length for the optimal kernel density estimation by constructing a grid search and utilizing the model accuracy;

[0041] Step 5.2.4: Use this likelihood estimate to replace the Gaussian distribution model of the Naive Bayes classifier to construct a flexible Bayesian model.

[0042] Furthermore, the specific steps for constructing the state machine filtering method in step six to further correct the stack state after median filtering and denoising are as follows:

[0043] After median filtering for denoising, the loading and unloading dock is continuously denoised using the state consistency and continuity criteria. When a step occurs in the sequence, it indicates that the state has changed. For example, a step from 0 to 1 indicates that the dock state has changed from idle to operational, and a step from 1 to 0 indicates that the dock state has changed from operational to idle. Using this characteristic, a state machine filtering method based on continuous sequences is constructed.

[0044] For the filtered sequence, a moving window is set with a window length of 30 seconds per sampling interval. When a step point occurs during the moving window, the mean of that point is compared with the mean of the window length after that point. If they are the same, it means that the change in state is reasonable. If they are not the same, it means that the current step point is a misjudgment and needs to be corrected to the state before the step. The second filtering result is then stored in the database.

[0045] Furthermore, the specific steps for generating the time-series-based job report in step seven are as follows:

[0046] After two filtering steps, a smooth and stable time-series square wave graph is obtained, with the horizontal axis representing time and the vertical axis representing the state of the stack. Based on the time series, the actual working time and idle time of the stack can be obtained. When the sequence jumps from 0 to 1, it means that the time is the start time of the stack operation. When 1 jumps to 0, it means that the time is the end time of the stack operation. Conversely, the idle time of the stack can be obtained.

[0047] Based on the special circumstances of pallet loading and unloading operations, noise reduction measures are implemented using the principles of consistency and continuity of status. The specific measures are as follows:

[0048] Step 7.1: The same vehicle continuously enters and exits the loading and unloading pallet.

[0049] During loading and unloading operations at the stack opening, there are situations where vehicles leave the stack opening and then reverse back into the stack. According to the sequence output of image recognition, the result is two operation times and one stack opening idle time, which is prone to misjudgment.

[0050] The vehicle information for each operation is detected by license plate recognition and text recognition technology. If the operation information of vehicles before and after the operation is consistent, the two time periods are merged to obtain the actual operation time of that operation.

[0051] Step 7.2: Loss of view caused by pulling down the roller shutter door at the loading / unloading point.

[0052] First, record the vehicle operation information at the loading and unloading dock before the roller shutter door is pulled down. When the status of the roller shutter door at the loading and unloading dock changes, determine the current status. If the dock is idle, the time when the roller shutter door is pulled down is the time when the vehicle stops operating. If a vehicle is operating at the loading and unloading dock at this time and the license plate matches the record, the vehicle operation is determined to be in progress. If the license plate does not match, the time when the roller shutter door is pulled down is the time when the vehicle stops operating, and this time point is also determined to be the start time point of the current vehicle's operation.

[0053] Step 7.3: Non-target vehicles stopping and passing by

[0054] When the decoration work area is idle, non-target vehicles may pass through the work area or stop briefly. The median filtering algorithm removes the step signal caused by vehicles passing through briefly, and sets 15 minutes as the threshold for the operation time. Operation time below this threshold is considered noise signal.

[0055] Step 7.4: People counting, license plate localization, and text detection algorithms

[0056] A training set was constructed by marking the license plate areas of vehicles at the loading and unloading points and the locations of workers. A YOLO v5 model was then trained to achieve target detection capabilities based on license plate areas and worker locations. The number of workers currently on the job can be counted based on their location. A text detection algorithm based on segmentation (DB) and a text detection algorithm based on convolutional recurrent neural networks (CRNN) were constructed for license plate text detection in target areas.

[0057] Compared with the prior art, the advantages of the present invention are as follows:

[0058] This invention achieves high-precision identification of the status of the stack by collecting real-time video data streams of the stack and improving the ResNet50 network model with median filtering algorithm. It overcomes the insufficient accuracy and generalization ability of traditional image classification algorithms, removes step noise in the observation sequence, realizes 24-hour high-precision monitoring, and restores the real operation of the stack.

[0059] This invention addresses the complexity and diversity of the stacking environment by utilizing continuity criteria and license plate recognition technology to enhance the model's anti-interference capabilities and obtain accurate data such as vehicle entry and exit times, operation time, and number of workers.

[0060] This invention integrates data such as vehicle docking time, departure time, unloading time, and number of workers identified by the system with the postal district center bureau's pallet operation information system, and uses this data for pallet operation monitoring. On one hand, it reduces the tedious manual record-keeping and information gaps of traditional methods, ensuring objectivity; on the other hand, it preserves evidence of pallet loading and unloading operations, providing objective information such as operation time and number of workers to support the analysis of pallet operation efficiency, and providing a basis for improving efficiency and analyzing anomalies. Attached Figure Description

[0061] Figure 1 This is a flowchart of the present invention;

[0062] Figure 2 These are the five stacking states trained during the specific implementation embodiment;

[0063] Figure 3 This refers to the newly added network layer designed in the specific embodiment;

[0064] Figure 4This is the confusion matrix of the test set in a specific embodiment;

[0065] Figure 5 The ROC curve is shown in the specific embodiment.

[0066] Figure 6 This is a real-time status recognition curve for the pallet opening based on ResNet50 in a specific embodiment;

[0067] Figure 7 The above are the state recognition accuracy curves after filtering based on different window values ​​in a specific embodiment.

[0068] Figure 8 This is a distribution diagram of step frequency and filter window length based on kernel density estimation in a specific embodiment;

[0069] Figure 9 This is a specific embodiment of the kernel density estimation bandwidth optimization graph based on the grid search method.

[0070] Figure 10 The real-time state monitoring curve after denoising using the adaptive filtering algorithm based on flexible Bayes in a specific embodiment;

[0071] Figure 11 The image shows a real-time state monitoring curve after state machine filtering based on a continuous sequence, as described in a specific embodiment. Detailed Implementation

[0072] The present invention will be further explained below with reference to the accompanying drawings and specific embodiments.

[0073] like Figure 1 As shown, the present invention provides a method for recognizing the status pattern of loading and unloading pallets, the specific steps of which are as follows:

[0074] 1. Data Sample Collection and Processing

[0075] The collection and processing of sample data is a fundamental step in the entire invention. In this case, the sample data is real video data of the pallet opening. The RTSP stream video data is decoded and analyzed. Due to the differences in the angle, orientation, and operating scenarios of different pallet opening cameras, the video data needs to be classified. In this case, it is divided into five categories: 1. Pallet opening idle state when the camera is low; 2. Pallet opening idle state; 3. Pallet opening roller shutter state; 4. Pallet opening operating state; 5. Pallet opening unloaded state. See the appendix for details of the states. Figure 2 Statuses 1-3 are idle states, while statuses 4-5 are stacking operation states.

[0076] During peak loading and unloading operations at the stacking point, vehicles enter and exit the stacking point in a short time, typically within 3-5 seconds. The sampling interval at the stacking point is set to be no less than 3 seconds per sampling, and at least 28,800 samples are collected in 24 hours to ensure the completeness of the samples.

[0077] 2. ResNet50 network model construction and training

[0078] (1) Construction of ResNet50 network image classification model

[0079] The initial ResNet50 network classifies 2048 categories, resulting in an output dimension of 2048. This invention adjusts the network output to accommodate five states of the stack, adding a new Dropout layer and logSoftmax, resulting in an output dimension of 5. The new network layers are added to the end of the original ResNet50 network structure; see attached diagram for details. Figure 3 Meanwhile, logSoftmax effectively solves the overflow problem, speeds up computation, and improves data stability. Assuming a total of k = 5 output nodes, the logSoftmax for the i-th node is:

[0080] L i =-logo(z i )

[0081]

[0082] Where L i Output the i-th node.

[0083] (2) Training the ResNet50 network model

[0084] The dataset is input into the network for training. In the initial stage, the model has been trained multiple times. Through transfer learning, residual convergence can be achieved quickly, resulting in weights with generalization capabilities. The test confusion matrix using the test set is attached. Figure 4 The ROC curve is attached. Figure 5 The initial model achieved an accuracy of 92%, a recall of 98%, and an F1-Score of 0.94. Therefore, the model has converged and possesses a certain degree of reliability.

[0085] 3. Model Deployment Scheme Based on Vida Edge Computing Box

[0086] The hardware utilizes an NVIDIA edge computing box as the deployment container, providing a low-cost, low-power solution that addresses the issues of expensive servers and the need for independent power supplies. Video stream parsing is achieved using the device's decoding and encoding chip through hardware decoding technology. Simultaneously, the device's core computing power meets the computational requirements of the system's intelligent algorithms.

[0087] 4. Using the RTSP protocol and edge computing boxes for decoding and encoding, the video stream is transmitted in real time to the trained ResNet50 model, recording the current time, pallet ID, and recognition results. A YOLO v5 algorithm model is constructed to locate the license plate region in the image and count the number of workers. A text detection algorithm is constructed to locate the license plate region, and a convolutional recurrent neural network is used to recognize the text sequence, extract the license plate information, and input it into the database.

[0088] Among them, the license plate localization and personnel localization algorithms based on YOLO v5 are:

[0089] Traditional license plate recognition cameras are well-positioned and produce high-quality images. They can locate the license plate area through contour extraction and edge sharpening, and then classify the segmented pixels to obtain the license plate information. However, the loading and unloading area in this case is complex. The camera is far from the vehicle's license plate, and factors such as lighting and pedestrian movement make license plate localization and text recognition difficult. Using image algorithms such as contour extraction and edge sharpening to locate and recognize license plates for vehicles entering and exiting the loading area is insufficient for extracting the target license plate area and segmenting the license plate text.

[0090] By marking the license plate locations of postal vehicle pallet trucks and constructing a training set based on the pallet loading and unloading personnel, a YOLOv5-based target detection model for license plate localization of pallet trucks was built. This method can achieve fast and accurate localization of vehicle license plates and personnel. The number of people currently working can be calculated by detecting the personnel area. License plate localization is achieved through target detection. Due to the small pixel size of the 1920*1080 camera image, traditional optical text recognition methods are difficult to use for license plate recognition. The area is enlarged to 200*128 pixels, and a text detection algorithm based on segmentation (DB) and convolutional recurrent neural network (CRNN) is constructed for license plate text detection in the target area.

[0091] 5. An improved ResNet50 pattern recognition network model based on median filtering

[0092] (1) Median filtering algorithm

[0093] The status of the stack, whether active or idle, is a continuous time series. A continuous time series does not transition from one state to another, and there are no abrupt changes. Based on these characteristics, median filtering can be used to denoise the recognition result sequence.

[0094] Median filtering: Let a sequence of video streams be f1, f2, ..., f n Take the window length s (s must be an odd number). Apply median filtering to the recognition results of this video stream:

[0095] Y i =Med{fi-t ,...,f i ,...f i+t}

[0096] Where Med is the formula for calculating the median of a sequence. s numbers are extracted from the video recognition sequence, and i is the median index of the sequence. The sequence is sorted in descending order, and the index is taken as the median value as the output of the entire sequence.

[0097] (2) Model optimization based on median filtering algorithm

[0098] The video feeds from each channel of the NVR device are transmitted in real time to the recognition system. Pattern recognition is then performed, and a time series of the stack status is output. The sequence 0, 1, and 2 are labeled as 0 (idle) and 3 and 4 as 1 (working). The final result is a square wave sequence of 0 and 1. See the attached output. Figure 6 .

[0099] The median filtering window size will affect the denoising effect. In this study, median filtering was applied to the experimental sequences, with the window size set from 1 to 69, and an interval of 2. Accuracy curves after filtering with different window sizes are shown in the appendix. Figure 7 Different window values ​​are required for different sequence lengths and specific sequences to achieve optimal denoising results. The window value setting is related to the filter length and the step frequency. In this case, the stacking scenario should be either continuous operation or continuous idle state, so a nonlinear estimation equation for the step frequency and filter window length needs to be constructed. Kernel density estimation is then performed on both:

[0100]

[0101] Where x (1) Indicates the step frequency, x (2) This represents the filter window length, and η is the smoothing parameter bandwidth. See the appendix for the sum-density curves of both. Figure 8 From the appendix Figure 8 It can be seen that the distributions of both are quite complex, and their joint distribution is relatively simple. The window length is odd and has a certain correlation with the step frequency and sequence length. Therefore, a window length selection model based on a flexible Bayesian classifier can be constructed to establish a median filter with an adaptive window length.

[0102] The steps are as follows:

[0103] (1) The dataset is segmented according to the window length. The features of the dataset include step frequency and sequence length.

[0104] (2) Construct a generative model for each dataset using kernel density estimation, and then calculate the likelihood estimate P(x|y), where x represents the feature and y represents the window length;

[0105] (3) The bandwidth length for finding the optimal kernel density estimate is obtained by constructing a grid search and utilizing the model accuracy (see Appendix). Figure 9 ;

[0106] (4) Use this distribution to replace the Gaussian distribution model of the Naive Bayes classifier to construct a flexible Bayes model.

[0107] The real-time monitoring curve of the post office barricade status after denoising using the flexible Bayesian adaptive filtering algorithm is attached. Figure 10 The unfiltered curve is shown in the appendix. Figure 6 In practical applications, if identification errors occur during the operation process, it will cause interruptions when calculating continuous operations, resulting in multiple operations and making it impossible to accurately calculate the vehicle approach time, departure time, and operation time. Median filtering can remove errors caused by misjudgments, thereby improving the accuracy of pallet status monitoring.

[0108] 6. State machine filtering method based on continuous sequences

[0109] Median filtering can improve accuracy to over 98% but not 100%. Therefore, continuous noise reduction at the loading / unloading point is still necessary using state consistency and continuity criteria. A step change in the sequence indicates a state change; for example, a step from 0 to 1 indicates the point transitions from idle to operational, and a step from 1 to 0 indicates the point transitions from operational to idle. This characteristic is used to construct a state machine filtering method based on continuous sequences.

[0110] For the filtered sequence, a moving window is set with a window length of 30 seconds per sampling interval. When a step point appears, the mean of that point is compared to the mean of the window length following it. If they match, the state change is considered reasonable; otherwise, the current step point is a misjudgment and needs to be corrected to the state before the step. The second filtering result is then stored in the database to improve pattern recognition accuracy. This method can improve accuracy to 100%, meeting the requirements for practical algorithm implementation. The filtering effect is shown in the attached figure. Figure 11 As shown.

[0111] 7. Generation of time-series-based job reports

[0112] After two filtering steps, a smooth and stable time-series square wave plot is obtained, with the horizontal axis representing time and the vertical axis representing the stacking gate state. Based on this sequence, the actual working time and idle time of the stacking gate can be obtained. When the sequence jumps from 0 to 1, it indicates that the time is the start time of stacking gate operation, and when it jumps from 1 to 0, it indicates that the time is the end time of stacking gate operation. Conversely, the idle time of the stacking gate can be obtained.

[0113] However, due to the complex environment and many special circumstances in the stacking operation, the principle of consistency and continuity of state is still used to reduce noise in the special circumstances of loading and unloading the stacks.

[0114] (1) The same train continuously enters and exits the loading and unloading dock.

[0115] During loading and unloading operations at the stack, vehicles may leave the stack and then reverse back into it due to added equipment or special circumstances. Based on the image recognition sequence output, the result is two segments of operation time and one segment of stack idle time, which can easily lead to misjudgments.

[0116] The vehicle information for each operation is detected by license plate recognition and text recognition technology. If the operation information of vehicles before and after the operation is consistent, the two time periods are merged to obtain the actual operation time of that operation.

[0117] (2) Loss of view caused by pulling down the roller shutter door at the loading and unloading point.

[0118] First, record the vehicle operation information at the loading / unloading dock before the roller shutter door is pulled down. When the status of the roller shutter door at the loading / unloading dock changes, determine the current status. If it is in an idle state, the time when the roller shutter door is pulled down is determined to be the vehicle's stop time. If a vehicle is operating at the loading / unloading dock at this time, and the license plate matches the record, it is determined that the vehicle's operation has not ended. If the license plate does not match, the time when the roller shutter door is pulled down is determined to be the vehicle's stop time, and this time point is also determined to be the start time point of the current vehicle's operation.

[0119] (3) Non-target vehicles stopping and passing through

[0120] When the construction site is idle, non-target vehicles may pass through or briefly stop at the site. A median filtering algorithm is used to remove the step signal caused by vehicles passing through for short periods. At the same time, a 15-minute threshold is set as the working time; working times below this threshold are considered noise signals.

[0121] (4) People counting, license plate location and text detection algorithm

[0122] To measure vehicle operation efficiency, it is necessary to identify the license plate information of the personnel and vehicles involved in the operation. Due to the presence of multiple uncertain text regions in the loading dock image, direct detection using text detection algorithms would lead to problems such as discontinuous and inaccurate license plate text localization and excessive noise data. First, the license plate information of the vehicles loading and unloading at the loading dock is labeled to construct a loading dock vehicle license plate dataset. Simultaneously, the personnel are labeled to construct a loading dock personnel dataset. This dataset is then trained using a YOLO v5 object detection model for locating license plate regions on loading dock vehicles and counting the number of personnel.

[0123] License plate region localization filters out noisy text areas, yielding target area text information that represents the license plate text features. In this case, the license plate area occupies less than 0.6% of the image, and the dense, stacked text pixels in the original area introduce noise interference to subsequent text detection, leading to missed text localization and thus preventing accurate acquisition of license plate information. The license plate localization area is scaled down to obtain a 200*128 pixel region to be recognized, a size close to the aspect ratio of large yellow vehicle license plates. A text detection network, DBNet, is then constructed, and the region to be recognized is input into this network for text target localization, obtaining the text sequence position. Finally, a convolutional recurrent neural network is constructed to extract feature maps, and a deep long short-term neural network combined with CTC loss is used to identify the sequence objects in the target image to complete license plate text recognition and obtain the license plate information.

[0124] The above description is merely an embodiment of the present invention and is not intended to limit the present invention. All equivalent substitutions made within the principles of the present invention should be included within the scope of protection of the present invention. Contents not described in detail in this invention belong to existing technology known to those skilled in the art.

Claims

1. A loading and unloading dock door status pattern recognition method, characterized by, Comprise the following steps: Step one: set the video collection interval time 1 second / frame, collect video data, make a data set through the collected video data, and pretreat the data set; Step two: divide the data into training set, test set, and cross-validation set, build a ResNet50 pattern recognition network model, and train the built ResNet50 pattern recognition network model; Step three: use a Nvidia edge computing box as a model deployment container, realize video stream analysis by using the decoding and encoding chips of the Nvidia edge computing box through hard decoding technology, and the core computing capacity of the Nvidia edge computing box meets the computing power requirements of the intelligent algorithm of the system; Step four: through the RTSP protocol, decode and encode by using the edge computing box, real-time transmit the video stream into the trained ResNet50 pattern recognition network model, record the current time, the stack port ID, and the recognition result; mark the stack vehicle license plate area and the stack operator position, build a yolo v5 algorithm model to locate the picture license plate area and count the number of operators; build a text detection algorithm to locate the license plate area, complete the recognition of the text sequence by combining the convolutional recurrent neural network, extract the license plate information, and input into the database; Step five: improve the ResNet50 pattern recognition network model based on median filtering, and correct the stack state; Step six: according to the stability of time sequence, build a state machine filtering method to correct the stack state sequence denoised by median filtering again; Step seven: calculate the corrected result, record the stack vehicle table, the in-table and out-table time, the time length, and the number of people, complete the pattern recognition of the stack state, and generate a work report based on time sequence.

2. The method of claim 1, wherein, The data set pretreatment step in step one is as follows: The data set is divided into two categories according to the camera angle, direction, and different work scenes: idle state and stack work state; the idle state includes stack idle state when the camera is low, stack idle state, and stack roller door state; the stack work state includes stack work state and stack empty state.

3. The method of claim 1, wherein, The specific steps of building a ResNet50 pattern recognition network model and training the built ResNet50 pattern recognition network model in step two are as follows: Step 2.1: ResNet50 pattern recognition network model construction The initial network of ResNet50 is to classify 2048 categories, and the output result dimension is 2048; adjust the network structure to adapt to the state of the stack: add a new Dropout layer and logSoftmax, and add a fully connected layer with a dimension of 5 at the tail of the original ResNet50 network structure; set a total of k=5 node outputs, then the logSoftmax of the i th node is: L i = - logo(z i ) where L i is the output of the i-th node; Step 2.2: ResNet50 pattern recognition network model training Input the data set into the network for training, first train the model multiple times in the initial stage, quickly reach residual convergence through transfer learning, and get the weight with generalization; then test by using the test set.

4. The method of claim 1, wherein, The specific steps of improving the ResNet50 pattern recognition network model based on median filtering in the step five are as follows: Step 5.1: Let a set of sequences of the video stream be f1, f2, …, f n , take the window length s, where s is an odd number; and median filter the recognition results of this video stream: Wherein Med is the sequence median calculation formula, s is the number of sequences extracted from the video recognition sequence, i is the sequence median index, the sequence is sorted in descending order, and the sequence number is taken as the median value as the output of the whole sequence. Step 5.2: Model optimization based on median filtering algorithm: Real-time transmission of each channel of the stack video to the ResNet50 pattern recognition network model is completed, the pattern recognition is completed, and the time recognition sequence of the stack state is output; wherein the time recognition sequence is: 0, the camera is low, the stack is idle; 1, the stack is idle; 2, the stack is the rolling shutter door state; 3, the stack is the operation state; 4, the stack is the empty state; Mark the above sequences 0, 1, 2 as 0 to represent idle, 3, 4 as 1 to represent operation, and the final result as 0, 1 sequence square wave; A nonlinear estimation equation of step frequency and filter window length is constructed, and kernel density estimation is performed on both of them: where x (1) represents the step frequency, x (2) represents the filter window length, and η is the smoothing parameter bandwidth; a window length selection model based on a flexible Bayesian classifier is constructed, thereby establishing a median filter with adaptive window length.

5. The method of claim 4, wherein, The specific steps of constructing the window length selection model based on flexible Bayesian classifier in the step 5.2 are as follows: Step 5.2.1: According to the window length, the data set is divided, and the data set characteristics include step frequency and sequence length; Step 5.2.2: Use kernel density estimation to construct the generating model of each data set, so as to calculate the likelihood estimate value P(x|y), wherein x represents the characteristics, and y represents the window length; Step 5.2.3: By constructing a grid search, the optimal bandwidth length of the model accuracy is found; Step 5.2.4: Use the likelihood estimate value to replace the Gaussian distribution model of the naive Bayesian classifier to construct the flexible Bayesian model.

6. The method of claim 1, wherein, The specific steps of constructing the state machine filtering method for further correction of the stack state after median filtering denoising in the step six are as follows: After median filtering denoising, the continuous denoising of the loading and unloading stack is performed by using the state consistency and continuity criterion; when the sequence jumps, it means that the state changes, such as 0 jumping to 1, which means that the stack state changes from idle to operation, and 1 jumping to 0, which means that the stack changes from operation to idle; by using this characteristic, a state machine filtering method based on continuous sequence is constructed; For the filtered sequence, a moving window is set, and the window length is set to 30 seconds / sampling interval; the moving window, when a jump point appears, compares the point with the mean value in the window length after the point, if they are consistent, it means that the state change is reasonable, if they are not consistent, it means that the current jump point is a false positive, which needs to be corrected to the state before the jump, and the second filtering result is stored in the database.

7. The method of claim 1, wherein, The specific steps of generating the operation report based on time sequence in the step seven are as follows: After two times of filtering, a smooth and stable time sequence square wave graph is obtained, the horizontal axis is time, and the vertical axis is the stack state; based on the time sequence, the real operation time and idle time of the stack can be obtained, when the sequence jumps from 0 to 1, it means that the time is the start time of the stack operation, when 1 jumps to 0, it means that the time is the end time of the stack operation, and vice versa, the idle time of the stack is obtained. According to the special situation of the stack port operation, the noise reduction processing of the loading and unloading stack port special situation is carried out by using the consistency and continuity criterion, as follows: Step 7.1: The same train continuously enters and exits the loading and unloading stack port During the loading and unloading stack port operation, there is a situation that the vehicle leaves the stack port and then reverses into the stack. According to the sequence output of image recognition, the result is two operation times and one idle time of the stack port, which is easy to misjudge; Through license plate recognition and text recognition technology, the vehicle information of each operation is detected. If the vehicle operation information before and after is consistent, the two time periods are combined, which is the real operation time of the train; Step 7.2: The loss of visual field caused by the pull-down of the loading and unloading stack port shutter First, record the vehicle operation information before the loading and unloading stack port shutter is pulled down. When the state of the loading and unloading stack port shutter changes, determine the state at this time. If it is idle at this time, the time when the shutter is pulled down is the vehicle stop operation time; If there is a vehicle operating at this time, the license plate is consistent with the record, it is determined that the vehicle operation has not ended. If the license plate is not consistent, it is determined that the time when the shutter is pulled down is the vehicle stop operation time, and the time point is determined as the starting time point of the current vehicle operation; Step 7.3: Non-target vehicle parking and passing In the idle state of the loading and unloading stack port, there are non-target vehicles passing through the stack port or short-time parking; The median filter algorithm removes the step signal caused by the short-time vehicle passing, and sets 15 minutes as the threshold of operation time. The operation time below the threshold is the noise signal; Step 7.4: Number of people, license plate positioning and text detection algorithm By marking the vehicle license plate area and the position of the operating personnel of the loading and unloading stack port, a training set is constructed to train the yolo v5 model, so that the model has the target detection capability of license plate area and personnel position. According to the personnel position area, the current number of operating personnel is counted. The text detection algorithm based on segmentation DB and convolutional recurrent neural network CRNN is constructed for license plate text detection in the target area.

Images