Crane remote control image delay detection method based on machine learning

Abstract

The invention discloses a crane remote control image delay detection method based on machine learning, which belongs to the technical field of crane remote control, and comprises the following steps: S1, data acquisition and preprocessing; s2, feature extraction; s3, training a machine learning model; s4, carrying out real-time image delay detection; and S5, adjusting the control strategy. According to the method, a machine learning algorithm is utilized, complex features and time sequence relations in image data can be automatically learned, compared with a traditional method, image time delay can be more accurately detected, the detection precision is effectively improved, reliable data support is provided for remote control of the crane, and by continuously updating training data and optimizing a model, the accuracy of remote control of the crane is improved. The method can adapt to different network environments, crane operation scenes and working condition changes. Good detection performance can be kept no matter in an environment with large network fluctuation or in a complex operation scene, and the universality and practicability of the method are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of remote control of cranes, and particularly relates to a remote control image delay detection method for cranes based on machine learning. BACKGROUND

[0002] With the development of industrial automation and remote control technology, remote control of cranes has gradually become a trend, which can effectively reduce the exposure of operators in dangerous environments, improve operation safety and efficiency. In the remote control process, image transmission is an important way for operators to obtain information about the crane operation site. However, image delay is inevitable in the transmission process. If the image delay cannot be detected and processed in time, the control instructions of the operator may not match the actual state of the crane, which may lead to operational errors, equipment damage and even safety accidents.

[0003] Existing image delay detection methods are mostly based on fixed network parameters or simple timestamp comparison, which are difficult to adapt to complex and variable network environments and crane operation scenarios. These methods often cannot accurately and timely detect image delay, and cannot provide reliable protection for remote control. In addition, traditional methods lack learning and utilization of historical data and operation scene characteristics, and have poor detection accuracy and stability when facing different working conditions and network fluctuations, which cannot meet the high-precision and real-time requirements of remote control of cranes for image delay detection.

[0004] Therefore, there is an urgent need for a method that can effectively detect the image delay of remote control of cranes to improve the accuracy and safety of remote control. SUMMARY

[0005] In view of the above problems, the present application provides a remote control image delay detection method for cranes based on machine learning, which effectively solves the problems in the current market.

[0006] The technical scheme adopted by the present application is as follows: the present application provides a remote control image delay detection method for cranes based on machine learning, which comprises the following steps: S1, data acquisition and preprocessing In the crane operation site, high-definition cameras are used to collect image data during crane operation, and the time stamp of image acquisition is recorded ; In the remote control end, the transmitted image data is received, and the time stamp of image reception is recorded ; S2, feature extraction Various features are extracted from the preprocessed image data, and the texture features of the image are extracted by using a gray level co-occurrence matrix; S3, machine learning model training Choose an appropriate machine learning model, such as a long short-term memory network, a model combining convolutional neural networks and LSTM, etc. During training, use a validation set to evaluate and adjust the model to prevent overfitting. S4, Real-time Image Delay Detection Image data is acquired and transmitted in real time during the remote control of the crane; S5, Control Strategy Adjustment Based on the detected image delay Adjust the remote control strategy: when the image latency is low, operate according to the normal control procedure; when the image latency exceeds a preset threshold... , and take corresponding measures.

[0007] Furthermore, the preprocessing in step S1 includes operations such as image denoising and size normalization. Principal component analysis or linear discriminant analysis is used to reduce the dimensionality of high-dimensional feature vectors, remove redundant information, and improve the computational efficiency of the model.

[0008] Furthermore, the features extracted in step S2 may include texture features, color features, edge features, etc.

[0009] Furthermore, the texture features extracted in step S2 may include contrast, entropy, energy, and correlation.

[0010] Furthermore, the formula for calculating contrast in step S2 is: , in The elements in the gray-level co-occurrence matrix The probability of; The formula for calculating entropy is: , in It is an element in the gray-level co-occurrence matrix. The probability of; The formula for calculating energy is: , in It is an element in the gray-level co-occurrence matrix. The probability of; The formula for calculating correlation is: , in, and The mean values ​​in the x and y directions, respectively; and These are the standard deviations in the x and y directions, respectively; It is an element in the gray-level co-occurrence matrix. The probability of.

[0011] Furthermore, in step S3, the dataset is divided into a training set, a validation set, and a test set. The machine learning model is trained using the training set, and the model parameters are adjusted through the backpropagation algorithm to minimize the error between the predicted image latency and the actual latency. The training set, validation set, and test set are divided in a ratio of 7:2:1.

[0012] Furthermore, the error function in step S3 uses the mean square error, and its calculation formula is as follows: , in For the sample size, For actual image delay, Delay the image predicted by the model.

[0013] Furthermore, in step S4, the real-time received image data is processed according to the above-mentioned data acquisition and preprocessing, feature extraction steps to obtain the corresponding feature vector. A sliding window mechanism is introduced to perform temporal smoothing of continuous frame features, reducing the interference of single-frame noise on the prediction results.

[0014] Furthermore, the feature vectors are input into the trained machine learning model, and the model outputs the predicted image latency. The actual delay is calculated by comparing it with the actual timestamp. .

[0015] Furthermore, it also includes recording and storing the real-time detected image delay data and control strategy adjustment information for subsequent model optimization and control strategy improvement.

[0016] The beneficial effects achieved by the present invention using the above structure are as follows: (1) This invention utilizes machine learning algorithms to automatically learn complex features and time-series relationships in image data. Compared with traditional methods, it can more accurately detect image delays, effectively improving detection accuracy and providing reliable data support for remote crane control. By continuously updating training data and optimizing the model, this invention can adapt to different network environments, crane operation scenarios, and working condition changes. Whether in environments with large network fluctuations or complex operation scenarios, it can maintain good detection performance, improving the versatility and practicality of the method. (2) By adopting real-time data acquisition and processing methods and combining them with efficient machine learning models, image delay can be detected quickly and control strategies can be adjusted in a timely manner to ensure the synchronization between remote control commands and the actual state of the crane. This greatly reduces the risk of operational errors caused by image delay and improves the real-time performance and safety of crane remote control. (3) By recording and analyzing historical data, machine learning models and control strategies can be continuously optimized. With the accumulation and learning of data, the detection accuracy of the model and the effectiveness of the control strategy will continue to improve, making remote control of cranes more intelligent and efficient. Attached Figure Description

[0017] Figure 1 The flowchart of the crane remote control image delay detection method based on machine learning proposed in this invention is as follows: Figure One ; Figure 2 The flowchart of the crane remote control image delay detection method based on machine learning proposed in this invention is as follows: Figure Two ; Figure 3 The flowchart of the crane remote control image delay detection method based on machine learning proposed in this invention is as follows: Figure Three ; Figure 4 The flowchart of the crane remote control image delay detection method based on machine learning proposed in this invention is as follows: Figure Four .

[0018] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of this invention, it should be understood that the terms "upper", "lower", "front", "rear", "left", "right", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0021] like Figure 1 As shown, this invention proposes a machine learning-based image delay detection method for remote control of cranes, comprising the following steps: S1. Data Acquisition and Preprocessing At the crane operation site, high-definition cameras capture image data of the crane operation process and record the timestamp of the image acquisition. ; At the remote control end, the transmitted image data is received, and the timestamp of image reception is recorded. ; S2, Feature Extraction Multiple features are extracted from the preprocessed image data, and the texture features of the image are extracted using the gray-level co-occurrence matrix; S3, Machine Learning Model Training Choose an appropriate machine learning model, such as a long short-term memory network, a model combining convolutional neural networks and LSTM, etc. During training, use a validation set to evaluate and adjust the model to prevent overfitting. S4, Real-time Image Delay Detection Image data is acquired and transmitted in real time during the remote control of the crane; S5, Control Strategy Adjustment Based on the detected image delay Adjust the remote control strategy: when the image latency is low, operate according to the normal control procedure; when the image latency exceeds a preset threshold... , and take corresponding measures.

[0022] The preprocessing in step S1 includes operations such as image denoising and size normalization. Principal component analysis or linear discriminant analysis is used to reduce the dimensionality of high-dimensional feature vectors, remove redundant information, and improve the model's computational efficiency.

[0023] The features extracted in step S2 can include texture features, color features, edge features, etc.

[0024] The texture features extracted in step S2 may include contrast, entropy, energy, and correlation.

[0025] The formula for calculating contrast in step S2 is: , in It is an element in the gray-level co-occurrence matrix. The probability of; The formula for calculating entropy is: , in It is an element in the gray-level co-occurrence matrix. The probability of; The formula for calculating energy is: , in It is an element in the gray-level co-occurrence matrix. The probability of; The formula for calculating correlation is:

[0026] in, and The mean values ​​in the x and y directions, respectively; and These are the standard deviations in the x and y directions, respectively; It is an element in the gray-level co-occurrence matrix. The probability of.

[0027] In step S3, the dataset is divided into a training set, a validation set, and a test set. The training set is used to train the machine learning model, and the parameters of the model are adjusted through the backpropagation algorithm to minimize the error between the predicted image latency and the actual latency. The training set, validation set, and test set are divided in a ratio of 7:2:1.

[0028] In step S3, the error function uses the mean square error, and its calculation formula is as follows: , in For the sample size, For actual image delay, Delay the image predicted by the model.

[0029] In step S4, the real-time received image data is processed according to the above steps of data acquisition and preprocessing, and feature extraction to obtain the corresponding feature vector. A sliding window mechanism is introduced to perform temporal smoothing of continuous frame features, reducing the interference of single-frame noise on the prediction results.

[0030] The feature vectors are input into a trained machine learning model, and the model outputs the predicted image latency. The actual delay is calculated by comparing it with the actual timestamp. .

[0031] It also includes recording and storing real-time detected image latency data and control strategy adjustment information for subsequent model optimization and control strategy improvement.

[0032] S1. Multimodal data acquisition and intelligent preprocessing Multi-view 4K high-definition cameras (3840×2160 resolution, 30fps) are deployed at the front end of the crane boom and in the working area, transmitting image data streams in real time via a 5G private network (peak bandwidth 1.2Gbps, theoretical latency ≤10ms). Data is recorded synchronously during acquisition. Image acquisition timestamp t (acquisition) (accuracy to milliseconds, based on GPS clock synchronization); The remote control terminal receives the timestamp t (receive) (deploy an NTP server to achieve clock calibration).

[0033] The preprocessing workflow has been upgraded to: Multi-stage noise reduction: First, Gaussian filtering (σ=1.5) is used to suppress Gaussian noise, and then median filtering (3×3 window) is used to remove salt-and-pepper noise; Geometric correction: Correction of fisheye lens distortion based on Zhang's calibration method, combined with perspective transformation to achieve image viewpoint normalization; Dynamic resizing: Adaptively scales images to 224×224 or 384×384 pixels based on the complexity of the task scenario, preserving key details.

[0034] S2, Spatiotemporal Feature Fusion and Dimensionality Reduction Representation Construct a three-layer feature extraction system: Spatial characteristics: Texture features: Contrast, entropy, energy, and correlation in four directions (0°, 45°, 90°, and 135°) are calculated based on the gray-level co-occurrence matrix (GLCM), and the average value is used as the global texture description. Color features: Extract chromaticity moments (first-order moments, second-order moments, and third-order moments) from the HSV space and color histograms (256 bins per channel) from the YCbCr space; Edge features: The improved Canny operator (with dual threshold adaptive adjustment) is used to extract the contours, and the Hough transform is combined with the statistical linear feature density.

[0035] Time characteristics: Calculate the mean optical flow vector magnitude between consecutive frames (characterizing the intensity of motion). Extract the texture feature difference and color histogram Bach distance between adjacent frames.

[0036] Feature dimensionality reduction: Kernel principal component analysis was used to reduce the dimensionality of high-dimensional features (dimension ≥ 500) to 128 dimensions while preserving nonlinear relationships. The feature contribution rate after dimensionality reduction was ≥ 95%.

[0037] Deep training of S3 and CNN-LSTM fusion models Building an end-to-end latency prediction framework: Network architecture: CNN layer: ResNet18 is used as the backbone network to extract spatial features of the image and output a 256-dimensional feature map; LSTM layer: Two-layer bidirectional LSTM (128 neurons each) to capture dynamic changes in temporal features; Fusion layer: It uses an attention mechanism to weight and fuse spatiotemporal features, and outputs a predicted delay value; Training strategy: Dataset partitioning: The training set, validation set, and test set are divided in a 7:2:1 ratio, including data from different lighting conditions (sunny / cloudy / night) and load conditions (idle / half-load / full-load). Optimized configuration: Use AdamW optimizer (β1=0.9, β2=0.999), learning rate with cosine annealing decay (initial 0.001, minimum 1e-5). Regularization: Combining L2 regularization (λ=1e-4), Dropout (probability 0.3) and early stopping strategy (terminating if the validation set error does not decrease for 5 consecutive rounds); Loss function: Mixed mean squared error (MSE) and mean absolute error (MAE), the formula is: ; in, , .

[0038] In practical use, a high-definition camera with a resolution of 4K and a frame rate of 30fps is installed on the crane to collect image data during the crane's operation. Data transmission between the remote control terminal and the crane is carried out via a 5G network. At the same time, a high-performance data processing server is provided to run machine learning models and perform data processing.

[0039] During normal crane operation, high-definition cameras continuously collect image data and record the acquisition timestamp of each image. After receiving the image data, the remote control terminal records the reception timestamp, preprocesses the collected and received images, uses Gaussian filtering for noise reduction, normalizes the image size to a fixed size (e.g., 224×224 pixels), and stores the processed image data and corresponding timestamp information in the database to form an initial dataset.

[0040] From the preprocessed image, texture features are extracted using the gray-level co-occurrence matrix, and parameters such as contrast, entropy, energy, and correlation are calculated. Color features are obtained by statistically analyzing the histograms of the RGB color channels. The edge features of the image are extracted using the Canny edge detection algorithm, and combined with timestamp information, temporal-spatial features such as time interval between adjacent images, texture feature difference, and color feature change are calculated. These features are then combined into a feature vector.

[0041] The CNN-LSTM model was selected as the image time-lapse detection model. The dataset was divided into training, validation, and test sets in a 7:2:1 ratio. The model was trained using the training set with a learning rate of 0.001 and a training period of 50 epochs. During training, the model's performance was evaluated using the validation set, and the model parameters were adjusted to prevent overfitting. Finally, the average detection error of the model on the test set was 50ms, which meets the requirements of practical applications.

[0042] During the remote control of the crane, image data is acquired and transmitted in real time. The received images are preprocessed and feature extracted. The feature vectors are input into the trained model to obtain the predicted image delay. When the detected image delay exceeds a preset threshold of 100ms, the remote control terminal automatically reduces the crane's operating speed and introduces a delay compensation mechanism into the control commands. The command sending time is adjusted according to the predicted delay. At the same time, the real-time detection data and control strategy adjustment information are recorded in the database for subsequent model optimization and strategy improvement. The above is the overall workflow of this invention. This step can be repeated for the next use. The actual operation process is very simple and easy to implement.

[0043] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0044] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0045] The present invention and its embodiments have been described above. This description is not restrictive, and the accompanying drawings are only one embodiment of the present invention; the actual structure is not limited thereto. In conclusion, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the invention, such designs should fall within the protection scope of the present invention.

Claims

1. A machine learning based crane remote control image time delay detection method, characterized by: Comprise the following steps: S1, data acquisition and pretreatment In the crane operation site, the high-definition camera is used to collect image data in the crane operation process, and the time stamp of image collection is recorded ; At the remote control end, the transmitted image data is received and a time stamp of the image reception is recorded ; S2, feature extraction A variety of features are extracted from the pretreated image data, and the texture features of the image are extracted using a gray level co-occurrence matrix; S3, machine learning model training Select a suitable machine learning model, such as a long short-term memory network, a convolutional neural network combined with an LSTM model, etc. In the training process, use the validation set to evaluate and adjust the model to prevent overfitting of the model; S4, real-time image delay detection In the process of remote control of the crane, real-time image data is collected and transmitted; S5, control strategy adjustment Adjusting remote control strategy according to detected image delay , when image delay is small, operating according to normal control flow; when image delay exceeds preset threshold , taking corresponding measures.

2. The machine learning based crane remote control image time delay detection method according to claim 1, characterized by: The pretreatment in step S1 includes image noise reduction, size normalization, etc. Principal component analysis or linear discriminant analysis is used to reduce the dimension of the high-dimensional feature vector, eliminate redundant information, and improve the operation efficiency of the model.

3. The machine learning based crane remote control image lag detection method according to claim 2, characterized by: The features extracted in step S2 can include texture features, color features, edge features, etc.

4. The machine learning based crane remote control image lag detection method according to claim 3, characterized by: The texture features extracted in step S2 can include contrast, entropy, energy, and correlation, etc.

5. The machine learning based crane remote control image lag detection method according to claim 4, characterized by: The calculation formula of contrast in step S2 is: ， wherein is the probability of element in the gray level co-occurrence matrix. The calculation formula of entropy is: ， wherein is the probability of the element in the gray level co-occurrence matrix. The calculation formula of energy is: ， wherein is the probability of element in the gray level co-occurrence matrix. The calculation formula of correlation is: ， wherein and the mean value of x and y direction, respectively; and are the standard deviations in the x and y directions, respectively; is the probability of the element in the gray level co-occurrence matrix .

6. The machine learning based crane remote control image lag detection method according to claim 5, characterized by: In step S3, the data set is divided into training set, validation set and test set. The training set is used to train the machine learning model, and the parameters of the model are adjusted through the back propagation algorithm to minimize the error between the predicted image delay and the actual delay. The training set, validation set and test set are divided in the ratio of 7:2:

1.

7. The machine learning based crane remote control image lag detection method according to claim 6, characterized by: In step S3, the error function uses mean square error, and its calculation formula is: ， wherein is the number of samples, is the actual image delay, is the model predicted image delay.

8. The machine learning based crane remote control image lag detection method according to claim 7, characterized by: In step S4, the real-time received image data is processed according to the above data acquisition and pretreatment, feature extraction steps to obtain the corresponding feature vector. The sliding window mechanism is introduced to perform time series smoothing processing on the continuous frame features, reducing the interference of single frame noise on the prediction result.

9. The machine learning based crane remote control image lag detection method according to claim 8, characterized by: inputting the feature vector into the trained machine learning model, the model outputting a predicted image latency , a true latency calculated from the actual time stamp .

10. The machine learning based crane remote control image lag detection method according to claim 9, characterized by: It also includes recording and storing the real-time detected image delay data and control strategy adjustment information for subsequent model optimization and control strategy improvement.

Images