A video prediction method based on an automatic driving remote control system

Abstract

The application provides a video prediction method based on an automatic driving remote control system, including the following steps: acquiring and decoding original images through camera shooting; collecting original image data acquired by the camera to generate a training set; training an MCNet model using the training set and performing data processing on the training set through the MCNet model; taking the position information of a spreader in a predicted image as an item in a loss function, adding a plurality of fully connected layers to the backbone network of the MCNet model to output the position information of the spreader, and adding a spreader regression loss function when the MCNet model is updated; copying the currently used MCNet model to obtain a mirrored MCNet model; performing small-step training using the original image of a previous historical frame as training data; performing performance verification on the original image of the second-to-last historical frame as verification data of the training data, if the accuracy of the verification data is higher, the MCNet model is updated and optimized; otherwise, the weights of the network of the MCNet model remain unchanged; and video streaming of the predicted image is generated and published.

Description

Technical Field

[0001] This invention relates to the field of industrial automation control, and in particular to a video prediction method based on an automated vehicle remote control system. Background Technology

[0002] Overhead cranes, as general-purpose logistics equipment with heavy loads, are used in most factory workshops, such as mining, steel, non-ferrous metals, and machinery manufacturing industries. Overhead crane operators need to possess certain technical skills and work long hours in the cramped cab.

[0003] Based on traditional anti-sway algorithms, programmable logic controllers (PLCs) and frequency converters are generally used. The operating speed of the trolley and crane that can eliminate load sway is calculated according to the operation instructions and the real-time operating status of the crane. Then, the frequency converter drives the trolley and crane to run at the required speed, thereby eliminating the crane load sway phenomenon.

[0004] While such methods are used, due to technical and cost issues, the level of automation across the industry is less than one-thousandth.

[0005] Remote control and video monitoring of automated vehicles are effective solutions for poor working environments. However, current remote control systems generally use relatively inexpensive IPC cameras as the primary sensor. IPC cameras typically suffer from significant image latency, which greatly increases the difficulty of operation and poses safety hazards.

[0006] Therefore, it is necessary to provide a novel video prediction method based on an automated vehicle remote control system to overcome the above-mentioned shortcomings. Summary of the Invention

[0007] The purpose of this invention is to provide a video prediction method based on an automated guided vehicle (AGV) remote control system, which solves the problem of video latency, is low in cost, has low requirements for network latency, allows users to complete fixed latency compensation through parameter settings, predicts the position of the AGV's lifting device, and can also greatly reduce the amount of computation.

[0008] To achieve the above objectives, the present invention provides a video prediction method based on an automated vehicle remote control system, comprising the following steps.

[0009] S1: Capture the original image by taking a picture with the camera and then decode it;

[0010] S2: Collect raw image data acquired by the camera under various working conditions to generate a training set;

[0011] S3: Train the MCNet model using the training set, and process the training set data using the MCNet model to obtain the predicted image;

[0012] S4: The position information of the lifting device in the predicted image is used as a term in the loss function. Multiple fully connected layers are added after the backbone network of the MCNet model to output the position information of the lifting device. The lifting device regression loss function is added when the MCNet model is updated.

[0013] A video prediction method based on an automated vehicle remote control system is characterized by the following steps: A mirrored MCNet model is obtained by copying the currently used MCNet model; the original image of the previous historical frame is used as training data for small-step training with a batch size of 1;

[0014] Copy the currently used MCNet model to obtain a mirrored MCNet model;

[0015] Using the original image of the previous historical frame as training data, we perform training with a small step size of 1 batch-size.

[0016] Using the original image of the second-to-last historical frame as validation data for the training data described in step S42, and utilizing the evaluation function... The performance of the MCNet model is validated. If the accuracy of the validation data is higher, the MCNet model is updated and optimized; otherwise, the weights of the MCNet model remain unchanged.

[0017] The evaluation function is as follows:

[0018] ,in, This represents the actual position of the automated vehicle. The predicted position of the autonomous vehicle. This represents the actual swing angle of the automated guided vehicle (AGV) crane. The value representing the predicted swing angle of the automated overhead crane. The pixel prediction value represents the predicted image. Represents the actual pixel value of the predicted image. Represents the weight of each item;

[0019] Repeat steps S42 and S43 to continuously update, and record the total number of updates;

[0020] S5: Convert the predicted image into an h265 format video stream and publish it in rtmp format.

[0021] Preferably, the specific steps for testing the prediction time of the predicted image of the remote control system are as follows:

[0022] Step a: Use two stopwatches with millisecond-level precision to synchronize the time, and record the start time S1 and S2 of the two stopwatches respectively;

[0023] Step b: Place one stopwatch under the camera and display it on the remote control screen, defining the stopwatch's time as E1; simultaneously, use the camera to photograph the remote control screen and the other stopwatch, defining the stopwatch's time as E2; calculate the remote control system's delay using the following formula.

[0024] Delay = (E1 - S1) - (E2 - S2); where Delay is the delay amount.

[0025] Compared with existing technologies, the advantages are as follows: the method of the present invention solves the problem of video latency, is low in cost, has low requirements for network latency, can meet the user's requirement to complete fixed latency compensation through parameter settings, predict the position of the crane's rigging, and can also greatly reduce the amount of computation. [Attached Image Description]

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 The flowchart shows the video prediction method based on the remote control system for automatic driving provided by the present invention.

[0028] Figure 2 This is a comparison chart showing the prediction performance of video prediction methods based on remote control systems for autonomous vehicles.

Detailed Implementation Methods

[0029] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described in this specification are merely for explaining the invention and are not intended to limit the invention.

[0030] Please see Figures 1 to 2 This invention provides a video prediction method based on an automated vehicle remote control system, comprising the following steps:

[0031] S1: Capture the original image by taking a picture with the camera and then decode it;

[0032] S2: Collect raw image data acquired by the camera under various working conditions to generate a training set;

[0033] S3: Train the MCNet model using the training set, and process the training set data using the MCNet model to obtain the predicted image;

[0034] Since the lifting device and the suspended object are the primary moving targets and the main objects of observation during remote control, their proportion in the image is generally maintained at around 60% through camera zoom. Therefore, the feature extraction layer eliminates the recognition layer that identifies small features, thereby accelerating the network's inference speed and reducing the required computational resources.

[0035] S4: The position information of the lifting device in the predicted image is used as a term in the loss function. Multiple fully connected layers are added after the backbone network of the MCNet model to output the position information of the lifting device. The lifting device regression loss function is added when the MCNet model is updated.

[0036] Because the camera can acquire continuous video data during operation, the MCNet model can be updated online in real time, thereby improving predictive ability and environmental adaptability.

[0037] Specifically, step S4 also includes step S41: copying the currently used MCNet model to obtain a mirrored MCNet model;

[0038] S42: Using the original image of the previous historical frame as training data, perform small-step training with a batch size of 1;

[0039] S43: Using the original image of the second-to-last historical frame as validation data for the training data described in step S42, and applying the evaluation function... The performance of the MCNet model is validated. If the accuracy of the validation data is higher, the MCNet model is updated and optimized; otherwise, the weights of the MCNet model remain unchanged.

[0040] The evaluation function is as follows:

[0041] ,in, This represents the actual position of the automated vehicle. The predicted position of the autonomous vehicle. This represents the actual swing angle of the automated guided vehicle (AGV) crane. This represents the predicted swing angle of the automated overhead crane. The pixel prediction value represents the predicted image. Represents the actual pixel value of the predicted image. Represents the weight of each item;

[0042] S44: Repeat steps S42 and S43 to continuously update, and record the total number of updates;

[0043] S5: Convert the predicted image into an h265 format video stream and publish it in rtmp format.

[0044] S44: Repeat steps S42 and S43 to continuously update, and record the total number of updates;

[0045] S5: Convert the predicted image into an h265 format video stream and publish it in rtmp format.

[0046] It should be noted that the specific steps for users to test the prediction time of the remote control system's predicted image by setting the delay based on the remote control system's settings are as follows:

[0047] Step a: Use two stopwatches with millisecond-level precision to synchronize the time, and record the start time S1 and S2 of the two stopwatches respectively;

[0048] Step b: Place one stopwatch under the camera and display it on the remote control screen, defining the stopwatch's time as E1; simultaneously, use the camera to photograph the remote control screen and the other stopwatch, defining the stopwatch's time as E2; calculate the remote control system's delay using the following formula.

[0049] Delay = (E1 - S1) - (E2 - S2); where Delay is the delay amount.

[0050] The present invention is not limited to the description in the specification and embodiments, and thus other advantages and modifications can be readily realized by those skilled in the art. Therefore, the present invention is not limited to the specific details, representative devices and examples shown and described herein without departing from the spirit and scope of the general concept as defined by the claims and their equivalents.

Claims

1. A video prediction method based on an automated vehicle remote control system, characterized in that, Includes the following steps, S1: Capture the original image by taking a picture with the camera and then decode it; S2: Collect raw image data acquired by the camera under various working conditions to generate a training set; S3: Train the MCNet model using the training set, and process the training set data using the MCNet model to obtain the predicted image; S4: The position information of the lifting device in the predicted image is used as a term in the loss function. Multiple fully connected layers are added after the backbone network of the MCNet model to output the position information of the lifting device. The lifting device regression loss function is added when the MCNet model is updated. Step S4 also includes step S41: copying the currently used MCNet model to obtain a mirrored MCNet model; S42: Using the original image of the previous historical frame as training data, perform small-step training with a batch size of 1; S43: Using the original image of the second-to-last historical frame as validation data for the training data described in step S42, and applying the evaluation function... The performance of the MCNet model is validated. If the accuracy of the validation data is higher, the MCNet model is updated and optimized; otherwise, the weights of the MCNet model remain unchanged. The evaluation function is as follows: ,in, This represents the actual position of the automated vehicle. The predicted position of the autonomous vehicle. This represents the actual swing angle of the automated guided vehicle (AGV) crane. The value representing the predicted swing angle of the automated overhead crane. The pixel prediction value represents the predicted image. Represents the actual pixel value of the predicted image. Represents the weight of each item; Repeat steps S42 and S43 to continuously update, and record the total number of updates; S5: Convert the predicted image into an h265 format video stream and publish it in rtmp format.

2. The video prediction method based on an automated vehicle remote control system as described in claim 1, characterized in that, The specific steps for testing the prediction time of the predicted image in the remote control system are as follows: Step a: Use two stopwatches with millisecond-level precision to synchronize the time, and record the start time S1 and S2 of the two stopwatches respectively; Step b: Place one stopwatch under the camera and display it on the remote control screen, defining the stopwatch's time as E1; simultaneously, use the camera to photograph the remote control screen and the other stopwatch, defining the stopwatch's time as E2; calculate the remote control system's delay using the following formula. Delay = (E1 - S1) - (E2 - S2); where Delay is the delay amount.

Images