Crane trolley positioning verification system and verification method based on visual detection

Abstract

The present invention provides a crane trolley positioning verification system and verification method based on visual detection, which relates to the field of crane trolley positioning technology. The method can effectively improve the positioning accuracy of the trolley in complex environments by applying deep learning algorithms to image recognition, and use laser ranging to provide real-time distance information, thereby achieving more accurate positioning verification. The core steps include real-time acquisition and denoising of image data, training of deep learning models, and multi-sensor data fusion based on Kalman filtering, which together constitute an efficient positioning system. The speed and angle are then corrected based on the deviation from the target position. The present invention not only reduces the potential safety hazards and economic losses that may occur in crane operations, but also improves the overall efficiency of trolley operations.

Description

Technical Field

[0001] The invention relates to the technical field of crane trolley positioning, and in particular to a crane trolley positioning verification system and verification method based on visual detection. Background Art

[0002] Crane trolley positioning technology plays a vital role in modern industry and construction. With the continuous improvement of automation and intelligence, traditional manual operation can no longer meet the needs of efficient, safe and precise construction. As one of the core components of the crane system, the accurate positioning of the crane trolley is not only related to the safe handling of materials, but also directly affects the construction efficiency of the entire project. In order to cope with complex working environments and dynamic operating conditions, advanced positioning technologies such as visual detection, laser ranging and fusion algorithms are widely used to improve the positioning accuracy and response speed of the trolley in various scenarios. Therefore, in-depth research and development of efficient crane trolley positioning systems have important practical significance and broad application prospects.

[0003] At present, the positioning verification of crane trolleys mainly relies on traditional visual detection and sensor technology. These methods usually use a single image recognition technology, which is affected by factors such as ambient light changes and object occlusion, resulting in low positioning accuracy. In addition, although laser ranging can provide relatively accurate distance information, in practical applications, it is often not closely combined with visual information, resulting in insufficient information fusion, which further restricts the positioning effect of the trolley. This single or simple information fusion method cannot effectively meet the high-precision and high-reliability positioning requirements in complex working environments.

[0004] The prior art has the following deficiencies:

[0005] Existing crane trolley positioning technologies mainly rely on traditional sensor systems, such as single laser ranging or encoders. These methods often face problems such as insufficient positioning accuracy, slow response speed, and susceptibility to environmental interference in complex and dynamic environments. In addition, traditional systems usually lack effective data fusion mechanisms when updating status, which affects the real-time and accuracy of position information, which may lead to operational risks and reduced efficiency. Summary of the invention

[0006] The purpose of the present invention is to provide a crane trolley positioning verification system and verification method based on visual detection to solve the problems raised in the above background technology.

[0007] The crane trolley positioning verification system based on visual inspection includes:

[0008] A data acquisition module is used to install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and to analyze and pre-process the collected data;

[0009] Localization model building module for convolutional neural network based models Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model;

[0010] The state fusion module is used to fuse the position coordinates output by the positioning model with the laser ranging data based on the Kalman filter algorithm, obtain the state vector, and update the state of the car to obtain the estimated state value of the car at the next moment;

[0011] The deviation detection module is used to compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the deviation correction module is used to correct the deviation.

[0012] The deviation correction module is used to obtain a correction control signal using a proportional controller, apply the correction control signal to the control input, update the state of the car again and compare it with the deviation threshold to obtain the updated state of the car.

[0013] Furthermore, the acquisition of the image in front of the crane trolley, the distance between the crane trolley and the obstacle in front, and the acceleration and angular velocity data of the crane trolley specifically includes:

[0014] A camera and a laser ranging sensor are arranged above the crane trolley to collect images in front of the crane trolley and the distance between the crane trolley and obstacles in front. An inertial measurement unit is installed at the center of gravity of the crane trolley to obtain the acceleration and angular velocity of the crane trolley.

[0015] Analyze the acceleration of the crane trolley and obtain the formula for the crane trolley speed:

[0016]

[0017] in, For the car The speed of time, is the initial velocity, for The acceleration of the moment;

[0018] When collecting image data in front of the crane trolley, the position coordinates of the crane trolley are marked, and the collected image data is denoised using a mean filter and a Gaussian filter. The filtered image is used to replace the image collected by the original timestamp, and the distance data collected by the laser ranging sensor is smoothed using the moving average method.

[0019] Furthermore, the construction of the positioning model specifically includes:

[0020] The image data with the crane trolley position coordinates marked in the collected image data are integrated into the image training set, and the image training set is input into the positioning model. The coordinate position of the crane trolley at the same timestamp is used as a label. Based on the convolutional neural network model, The model trains the positioning model and defines the mean square error as the loss function of the model. When the loss function is lower than 0.01, it indicates that the positioning model training is completed.

[0021] Furthermore, obtaining the estimated value of the state of the car at the next moment specifically includes:

[0022] The acceleration and angular velocity of the inertial measurement unit are converted from the inertial measurement unit coordinate system to the reference coordinate system of the crane trolley through the Euler angle. The velocity component is obtained by integrating the acceleration data. The formula is:

[0023]

[0024]

[0025] in, Respectively in The velocity component in the direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates that the crane trolley is The last timestamp of the moment Moving speed in the axis direction, Indicates that the crane trolley is The previous timestamp of the moment Moving speed in the axis direction, is the acceleration component measured by the inertial measurement unit, Indicates crane trolley Always The acceleration in the axis direction, Indicates crane trolley Always The acceleration in the axis direction, for Time has come The time interval between moments;

[0026] The position coordinates, velocity and angular velocity data of the crane trolley are integrated and the state vector is set as:

[0027]

[0028] in, is the position coordinate of the crane trolley, is the velocity component of the crane trolley, is the angular velocity of the crane trolley;

[0029] The state transfer matrix and observation vector are obtained by establishing the state transfer model and observation model. The distance data between the crane trolley and the obstacle in front collected by the laser ranging sensor is used as the observation value, which is entered into the observation model for fusion, and the initial state and initial error covariance are estimated. The state is predicted through the state transfer model to obtain the current state vector, and the state error covariance is updated. The Kalman gain is calculated to obtain the state update formula. The formula is as follows:

[0030] The state transition model of the car can be described by the following linear model:

[0031]

[0032] Among them, the state transfer matrix for:

[0033]

[0034] in, for The state of the car at the moment, is the state transfer matrix, for The state of the car at the moment, is the process noise, which is set to zero-mean Gaussian white noise;

[0035] The observation model maps the state to the observation space. The model is described as:

[0036]

[0037] Among them, the observation matrix for:

[0038]

[0039] in, is the observation vector, is the current state vector, is the observation noise, which is set to zero-mean Gaussian white noise;

[0040] The initial state estimate and initial error covariance are set as:

[0041]

[0042] in, is the initial state estimate, is the initial error covariance, is the set initial error, is the identity matrix;

[0043] State prediction through state transition model:

[0044]

[0045] in, for The predicted state at the moment is based only on The state and control input at the moment, without considering the current observation, for The state of the car at the moment;

[0046] Update state error covariance:

[0047]

[0048] in, for The error covariance of the moment prediction is based only on The state and control input at the moment, without considering the current observation, for The error covariance at time t, is the transpose of the state transfer matrix, is the process noise covariance matrix;

[0049] Calculate the Kalman gain:

[0050]

[0051] in, is the transpose of the observation matrix, is the observation noise covariance matrix;

[0052] Get status update formula:

[0053]

[0054] in, for The estimated value of the state at time , conditioned on the current observation has been taken into consideration, is the Kalman gain, for The observation vector at time, is the observation matrix.

[0055] Furthermore, the obtained position information of the car at the next moment is compared with the target position specifically including:

[0056] The formula for obtaining the position deviation is:

[0057]

[0058] in, is the position deviation, is the target location;

[0059] The formula for obtaining the angle deviation is:

[0060]

[0061] in, is the angle deviation, The target location is Position coordinates in direction, The target location is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, is the angular velocity of the crane trolley;

[0062] Normalize the angular deviation to The range is expressed as , compare the position deviation and angle deviation with the preset position deviation threshold and angle deviation threshold. If the position deviation and angle deviation are both within the preset deviation threshold range, the crane trolley continues to work. If the position deviation or angle deviation exceeds the preset threshold, the next step is implemented to correct the deviation.

[0063] Furthermore, the step of obtaining the updated status of the vehicle specifically includes:

[0064] The formula for obtaining the corrected control signal is:

[0065]

[0066]

[0067] in, They are position correction control signal and angle correction control signal respectively. They are the proportional gain of position and the proportional gain of angle respectively;

[0068] The modified control signal is applied to the control input to update the motion instructions of the crane trolley according to the formula:

[0069]

[0070] in, The updated Axis and The velocity components on the axes and the updated angular velocity, , The position correction control signal is Axis and The weight on the axis, is the angle correction control signal;

[0071] After the control input is updated, the state of the crane trolley is updated again using the new velocity and angular velocity, based on the formula:

[0072]

[0073] in, is the state transfer matrix, The state of the car after applying the corrected control signal;

[0074] At this time, repeat the previous step to obtain the position deviation and angle deviation of the crane trolley until the deviation data are lower than the preset deviation threshold, and then start the crane trolley to continue working.

[0075] The present invention further provides a crane trolley positioning verification method based on visual detection, which is performed by the above-mentioned crane trolley positioning verification system based on visual detection, and the specific steps include:

[0076] Step 1: Install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and analyze and pre-process the collected data;

[0077] Step 2: Based on the convolutional neural network model Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model;

[0078] Step 3: Based on the Kalman filter algorithm, the position coordinates output by the positioning model are fused with the laser ranging data to obtain the state vector, and the state of the car is updated to obtain the estimated state value of the car at the next moment;

[0079] Step 4: Compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the next step is implemented to correct the deviation.

[0080] Step 5: Use a proportional controller to obtain a corrected control signal, apply the corrected control signal to the control input, update the state of the car again, and compare it with the deviation threshold to obtain the updated state of the car.

[0081] In the above technical solution, the technical effects and advantages provided by the present invention are:

[0082] This solution significantly improves the positioning accuracy and real-time response capability of the crane trolley by combining convolutional neural networks and Kalman filtering technology. By using visual recognition technology, key image features can be extracted from complex working environments to accurately determine the position of the trolley. At the same time, Kalman filtering realizes the effective fusion of different sensor data, reduces the errors that may be caused by a single sensor, and improves the reliability of state estimation. In addition, the intelligent characteristics of the system enable it to dynamically adapt to environmental changes and adjust the positioning results in real time, thereby optimizing the operation process and reducing potential operational risks. Overall, this solution not only improves the operating efficiency of the crane trolley in dynamic and complex environments, but also provides strong technical support for intelligent logistics and automated construction, and promotes the continuous development and progress of related fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a schematic diagram of the system structure of the present invention;

[0084] Figure 2 It is a schematic diagram of the overall method flow of the present invention. DETAILED DESCRIPTION

[0085] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments.

[0086] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the present invention should be understood by people with ordinary skills in the field to which the present invention belongs. The words "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0087] Example:

[0088] See also Figure 1 The present invention provides a crane trolley positioning verification system based on visual detection, comprising:

[0089] A data acquisition module is used to install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and to analyze and pre-process the collected data;

[0090] In this embodiment, the acquisition of the image in front of the crane trolley, the distance between the crane trolley and the obstacle in front, and the acceleration and angular velocity data of the crane trolley specifically includes:

[0091] A camera and a laser ranging sensor are arranged above the crane trolley to collect images in front of the crane trolley and the distance between the crane trolley and obstacles in front. An inertial measurement unit is installed at the center of gravity of the crane trolley to obtain the acceleration and angular velocity of the crane trolley.

[0092] The center of gravity is the point of equilibrium of the overall mass of the vehicle, where a force applied does not result in any rotation, only linear motion. The center of gravity can be determined by computer-aided design software by inputting the geometry and mass properties of the various components of the vehicle.

[0093] Analyze the acceleration of the crane trolley and obtain the formula for the crane trolley speed:

[0094]

[0095] in, For the car The speed of time, is the initial velocity, for The acceleration of the moment;

[0096] When collecting image data in front of the crane trolley, the position coordinates of the crane trolley are marked, and the collected image data is denoised using a mean filter and a Gaussian filter. The filtered image is used to replace the image collected by the original timestamp, and the distance data collected by the laser ranging sensor is smoothed using the moving average method.

[0097] Data fusion of multiple sensors can reduce the error of a single sensor and provide more reliable environmental perception and state estimation, which is particularly important in complex or dynamic environments. Accurate speed and position estimation enables the car to perform more complex path planning, effectively deal with obstacles ahead, reduce the risk of collision, and improve the safety of autonomous navigation. De-noising and data smoothing can reduce the computational burden of the system, improve real-time performance, and enable the car to respond promptly in a rapidly changing environment. By collecting and processing multiple types of data, the system can be flexibly adjusted to different task requirements and environmental changes to adapt to different application scenarios. Through data analysis and processing, the system can extract important feature information, combined with machine learning or other intelligent algorithms, to further enhance the car's intelligent decision-making capabilities, such as automatic obstacle avoidance and path optimization.

[0098] Localization model building module for convolutional neural network based models Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model;

[0099] In this embodiment, the construction of the positioning model specifically includes:

[0100] The image data with the crane trolley position coordinates marked in the collected image data are integrated into the image training set, and the image training set is input into the positioning model. The coordinate position of the crane trolley at the same timestamp is used as a label. Based on the convolutional neural network model, The model trains the positioning model and defines the mean square error as the loss function of the model. When the loss function is lower than 0.01, it indicates that the positioning model training is completed.

[0101] The model effectively alleviates the gradient vanishing problem in deep network training through its deep hierarchical structure and jump connections, enabling the model to learn more complex features, thereby enhancing the model's ability to identify and locate the crane trolley in the image. The use of deep learning models to automatically extract image features reduces the need for manual feature engineering and improves processing efficiency. At the same time, this automated approach enables the system to adapt and learn in different environments, further improving robustness. Using mean square error as the loss function can effectively evaluate the difference between the model's prediction and the true value. When the loss function value is lower than the set threshold (such as 0.01), it indicates that the model has achieved the expected accuracy, which facilitates model tuning and selection.

[0102] The state fusion module is used to fuse the position coordinates output by the positioning model with the laser ranging data based on the Kalman filter algorithm, obtain the state vector, and update the state of the car to obtain the estimated state value of the car at the next moment;

[0103] In this embodiment, obtaining the estimated value of the state of the car at the next moment specifically includes:

[0104] The acceleration and angular velocity of the inertial measurement unit are converted from the inertial measurement unit coordinate system to the reference coordinate system of the crane trolley through the Euler angle. The velocity component is obtained by integrating the acceleration data. The formula is:

[0105]

[0106]

[0107] in, Respectively in The velocity component in the direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates that the crane trolley is The last timestamp of the moment Moving speed in the axis direction, Indicates that the crane trolley is The previous timestamp of the moment Moving speed in the axis direction, is the acceleration component measured by the inertial measurement unit, Indicates crane trolley Always The acceleration in the axis direction, Indicates crane trolley Always The acceleration in the axis direction, for Time has come The time interval between moments;

[0108] The position coordinates, velocity and angular velocity data of the crane trolley are integrated and the state vector is set as:

[0109]

[0110] in, is the position coordinate of the crane trolley, is the velocity component of the crane trolley, is the angular velocity of the crane trolley;

[0111] The state transfer matrix and observation vector are obtained by establishing the state transfer model and observation model. The distance data between the crane trolley and the obstacle in front collected by the laser ranging sensor is used as the observation value, which is entered into the observation model for fusion, and the initial state and initial error covariance are estimated. The state is predicted through the state transfer model to obtain the current state vector, and the state error covariance is updated. The Kalman gain is calculated to obtain the state update formula. The formula is as follows:

[0112] The state transition model of the car can be described by the following linear model:

[0113]

[0114] Among them, the state transfer matrix for:

[0115]

[0116] in, for The state of the car at the moment, is the state transfer matrix, for The state of the car at the moment, is the process noise, which is set to zero-mean Gaussian white noise;

[0117] The observation model maps the state to the observation space. The model is described as:

[0118]

[0119] Among them, the observation matrix for:

[0120]

[0121] in, is the observation vector, is the current state vector, is the observation noise, which is set to zero-mean Gaussian white noise;

[0122] The initial state estimate and initial error covariance are set as:

[0123]

[0124] in, is the initial state estimate, is the initial error covariance, is the set initial error, is the identity matrix;

[0125] State prediction through state transition model:

[0126]

[0127] in, for The predicted state at the moment is based only on The state and control input at the moment, without considering the current observation, for The state of the car at the moment;

[0128] Update state error covariance:

[0129]

[0130] in, for The error covariance of the moment prediction is based only on The state and control input at the moment, without considering the current observation, for The error covariance at time t, is the transpose of the state transfer matrix, is the process noise covariance matrix;

[0131] Calculate the Kalman gain:

[0132]

[0133] in, is the transpose of the observation matrix, is the observation noise covariance matrix;

[0134] Get status update formula:

[0135]

[0136] in, for The estimated value of the state at time , conditioned on the current observation has been taken into consideration, is the Kalman gain, for The observation vector at time, is the observation matrix.

[0137] Through Kalman filtering, data from different sources (such as the position coordinates output by the positioning model and the data from the laser ranging sensor) are fused to obtain a more accurate state estimate, which can help the car maintain high-precision navigation in a dynamic environment. Kalman filtering can effectively reduce the noise and error between different sensor data through weighted updates, providing more accurate position information. This is especially important in dynamic environments because sensor measurements are often affected by environmental changes. Kalman filtering is a recursive algorithm that can quickly calculate new state estimates at each data update. This real-time performance is critical for autonomous navigation systems, allowing the car to adjust its motion state in time to adapt to environmental changes. The algorithm can effectively handle the uncertainty in sensor measurements, including measurement noise and model uncertainty, and provide probability-based state estimates. This capability makes the system more robust in the face of uncertain environments. By combining prediction and observation, Kalman filtering provides a powerful framework that allows the system to not only make state estimates based on current observations, but also use prediction information to improve estimates. This feedback mechanism effectively improves the accuracy of state updates. This method allows the fusion of multiple sensor data, including visual information, laser ranging information, and IMU information, which can maximize the use of available information sources and improve the overall performance of the system.

[0138] The deviation detection module is used to compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the deviation correction module is used to correct the deviation.

[0139] In this embodiment, the obtained position information of the vehicle at the next moment is compared with the target position specifically including:

[0140] The formula for obtaining the position deviation is:

[0141]

[0142] in, is the position deviation, is the target location;

[0143] The formula for obtaining the angle deviation is:

[0144]

[0145] in, is the angle deviation, The target location is Position coordinates in direction, The target location is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, is the angular velocity of the crane trolley;

[0146] Normalize the angular deviation to The range is expressed as , compare the position deviation and angle deviation with the preset position deviation threshold and angle deviation threshold. If the position deviation and angle deviation are both within the preset deviation threshold range, the crane trolley continues to work. If the position deviation or angle deviation exceeds the preset threshold, the next step is implemented to correct the deviation.

[0147] By instantly detecting and correcting position and angle deviations, the robot can maintain high-precision navigation in complex environments. This is essential for performing refined tasks such as handling, loading and unloading. This step allows the system to monitor the status of the robot at every moment, so that any deviation can be captured and processed in time. This real-time feedback mechanism enhances the adaptability and flexibility of the system, especially in dynamic environments. Through frequent deviation correction, even small deviations can be corrected in time to prevent them from accumulating into larger errors in subsequent operations, thereby improving the stability and reliability of long-term operation. Moreover, under different working environments and task requirements, the system can adjust the target position and motion strategy based on real-time feedback, so that the robot can better adapt to different working conditions. Timely deviation correction helps to avoid potential collisions and accidents and ensure the safety of equipment and people around it. Especially in industrial environments, ensuring safety is always the first priority.

[0148] The deviation correction module is used to obtain a correction control signal using a proportional controller, apply the correction control signal to the control input, update the state of the car again and compare it with the deviation threshold to obtain the updated state of the car.

[0149] In this embodiment, obtaining the updated status of the vehicle specifically includes:

[0150] The formula for obtaining the corrected control signal is:

[0151]

[0152]

[0153] in, They are position correction control signal and angle correction control signal respectively. They are the proportional gain of position and the proportional gain of angle respectively;

[0154] The modified control signal is applied to the control input to update the motion instructions of the crane trolley according to the formula:

[0155]

[0156] in, The updated Axis and The velocity components on the axes and the updated angular velocity, , The position correction control signal is Axis and The weight on the axis, is the angle correction control signal;

[0157] After the control input is updated, the state of the crane trolley is updated again using the new velocity and angular velocity, based on the formula:

[0158]

[0159] in, is the state transfer matrix, The state of the car after applying the corrected control signal;

[0160] At this time, repeat the previous step to obtain the position deviation and angle deviation of the crane trolley until the deviation data are lower than the preset deviation threshold, and then start the crane trolley to continue working.

[0161] The proportional controller calculates the position and angle correction control signals to adjust the control input of the car to ensure that it can move towards the target position, using the proportional gain and , which can accurately control the position and angle deviations and form targeted correction control signals. This helps the car to gradually adjust when approaching the target position to avoid over-correction or oscillation. After detecting the deviation, the system can quickly generate a correction control signal to achieve real-time state adjustment. This fast feedback mechanism enables the car to dynamically adapt to environmental changes and improve overall operating efficiency. By iteratively updating the state of the car and continuously monitoring the deviation, the error during the movement can be effectively reduced. This method can ensure stability and reliability during long-term operation and prevent the accumulation of errors. Through continuous state updates and deviation correction, the system has stronger anti-interference ability and is not easily affected by changes in the external environment. This robustness is very important for improving the adaptability of the car, especially in dynamic or unpredictable environments.

[0162] See also Figure 2 The present invention further provides a crane trolley positioning verification method based on visual detection, the crane trolley positioning verification method based on visual detection is performed by the above-mentioned crane trolley positioning verification system based on visual detection, and the specific steps include:

[0163] Step 1: Install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and analyze and pre-process the collected data;

[0164] Step 2: Based on the convolutional neural network model Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model;

[0165] Step 3: Based on the Kalman filter algorithm, the position coordinates output by the positioning model are fused with the laser ranging data to obtain the state vector, and the state of the car is updated to obtain the estimated state value of the car at the next moment;

[0166] Step 4: Compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the next step is implemented to correct the deviation.

[0167] Step 5: Use a proportional controller to obtain a corrected control signal, apply the corrected control signal to the control input, update the state of the car again, and compare it with the deviation threshold to obtain the updated state of the car.

[0168] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0169] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. Those skilled in the art may appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein may be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed by hardware or software methods depends on the specific application and design constraints of the technical solution.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. The crane trolley positioning verification system based on visual detection is characterized by: include: A data acquisition module is used to install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and to analyze and pre-process the collected data; Localization model building module for convolutional neural network based models Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model; The state fusion module is used to fuse the position coordinates output by the positioning model with the laser ranging data based on the Kalman filter algorithm, obtain the state vector, and update the state of the car to obtain the estimated state value of the car at the next moment; The deviation detection module is used to compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the deviation correction module is used to correct the deviation. The deviation correction module is used to obtain a correction control signal using a proportional controller, apply the correction control signal to the control input, update the state of the car again and compare it with the deviation threshold to obtain the updated state of the car.

2. The crane trolley positioning and verification system based on visual detection according to claim 1 is characterized in that: The collecting of the image in front of the crane trolley, the distance between the crane trolley and the obstacle in front, and the acceleration and angular velocity data of the crane trolley specifically includes: A camera and a laser ranging sensor are arranged above the crane trolley to collect images in front of the crane trolley and the distance between the crane trolley and obstacles in front. An inertial measurement unit is installed at the center of gravity of the crane trolley to obtain the acceleration and angular velocity of the crane trolley. Analyze the acceleration of the crane trolley and obtain the formula for the crane trolley speed: in, For the car The speed of time, is the initial velocity, for The acceleration of the moment; When collecting image data in front of the crane trolley, the position coordinates of the crane trolley are marked, and the collected image data is denoised using a mean filter and a Gaussian filter. The filtered image is used to replace the image collected by the original timestamp, and the distance data collected by the laser ranging sensor is smoothed using the moving average method.

3. The crane trolley positioning verification system based on visual detection according to claim 2 is characterized in that: The construction of the positioning model specifically includes: The image data with the crane trolley position coordinates marked in the collected image data are integrated into the image training set, and the image training set is input into the positioning model. The coordinate position of the crane trolley at the same timestamp is used as a label. Based on the convolutional neural network model, The model trains the positioning model and defines the mean square error as the loss function of the model. When the loss function is lower than 0.01, it indicates that the positioning model training is completed.

4. The crane trolley positioning and verification system based on visual detection according to claim 3 is characterized in that: The obtaining of the estimated state value of the car at the next moment specifically includes: The acceleration and angular velocity of the inertial measurement unit are converted from the inertial measurement unit coordinate system to the reference coordinate system of the crane trolley through the Euler angle. The velocity component is obtained by integrating the acceleration data. The formula is: in, Respectively in The velocity component in the direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates crane trolley Always Moving speed in the axis direction, Indicates that the crane trolley is The last timestamp of the moment Moving speed in the axis direction, Indicates that the crane trolley is The previous timestamp of the moment Moving speed in the axis direction, is the acceleration component measured by the inertial measurement unit, Indicates crane trolley Always The acceleration in the axis direction, Indicates crane trolley Always The acceleration in the axis direction, for Time has come The time interval between moments; The position coordinates, velocity and angular velocity data of the crane trolley are integrated and the state vector is set as: in, is the position coordinate of the crane trolley, is the velocity component of the crane trolley, is the angular velocity of the crane trolley; The state transfer matrix and observation vector are obtained by establishing the state transfer model and observation model. The distance data between the crane trolley and the obstacle in front collected by the laser ranging sensor is used as the observation value, which is entered into the observation model for fusion, and the initial state and initial error covariance are estimated. The state is predicted through the state transfer model to obtain the current state vector, and the state error covariance is updated. The Kalman gain is calculated to obtain the state update formula. The formula is as follows: The state transition model of the car can be described by the following linear model: Among them, the state transfer matrix for: in, for The state of the car at the moment, is the state transfer matrix, for The state of the car at the moment, is the process noise, which is set to zero-mean Gaussian white noise; The observation model maps the state to the observation space. The model is described as: Among them, the observation matrix for: in, is the observation vector, is the current state vector, is the observation noise, which is set to zero-mean Gaussian white noise; The initial state estimate and initial error covariance are set as: in, is the initial state estimate, is the initial error covariance, is the set initial error, is the identity matrix; State prediction through state transition model: in, for The predicted state at the moment is based only on The state and control input at the moment, without considering the current observation, for The state of the car at the moment; Update state error covariance: in, for The error covariance of the moment prediction is based only on The state and control input at the moment, without considering the current observation, for The error covariance at time t, is the transpose of the state transfer matrix, is the process noise covariance matrix; Calculate the Kalman gain: in, is the transpose of the observation matrix, is the observation noise covariance matrix; Get status update formula: in, for The estimated value of the state at time , conditioned on the current observation has been taken into consideration, is the Kalman gain, for The observation vector at time, is the observation matrix.

5. The crane trolley positioning and verification system based on visual detection according to claim 4 is characterized in that: The obtained next moment position information of the vehicle is compared with the target position and specifically includes: The formula for obtaining the position deviation is: in, is the position deviation, is the target location; The formula for obtaining the angle deviation is: in, is the angle deviation, The target location is Position coordinates in direction, The target location is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, The crane trolley after the status update is Position coordinates in direction, is the angular velocity of the crane trolley; Normalize the angular deviation to The range is expressed as , compare the position deviation and angle deviation with the preset position deviation threshold and angle deviation threshold. If the position deviation and angle deviation are both within the preset deviation threshold range, the crane trolley continues to work. If the position deviation or angle deviation exceeds the preset threshold, the next step is implemented to correct the deviation.

6. The crane trolley positioning verification system based on visual detection according to claim 5 is characterized in that: The step of obtaining the updated status of the car specifically includes: The formula for obtaining the corrected control signal is: in, They are position correction control signal and angle correction control signal respectively. They are the proportional gain of position and the proportional gain of angle respectively; The modified control signal is applied to the control input to update the motion instructions of the crane trolley according to the formula: in, The updated Axis and The velocity components on the axes and the updated angular velocity, , The position correction control signal is Axis and The weight on the axis, is the angle correction control signal; After the control input is updated, the state of the crane trolley is updated again using the new velocity and angular velocity, based on the formula: in, is the state transfer matrix, The state of the car after applying the corrected control signal; At this time, repeat the previous step to obtain the position deviation and angle deviation of the crane trolley until the deviation data are lower than the preset deviation threshold, and then start the crane trolley to continue working.

7. A crane trolley positioning verification method based on visual detection, characterized in that: The crane trolley positioning verification method based on visual detection is performed by the crane trolley positioning verification system based on visual detection according to any one of claims 1 to 6, and the specific steps include: Step 1: Install relevant equipment and sensors on the crane trolley to collect images in front of the crane trolley, the distance between the crane trolley and obstacles in front, the acceleration and angular velocity data of the crane trolley, and analyze and pre-process the collected data; Step 2: Based on the convolutional neural network model Build a positioning model, input the preprocessed image into the positioning model, use the position coordinates of the crane trolley in the image as labels, and train the positioning model; Step 3: Based on the Kalman filter algorithm, the position coordinates output by the positioning model are fused with the laser ranging data to obtain the state vector, and the state of the car is updated to obtain the estimated state value of the car at the next moment; Step 4: Compare the position information of the car at the next moment with the target position to obtain the position deviation and angle deviation. If any deviation is higher than the preset deviation threshold, the next step is implemented to correct the deviation. Step 5: Use a proportional controller to obtain a corrected control signal, apply the corrected control signal to the control input, update the state of the car again, and compare it with the deviation threshold to obtain the updated state of the car.

Images