Swing angle determination method of lifting appliance, electronic equipment, storage medium and overhead lifting equipment

Abstract

The embodiment of the invention provides a method for determining the swing angle of a lifting appliance, electronic equipment, a storage medium and overhead lifting equipment. Relates to the technical field of sling swing angle measurement. The method comprises the steps of collecting a first image through a preset first camera; acquiring a second image through a preset second camera; wherein the first image and the second image both comprise the same lifting appliance with the angle to be measured. Determining three-dimensional coordinates of the lifting appliance according to the first image and the second image; wherein the three-dimensional coordinates are actual coordinates of the lifting appliance. And according to the three-dimensional coordinates and preset standard coordinates, the swing angle of the lifting appliance is determined. The method is used for achieving the effect of improving the measurement and calculation precision of the swing angle of the lifting appliance.

Description

Technical Field

[0001] The present application relates to the technical field of swing angle measurement of a spreader, and in particular to a method for determining the swing angle of a spreader, an electronic device, a storage medium and an overhead lifting device. Background Art

[0002] At present, overhead cranes are common lifting machinery in freight terminals. Due to the long lowering distance of the overhead crane's spreader, the lifting load will swing to a large extent due to the inertial force during operation, which seriously affects the operating efficiency and stability of the overhead crane. Therefore, it is necessary to detect the swing angle of the spreader.

[0003] In the prior art, the sling sway angle detection method is often combined with the anti-sway system and applied to lifting machinery. Currently, most commonly used sling sway angle detection methods require the use of laser rangefinders for distance measurement and scanning modeling, or combine total stations, laser trackers, angle sensors and other traditional measurement tools to measure specific positions.

[0004] However, in the prior art, the method based on the laser rangefinder has high requirements on the environment. Due to the influence of environmental factors, it is difficult to measure the spreader with a long lowering distance. Therefore, it is impossible to stably and accurately measure the spreader during the overhead crane operation. Secondly, the method based on the combination of multiple traditional measurement tools is not only expensive, but also complicated to operate. Since it cannot be guaranteed that the captured points are in the same position, the accuracy cannot be guaranteed. Therefore, the existing spreader swing angle detection method cannot achieve a high-precision detection effect when applied to the overhead crane. Summary of the invention

[0005] The embodiments of the present application provide a method for determining the swing angle of a spreader, an electronic device, a storage medium and an overhead lifting device, so as to achieve the effect of improving the measurement accuracy of the swing angle of the spreader.

[0006] In a first aspect, an embodiment of the present application provides a method for determining a swing angle of a spreader, comprising:

[0007] A first image is captured by a preset first camera; and a second image is captured by a preset second camera; wherein the first image and the second image both include the same sling at the angle to be measured;

[0008] Determine the three-dimensional coordinates of the spreader according to the first image and the second image; wherein the three-dimensional coordinates are actual coordinates of the spreader;

[0009] The swing angle of the hanger is determined according to the three-dimensional coordinates and the preset standard coordinates.

[0010] In a possible implementation manner, determining the three-dimensional coordinates of the spreader according to the first image and the second image includes:

[0011] Performing image stereo matching processing on the first image and the second image to generate a depth map;

[0012] Performing target detection on the first image according to a preset target detection model to obtain two-dimensional pixel coordinates of the spreader;

[0013] The three-dimensional coordinates of the spreader are determined according to the two-dimensional pixel coordinates and the depth map.

[0014] In a possible implementation manner, performing stereo matching processing on the first image and the second image to generate a depth map includes:

[0015] Using a preset image stereo matching algorithm, performing image stereo matching processing on points at corresponding positions on the first image and the second image to generate a minimum cost path;

[0016] Converting the minimum cost path into a disparity map;

[0017] The disparity map is converted according to a preset disparity formula to generate a depth map.

[0018] In a possible implementation, performing target detection on the first image according to a preset target detection model to obtain the two-dimensional pixel coordinates of the spreader includes:

[0019] Performing target detection on the first image according to a preset target detection model to obtain a detection frame including the sling, and determining a detection frame information vector of the detection frame;

[0020] The center point coordinates of the detection frame information vector are determined, and the center point coordinates are determined to be the two-dimensional pixel coordinates of the hanger.

[0021] In a possible implementation, determining the three-dimensional coordinates of the spreader according to the two-dimensional pixel coordinates and the depth map includes:

[0022] Determining in the depth map a depth value at a coordinate position corresponding to the two-dimensional pixel coordinate;

[0023] Determine the coordinates of the sling in the camera coordinate system by using a projection relationship between a preset camera coordinate system and an image coordinate system; wherein the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system in which the depth map is located;

[0024] According to the projection relationship between the preset lifting rope coordinate system and the camera coordinate system, the coordinates in the camera coordinate system are projected and transformed to determine the three-dimensional coordinates of the lifting device; wherein the lifting rope coordinate system is the coordinate system in which the lifting device is located.

[0025] In a possible implementation manner, determining the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates includes:

[0026] Modeling is performed according to the three-dimensional coordinates and preset standard coordinates to determine the swing angle of the spreader; wherein the standard coordinates are the coordinates of the spreader in a standard vertical state; and the swing angle includes a pitch swing angle and a horizontal swing angle.

[0027] In a possible implementation manner, before acquiring the first image by using a preset first camera and acquiring the second image by using a preset second camera, the method further includes:

[0028] The first camera and the second camera are calibrated to determine calibration parameters, wherein the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera.

[0029] In a possible implementation manner, before determining the three-dimensional coordinates of the spreader according to the first image and the second image, the method further includes:

[0030] According to calibration parameters, epipolar correction processing is performed on the first camera and the second camera to obtain intermediate values; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera;

[0031] Determining a projection matrix for distortion correction and a projection matrix for polar image alignment according to the intermediate value;

[0032] According to the distortion-corrected projection matrix and the polar-aligned projection matrix, the first image and the second image are mapped and transformed to obtain the transformed first image and the second image.

[0033] In a second aspect, an embodiment of the present application provides a device for determining a swing angle of a spreader, comprising:

[0034] An acquisition module, used to acquire a first image through a preset first camera; and to acquire a second image through a preset second camera; wherein the first image and the second image both include the same sling at the angle to be measured;

[0035] A first determination module, configured to determine the three-dimensional coordinates of the spreader according to the first image and the second image; wherein the three-dimensional coordinates are actual coordinates of the spreader;

[0036] The second determination module is used to determine the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates.

[0037] In a possible implementation manner, the first determining module includes:

[0038] A matching unit, configured to perform stereo matching processing on the first image and the second image to generate a depth map;

[0039] A first determination unit, configured to perform target detection on the first image according to a preset target detection model to obtain two-dimensional pixel coordinates of the spreader;

[0040] The second determining unit is used to determine the three-dimensional coordinates of the spreader according to the two-dimensional pixel coordinates and the depth map.

[0041] In a possible implementation manner, the matching unit is specifically configured to:

[0042] Using a preset image stereo matching algorithm, performing image stereo matching processing on points at corresponding positions on the first image and the second image to generate a minimum cost path;

[0043] Converting the minimum cost path into a disparity map;

[0044] The disparity map is converted according to a preset disparity formula to generate a depth map.

[0045] In a possible implementation manner, the first determining unit is specifically configured to:

[0046] Performing target detection on the first image according to a preset target detection model to obtain a detection frame including the sling, and determining a detection frame information vector of the detection frame;

[0047] The center point coordinates of the detection frame information vector are determined, and the center point coordinates are determined to be the two-dimensional pixel coordinates of the hanger.

[0048] In a possible implementation manner, the second determining unit is specifically configured to:

[0049] Determining in the depth map a depth value at a coordinate position corresponding to the two-dimensional pixel coordinate;

[0050] Determine the coordinates of the sling in the camera coordinate system by using a projection relationship between a preset camera coordinate system and an image coordinate system; wherein the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system in which the depth map is located;

[0051] According to the projection relationship between the preset lifting rope coordinate system and the camera coordinate system, the coordinates in the camera coordinate system are projected and transformed to determine the three-dimensional coordinates of the lifting device; wherein the lifting rope coordinate system is the coordinate system in which the lifting device is located.

[0052] In a possible implementation manner, the second determining module is specifically configured to:

[0053] Modeling is performed according to the three-dimensional coordinates and preset standard coordinates to determine the swing angle of the spreader; wherein the standard coordinates are the coordinates of the spreader in a standard vertical state; and the swing angle includes a pitch swing angle and a horizontal swing angle.

[0054] In a possible implementation manner, the device is further specifically used for:

[0055] Before acquiring a first image through a preset first camera and acquiring a second image through a preset second camera, the first camera and the second camera are calibrated to determine calibration parameters; wherein the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera.

[0056] In a possible implementation manner, the device is further specifically used for:

[0057] Before determining the three-dimensional coordinates of the spreader according to the first image and the second image, the first camera and the second camera are subjected to epipolar correction processing according to calibration parameters to obtain intermediate values; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include the intrinsic parameters of the first camera, the intrinsic parameters of the second camera, and the mapping matrix of the second camera to the first camera;

[0058] Determining a projection matrix for distortion correction and a projection matrix for polar image alignment according to the intermediate value;

[0059] According to the distortion-corrected projection matrix and the polar-aligned projection matrix, the first image and the second image are mapped and transformed to obtain the transformed first image and the second image.

[0060] In a third aspect, an embodiment of the present application provides an overhead lifting device, including a lifting device and a swing angle detection device; wherein the swing angle detection device is used to determine the swing angle of the lifting device; and the detection method of the swing angle detection device is the swing angle determination method of the lifting device as described in the first aspect.

[0061] In a fourth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0062] The memory stores computer-executable instructions;

[0063] The processor executes the computer-executable instructions stored in the memory, so that the processor executes various possible implementations of the first aspect as described above.

[0064] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement various possible implementation methods of the first aspect above.

[0065] In a sixth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements various possible implementation methods of the first aspect above.

[0066] The swing angle determination method, electronic device, storage medium and overhead crane provided by the embodiment of the present application, collects a first image through a preset first camera; and collects a second image through a preset second camera; wherein the first image and the second image both include the same crane of the angle to be measured. According to the first image and the second image, the three-dimensional coordinates of the crane are determined; wherein the three-dimensional coordinates are the actual coordinates of the crane. According to the three-dimensional coordinates and the preset standard coordinates, the swing angle of the crane is determined. In this scheme, the first image and the second image are respectively collected vertically downward according to the first camera and the second camera built, and then the actual three-dimensional coordinates of the crane are determined according to the acquired first image and the second image. Finally, the current swing angle of the crane is determined by combining the current three-dimensional coordinates and the standard coordinates. Therefore, only two cameras are used to accurately detect the swing angle of the crane with a long lowering distance. Compared with the existing method, the present application does not need to use expensive detection equipment, the detection equipment used is low in cost, the angle measurement is safe, reliable, efficient and accurate, and provides accurate swing angle of the crane for subsequent overhead crane crane to reduce the swing and prevent the swing, realizes automatic angle detection, has high practicality, and is used to achieve the effect of improving the measurement accuracy of the swing angle of the crane. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0068] Figure 1 A schematic diagram of a method for determining a swing angle of a sling provided in an embodiment of the present application Figure 1 ;

[0069] Figure 2 A scenario diagram of a method for determining the swing angle of a spreader provided in this application Figure 1 ;

[0070] Figure 3 A schematic diagram of another method for determining the swing angle of a sling provided in an embodiment of the present application Figure 2 ;

[0071] Figure 4 A scenario diagram of a method for determining the swing angle of a spreader provided in this application Figure 2 ;

[0072] Figure 5 A schematic diagram of a method for determining the swing angle of a sling provided in this application Figure 3 ;

[0073] Figure 6 A schematic diagram of the structure of a device for determining a swing angle of a sling provided in an embodiment of the present application;

[0074] Figure 7 A schematic structural diagram of another device for determining the swing angle of a sling provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0076] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0078] At present, overhead cranes are common lifting machinery in freight terminals. Due to the long lowering distance of the overhead crane's spreader, the lifting load will swing to a large extent due to the inertial force during operation, which seriously affects the operating efficiency and stability of the overhead crane. Therefore, it is necessary to detect the swing angle of the spreader.

[0079] In one example, the sling swing angle detection method is often combined with the anti-sway system and applied to lifting machinery. Currently, most commonly used sling swing angle detection methods require the use of laser rangefinders for distance measurement and scanning modeling, or combine total stations, laser trackers, angle sensors and other traditional measuring tools to measure specific positions. However, in the prior art, the method based on the laser rangefinder has very high requirements for the environment. Affected by environmental factors, it is difficult to measure slings with a long lowering distance. Therefore, it is impossible to stably and accurately measure the slings during overhead crane operations. Secondly, the method based on the combination of multiple traditional measuring tools is not only expensive, but also complicated to operate. Since there is no guarantee that the captured points are in the same position, the accuracy cannot be guaranteed. Therefore, the existing sling swing angle detection method cannot obtain a high-precision detection effect when applied to overhead cranes.

[0080] In combination with the above-mentioned scenarios, it can be known that in the prior art, there is a technical problem that the existing method for detecting the swing angle of the hoist cannot obtain a high-precision detection effect when applied to an overhead crane.

[0081] The method for determining the swing angle of a spreader provided in the present application can accurately detect the swing angle of a spreader with a long lowering distance using only two cameras, thereby solving the technical problem of low accuracy in measuring the swing angle of the spreader.

[0082] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0083] Figure 1 A schematic diagram of a method for determining the swing angle of a sling provided in this application Figure 1 ,like Figure 1 As shown, the method includes:

[0084] S101, collecting a first image through a preset first camera; and collecting a second image through a preset second camera; wherein both the first image and the second image include the same hanger at an angle to be measured.

[0085] For example, the execution subject of this embodiment can be an electronic device, or a terminal device, or a swing angle determination device or device of a sling, or other devices or equipment that can execute this embodiment, and there is no limitation on this. In this embodiment, the execution subject is introduced as an electronic device.

[0086] First, the preset first camera and second camera are used to realize binocular vision. The first camera and the second camera can be two monocular cameras or a binocular camera including two cameras. For example, the overall camera is a damped binocular camera, etc., which is not limited. The following takes the damped binocular camera as an example for introduction. Figure 2 A scenario diagram of a method for determining the swing angle of a spreader provided in this application Figure 1 ,like Figure 2 As shown, it includes an overhead crane, an arm, a damped binocular camera and a hoist. The damped bracket is fixed to the top of the arm of the overhead crane. The damped binocular camera is built on the damped bracket. When installed, the camera shoots vertically downward. The damped bracket can maintain the camera shooting direction vertically downward when the arm swings up and down. It should be noted that in addition to the damped bracket, other devices that can maintain the binocular camera shooting direction vertically downward when the arm moves can be used to build the binocular camera, such as a gimbal, etc., and there is no limitation on this.

[0087] In this step, a first image of the spreader is captured vertically downward by a first camera, and a second image of the spreader is captured vertically downward by a second camera, that is, the first image and the second image both include the same spreader with a swing angle to be measured.

[0088] S102, determining the three-dimensional coordinates of the spreader according to the first image and the second image; wherein the three-dimensional coordinates are the actual coordinates of the spreader.

[0089] Exemplarily, firstly, stereo matching is performed on the first image and the second image to generate a depth map. Then, target detection is performed on the first image according to a preset target detection model to obtain the two-dimensional pixel coordinates of the spreader. Combining the two-dimensional pixel coordinates and the depth map, the three-dimensional coordinates of the spreader are determined; wherein the three-dimensional coordinates are the actual coordinates of the spreader.

[0090] S103, determining the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates.

[0091] Exemplarily, the preset standard coordinates are coordinates in an ideal vertical state, and modeling is performed based on the three-dimensional coordinates of the current sling and the standard coordinates to determine the current swing angle of the sling; wherein the swing angle includes a pitch swing angle and a horizontal swing angle.

[0092] The swing angle determination method of the sling provided in the embodiment of the present application, collects a first image through a preset first camera; and collects a second image through a preset second camera; wherein the first image and the second image both include the same sling of the angle to be measured. According to the first image and the second image, the three-dimensional coordinates of the sling are determined; wherein the three-dimensional coordinates are the actual coordinates of the sling. According to the three-dimensional coordinates and the preset standard coordinates, the swing angle of the sling is determined. In this scheme, the first image and the second image are respectively collected vertically downward according to the first camera and the second camera built, and then the actual three-dimensional coordinates of the sling are determined according to the acquired first image and the second image. Finally, the current swing angle of the sling is determined by combining the current three-dimensional coordinates and the standard coordinates. Therefore, only two cameras are used to accurately detect the swing angle of the sling with a long lowering distance. Compared with the existing method, the present application does not need to use expensive detection equipment, the detection equipment used is low-cost, the angle measurement is safe, reliable, efficient and accurate, and provides accurate swing angle of the sling for subsequent overhead sling swing reduction and anti-sway, realizes automated angle detection, has high practicality, and is used to achieve the effect of improving the measurement accuracy of the swing angle of the sling.

[0093] Figure 3 A schematic diagram of a method for determining the swing angle of a sling provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment Figure 1 Based on the embodiment, a method for determining the swing angle of the spreader is described in detail, and the method includes:

[0094] S201, calibrate the first camera and the second camera to determine calibration parameters; wherein the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera.

[0095] Exemplarily, first, the electronic device can use a checkerboard rectangular calibration plate to repeatedly move the calibration plate within the common field of view of the first camera and the second camera to collect two groups of image data on the left and right, each group is a plurality of checkerboard images, for example, each group is 25 checkerboard images, etc., which is not limited to this; wherein, the image collected by the first camera is the left image, and the image collected by the second camera is the right image. Secondly, using the preset calibrateCamera function of the opencv toolkit, the first camera is calibrated separately with the left image, and the second camera is calibrated separately with the right image, and the internal parameters of the first camera and the internal parameters of the second camera are obtained. Finally, the stereoCalibrate function of the opencv toolkit is used to perform stereo calibration processing on the first camera and the second camera, and the mapping matrix of the second camera to the first camera is obtained, and the internal parameters of the first camera, the internal parameters of the second camera, and the mapping matrix of the second camera to the first camera are obtained, which are the calibration parameters.

[0096] Furthermore, the rope coordinate system can also be established by the preset PnP algorithm. Specifically, a static vertical downward sling image is captured by the first camera, and the current rope lowering distance is recorded as L'. The rope coordinate system is established with the rope outlet at the top of the boom as the origin. The vertical downward direction of the rope is the positive direction of the z-axis of the rope coordinate system, the forward direction of the overhead crane is the positive direction of the y-axis of the rope coordinate system, and the horizontal swing plane of the boom is the x-axis plane of the rope coordinate system. Therefore, the coordinates of the target center point of the sling in the sling image in the rope coordinate system can be obtained as (0,0,L'). Similarly, the coordinates of multiple points on the sling plane in the rope coordinate system can be obtained according to the size of the sling. Integrate the pixel points in multiple sling images and the corresponding coordinate points in the rope coordinate system. Based on the PnP algorithm, the external parameter matrix of the rope coordinate system converted to the camera coordinate system of the first camera is calculated through the three-dimensional coordinates and two-dimensional projection coordinates of multiple pairs of points.

[0097] S202, collecting a first image through a preset first camera; and collecting a second image through a preset second camera; wherein both the first image and the second image include the same hanger at the angle to be measured.

[0098] For example, this step can be referred to Figure 1 Step 101 in the above is not described in detail.

[0099] S203, performing epipolar correction processing on the first camera and the second camera according to calibration parameters to obtain intermediate values; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include intrinsic parameters of the first camera, intrinsic parameters of the second camera, and a mapping matrix of the second camera to the first camera.

[0100] Exemplarily, the electronic device can transmit both the first image and the second image to the computing platform to perform distortion correction and epipolar correction to obtain aligned left and right images; wherein the computing platform can be a platform deployed on the electronic device or a third-party platform, and there is no limitation on this.

[0101] Specifically, based on the intrinsic parameters of the first camera, the intrinsic parameters of the second camera, and the mapping matrix of the second camera to the first camera, the stereoRectify function of the opencv toolkit is used to perform epipolar correction on the first camera and obtain an intermediate value, so as to align the first camera and the second camera epipolarly and mathematically align them to the same observation plane.

[0102] S204: Determine a projection matrix for distortion correction and a projection matrix for image alignment according to the intermediate value.

[0103] Exemplarily, the electronic device may calculate the projection matrix for distortion correction and the projection matrix for polar image alignment by combining the initUndistortRectifyMap function of the opencv toolkit and the intermediate values.

[0104] S205 , performing a mapping transformation on the first image and the second image according to the distortion-corrected projection matrix and the pole-aligned projection matrix to obtain the transformed first image and the second image.

[0105] Exemplarily, the electronic device can use the remap function of the opencv toolkit, the projection matrix of the distortion correction and the projection matrix of the epipolar alignment to perform a mapping transformation on the first image and the second image, obtain the transformed first image and the second image, and then complete the mapping transformation of the first image and the second image. It should be noted that the epipolar correction provides constraints for the subsequent corresponding point matching of the first image and the second image, so that the corresponding point matching is compressed from the search in the entire image to the search on a straight line, which greatly reduces the calculation time of the subsequent image stereo matching.

[0106] S206: Perform stereo matching processing on the first image and the second image to generate a depth map.

[0107] In one example, step S206 includes: using a preset image stereo matching algorithm to perform image stereo matching processing on points at corresponding positions on the first image and the second image to generate a minimum cost path; converting the minimum cost path into a disparity map; converting the disparity map according to a preset disparity formula to generate a depth map.

[0108] Exemplarily, the electronic device may use a preset image stereo matching algorithm to determine points at corresponding positions between the first image and the second image, and perform image stereo matching processing on the points at corresponding positions, construct a minimum cost path between matching pixel points of the first camera and the second camera, and convert the minimum cost path into a disparity map, where the disparity map is a single-channel image whose length and width dimensions are consistent with those of the two images. According to a preset disparity formula, each coordinate point in the disparity map is converted to obtain a depth map, and the preset disparity formula is as follows:

[0109]

[0110] Wherein, d represents the disparity value of a coordinate point in the disparity map; z represents the depth value of the corresponding coordinate in the depth map; f represents the focal length of the first camera, and b represents the baseline between the optical centers of the first camera and the second camera.

[0111] S207 . Perform target detection on the first image according to a preset target detection model to obtain two-dimensional pixel coordinates of the spreader.

[0112] In one example, step S207 includes: performing target detection on the first image according to a preset target detection model to obtain a detection frame including the sling, and determining a detection frame information vector of the detection frame; determining the center point coordinates of the detection frame information vector, and determining the center point coordinates as the two-dimensional pixel coordinates of the sling.

[0113] Exemplarily, the electronic device may input the first image into a preset target detection model for target detection, output a detection frame containing the sling in the first image, and determine a detection frame information vector of the detection frame; wherein the detection frame information vector is expressed as [u1, v1, u2, v2] T Indicates that u1, v1, u2, v2 in the vector represent the upper left corner horizontal coordinate value, upper left corner vertical coordinate value, lower right corner horizontal coordinate value and lower right corner vertical coordinate value respectively. Finally, determine the center point coordinate of the detection frame information vector, that is, the center point coordinate of the detection frame, and use the center point coordinate (u, v) as the two-dimensional pixel coordinate of the hanger; where,

[0114]

[0115] S208. Determine the three-dimensional coordinates of the spreader according to the two-dimensional pixel coordinates and the depth map.

[0116] In one example, step S208 includes: determining the depth value at the coordinate position corresponding to the two-dimensional pixel coordinate in the depth map; determining the coordinates of the sling in the camera coordinate system by using the projection relationship between the preset camera coordinate system and the image coordinate system; wherein the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system in which the depth map is located; according to the projection relationship between the preset lifting rope coordinate system and the camera coordinate system, performing projection transformation processing on the coordinates in the camera coordinate system to determine the three-dimensional coordinates of the sling; wherein the lifting rope coordinate system is the coordinate system in which the sling is located.

[0117] For example, for the two-dimensional pixel coordinates (u, v) of the spreader, the electronic device may determine the coordinate position corresponding to the two-dimensional pixel coordinates (u, v) in the depth map, and obtain the depth value z at the coordinate position. c Using the preset projection relationship from the camera coordinate system to the image coordinate system, the coordinates (x c ,y c ,z c ):

[0118]

[0119] Among them, f x 、f y, u0, v0 are all elements in the intrinsic parameters (i.e., the intrinsic parameter matrix) of the first camera obtained in step 201, respectively representing the horizontal focal length of the camera, the vertical focal length of the camera, the horizontal offset of the principal point, and the vertical offset; the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system of the depth map.

[0120] Finally, the coordinates (x c ,y c ,z c ) to perform projection transformation and obtain the three-dimensional coordinates (x l ,y l ,z l ) to achieve spatial positioning. The projection transformation formula is as follows:

[0121]

[0122] Among them, R 3×3 and T 3×1 They are the rotation matrix and translation vector of the rope coordinate system converted to the camera coordinate system of the first camera, obtained by the PnP algorithm of step 201; the rope coordinate system is the coordinate system where the sling is located. It should be noted that the formulas in this step are all preset camera model projection formulas.

[0123] S209: Determine the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates.

[0124] In one example, step S209 includes: performing modeling processing according to three-dimensional coordinates and preset standard coordinates to determine the swing angle of the spreader; wherein the standard coordinates are the coordinates of the spreader in a standard vertical state; the swing angle includes a pitch swing angle and a horizontal swing angle.

[0125] For example, Figure 4 A scenario diagram of a method for determining the swing angle of a spreader provided in this application Figure 2 ,like Figure 4 As shown, including the suspension rope coordinate system (X l ,Y l ,Z l ), camera coordinate system (X c ,Y c ,Z c ). First, assume that the pitch swing of the current hoist is the Y axis of the rope around the rope coordinate system. l The axis rotates α degrees, and the horizontal swing is the X axis of the rope around the rope coordinate system. l The axis rotates β degrees, and the rotation matrix R of the overall swing of the spreader can be expressed as follows:

[0126]

[0127] Here, Rot(·) is a rotation function. Based on this, the three-dimensional coordinates of the current sling and the preset standard coordinates in the ideal vertical state are modeled in the rope coordinate system. w ,y w ,z w ), we can establish the following equation:

[0128]

[0129] Where L is the current length of the spreader, [0 0L] T It is expressed as the coordinate vector of the standard coordinate of the spreader in the ideal vertical state. Therefore, the pitching angle α and the horizontal swing angle β of the spreader can be calculated by the following formula:

[0130]

[0131] The swing angle determination method of the sling provided in the embodiment of the present application performs calibration processing on the first camera and the second camera to determine the calibration parameters; wherein the calibration parameters include the internal parameters of the first camera, the internal parameters of the second camera, and the mapping matrix of the second camera to the first camera. The first image is acquired through the preset first camera; and the second image is acquired through the preset second camera; wherein the first image and the second image both include the same sling of the angle to be measured. According to the calibration parameters, the first camera and the second camera are subjected to epipolar correction processing to obtain an intermediate value; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include the internal parameters of the first camera, the internal parameters of the second camera, and the mapping matrix of the second camera to the first camera. According to the intermediate value, the projection matrix of distortion correction and the projection matrix of epipolar alignment are determined. According to the projection matrix of distortion correction and the projection matrix of epipolar alignment, the first image and the second image are mapped and transformed to obtain the transformed first image and the second image. The first image and the second image are subjected to image stereo matching processing to generate a depth map. According to the preset target detection model, the first image is subjected to target detection to obtain the two-dimensional pixel coordinates of the sling. The three-dimensional coordinates of the spreader are determined based on the two-dimensional pixel coordinates and the depth map. The swing angle of the spreader is determined based on the three-dimensional coordinates and the preset standard coordinates. Therefore, only two cameras are used to accurately detect the swing angle of the spreader with a long lowering distance. Compared with the existing methods, this application does not need to use expensive detection equipment. The detection equipment used is low-cost, and the angle measurement is safe, reliable, efficient and accurate. It provides accurate spreader swing angles for subsequent overhead crane spreader swing reduction and anti-sway, realizes automated angle detection, has high practicality, and is used to achieve the effect of improving the measurement accuracy of the spreader's swing angle.

[0132] In one example, according to Figure 5 The above embodiments are described in detail.

[0133] Figure 5 A schematic diagram of a method for determining the swing angle of a sling provided in this application Figure 3 ,like Figure 5 As shown, the first camera and the second camera take a damped binocular camera as an example, the first image captured by the first camera is the left image, and the second image captured by the second camera is the right image; the method includes:

[0134] Step 1: Install the camera and build the damped binocular camera on the top of the boom of the overhead crane; calibrate the camera to obtain the intrinsic parameters of the first camera and the second camera; establish the rope coordinate system through the preset PnP algorithm.

[0135] Step 2: Image acquisition: The damped binocular camera collects the left image and the right image vertically downward; distortion correction and epipolar correction are performed to obtain the aligned left image and the right image.

[0136] Step 3: Perform image stereo matching on the left image and the right image to obtain a depth map.

[0137] Step 4: Use the preset target detection model to detect the two-dimensional pixel coordinates of the spreader in the left image, use the depth value at the coordinate corresponding to the two-dimensional pixel coordinate in the depth map, combine the intrinsic parameters, extrinsic parameters of the first camera and the second camera, and the preset camera model projection formula to obtain the three-dimensional coordinates of the current spreader and achieve spatial positioning.

[0138] Step 5: Modeling is performed based on the three-dimensional coordinates of the spreader and the preset standard coordinates in the ideal vertical state to determine the current pitching and horizontal swing angles of the spreader.

[0139] Figure 6 A structural schematic diagram of a swing angle determination device for a sling provided in this application, such as Figure 6 As shown, the swing angle determination device 30 of the spreader provided in this embodiment includes:

[0140] The acquisition module 31 is used to acquire a first image through a preset first camera; and to acquire a second image through a preset second camera; wherein both the first image and the second image include the same sling at the angle to be measured.

[0141] The first determination module 32 is used to determine the three-dimensional coordinates of the spreader according to the first image and the second image; wherein the three-dimensional coordinates are the actual coordinates of the spreader.

[0142] The second determination module 33 is used to determine the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates.

[0143] Figure 7A schematic diagram of the structure of another device for determining the swing angle of a sling provided in an embodiment of the present application, wherein Figure 6 Based on the embodiment shown, Figure 7 As shown, the first determination module 32 includes:

[0144] The matching unit 321 is used to perform stereo matching processing on the first image and the second image to generate a depth map.

[0145] The first determination unit 322 is used to perform target detection on the first image according to a preset target detection model to obtain the two-dimensional pixel coordinates of the spreader.

[0146] The second determining unit 323 is used to determine the three-dimensional coordinates of the spreader according to the two-dimensional pixel coordinates and the depth map.

[0147] In a possible implementation, the matching unit 321 is specifically configured to:

[0148] Using a preset image stereo matching algorithm, points at corresponding positions on the first image and the second image are processed for image stereo matching to generate a minimum cost path.

[0149] Convert the minimum cost path into a disparity map.

[0150] The disparity map is converted according to a preset disparity formula to generate a depth map.

[0151] In a possible implementation, the first determining unit 322 is specifically configured to:

[0152] Perform object detection on the first image according to a preset object detection model to obtain a detection frame including the sling, and determine a detection frame information vector of the detection frame.

[0153] The center point coordinates of the detection frame information vector are determined, and the center point coordinates are determined to be the two-dimensional pixel coordinates of the spreader.

[0154] In a possible implementation manner, the second determining unit 323 is specifically configured to:

[0155] A depth value at a coordinate position corresponding to the two-dimensional pixel coordinate is determined in the depth map.

[0156] The coordinates of the sling in the camera coordinate system are determined by using the projection relationship between the preset camera coordinate system and the image coordinate system; wherein the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system where the depth map is located.

[0157] According to the projection relationship between the preset rope coordinate system and the camera coordinate system, the coordinates in the camera coordinate system are projected and transformed to determine the three-dimensional coordinates of the sling; wherein the rope coordinate system is the coordinate system in which the sling is located.

[0158] In a possible implementation, the second determining module 33 is specifically configured to:

[0159] Modeling is performed according to the three-dimensional coordinates and the preset standard coordinates to determine the swing angle of the spreader; wherein the standard coordinates are the coordinates of the spreader in a standard vertical state; the swing angle includes a pitch swing angle and a horizontal swing angle.

[0160] In a possible implementation, the device is further specifically used for:

[0161] Before acquiring a first image through a preset first camera and acquiring a second image through a preset second camera, the first camera and the second camera are calibrated to determine calibration parameters; wherein the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera.

[0162] In a possible implementation, the device is further specifically used for:

[0163] Before determining the three-dimensional coordinates of the sling according to the first image and the second image, the first camera and the second camera are subjected to epipolar correction processing according to calibration parameters to obtain intermediate values; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include the intrinsic parameters of the first camera, the intrinsic parameters of the second camera, and the mapping matrix of the second camera to the first camera.

[0164] Based on the intermediate values, the projection matrix for distortion correction and the projection matrix for image alignment are determined.

[0165] According to the distortion-corrected projection matrix and the polar-image-aligned projection matrix, the first image and the second image are mapped and transformed to obtain the transformed first image and the second image.

[0166] The device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0167] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 50 provided in this embodiment includes: at least one processor 501 and a memory 502. Optionally, the device 50 also includes a communication component 503. The processor 501, the memory 502 and the communication component 503 are connected via a bus 504.

[0168] In a specific implementation process, at least one processor 501 executes the computer-executable instructions stored in the memory 502, so that at least one processor 501 executes the above method.

[0169] The specific implementation process of the processor 501 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0170] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0171] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk storage.

[0172] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0173] An embodiment of the present application also provides an overhead lifting device, including a lifting device and a swing angle detection device; wherein the swing angle detection device is used to detect and determine the swing angle of the lifting device; the detection method of the swing angle detection device is the swing angle determination method of the lifting device in the above-mentioned embodiment.

[0174] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0175] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0176] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0177] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0178] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0179] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0180] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0181] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0182] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0183] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for determining the swing angle of a spreader, characterized in that: include: Capturing a first image through a preset first camera; and collecting a second image through a preset second camera; wherein the first image and the second image both include the same sling at the angle to be measured; Determine the three-dimensional coordinates of the spreader according to the first image and the second image; wherein the three-dimensional coordinates are actual coordinates of the spreader; The swing angle of the hanger is determined according to the three-dimensional coordinates and the preset standard coordinates.

2. The method according to claim 1, characterized in that Determining the three-dimensional coordinates of the spreader according to the first image and the second image includes: Performing image stereo matching processing on the first image and the second image to generate a depth map; Performing target detection on the first image according to a preset target detection model to obtain two-dimensional pixel coordinates of the spreader; The three-dimensional coordinates of the spreader are determined according to the two-dimensional pixel coordinates and the depth map.

3. The method according to claim 2, characterized in that Performing image stereo matching processing on the first image and the second image to generate a depth map includes: Using a preset image stereo matching algorithm, performing image stereo matching processing on points at corresponding positions on the first image and the second image to generate a minimum cost path; Converting the minimum cost path into a disparity map; The disparity map is converted according to a preset disparity formula to generate a depth map.

4. The method according to claim 2, characterized in that: The performing target detection on the first image according to a preset target detection model to obtain the two-dimensional pixel coordinates of the spreader includes: Performing target detection on the first image according to a preset target detection model to obtain a detection frame including the sling, and determining a detection frame information vector of the detection frame; The center point coordinates of the detection frame information vector are determined, and the center point coordinates are determined to be the two-dimensional pixel coordinates of the hanger.

5. The method according to claim 2, characterized in that: Determining the three-dimensional coordinates of the spreader according to the two-dimensional pixel coordinates and the depth map includes: Determining in the depth map a depth value at a coordinate position corresponding to the two-dimensional pixel coordinate; Determine the coordinates of the sling in the camera coordinate system by using a projection relationship between a preset camera coordinate system and an image coordinate system; wherein the camera coordinate system is the coordinate system of the first camera, and the image coordinate system is the coordinate system in which the depth map is located; According to the projection relationship between the preset lifting rope coordinate system and the camera coordinate system, the coordinates in the camera coordinate system are projected and transformed to determine the three-dimensional coordinates of the lifting device; wherein the lifting rope coordinate system is the coordinate system in which the lifting device is located.

6. The method according to claim 1, characterized in that Determining the swing angle of the spreader according to the three-dimensional coordinates and the preset standard coordinates includes: Modeling is performed according to the three-dimensional coordinates and preset standard coordinates to determine the swing angle of the spreader; wherein the standard coordinates are the coordinates of the spreader in a standard vertical state; and the swing angle includes a pitch swing angle and a horizontal swing angle.

7. The method according to any one of claims 1 to 6, characterized in that: Capturing a first image through a preset first camera; Before acquiring the second image by the preset second camera, the method further includes: The first camera and the second camera are calibrated to determine calibration parameters, wherein the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera.

8. The method according to any one of claims 1 to 6, characterized in that: Before determining the three-dimensional coordinates of the spreader according to the first image and the second image, the method further includes: According to calibration parameters, epipolar correction processing is performed on the first camera and the second camera to obtain intermediate values; wherein the calibration parameters are obtained by calibrating the first camera and the second camera, and the calibration parameters include an intrinsic parameter of the first camera, an intrinsic parameter of the second camera, and a mapping matrix of the second camera to the first camera; Determining a projection matrix for distortion correction and a projection matrix for polar image alignment according to the intermediate value; According to the distortion-corrected projection matrix and the polar-aligned projection matrix, the first image and the second image are mapped and transformed to obtain the transformed first image and the second image.

9. An overhead hanging device, characterized in that: It comprises a spreader and a swing angle detection device; wherein the swing angle detection device is used to determine the swing angle of the spreader; and the detection method of the swing angle detection device is the swing angle determination method of the spreader as described in any one of claims 1-8.

10. An electronic device / computer-readable storage medium, characterized in that: The electronic device comprises: a memory, a processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory, so that the processor executes the method according to any one of claims 1 to 8; and / or, The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 8 when executed by a processor.

Images