A global three-dimensional reconstruction method and device for metallurgical storage area based on depth vision

Abstract

The present invention provides a method and device for global three-dimensional reconstruction of a metallurgical storage area based on depth vision, and relates to the field of intelligent warehousing and logistics technology. It includes: taking the original image and depth image captured by the camera as input, calculating the inter-frame pose conversion relationship by matching the feature points between adjacent images and combining the depth information, thereby splicing the local spatial three-dimensional information collected at different positions. By using the global three-dimensional reconstruction system and method of the metallurgical storage area that integrates deep vision, the present invention can obtain the global spatial three-dimensional information of the metallurgical storage area by splicing the spatial three-dimensional information, so as to perform a more complete measurement of the stack.

Description

Technical Field

[0001] The present invention relates to the field of intelligent warehousing and logistics technology, and in particular to a method and device for global three-dimensional reconstruction of a metallurgical storage area based on depth vision. Background Art

[0002] The positioning of the crane in the unmanned crane garage area mainly relies on the Gray busbar measurement device, and the current crane position is obtained through the PLC feedback callback system during the crane movement. In the prior art, the detection information obtained by fixed position detection for the three-dimensional reconstruction of stacks and vehicles in the metallurgical storage area with a large field of view is extremely limited. Summary of the invention

[0003] In view of the problem in the prior art that for the three-dimensional reconstruction of stacks and vehicles in a metallurgical storage area with a large field of view, the detection information obtained by using fixed position detection is extremely limited, the present invention proposes a global three-dimensional reconstruction method and device for a metallurgical storage area based on depth vision.

[0004] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0005] On the one hand, a global 3D reconstruction method of a metallurgical storage area based on deep vision is provided, including:

[0006] S1: Collect original images of consecutive frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; respectively detect ORB feature points in two adjacent frames of original images;

[0007] S2: Match the ORB feature points between two adjacent frames of original images to obtain initial matching point pairs; filter the initial matching point pairs to obtain feature point pair matching information and feature point pair pixel coordinates;

[0008] S3: extracting depth information of the depth image corresponding to the original image through the depth image sequence; combining the pixel coordinates of the matched feature point pairs and the corresponding depth information to obtain local spatial three-dimensional information;

[0009] S4: Calculate the inter-frame pose transformation matrix of the original image;

[0010] S5: Repeat S1-S4 until all inter-frame pose transformation matrices in the original image sequence are calculated; the local spatial three-dimensional information collected at different positions is combined with the pose transformation matrix to form global spatial three-dimensional information;

[0011] S6: The coordinates of the three-dimensional information of the global space are converted to the crane coordinate system in the storage area through the coordinate system, and provided to the crane for use.

[0012] Optionally, in step S1, the depth camera is used to collect original images of consecutive frames and depth images corresponding to the original images to form an original image sequence and a depth image sequence; and the ORB feature points in two adjacent frames of original images are respectively detected, including:

[0013] S11: The depth camera disposed on the overhead crane moves with the overhead crane to collect original images and depth images of consecutive frames, thereby forming an original image sequence and a depth image sequence;

[0014] S12: Select the current original image and the previous original image to extract ORB feature points, and obtain the pixel coordinates of the ORB feature points;

[0015] S13: With the ORB feature point as the center, the window size is selected as 13 pixels, 128 comparison point pairs are selected in the window, and the binary code string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel in the window through formula (1):

[0016]

[0017] Among them, I(p), I(q) represent the grayscale values ​​of the center pixel and the pixels in the window; N p Indicates the pixel points contained in the neighborhood of the central pixel within the set window size;

[0018] If I(q)≥I(p), the binary code string is recorded as 0, and if I(q)<I(p), the binary code string is recorded as 1.

[0019] Optionally, in step S2, ORB feature points between two adjacent frames of original images are matched to obtain initial matching point pairs; the initial matching point pairs are screened to obtain feature point pair matching information and feature point pair pixel coordinates, including:

[0020] S21: Match the ORB feature points detected in the current original image and the previous original image to form a matching point pair; use the Hamming distance percentage to measure the consistency of the matching point pair;

[0021] S22: Using the random consensus algorithm RANSAC, the initial matching point pair obtained in S21 is substituted into formula (2):

[0022]

[0023] Where (x', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position in the previous frame of the original image; s represents the scale parameter;

[0024] Determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers;

[0025] S23: Collect the position coordinate information of the overhead crane in different images and at different times, calculate the movement direction of the overhead crane, and remove the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the overhead crane; obtain the final feature point pair matching information and the pixel coordinates of the feature point pairs.

[0026] Optionally, in step S3, the depth information of the depth image corresponding to the original image is extracted through the depth image sequence; the local space three-dimensional information is obtained by combining the pixel coordinates of the matched feature point pairs and the corresponding depth information, including:

[0027] S31: extracting a depth value D of a depth image corresponding to the original image through the depth image sequence;

[0028] S32: According to the pixel coordinates of the feature point obtained in step S2, combined with the depth value D, the pixel coordinates of the feature point and the depth value D are substituted into formula (3) to calculate the local spatial three-dimensional coordinates of the feature point and obtain the local spatial three-dimensional information:

[0029]

[0030] Where: (u, v) represents the pixel coordinates of the feature point; C x ,C y are the camera calibration parameters, which represent the position coordinates of the principal point of the image; f x ,f y is the camera calibration parameter, f x represents the component of focal length along the X-axis in the image pixel coordinate system; f y It represents the component of focal length along the Y axis in the image pixel coordinate system; X, Y, D represent the calculated three-dimensional coordinates in space.

[0031] Optionally, in step S4, calculating the inter-frame pose transformation matrix of the original image includes:

[0032] S41: construct the spatial point set error term between the current original image and the previous frame original image, and calculate a set of spatial transformation matrices with the minimum spatial position error through the singular value decomposition SVD of formula (4).

[0033]

[0034] Where: p i Represents the spatial three-dimensional point set obtained after feature point matching of the current original image, q i It represents the spatial three-dimensional point set obtained after feature point matching of the previous frame of the original image; E iRepresents the set of spatial position errors of feature point pairs corresponding to the spatial transformation matrix [R|t]; R represents the spatial rotation matrix that minimizes the spatial position error, and t represents the spatial transformation matrix;

[0035] S42: Will E i Convert to quaternion format for storage and obtain the inter-frame pose transformation matrix.

[0036] Optionally, in step S5, the local spatial three-dimensional information collected at different positions is combined with the posture transformation matrix to form the global spatial three-dimensional information, including:

[0037] Combine the local spatial 3D information of images at different times with the inter-frame pose transformation matrix; use the spatial transformation matrix to stitch the spatial 3D information through formula (5) to form the global spatial 3D information:

[0038] P i =T*P' (5)

[0039] Where: P i Represents the set of three-dimensional points in the current original image, P′ i Represents the set of spatial three-dimensional points in the previous frame image; T represents the calculated inter-frame pose transformation matrix.

[0040] Optionally, in step S6, the coordinates in the global space three-dimensional information are converted to the warehouse area crane coordinate system through the coordinate system, and provided to the crane for use, including:

[0041] By obtaining the position coordinate information of the same detection point in the same scene, the coordinate system is fitted and the coordinate transformation relationship is calculated; the coordinates in the global space three-dimensional information are converted to the warehouse area coordinate system and provided to the overhead crane for use.

[0042] On the one hand, a global three-dimensional reconstruction device for a metallurgical storage area based on depth vision is provided, and the device is applied to any of the above methods, and the device comprises:

[0043] The ORB detection module is used to collect the original images of continuous frames and the depth images corresponding to the original images through the depth camera to form the original image sequence and the depth image sequence; respectively detect the ORB feature points in two adjacent frames of the original image;

[0044] The similarity matching module is used to match the ORB feature points between two adjacent frames of original images to obtain the initial matching point pairs; the initial matching point pairs are screened to obtain the matching information of the feature point pairs and the pixel coordinates of the feature point pairs;

[0045] The local information calculation module is used to extract the depth information of the depth image corresponding to the original image through the depth image sequence; combine the pixel coordinates of the matched feature point pairs and the corresponding depth information to obtain the local space three-dimensional information;

[0046] A posture transformation calculation module, used to calculate the inter-frame posture transformation matrix;

[0047] The global information stitching module is used to calculate the inter-frame pose transformation matrix of all frames in the original image sequence; the local spatial 3D information collected at different positions is combined with the pose transformation matrix for stitching to form the global spatial 3D information;

[0048] The coordinate system conversion module is used to convert the coordinates in the global space three-dimensional information into the warehouse area crane coordinate system through the coordinate system, and provide it to the crane.

[0049] Optionally, the ORB detection module is used to collect original images and depth images of continuous frames through a depth camera arranged on the overhead crane as the overhead crane moves, thereby forming an original image sequence and a depth image sequence;

[0050] Select the current original image and the previous original image to extract ORB feature points and obtain the pixel coordinates of the ORB feature points;

[0051] After determining the pixel coordinate position of the ORB feature point, the window size is selected as 13 pixels with the ORB feature point as the center, and 128 comparison point pairs are selected in the window. The binary code string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel in the window.

[0052] Optionally, a similarity matching module is used to match the ORB feature points detected in the current original image and the previous original image to form a matching point pair; and the consistency of the matching point pair is measured using the Hamming distance percentage;

[0053] Select the matching point pairs in the binary code string whose encoding XOR value is less than the Hamming distance percentage threshold, and set this group of feature points as the initial matching point pair:

[0054] The random consensus algorithm RANSAC is used to determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers.

[0055] The position coordinate information of the overhead crane in different images and at different times is collected, the movement direction of the overhead crane is calculated, and the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the overhead crane are eliminated; the final feature point pair matching information and the pixel coordinates of the feature point pairs are obtained.

[0056] The above technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0057] In the above scheme, the global three-dimensional reconstruction method and device of the metallurgical storage area integrating depth vision adopted by the present invention collects the original image and depth image covering the entire metallurgical storage area in real time by means of surface scanning, and obtains the global spatial three-dimensional information of the metallurgical storage area by means of spatial three-dimensional information splicing, thereby achieving a more comprehensive measurement effect of the stack. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0059] Figure 1 It is a flow chart of a method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision provided by an embodiment of the present invention;

[0060] Figure 2 It is a feature point detection flow chart of a global three-dimensional reconstruction method of a metallurgical storage area based on deep vision provided by an embodiment of the present invention;

[0061] Figure 3 It is a feature point matching flow chart of a global three-dimensional reconstruction method of a metallurgical storage area based on depth vision provided by an embodiment of the present invention;

[0062] Figure 4 This is the effect of the feature point matching algorithm of the global three-dimensional reconstruction method of the metallurgical storage area based on deep vision provided by the embodiment of the present invention;

[0063] Figure 5 It is a flow chart of local space three-dimensional information calculation of a global three-dimensional reconstruction method of a metallurgical storage area based on depth vision provided by an embodiment of the present invention;

[0064] Figure 6 It is a flow chart of inter-frame pose transformation matrix calculation of a global three-dimensional reconstruction method of a metallurgical storage area based on deep vision provided by an embodiment of the present invention;

[0065] Figure 7 It is a device block diagram of a global three-dimensional reconstruction device for a metallurgical storage area based on depth vision provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the technical problems, technical solutions and advantages to be solved by the present invention more clear, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0067] like Figure 1 As shown, an embodiment of the present invention provides a global three-dimensional reconstruction method of a metallurgical storage area based on depth vision, comprising:

[0068] S1: The depth camera is used to collect original images of continuous frames and depth images corresponding to the original images to form an original image sequence and a depth image sequence; the ORB (Oriented FAST and Rotated BRIEF, fast feature point extraction and description algorithm) feature points in two adjacent original image frames are detected respectively.

[0069] S2: Match the ORB feature points between two adjacent frames of original images to obtain initial matching point pairs; filter the initial matching point pairs to obtain feature point pair matching information and feature point pair pixel coordinates.

[0070] S3: extracting the depth information of the depth image corresponding to the original image through the depth image sequence; combining the pixel coordinates of the matched feature point pairs and the corresponding depth information to obtain the local space three-dimensional information.

[0071] S4: Calculate the inter-frame pose transformation matrix of the original image.

[0072] S5: Repeat S1-S4 until all inter-frame pose transformation matrices in the original image sequence are calculated; the local spatial three-dimensional information collected at different positions is combined with the pose transformation matrix to form the global spatial three-dimensional information.

[0073] S6: The coordinates of the three-dimensional information of the global space are converted to the crane coordinate system in the storage area through the coordinate system, and provided to the crane for use.

[0074] In this embodiment, by using a global three-dimensional reconstruction system and method for a metallurgical storage area that integrates depth vision, the global spatial three-dimensional information of the metallurgical storage area can be obtained by splicing spatial three-dimensional information, thereby performing a more complete measurement of the stacks.

[0075] like Figure 2 As shown, in step S1, the depth camera collects the original images of consecutive frames and the depth images corresponding to the original images to form an original image sequence and a depth image sequence; and detects the ORB feature points in two adjacent frames of the original images respectively, including:

[0076] S11: The depth camera disposed on the overhead crane moves with the overhead crane to collect original images and depth images of continuous frames, thereby forming an original image sequence and a depth image sequence.

[0077] S12: Select the current original image and the previous original image to extract ORB feature points to obtain the pixel coordinates of the ORB feature points.

[0078] S13: With the ORB feature point as the center, the window size is selected as 13 pixels, 128 comparison point pairs are selected in the window, and the binary code string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel in the window through formula (1):

[0079]

[0080] Among them, I (p) Represents the gray value of the center pixel, I (q) Represents the gray value of the pixel in the window; N p Indicates the pixel points contained in the neighborhood of the central pixel within the set window size;

[0081] If I(q)≥I(p), the binary code string is recorded as 0, and if I(q)<I(p), the binary code string is recorded as 1.

[0082] In this embodiment, it is assumed that the camera used is an area array CCD camera that can simultaneously acquire an original image and a depth image.

[0083] By calibrating the camera parameters and collecting the depth image information, the correspondence between the image pixel plane and the spatial three-dimensional information can be established. In this embodiment, the intrinsic parameter matrix of the depth camera can be expressed as:

[0084]

[0085] Among them, f x ,f y is the camera calibration parameter, f x represents the component of focal length along the X-axis in the image pixel coordinate system; f y Represents the component of focal length along the Y axis in the image pixel coordinate system; C x ,C y Represent the camera calibration parameters, and respectively represent the position coordinates of the principal point of the image.

[0086] In this embodiment, the intrinsic calibration result of the depth camera is:

[0087]

[0088] For ORB feature point extraction: first select the current original image and the previous frame original image to perform ORB feature point extraction to obtain the pixel coordinates (u, v) of the feature points.

[0089] In this embodiment, corresponding to the central pixel point p, if 16 neighboring pixel points q on a circle with a radius of 3 are selected, if the conditions in the neighborhood are met and can be used as feature points, the judgment expression is:

[0090]

[0091] Where N represents the number of pixels that match; λ d Represents the pixel difference threshold.

[0092] In this embodiment, the number of matching pixels is set to 12, and the pixel difference threshold is set to 15.

[0093] In this embodiment, the ORB feature points in each frame image are extracted, and the number of ORB feature points in each frame image is shown in Table 1:

[0094] Table 1

[0095]

[0096]

[0097] like Figure 3 As shown, in step S2, ORB feature points between two adjacent frames of original images are matched to obtain initial matching point pairs; the initial matching point pairs are screened to obtain feature point pair matching information and feature point pair pixel coordinates, including:

[0098] S21: Match the ORB feature points detected in the current original image and the previous original image to form matching point pairs; use the Hamming distance percentage to measure the consistency of the matching point pairs.

[0099] S22: Substitute the initial matching point pair obtained in S21 into formula (2) through RANSAC (RANdom SAmple Consensus):

[0100]

[0101] Among them, (x', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position in the previous frame image; s represents the scale parameter.

[0102] Determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers.

[0103] S23: Collect the position coordinate information of the overhead crane in different images and at different times, calculate the movement direction of the overhead crane, and remove the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the overhead crane; obtain the final feature point pair matching information and the pixel coordinates of the feature point pairs.

[0104] In this embodiment, the Hamming distance percentage threshold is set to 0.6, and the feature points in the current original image and the previous original image are matched, and the consistency of the feature points is measured using the Hamming distance percentage:

[0105] η=Hamming(Sl (x,y),S r (x',y'))

[0106] If the XOR value of the descriptor code is less than a certain threshold, the group of feature points is determined to be a matching point pair;

[0107] Among them, S l (x, y) represents the ORB feature point descriptor of the current original image, S r (x′, y′) represents the ORB feature point descriptor of the previous frame image; η represents the Hamming distance percentage threshold.

[0108] In this embodiment, the number of ORB feature point pairs initially matched by Hamming distance is shown in Table 2:

[0109] Table 2

[0110]

[0111]

[0112] In this embodiment, RANSAC is used to iteratively bring the initial matching point pairs into the matrix, determine the optimal transformation matrix that satisfies the maximum number of matching point pairs, and remove outliers; the final feature point pair matching information and feature point pair pixel coordinates are obtained through motion trend constraints. In this embodiment, the ORB feature point matching algorithm based on motion trend constraints is as follows: Figure 4 shown.

[0113] In this embodiment, the number of completely consistent ORB feature point matching pairs obtained after RANSAC random consistency constraints and motion trend constraints is shown in Table 3:

[0114] Table 3

[0115] Matching images Matching feature points (pairs) Matching images Matching feature points (pairs) 1-2 22 11-12 10 2-3 20 12-13 18 3-4 22 13-14 20 4-5 30 14-15 18 5-6 32 15-16 41 6-7 11 16-17 54 7-8 9 17-18 85 8-9 6 18-19 99 9-10 10 19-20 180 10-11 8

[0116] like Figure 5 As shown, in step S3, the depth information of the depth image corresponding to the original image is extracted through the depth image sequence; the local space three-dimensional information is obtained by combining the pixel coordinates of the matched feature point pairs and the corresponding depth information, including:

[0117] S31: extracting a depth value D of a depth image corresponding to the original image through the depth image sequence.

[0118] S32: According to the pixel coordinates of the feature point obtained in step S2, combined with the depth value D, the pixel coordinates of the feature point and the depth value D are substituted into formula (3) to calculate the local spatial three-dimensional coordinates of the feature point and obtain the local spatial three-dimensional information:

[0119]

[0120] Where: (u, v) represents the pixel coordinates of the feature point; C x ,C y are the camera calibration parameters, which represent the position coordinates of the principal point of the image; f x ,f y is the camera calibration parameter, f x represents the component of focal length along the X-axis in the image pixel coordinate system; f y It represents the component of focal length along the Y axis in the image pixel coordinate system; X, Y, D represent the calculated three-dimensional coordinates in space.

[0121] In this embodiment, the Rodriguez formula is used to convert the space transformation matrix into a quaternion for storage.

[0122] The inter-frame pose transformation values ​​between each group of images can be expressed using quaternions as shown in Table 4:

[0123] Table 4

[0124]

[0125]

[0126] like Figure 6 As shown, in step S4, the inter-frame pose transformation matrix of the original image is calculated, including:

[0127] S41: construct the spatial point set error term between the current original image and the previous frame original image, and calculate a set of spatial transformation matrices with the minimum spatial position error through formula (4) SVD (Singular Value Decomposition).

[0128]

[0129] Where: p i Represents the spatial three-dimensional point set obtained after feature point matching of the current original image, q i It represents the spatial three-dimensional point set obtained after feature point matching of the previous frame of the original image; E i Represents the set of spatial position errors of feature point pairs corresponding to the spatial transformation matrix [R|t]; R represents the spatial rotation matrix that minimizes the spatial position error, and t represents the spatial transformation matrix.

[0130] S42: Will E i Convert to quaternion format for storage and obtain the inter-frame pose transformation matrix.

[0131] In step S5, the local spatial three-dimensional information collected at different positions is combined with the posture transformation matrix to form the global spatial three-dimensional information, including:

[0132] Combine the local spatial 3D information of images at different times with the inter-frame pose transformation matrix; use the spatial transformation matrix to stitch the spatial 3D information through formula (5) to form the global spatial 3D information:

[0133] P i =T*P' (5)

[0134] Where: P i Represents the set of spatial three-dimensional points in the current original image, P i ′ represents the set of spatial three-dimensional points in the previous frame image; T represents the calculated inter-frame pose transformation matrix.

[0135] In this embodiment, the image acquisition resolution is 1280×720, and the maximum number of spatial three-dimensional points that can be obtained by a single image is 921600. The corresponding actual coverage area calculated according to the installation position height is 10m×7m.

[0136] The 3D points of the controls in each group of images are spliced ​​together, and the spatial 3D point information under each group of images is statistically shown in Table 5:

[0137] Table 5

[0138]

[0139]

[0140] In this embodiment, the global point cloud image obtained by splicing spatial three-dimensional information is used to increase the detection coverage to 20m×7m.

[0141] In step S6, the global space three-dimensional information is converted to the warehouse area crane coordinate system through the coordinate system and provided to the crane for use, including:

[0142] By obtaining the position coordinate information of the same detection point in the same scene, the coordinate system is fitted and the coordinate transformation relationship is calculated; the global space three-dimensional information is converted to the warehouse area coordinate system and provided to the overhead crane for use.

[0143] In this embodiment, by acquiring the coordinate information of the same detection point of the visual detection system and the spatial detection system in the same scene, the coordinate system is fitted and the coordinate transformation relationship is calculated. Finally, the detection results based on depth vision are converted to the warehouse area coordinate system and provided to the overhead crane for operation.

[0144] In this embodiment, the coordinate values ​​of the same points in the first image and the warehouse area coordinate system in the continuous image sequence captured by the depth camera in their respective coordinate systems are obtained, as shown in Table 6:

[0145] Table 6

[0146]

[0147]

[0148] In this embodiment, the spatial transformation matrix fitting result of the visual coordinate system and the warehouse coordinate system is:

[0149]

[0150] The embodiment of the present invention provides a device 700 for global three-dimensional reconstruction of a metallurgical storage area based on depth vision, and the device 700 is used to implement the above-mentioned method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision, such as Figure 7 As shown, the device comprises:

[0151] The ORB detection module 701 is used to collect original images of consecutive frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; and respectively detect ORB feature points in two adjacent frames of original images;

[0152] The similarity matching module 702 is used to match the ORB feature points between two adjacent frames of original images to obtain initial matching point pairs; screen the initial matching point pairs to obtain feature point pair matching information and feature point pair pixel coordinates.

[0153] The local information calculation module 703 is used to extract the depth information of the depth image corresponding to the original image through the depth image sequence; and obtain the local spatial three-dimensional information by combining the pixel coordinates of the matched feature point pairs and the corresponding depth information.

[0154] The posture transformation calculation module 704 is used to calculate the inter-frame posture transformation matrix.

[0155] The global information stitching module 705 is used to calculate all inter-frame pose transformation matrices in the original image sequence; stitch the local spatial three-dimensional information collected at different positions in combination with the pose transformation matrix to form global spatial three-dimensional information.

[0156] The coordinate system conversion module 706 is used to convert the global space three-dimensional information into the warehouse area crane coordinate system through the coordinate system, and provide it to the crane.

[0157] In this embodiment, a depth camera is mounted on an overhead crane and connected to a ground server via Ethernet. As the overhead crane moves, original images and depth images covering the entire metallurgical storage area are collected in real time in a surface scanning manner.

[0158] The ORB detection module 701 is used to collect original images and depth images of consecutive frames through a depth camera arranged on the overhead crane as the overhead crane moves, thereby forming an original image sequence and a depth image sequence;

[0159] Select the current original image and the previous original image to extract ORB feature points and obtain the pixel coordinates of the ORB feature points;

[0160] After determining the pixel coordinate position of the ORB feature point, the window size is selected as 13 pixels with the ORB feature point as the center, and 128 comparison point pairs are selected in the window. The binary code string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel in the window.

[0161] The similarity matching module 702 is used to match the ORB feature points detected in the current original image and the previous original image to form a matching point pair; the consistency of the matching point pair is measured using the Hamming distance percentage;

[0162] Select the matching point pairs in the binary code string whose encoding XOR value is less than the Hamming distance percentage threshold, and set this group of feature points as the initial matching point pair:

[0163] The random consensus algorithm RANSAC is used to determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers.

[0164] The position coordinate information of the overhead crane in different images and at different times is collected, the movement direction of the overhead crane is calculated, and the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the overhead crane are eliminated; the final feature point pair matching information and the pixel coordinates of the feature point pairs are obtained.

[0165] The present invention uses a global three-dimensional reconstruction system and method for a metallurgical storage area that integrates depth vision, and can obtain global spatial three-dimensional information of the metallurgical storage area by splicing spatial three-dimensional information, thereby performing a more complete measurement of the stack.

[0166] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0167] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A global three-dimensional reconstruction method for metallurgical storage area based on deep vision, characterized in that: include: S1: Collecting original images of continuous frames and depth images corresponding to the original images through a depth camera to form an original image sequence and a depth image sequence; Detect the ORB feature points in two adjacent frames of original images respectively; In the step S1, the depth camera collects original images of consecutive frames and depth images corresponding to the original images to form an original image sequence and a depth image sequence; Detect ORB feature points in two adjacent frames of original images respectively, including: S11: The depth camera disposed on the overhead crane moves with the overhead crane to collect original images and depth images of consecutive frames, thereby forming an original image sequence and a depth image sequence; S12: Select the current original image and the previous original image to extract ORB feature points, and obtain the pixel coordinates of the ORB feature points; S13: With the ORB feature point as the center, the window size is selected as 13 pixels, 128 comparison point pairs are selected in the window, and the binary code string of the ORB feature point descriptor is calculated according to formula (1) by comparing the gray value of the central pixel with the gray value of the pixel in the window: Among them, I (p) Represents the gray value of the center pixel, I (q) Represents the gray value of the pixel in the window; N p Indicates the pixels contained in the neighborhood of the central pixel within the set window size; If I(q)≥I(p), the binary code string is recorded as 0, and if I(q)<I(p), the binary code string is recorded as 1; S2: Match the ORB feature points between two adjacent frames of original images to obtain initial matching point pairs; filter the initial matching point pairs to obtain feature point pair matching information and feature point pair pixel coordinates; S3: extracting depth information of the depth image corresponding to the original image through the depth image sequence; combining the pixel coordinates of the matched feature point pairs and the corresponding depth information to obtain local spatial three-dimensional information; In step S3, the depth information of the depth image corresponding to the original image is extracted through the depth image sequence; the local space three-dimensional information is obtained by combining the pixel coordinates of the matched feature point pairs and the corresponding depth information, including: S31: extracting a depth value D of a depth image corresponding to the original image through the depth image sequence; S32: According to the pixel coordinates of the feature point obtained in step S2, combined with the depth value D, the pixel coordinates of the feature point and the depth value D are substituted into formula (3) to calculate the local spatial three-dimensional coordinates of the feature point and obtain the local spatial three-dimensional information: Where: (u, v) represents the pixel coordinates of the feature point; C x ,C y represents the camera calibration parameters, and represents the position coordinates of the main point of the image; f x ,f y is the camera calibration parameter, f x represents the component of focal length along the X-axis in the image pixel coordinate system; f y Represents the component of focal length along the Y axis in the image pixel coordinate system; X, Y, D represent the calculated three-dimensional spatial coordinates; S4: Calculate the inter-frame pose transformation matrix of the original image; S5: Repeat S1-S4 until all inter-frame pose transformation matrices in the original image sequence are calculated; the local spatial three-dimensional information collected at different positions is combined with the pose transformation matrix to form global spatial three-dimensional information; S6: The coordinates of the three-dimensional information of the global space are converted to the crane coordinate system in the storage area through the coordinate system, and provided to the crane for use.

2. The method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision according to claim 1 is characterized in that: In step S2, ORB feature points between two adjacent frames of original images are matched to obtain an initial matching point pair; Filter the initial matching point pairs to obtain feature point pair matching information and feature point pair pixel coordinates, including: S21: Match the ORB feature points detected in the current original image and the previous original image to form a matching point pair; use the Hamming distance percentage to measure the consistency of the matching point pair; S22: Using the random consensus algorithm RANSAC, the initial matching point pair obtained in S21 is substituted into formula (2): Where (x', y') represents the initial matching point pair position in the current original image; (x, y) represents the initial matching point pair position in the previous frame image; s represents the scale parameter; Determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers; S23: Collect the position coordinate information of the overhead crane in different images and at different times, calculate the movement direction of the overhead crane, and remove the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the overhead crane; obtain the final feature point pair matching information and the pixel coordinates of the feature point pairs.

3. The method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision according to claim 1 is characterized in that: In step S4, calculating the inter-frame pose transformation matrix of the original image includes: S41: construct the spatial point set error term between the current original image and the previous frame original image, and calculate a set of spatial transformation matrices with the minimum spatial position error through the singular value decomposition SVD of formula (4): Where: p i represents the spatial 3D point set obtained after feature point matching of the current original image, and qi represents the spatial 3D point set obtained after feature point matching of the previous frame original image; E i Represents the set of spatial position errors of feature point pairs corresponding to the spatial transformation matrix [R|t]; R represents the spatial rotation matrix that minimizes the spatial position error, and t represents the spatial transformation matrix; S42: Will E i Convert to quaternion format for storage and obtain the inter-frame pose transformation matrix.

4. The method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision according to claim 3 is characterized in that: In step S5, the local spatial three-dimensional information collected at different positions is combined with the posture transformation matrix to form the global spatial three-dimensional information, including: Combine the local spatial 3D information of images at different times with the inter-frame pose transformation matrix; use the spatial transformation matrix to stitch the spatial 3D information through formula (5) to form the global spatial 3D information: P i =T*P' (5) Where: P i Represents the set of spatial three-dimensional points in the current original image, P′ i Represents the set of spatial three-dimensional points in the previous frame image; T represents the calculated inter-frame pose transformation matrix.

5. The method for global three-dimensional reconstruction of a metallurgical storage area based on depth vision according to claim 1 is characterized in that: In the step S6, the coordinates in the global space three-dimensional information are converted to the warehouse area crane coordinate system through the coordinate system, and provided to the crane for use, including: By obtaining the position coordinate information of the same detection point in the same scene, the coordinate system is fitted and the coordinate transformation relationship is calculated; the coordinates in the global space three-dimensional information are converted to the warehouse area coordinate system and provided to the overhead crane for use.

6. A global three-dimensional reconstruction device for metallurgical storage area based on deep vision, characterized in that: The device is applied to the method according to any one of claims 1 to 5, and the device comprises: The ORB detection module is used to collect the original images of continuous frames and the depth images corresponding to the original images through the depth camera to form the original image sequence and the depth image sequence; respectively detect the ORB feature points in two adjacent frames of the original image; The similarity matching module is used to match the ORB feature points between two adjacent frames of original images to obtain the initial matching point pairs; the initial matching point pairs are screened to obtain the matching information of the feature point pairs and the pixel coordinates of the feature point pairs; The local information calculation module is used to extract the depth information of the depth image corresponding to the original image through the depth image sequence; combine the pixel coordinates of the matched feature point pairs and the corresponding depth information to obtain the local space three-dimensional information; A posture transformation calculation module, used to calculate the inter-frame posture transformation matrix; A global information stitching module is used to calculate the inter-frame pose transformation matrix of all frames in the original image sequence; stitch the local spatial three-dimensional information collected at different positions in combination with the pose transformation matrix to form global spatial three-dimensional information; The coordinate system conversion module is used to convert the coordinates in the global space three-dimensional information into the warehouse area crane coordinate system through the coordinate system, and provide it to the crane.

7. The global three-dimensional reconstruction device for metallurgical storage area based on depth vision according to claim 6 is characterized in that: The ORB detection module is used to collect original images and depth images of continuous frames through a depth camera arranged on the overhead crane as the overhead crane moves, thereby forming an original image sequence and a depth image sequence; Select the current original image and the previous original image to extract ORB feature points and obtain the pixel coordinates of the ORB feature points; After determining the pixel coordinate position of the ORB feature point, the window size is selected as 13 pixels with the ORB feature point as the center, 128 groups of comparison point pairs are selected in the window, and the binary code string of the ORB feature point descriptor is calculated by comparing the gray value of the central pixel with the gray value of the pixel in the window.

8. The global three-dimensional reconstruction device for metallurgical storage area based on depth vision according to claim 7 is characterized in that: The similarity matching module is used to match the ORB feature points detected in the current original image and the previous original image to form a matching point pair; the consistency of the matching point pair is measured using the Hamming distance percentage; The random consensus algorithm RANSAC is used to determine the optimal transformation matrix that satisfies the maximum number of matching point pairs and remove outliers. Collect the position coordinate information of the crane in different images and at different times, calculate the movement direction of the crane, and remove the feature point pairs whose relative positions between the current original image and the previous original image do not conform to the movement trend of the crane; The final feature point pair matching information and feature point pair pixel coordinates are obtained.

Images