Near-surface ranging method based on virtual large-baseline four-eye vision, medium and equipment

Abstract

The invention discloses a near-ground distance measurement method based on virtual large baseline four-eye vision, a medium and equipment, and the method comprises the steps: synchronously obtaining four images when a lifting appliance enters a near-ground operation range; based on the calibration relation between the cameras, the high-altitude images are respectively projected and transformed to the visual angles of the corresponding lifting appliance cameras and fused, two virtual camera images are generated, and therefore a virtual large base line far exceeding the physical distance is constructed; performing three-dimensional correction and cutting on the virtual image pair to obtain a row-aligned three-dimensional image pair; performing feature matching and triangulation on the virtual image pair to generate a sparse reference depth map; and inputting the stereo image pair and the sparse depth map into a pre-trained depth estimation neural network model together, outputting a dense depth map, and converting the dense depth map into a coordinate system taking the lifting appliance as an original point to obtain a vertical distance. According to the invention, the ultra-large baseline is virtually synthesized by using the existing camera, the remote distance measurement precision is improved, the cost is low, and the reliability is high.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of port automation, and particularly relates to a near-ground ranging method based on virtual large-baseline four-eye vision, a medium and an equipment. BACKGROUND

[0002] In modern port operations, the spreader of a shore-based container crane needs to accurately perceive the vertical distance between itself and the target below (such as a container or a vehicle), which is directly related to the operation efficiency, safety and automation level.

[0003] At present, the following camera systems are usually installed on port cranes: (1) High camera group: installed at a high position (first preset height) of the crane gantry structure, containing two cameras (first camera and second camera). These two cameras have a wide field of view and can overlook the entire operation area.

[0004] (2) Spreader camera group: installed on the spreader that moves vertically with the spreader (second preset height, lower than the first preset height), containing two cameras (third camera and fourth camera).

[0005] However, the existing ranging schemes have significant defects: (1) Laser radar scheme: although it has high accuracy, the device cost is high, and in the harsh environment of high salt, high humidity and high vibration in the port, the service life is short and the maintenance cost is extremely high.

[0006] (2) Traditional binocular camera scheme: limited by the physical installation space on the spreader, its baseline length is usually only 0.1-0.3 meters. According to the stereo vision depth error calculation formula, when the working distance reaches dozens of meters, the tiny parallax measurement error will be amplified sharply, resulting in a serious lack of long-distance ranging accuracy, which cannot meet the requirements of accurate operations.

[0007] Therefore, how to make full use of the existing and widely deployed camera hardware of the crane, overcome the limitation of physical space on the baseline, and realize a low-cost, high-precision and high-reliability long-distance vertical ranging scheme has become a technical problem to be solved in the field. SUMMARY

[0008] In view of the above problems, the present application provides a near-ground ranging technical scheme based on virtual large-baseline four-eye vision to solve the technical problems of high cost and short service life of laser radar, and the serious lack of long-distance ranging accuracy caused by the short physical baseline of traditional binocular cameras.

[0009] To achieve the above object, in a first aspect, the application provides a near-ground ranging method based on virtual large-baseline four-eye vision, applied to a container crane, the crane comprising a portal structure and a spreader vertically movable along the portal structure, the portal structure being provided with a first camera and a second camera at a first preset height, and the spreader being provided with a third camera and a fourth camera at a second preset height, the first preset height being higher than the second preset height; The method comprises: S1: when the spreader enters a preset near-ground operation range, synchronously triggering and acquiring a first image collected by the first camera, a second image collected by the second camera, a third image collected by the third camera, and a fourth image collected by the fourth camera; S2: based on a calibration relationship between the first camera and the third camera, projecting and transforming the first image to the visual angle of the third camera, and fusing with the third image to generate a first virtual camera image, and based on a calibration relationship between the second camera and the fourth camera, projecting and transforming the second image to the visual angle of the fourth camera, and fusing with the fourth image to generate a second virtual camera image; S3: performing stereo correction on the first virtual camera image and the second virtual camera image to obtain a pair of corrected images in line alignment; cropping a common field of view region from the pair of corrected images to obtain a first visual angle image and a second visual angle image in the same size and in line alignment; S4: performing feature point matching and triangulation on the first virtual camera image and the second virtual camera image to obtain coordinates and corresponding depth values of a plurality of feature points; filling the depth values into a two-dimensional matrix in the same size as the first visual angle image according to the coordinate mapping relationship between the plurality of feature points in the first virtual camera image and the first visual angle image, to generate a sparse reference depth map in pixel alignment with the first visual angle image, wherein the matrix elements without filled depth values are empty or zero; S5: jointly inputting the first visual angle image, the second visual angle image and the sparse reference depth map into a pre-trained depth estimation neural network model, and outputting a dense depth map in pixel alignment with the first visual angle image from the model; S6: based on the depth values of each pixel in the dense depth map, the intrinsic matrix of the first virtual camera, the geometric relationship between the first virtual camera and the third camera, and the installation pose of the third camera relative to the spreader, converting the coordinates corresponding to each pixel to a coordinate system with the spreader as the origin, and taking the vertical component of the coordinates as the vertical distance from the pixel to the spreader, to obtain the near-ground ranging result.

[0010] Further, the projecting and transforming the first image to the visual angle of the third camera based on the calibration relationship between the first camera and the third camera comprises: based on a first homography matrix between the first camera and the third camera performing a first perspective transformation on the first image, the first homography matrix The calculation formula of the first homography matrix is as follows: ; wherein G1 represents the first camera, A1 represents the third camera, h represents the height when the spreader enters the preset near-ground operation range, K A1 represents the intrinsic matrix of the third camera, represents the rotation matrix of the first camera to the third camera at the height h, represents the translation vector from the coordinate system origin of the first camera to the coordinate system origin of the third camera at the height h, and n represents the unit normal vector of the preset reference plane, represents the vertical distance from the origin of the world coordinate system to the preset reference plane, represents the inverse matrix of the intrinsic matrix of the first camera. The projecting and transforming the second image to the visual angle of the fourth camera based on the calibration relationship between the second camera and the fourth camera comprises: projecting and transforming the second image to the visual angle of the fourth camera based on a second homography matrix performing a second perspective transformation on the second image, the second homography matrix The calculation formula of the second homography matrix is as follows: ; wherein G2 represents the second camera, A2 represents the fourth camera, K A2 represents the intrinsic matrix of the fourth camera, represents the rotation matrix of the second camera to the fourth camera at the height h, represents the translation vector from the coordinate system origin of the second camera to the coordinate system origin of the fourth camera at the height h, represents the inverse matrix of the intrinsic matrix of the second camera.

[0011] Further, the first virtual camera image is generated according to the following manner: aligning the first image after the first perspective transformation with the original image of the third image in the image space, and determining the overlapping area between the two images; performing pixel-by-pixel fusion on the corresponding pixels of the first image after the first perspective transformation and the original image of the third image in the overlapping area by using a weight-based fusion algorithm, to generate the first virtual camera image with an expanded field of view; The second virtual camera image is generated according to the following manner: aligning the second image after the second perspective transformation with the original image of the fourth image in the image space, and determining the overlapping area between the two images; In the overlapping region, corresponding pixels of the second perspective-transformed second image and the original fourth image are fused pixel by pixel by using a weight-based fusion algorithm to generate the second virtual camera image with an extended field of view.

[0012] Further, the weight-based fusion algorithm includes a multi-band fusion algorithm. The weight-based fusion algorithm is used for pixel-by-pixel fusion to generate the first virtual camera image with an extended field of view, specifically including: The first perspective-transformed first image and the original third image are respectively decomposed by a Laplacian pyramid; In each layer of the Laplacian pyramid, a first fusion weight map is calculated according to the position of the pixel in the overlapping region; The first fusion weight map is used to weight and fuse the pyramid coefficients of the corresponding layers of the original first image and the original third image; The weighted and fused Laplacian pyramid is reconstructed to generate the first virtual camera image with an extended field of view; The weight-based fusion algorithm is used for pixel-by-pixel fusion to generate the second virtual camera image with an extended field of view, specifically including: The second perspective-transformed second image and the original fourth image are respectively decomposed by a Laplacian pyramid; In each layer of the Laplacian pyramid, a second fusion weight map is calculated according to the position of the pixel in the overlapping region; The second fusion weight map is used to weight and fuse the pyramid coefficients of the corresponding layers of the original second image and the original fourth image; The weighted and fused Laplacian pyramid is reconstructed to generate the second virtual camera image with an extended field of view.

[0013] Further, the first virtual camera image and the second virtual camera image are rectified to obtain a pair of row-aligned rectified images, including the following steps: S31: based on the camera parameters associated with the first virtual camera image and the second virtual camera image, calculate the stereo rectification parameters; the camera parameters include the relative pose between the first virtual camera and the second virtual camera and the respective intrinsic matrices; S32: using the calculated stereo rectification parameters, respectively performing geometric transformation on the first virtual camera image and the second virtual camera image to obtain a pair of row-aligned rectified images, the calculation formula of the geometric transformation is as follows: ; ; wherein, (x, y) represents pixel coordinates of the first virtual camera image before rectification, (x', y') represents pixel coordinates of the first virtual camera image after rectification, (x'', y'') represents pixel coordinates of the second virtual camera image before rectification, (x''', y''') represents pixel coordinates of the second virtual camera image after rectification, , , , , represents an inverse matrix of an intrinsic matrix of the first virtual camera , represents an inverse matrix of an intrinsic matrix of the second virtual camera , represents a rectification rotation matrix of the first virtual camera, represents a rectification rotation matrix of the second virtual camera, represents a rectified and unified intrinsic matrix; Further, the feature point matching and triangulation on the first virtual camera image and the second virtual camera image are performed to obtain coordinates of a plurality of feature points and corresponding depth values, specifically comprising: extracting feature points and generating corresponding feature descriptors on the first virtual camera image and the second virtual camera image, respectively; matching the feature points in the first virtual camera image and the second virtual camera image through the feature descriptors to obtain an initial matching point pair set; screening the initial matching point pair set to obtain a high-confidence matching point pair set, wherein any high-confidence matching point in the first virtual camera image is denoted as p1, the coordinates of p1 are denoted as (u1, v1), and the corresponding matching point in the second virtual camera image is denoted as p2, the coordinates of p2 are denoted as (u2, v2); for each matching point pair (p1, p2) in the high-confidence matching point pair set, a homogeneous linear equation set is constructed, and a singular value decomposition algorithm is used to solve the homogeneous linear equation set to obtain the coordinates P=(X, Y, Z) of the matching point pair (p1, p2) in the three-dimensional space, and the homogeneous linear equation set is represented as follows: ; wherein, represents the first row of a projection matrix , M v1 and M v2 represent projection matrices of the first virtual camera and the second virtual camera, respectively, M v1 and M​​​​v2 The expression formula is as follows: ; ; wherein, represents an intrinsic matrix of the first virtual camera, represents an intrinsic matrix of the second virtual camera, represents an extrinsic matrix when the first virtual camera coordinate system is the world coordinate system, represents a relative pose matrix for transforming a point from the first virtual camera coordinate system to the second virtual camera coordinate system; convert the coordinate P=(X, Y, Z) into the camera coordinate system of the first virtual camera to obtain the coordinate P v1 The conversion formula is as follows: wherein, is the optical center coordinate of the first virtual camera, represents a rotation matrix for converting the coordinate P into the camera coordinate system of the first virtual camera; take the Z-axis component of P v1 as the depth value Z depth corresponding to the matching point p1. Iterate through all high-confidence matching point pairs to obtain a set of coordinates (u1, v1) of the feature points in the first virtual camera image and corresponding depth values Z depth .

[0014] Further, step S6 specifically comprises: S61: Based on the depth values of the pixels in the dense depth map and the intrinsic matrix of the first virtual camera, project each pixel back to the coordinate system of the first virtual camera to obtain a first set of three-dimensional point clouds P d . S62: Based on the geometric relationship between the first virtual camera and the third camera, convert the first set of three-dimensional point clouds P d to the coordinate system of the third camera to obtain a second set of three-dimensional point clouds P c . S63: Based on the mounting pose of the third camera relative to the spreader, convert the second set of three-dimensional point clouds P c to the coordinate system with the spreader as the origin to obtain a third set of three-dimensional point clouds P h . S64: Extract the vertical direction coordinate component of each point in the third set of three-dimensional point clouds P h as the vertical distance from the corresponding pixel to the spreader to obtain the near-ground ranging result.

[0015] Further, the depth estimation neural network model is trained by the following manner: S81: A training sample set is constructed, each sample in the training sample set comprising a stereo image pair composed of a first training perspective image and a second training perspective image, and a corresponding sparse training depth map; wherein the sparse training depth map is generated by randomly sampling or noise simulation on a dense ground truth depth map aligned with pixels of the first training perspective image; S82: inputting the training sample set into the depth estimation neural network model, the model performing the following operations: performing feature extraction and stereo matching on the stereo image pair to generate an initial depth estimation; fusing the initial depth estimation with the sparse training depth map as guiding information in the process of stereo matching or in a subsequent optimization process; outputting depth maps at different stages or different resolutions, including a first dense depth map and a second dense depth map; wherein the first dense depth map is an intermediate coarse depth map, and the second dense depth map is a final output fine depth map; S83: for the first dense depth map, calculating the difference between the first dense depth map and a first reference dense depth map to obtain a first loss L coarse , the first reference dense depth map being obtained by spatially downsampling a second reference dense depth map; for the second dense depth map, calculating the difference between the second dense depth map and a second reference dense depth map, a second loss L final , the second reference dense depth map being the dense ground truth depth map; for the sparse pixel points in the sparse training depth map with valid depth values, calculating the difference between the predicted depth value of the second dense depth map on the corresponding pixel point and the sparse ground truth depth value, a third loss L sparse ; constructing a total loss function based on the first loss L coarse , the second loss L final and the third loss L sparse , the formula of the total loss function being as follows: L total =L final +α·L coarse +β·L sparse ; wherein L total represents the total loss, and α and β are preset weight coefficients; S84: using the total loss L total to update the parameters of the depth estimation neural network model through a back propagation algorithm; S85: repeat steps S82 to S84 until the model converges, to obtain a pre-trained depth estimation neural network model.

[0016] In a second aspect, the present application provides a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method for near-ground distance measurement based on virtual large-baseline four-vision as described in the first aspect of the present application.

[0017] In a third aspect, the present application provides an electronic device having stored thereon a computer program, comprising a processor and a storage medium having stored thereon a computer program which, when executed by the processor, implements the method for near-ground distance measurement based on virtual large-baseline four-vision as described in the first aspect of the present application.

[0018] Distinguished from the prior art, the above technical solution relates to a method for near-ground distance measurement based on virtual large-baseline four-vision, medium and device. The method uses a first camera and a second camera installed on a high gantry, and a third camera and a fourth camera installed on a moving hoist. When the hoist enters the near-ground working range, four images are synchronously acquired. First, based on the calibration relationship between the cameras, the high images are projected and transformed to the visual angle of the corresponding hoist cameras, and fused with the hoist images to generate two virtual camera images, thereby constructing a virtual large baseline far beyond the physical installation distance. Then, the virtual image pair is subjected to stereo rectification and cropping to obtain a row-aligned stereo image pair. Next, feature matching and triangulation are performed on the virtual image pair to generate a sparse reference depth map aligned with one of the stereo image pair. Finally, the stereo image pair and the sparse depth map are jointly input into a pre-trained depth estimation neural network model to output a dense depth map, and the vertical distance is obtained in the coordinate system with the hoist as the origin. The present application ingeniously utilizes the existing cameras to virtually synthesize an ultra-large baseline, which fundamentally improves the long-distance ranging accuracy, while having the advantages of low cost and high reliability.

[0019] The above summary of the invention is only a summary of the technical solutions of the present application. In order for those skilled in the art to more clearly understand the technical solutions of the present application, and to implement the contents recorded in the specification and drawings, and in order for the above and other purposes, features and advantages of the present application to be more easily understood, the following describes the specific embodiments of the present application in conjunction with the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] The accompanying drawings are only used to illustrate the principles, implementation manners, applications, characteristics and effects of the specific embodiments of the present application and other related contents, and cannot be considered as limitations of the present application.

[0021] In the drawings of the specification: Figure 1This is a flowchart illustrating a near-ground ranging method based on virtual large baseline quad-vision, which is an exemplary embodiment of this application. Figure 2 This is a flowchart illustrating a first virtual camera image generation method according to an exemplary embodiment of this application; Figure 3 This is a flowchart illustrating a second virtual camera image generation method according to an exemplary embodiment of this application; Figure 4 A flowchart illustrating a method for generating a first virtual camera image using a weighted fusion algorithm, as described in an exemplary embodiment of this application. Figure 5 A flowchart illustrating a method for generating a second virtual camera image using a weighted fusion algorithm, as described in an exemplary embodiment of this application. Figure 6 A flowchart illustrating a stereoscopic correction method for a first virtual camera image and a second virtual camera image, as described in an exemplary embodiment of this application; Figure 7 This is a flowchart illustrating the process of performing feature point matching and triangulation on a first virtual camera image and a second virtual camera image, as described in an exemplary embodiment of this application. Figure 8 This is a flowchart illustrating a near-ground ranging method based on the depth values ​​of each pixel in a dense depth map and the intrinsic parameter matrix of a first virtual camera, which is an exemplary embodiment of this application. Figure 9 This is a schematic diagram of an electronic device according to an exemplary embodiment of this application; The reference numerals used in the above figures are explained as follows: 10. Electronic equipment; 101. Processor; 102. Storage medium. Detailed Implementation

[0022] To explain in detail the possible application scenarios, technical principles, specific feasible solutions, and the objectives and effects that this application can achieve, the following detailed description is provided in conjunction with the listed specific embodiments and accompanying drawings. The embodiments described herein are only used to more clearly illustrate the technical solutions of this application, and are therefore only examples, and should not be used to limit the scope of protection of this application.

[0023] In a first aspect, this application provides a near-ground ranging method based on virtual large baseline four-eye vision, applied to a container crane. The crane includes a gantry structure and a spreader that can move vertically along the gantry structure. A first camera and a second camera are installed on the gantry structure at a first preset height, and a third camera and a fourth camera are installed on the spreader at a second preset height. The first preset height is higher than the second preset height. like Figure 1 As shown, the method includes: S1: when the spreader enters the preset near-ground operation range, simultaneously trigger and obtain a first image collected by a first camera, a second image collected by a second camera, a third image collected by a third camera, and a fourth image collected by a fourth camera; S2: based on a calibration relationship between the first camera and the third camera, project and transform the first image to the visual angle of the third camera, and fuse with the third image to generate a first virtual camera image, and based on a calibration relationship between the second camera and the fourth camera, project and transform the second image to the visual angle of the fourth camera, and fuse with the fourth image to generate a second virtual camera image; S3: perform stereo rectification on the first virtual camera image and the second virtual camera image to obtain a pair of rectified images that are row-aligned; crop a common field of view region from the pair of rectified images to obtain a first visual angle image and a second visual angle image that are the same in size and row-aligned; S4: perform feature point matching and triangulation on the first virtual camera image and the second virtual camera image to obtain coordinates and corresponding depth values of a plurality of feature points; according to the coordinate mapping relationship between the plurality of feature points in the first virtual camera image and the first visual angle image, fill the depth values into a two-dimensional matrix that is the same in size as the first visual angle image to generate a sparse reference depth map that is pixel-aligned with the first visual angle image, wherein the matrix elements without filled depth values are empty or zero; S5: input the first visual angle image, the second visual angle image, and the sparse reference depth map into a pre-trained depth estimation neural network model, and output a dense depth map that is pixel-aligned with the first visual angle image from the model; S6: based on the depth values of each pixel in the dense depth map, the intrinsic matrix of the first virtual camera, the geometric relationship between the first virtual camera and the third camera, and the installation pose of the third camera relative to the spreader, convert the coordinates corresponding to each pixel to a coordinate system with the spreader as the origin, and take the vertical component of the coordinates as the vertical distance from the pixel to the spreader to obtain the near-ground ranging result.

[0024] In step S1, when the height h of the spreader fed back by the crane PLC system PLC ≤ 5 meters (i.e., entering the near-ground operation range), trigger the synchronous acquisition mechanism. This mechanism links the signals of the PLC and each camera to ensure that the first camera (G1), the second camera (G2), the third camera (A1), and the fourth camera (A2) simultaneously capture a frame of static image, respectively obtaining the first image, the second image, the third image, and the fourth image. The core purpose of synchronous acquisition is to avoid scene changes caused by time differences in acquisition, ensuring that the physical scenes corresponding to the four images are completely consistent, providing a unified data source in time and space for subsequent image fusion and stereo matching.

[0025] In step S2, the first camera and the third camera, the second camera and the fourth camera need to be calibrated in advance. Among them, the third camera and the fourth camera are fixed on the spreader to form a rigid binocular system, and the relative pose (rotation matrix, translation vector) is constant. Preferably, the length of the translation vector is about 2.8 meters; the first camera and the second camera are zooming camera heads, and for each discrete height h (such as 5, 4, 3, 2, 1, 0.5 meters) of near-ground operation, the corresponding pan-tilt angle and focal length are preset, the internal and external parameters of each point are calibrated and stored, and a lookup table is formed for real-time calling.

[0026] Then, based on the homography matrix, the high camera image is projected and transformed to the spreader camera view, and the high camera image after projection and transformation is weighted and fused with the original spreader camera image in the overlapping area. The fusion algorithm can use multi-band fusion or Poisson fusion, etc. to ensure that the generated virtual camera image (first virtual camera image, second virtual camera image) has an extended and smooth transition view, and has the advantages of both views and clear details.

[0027] In step S3, based on the relative pose and internal parameter matrix of the first virtual camera and the second virtual camera, the Bouguet stereo correction algorithm is used to calculate the correction rotation matrix and the unified internal parameter matrix. The two virtual camera images are re-projected to a common image plane through geometric transformation, so that the projection points of the same physical point in the two images are located on the same horizontal line (epipolar line horizontal), and the two-dimensional search of feature matching is simplified to one-dimensional search. After correction, the maximum overlapping observation area is determined by calculating the intersection of the effective pixels of the two corrected images. With the spreader projection center as the reference, a rectangular area of a fixed size (such as 640x480 pixels) is cropped to obtain the first and second view images, which have the same size and pixel rows strictly aligned, meeting the standardized input requirements of stereo matching.

[0028] In step S4, feature points are extracted from the first virtual camera image and the second virtual camera image using ORB, SIFT or SURF feature point detection algorithms, and matched by feature descriptors, and then screened for high-confidence matching point pairs through ratio testing (typical threshold 0.75) and cross-validation. Then, based on the projection matrix of the first virtual camera and the second virtual camera, a homogeneous linear equation set is constructed using the triangulation principle of stereo vision, and the three-dimensional space coordinates of the matching point pairs are solved by singular value decomposition (SVD), and then the depth values of the feature points are calculated. This process relies on the high depth sensitivity of the virtual large baseline (about 2.8 meters) to ensure the geometric accuracy of the depth values. Then, through homographic transformation or direct projection calculation, the coordinate mapping relationship between the feature points in the first virtual camera image and the first view image is established, the depth values of the feature points are filled into a two-dimensional matrix with the same size as the first view image, and the unfilled area is set to zero, forming a sparse reference depth map, which provides reliable geometric prior for the subsequent neural network.

[0029] In step S5, the depth estimation neural network model is an end-to-end architecture optimized for a specific near-ground height h, and the input is a first-view image (a grayscale image), a second-view image (a grayscale image), and a sparse reference depth map. The model fuses image features and depth prior features through a multi-modal feature extraction network, constructs a 4D cost space and is regularized by 3D convolution, then obtains a coarse disparity map through a differentiable disparity regression, and finally converts the dense depth map through optimization and upsampling to realize pixel-level accurate depth estimation.

[0030] In step S5, based on the depth value of each pixel in the dense depth map and the first virtual camera internal parameter, it is back projected to the first virtual camera coordinate system; combined with the geometric relationship between the first virtual camera and the third camera, it is converted to the third camera coordinate system; and finally converted to the coordinate system with the spreader as the origin by using the installation pose of the third camera relative to the spreader. In the spreader coordinate system, the vertical component of the three-dimensional coordinate is taken, that is, the vertical distance from the corresponding pixel to the spreader, and the vertical distance of all pixels is integrated to obtain the complete near-ground ranging result, which directly provides decision basis for the automatic control system of the crane.

[0031] The above scheme directly calls the four cameras already installed on the crane, avoiding the high equipment cost and maintenance cost of the laser radar scheme, and also does not need to deploy special ranging equipment. The virtual large baseline (about 2.8 meters) constructed by image fusion improves the depth measurement accuracy compared with the short baseline (0.1-0.3 meters) of the traditional binocular camera, and solves the accurate ranging demand in the near-ground operation scene. The final output vertical distance in the spreader coordinate system does not need additional conversion and can be directly connected to the automatic control system of the crane, improving the operation safety and automation level and effectively avoiding the collision risk.

[0032] In some embodiments, the projecting and transforming the first image to the perspective of the third camera based on the calibration relationship between the first camera and the third camera comprises: performing a first perspective transformation on the first image, the first homography matrix is calculated according to the following formula: ; wherein G1 represents the first camera, A1 represents the third camera, h represents the height when the spreader enters the preset near-ground operation range, K A1 represents the internal parameter matrix of the third camera, represents the rotation matrix from the first camera to the third camera at the height h, represents the translation vector from the coordinate system origin of the first camera to the coordinate system origin of the third camera at the height h, and n represents the unit normal vector of the preset reference plane. represents the vertical distance from the origin of the world coordinate system to the preset reference plane, represents the inverse matrix of the intrinsic matrix of the first camera; The second image is projected and transformed to the perspective of the fourth camera based on the calibration relationship between the second camera and the fourth camera, including: based on the second homography matrix , the second perspective transformation is performed on the second image. The calculation formula of is as follows: ; Wherein, G2 represents the second camera, A2 represents the fourth camera, K A2 represents the intrinsic matrix of the fourth camera, represents the rotation matrix of the second camera to the fourth camera at height h, represents the translation vector from the origin of the coordinate system of the second camera to the origin of the coordinate system of the fourth camera at height h, represents the inverse matrix of the intrinsic matrix of the second camera.

[0033] In this embodiment, the homography matrix is a 3x3 matrix describing the projection transformation relationship between two image planes, which is a core mathematical tool for realizing the projection of the high camera image to the sling camera perspective, including the first homography matrix and the second homography matrix .

[0034] The intrinsic matrix is a matrix representing the internal optical characteristics of the camera, including focal length (f x , f y ) and principal point coordinates (cx, cy), the third camera intrinsic matrix K A1 , the fourth camera intrinsic matrix K A2 is a fixed value (one-time calibration), the first camera intrinsic matrix K G1 , the second camera intrinsic matrix K G2 changes dynamically with the preset point position, and is stored in the lookup table.

[0035] The reference plane can be selected as the horizontal plane where the target container top surface is located, and its equation is n T •X=d plane , wherein, n=(0, 1, 0) T is the unit normal vector (vertically upward), X is the coordinate of any point on the horizontal plane, and d planeis the vertical distance from the origin of the world coordinate system to the horizontal plane, which is usually set as the standard top surface height of the target container (e.g., 2.5 meters). The main operation target of the spreader in the near-ground operation is the container, and the top surface thereof is used as the reference plane to maximize the accuracy of the projection transformation. Meanwhile, the plane is approximately an ideal plane, which meets the application prerequisite of the homography matrix. If the operation target is a deck of a cargo ship, a trailer or other planar targets, the value of d plane can be adjusted through system parameter configuration to adapt to different operation scenarios.

[0036] When the height of the spreader is h, the intrinsic matrix K G1 and the extrinsic parameters (rotation matrix, position coordinates) of the first camera (G1) corresponding to the preset point are read from the lookup table; through the PLC and the kinematics model of the crane, combined with the fixed calibration relationship between the third camera (A1) and the spreader, the real-time extrinsic parameters (rotation matrix R A1 (h), position coordinates P A1 (h)) of A1 in the world coordinate system are calculated.

[0037] The relative pose conversion formula of the first camera to the third camera is as follows: =R A1 (h)•R G1 (h) T , wherein R G1 (h) is the rotation matrix of the first camera at height h; the calculation formula of the translation vector from the origin of the coordinate system of the first camera to the origin of the coordinate system of the third camera is as follows: =P A1 (h)-R A1 (h)•R G1 (h) T •P G1 (h), wherein P G1 (h) is the position coordinates of the first camera at height h. Then each pixel coordinate of the first image is substituted into the homography matrix , and the projection transformation of the pixel coordinates is realized through matrix multiplication to obtain the transformed image of the first image under the perspective of the third camera, and the perspective alignment is completed. The calculation process of the second homography matrix is the same.

[0038] The spreader height h is included in the homography matrix as a variable, so that the rotation matrix and the translation vector are dynamically adjusted with h. No matter the specific height (e.g., 5 meters, 3 meters, 0.5 meters) of the spreader in the near-ground operation range, the corresponding homography matrix can be calculated to realize accurate perspective projection at different heights and ensure the consistency of the ranging accuracy in the entire operation range.

[0039] In the above scheme, the homography matrix strictly follows the principle of projective geometry, ensuring that the high camera image is completely matched in geometric structure after perspective transformation, providing a solid foundation for subsequent image fusion. By introducing the height h of the spreader into the homography matrix calculation, the projection transformation can adapt to different working heights of the spreader in real time, avoiding the height adaptation limitations caused by fixed transformation matrix, and ensuring stable performance within the 0-5 meter working range. For the dynamic parameters of the first camera and the second camera (zooming gimbal camera), the preset point calibration and lookup table storage method not only solves the problem of dynamic changes of internal and external parameters, but also ensures the accuracy of homography matrix calculation, without additional complex processing.

[0040] In some embodiments, as shown in Figure 2 The first virtual camera image is generated according to the following manner: S201: Align the first image after the first perspective transformation with the original image of the third image in the image space, and determine the overlapping area between the two images; S202: In the overlapping area, the corresponding pixels of the first image after the first perspective transformation and the original image of the third image are fused pixel by pixel using a weight-based fusion algorithm to generate the first virtual camera image with an expanded field of view; As shown in Figure 3 The second virtual camera image is generated according to the following manner: S301: Align the second image after the second perspective transformation with the original image of the fourth image in the image space, and determine the overlapping area between the two images; S302: In the overlapping area, the corresponding pixels of the second image after the second perspective transformation and the original image of the fourth image are fused pixel by pixel using a weight-based fusion algorithm to generate the second virtual camera image with an expanded field of view.

[0041] In the embodiment, the first image after the first perspective transformation (G1 transformed image) and the third image (A1 original image) are introduced into the same pixel coordinate system, the pixel coordinates of corresponding physical points in the two images are accurately aligned based on the geometric mapping relationship of the homography matrix, and the scene consistency is ensured. Through pixel coordinate traversal and scene feature matching, the pixel region corresponding to the physical scene simultaneously contained in the two images, i.e., the overlapping region, is identified. In the overlapping region, the G1 transformed image provides the far-end scene information, and the A1 original image provides the near-end detail information, and then a fusion weight is assigned to each pixel in the overlapping region. The weight assignment rule can be based on the distance of the pixel to the image edge (for example, the pixel close to the edge of the G1 transformed image has a reduced G1 image weight and an increased A1 image weight, and vice versa for the pixel close to the edge of the A1 image), or based on the image definition index (the pixel weight of the image with high definition is higher). Through weighted average calculation, the fused pixel value is obtained, the pixel value of the corresponding image is directly retained in the non-overlapping region, and finally the first virtual camera image with an expanded field of view is generated. The generation method of the second virtual camera image is the same.

[0042] The fusion weight can be dynamically adjusted according to the actual scene. For example, in a strong light environment in a port, if the image of the high camera is overexposed, the weight of the image in the overlapping region can be reduced; if the image of the spreader camera is too close to the distance, resulting in blurred far-end details, the weight of the transformed image of the high camera can be increased to ensure the overall quality of the fused image and provide high-quality input for subsequent feature extraction and matching.

[0043] The above scheme avoids the problems of seams, ghosting and brightness mutation caused by simple splicing through weighted fusion, so that the visual effect of the virtual camera image is close to the natural image taken by a single camera, and the reliability of subsequent image processing is improved. The fused image contains the near-end clear details of the spreader camera and the far-end wide field of view of the high camera, solving the problem of limited field of view of a single camera and providing more comprehensive scene information for stereo matching.

[0044] In some embodiments, the weight-based fusion algorithm includes a multi-band fusion algorithm; as shown in Figure 4 The weight-based fusion algorithm is used for pixel-by-pixel fusion to generate the first virtual camera image with an expanded field of view, and specifically includes: S401: Laplacian pyramid decomposition is performed on the first image after the first perspective transformation and the original image of the third image, respectively; S402: In each layer of the Laplacian pyramid, a first fusion weight map is calculated according to the position of the pixel in the overlapping region; S403: The first fusion weight map is used to perform weighted fusion on the pyramid coefficients of the corresponding layers of the first image and the original image of the third image; S404: Reconstruct the Laplacian pyramid after the weighted fusion to generate the first virtual camera image with an extended field of view.

[0045] As shown in the embodiment, the pixel-by-pixel fusion using the weight-based fusion algorithm to generate the second virtual camera image with an extended field of view specifically includes: Figure 5 S501: Perform Laplacian pyramid decomposition on the second image after the second perspective transformation and the original image of the fourth image respectively; S502: At each layer of the Laplacian pyramid, calculate a second fusion weight map according to the position of the pixel in the overlapping area; S503: Use the second fusion weight map to perform weighted fusion on the pyramid coefficients of the corresponding layers of the original image of the second image and the fourth image; S504: Reconstruct the Laplacian pyramid after the weighted fusion to generate the second virtual camera image with an extended field of view.

[0046] In this embodiment, the multi-band fusion algorithm is a fusion algorithm based on image frequency decomposition, which decomposes the image into subbands of different spatial frequencies (low-frequency layer, high-frequency layer), and reconstructs after independent fusion at each layer, which can ensure the overall structure of the image to be smooth and the details and textures to be clear. In this application, the Laplacian pyramid fusion algorithm is specifically used.

[0047] Laplacian pyramid decomposition refers to decomposing an image into a series of sub-images of different resolutions (pyramid layers). The bottom layer (low-frequency layer) contains the overall structure and brightness information of the image, and the high layer (high-frequency layer) contains the edge, texture and other detail information of the image. The decomposition process is realized through Gaussian filtering and down-sampling.

[0048] The fusion weight map is a weight matrix generated for each layer of the Laplacian pyramid. Each element in the matrix represents the weight of the corresponding pixel during fusion, which is dynamically calculated based on the position of the pixel in the overlapping area to ensure smooth transition of each layer fusion.

[0049] The pyramid coefficient is the pixel value of each layer of the Laplacian pyramid, which represents the image information of the corresponding frequency band. During the fusion process, the corresponding layer coefficients of the two images are weighted and summed through the weight map to realize the fusion of the information of the frequency band.

[0050] Pyramid reconstruction refers to combining the coefficients of each layer of the fused Laplacian pyramid according to the inverse process, restoring the image to the original resolution through up-sampling and Gaussian filtering, and obtaining the final fusion result.

[0051] ​The specific procedure of Laplacian pyramid fusion of the first virtual camera image is as follows: the original images of the first image after the first perspective transformation and the third image are respectively subjected to Laplacian pyramid decomposition to obtain respective pyramid hierarchical structures. The number of decomposition layers can be adjusted according to the image resolution, and the typical number of decomposition layers is 4-6 layers, which ensures full-band information covering from the overall structure to the detailed texture. For each layer of the pyramid, a first fusion weight map is calculated based on the position of the pixel in the overlapping area. For example, in a certain layer, the closer the pixel is to the edge of the original image of the third image, the higher the weight of the third image corresponding to the pixel, and the closer the pixel is to the edge of the first image after the perspective transformation, the higher the weight of the first image corresponding to the pixel, and the weight value changes in a smooth gradient in the overlapping area. For non-overlapping areas, the weight value is set to 1 (corresponding to the image itself) and 0 (the other image). Using the first fusion weight map, the pyramid coefficients of the corresponding layers of the first image and the original image of the third image are weighted and fused, that is, the fusion coefficient of each pixel = the first image layer coefficient x the corresponding weight + the third image layer coefficient x (1-the corresponding weight), to realize accurate fusion of each frequency band information. Then, the Laplacian pyramid coefficients of all levels after fusion are reconstructed according to the inverse process of the Laplacian pyramid, the resolution of each layer is restored by upsampling, and then it is subjected to Gaussian filtering and smoothing processing, and finally a first virtual camera image with an expanded field of view, smooth structure and clear details is generated. The generation mode of the second virtual camera image is similar to that of the first virtual camera image, and only the corresponding parameters need to be replaced in the calculation process.

[0052] Different levels of the Laplacian pyramid correspond to different frequency information, and the fusion strategy can be adjusted according to the frequency characteristics: the low-frequency layer (image structure) adopts a smoother weight transition to ensure the consistency of the overall brightness and structure; the high-frequency layer (edge details) can appropriately increase the weight gradient to retain clear edge texture, so that the fusion image achieves optimal results in structure and details.

[0053] Compared with traditional pixel domain fusion, the Laplacian pyramid fusion can independently optimize the fusion effect in different frequency bands, ensuring smooth transition of the overall structure of the image and effectively preserving the edge, texture and other detail information, avoiding the problem of blurred or false images in the fusion image. Different fusion rules can be designed for different frequency bands, for example, a higher weight resolution is used for high-frequency detail layers, and a wider weight transition band is used for low-frequency structure layers, which adapts to the quality difference of high-altitude images and spreader images in different frequency bands. The high-quality virtual camera image contains clear feature points and rich texture information, making the subsequent feature extraction more stable, the matching more accurate, and reducing matching errors, thereby providing reliable image input for triangulation and depth estimation. Under complex lighting conditions in the port (such as strong light and shadow), the frequency characteristics of different images differ greatly, and multi-band fusion can effectively balance the frequency band advantages of each image, reduce the influence of lighting changes on the fusion effect, and improve the environmental robustness of the system.

[0054] In other embodiments, the weight-based fusion algorithm is a Poisson fusion algorithm, specifically including: using the original image of the third image as the fusion target region, using the image gradient of the first image after the first perspective transformation within the overlapping region as a guiding field, and integrating the gradient information of the guiding field into the target region by solving the Poisson equation, while keeping the boundary conditions of the original image of the third image unchanged, to generate the image with an expanded field of view. The Poisson fusion algorithm and the Laplacian pyramid fusion algorithm can be dynamically switched according to the actual operating environment and image quality, improving the system's flexibility and adaptability.

[0055] In some embodiments, such as Figure 6 As shown, the process of performing stereo correction on the first virtual camera image and the second virtual camera image to obtain a pair of row-aligned corrected images includes the following steps: S31: Calculate stereo correction parameters based on camera parameters associated with the first virtual camera image and the second virtual camera image; the camera parameters include the relative pose between the first virtual camera and the second virtual camera and their respective intrinsic parameter matrices; S32: Using the calculated stereo correction parameters, perform geometric transformations on the first virtual camera image and the second virtual camera image respectively to obtain a pair of row-aligned corrected images. The calculation formula for the geometric transformation is as follows: ; ; in,( , ) represents the pixel coordinates of the first virtual camera image before correction, ( , ) represents the pixel coordinates of the corrected first virtual camera image, ( , ) represents the pixel coordinates of the second virtual camera image before correction, ( , () represents the pixel coordinates of the corrected second virtual camera image. The intrinsic parameter matrix of the first virtual camera The inverse matrix, The intrinsic parameter matrix representing the second virtual camera The inverse matrix, This represents the correction rotation matrix of the first virtual camera. This represents the correction rotation matrix of the second virtual camera. This represents the unified intrinsic parameter matrix after correction. In step S31, the relative poses of the first virtual camera and the second virtual camera are directly determined by the relative poses of the third camera A1 and the fourth camera A2 located on the spreader, because the virtual cameras are generated by fusing A1, A2 and the gantry structure, and their relative poses are consistent with the relative poses of A1, A2; the intrinsic matrix 、 of A1 and A2 are respectively inherited 、 .

[0056] Then the Bouguet stereo rectification algorithm is used, which solves the rotation matrix 、 so that the rectified two virtual camera images satisfy the following constraints: epipolar line is horizontal, that is, the projection points of the same physical point in the two images have the same vertical coordinates ( = ); the principal point coordinates are consistent and located on the same horizontal line; the focal lengths are the same to ensure the uniformity of the image scale. The output results are the rectification rotation matrix (corresponding to the first virtual camera), (corresponding to the second virtual camera), and the rectified uniform intrinsic matrix .

[0057] In step S32, for the pixels of the first virtual camera image ( , ), first, back-project to the normalized camera coordinate system of A1 through (that is, ); then rotate through to make the image epipolar line horizontal; finally, re-project to the rectified pixel coordinate system through to obtain ( , ).

[0058] For the pixels of the second virtual camera image ( , ), similarly, back-project, rotate and re-project through (that is, ) to obtain ( , ). The pixel rows of the two rectified images (Rectified1, Rectified2) are strictly aligned, and the projection points of the same physical point only have a horizontal offset (parallax), which greatly facilitates subsequent stereo matching.

[0059] The corrected image has uniform intrinsic parameters, horizontal epipolar lines and consistent scales, so that subsequent feature matching algorithms (including neural networks) do not need to adapt to the attitude differences of different cameras, improving the universality and stability of the algorithm. The horizontal epipolar line constraint avoids the mismatch of non-corresponding points, and combined with the high parallax sensitivity of the virtual large baseline, the confidence of feature point matching is further improved, providing high-quality matching point pairs for subsequent triangulation. The stereo correction parameters are dynamically calculated based on the camera calibration parameters, and no matter the spreader is at which height in the near-ground operation range, the standardized corrected image can be generated to ensure the stability of the ranging accuracy in the whole operation range.

[0060] In some embodiments, as shown in Figure 7 , the feature point matching and triangulation on the first virtual camera image and the second virtual camera image to obtain the coordinates of the plurality of feature points and the corresponding depth values specifically include: S701: Extract feature points and generate corresponding feature descriptors on the first virtual camera image and the second virtual camera image, respectively; S702: Match the feature points in the first virtual camera image and the second virtual camera image through the feature descriptors to obtain an initial matching point pair set; S703: Screen the initial matching point pair set to obtain a high-confidence matching point pair set, wherein any high-confidence matching point in the first virtual camera image is denoted as p1, the coordinates of p1 are denoted as (u1, v1), and the corresponding matching point in the second virtual camera image is denoted as p2, the coordinates of p2 are denoted as (u2, v2); S704: For each matching point pair (p1, p2) in the high-confidence matching point pair set, construct a homogeneous linear equation set, and use a singular value decomposition algorithm to solve the homogeneous linear equation set to obtain the coordinates P=(X, Y, Z) of the matching point pair (p1, p2) in the three-dimensional space, the homogeneous linear equation set is represented as follows: ; wherein, represents the first row of the projection matrix ; M v1 and M v2 represent the projection matrices of the first virtual camera and the second virtual camera, respectively, and the expression formulas of M v1 and M v2 are as follows: ; ; wherein, represents the intrinsic matrix of the first virtual camera, represents the intrinsic matrix of the second virtual camera, represents an extrinsic matrix when the first virtual camera coordinate system is the world coordinate system, represents a relative pose matrix for transforming a point from the first virtual camera coordinate system to the second virtual camera coordinate system; S705: convert the coordinate P = (X, Y, Z) to the camera coordinate system of the first virtual camera to obtain the coordinate P v1 , and the conversion formula is as follows: , wherein, is the optical center coordinate of the first virtual camera, represents a rotation matrix for converting the coordinate P to the camera coordinate system of the first virtual camera; S706: take the Z-axis component of P v1 as the depth value Z depth corresponding to the matching point p1. S707: traverse all high-confidence matching point pairs to obtain a set of coordinates (u1, v1) of the feature points in the first virtual camera image and the corresponding depth values Z depth .

[0061] In feature point extraction and matching, the balance between real-time performance and robustness is considered, and the ORB algorithm (efficient and real-time) is preferentially used. If the scale invariance requirement is high, the SIFT or SURF algorithm can be selected. In the extraction process, the feature points with high recognition are selected by setting the feature point response threshold, and the low-quality feature points in the weak texture area are avoided. Then, the Hamming distance (binary descriptor) or Euclidean distance (floating-point descriptor) of the feature descriptor of the first virtual camera image and the feature descriptor of the second virtual camera image is calculated, and the similar feature points are preliminarily matched. For each initial matching point pair, the distance ratio of the optimal matching and the suboptimal matching is calculated, and only the matching point pairs with a ratio less than 0.75 are retained, and the fuzzy matching is eliminated. If the feature point p1 in the first virtual camera image matches the feature point p2 in the second virtual camera image, and the optimal matching of the feature point p2 in the second virtual camera image is the feature point p1 in the first virtual camera image, the matching point pair is retained, and the one-way false matching is further eliminated. The output result is a high-confidence matching point pair set, and each point pair contains (p1, p2).

[0062] Then, based on the projection matrix M v1 and M v2For each matching point pair (p1, p2), a 4x4 overdetermined system of equations is constructed. The core constraint of the system is that "the 3D point lies on the imaging rays of the two cameras," i.e., it satisfies the projection relationship. Singular Value Decomposition (SVD) is used to solve the homogeneous linear system of equations. SVD can handle overdetermined equations and obtain the optimal solution, avoiding the numerical instability problems caused by direct solution. After solving, the 3D point P = (X, Y, Z) (world coordinate system, i.e., the first virtual camera coordinate system) is obtained. Then, the 3D point P is transformed to the first virtual camera coordinate system (because M... v1 Using itself as the world coordinate system (the coordinates remain unchanged after transformation), its Z-axis component is the depth value Z corresponding to feature point p1. depth This value represents the physical distance along the camera's optical axis.

[0063] Then, all high-confidence matching point pairs are traversed, and the above triangulation process is repeated to obtain the coordinates of each matching point pair in the first virtual camera image and its corresponding depth value, forming a complete set of depth values, which provides data support for the generation of sparse reference depth maps.

[0064] The above scheme, based on virtual large baseline triangulation, significantly reduces depth measurement error and provides reliable geometric constraints for subsequent neural networks compared to traditional short baseline binocular cameras. Ratio testing and cross-validation effectively eliminate mismatched point pairs, ensuring the accuracy of depth values ​​and avoiding depth estimation bias caused by mismatches. The feature point extraction algorithm is rotation- and scale-invariant, enabling the extraction of stable feature points even in complex scenarios such as port lighting variations and target occlusion, ensuring robustness of matching and triangulation.

[0065] In some embodiments, such as Figure 8 As shown, step S6 specifically includes: S61: Based on the depth values ​​of each pixel in the dense depth map and the intrinsic parameter matrix of the first virtual camera, back-project each pixel onto the coordinate system of the first virtual camera to obtain the first set of 3D point clouds P. d ; S62: Based on the geometric relationship between the first virtual camera and the third camera, the first set of 3D point cloud P d Transform to the coordinate system of the third camera to obtain the second set of 3D point cloud P. c ; S63: Based on the installation pose of the third camera relative to the hoist, the second set of three-dimensional point cloud P c Transform to the coordinate system with the lifting device as the origin to obtain the third set of three-dimensional point clouds P. h ; S64: Extract the third set of three-dimensional point cloud P hThe vertical coordinate component of each point is taken as the vertical distance from the corresponding pixel to the sling, and the ground proximity measurement result is obtained.

[0066] In step S61, for a pixel with coordinates (x, y) and depth value depth in the dense depth map, the three-dimensional coordinates P d =(X d ,Y d ,Z d ) of the pixel in the first virtual camera coordinate system are calculated as follows: X d =(x-c x )×depth / f x ; Y d =(y-c y )×depth / f y ; Z d =depth; where (c x , c y ) are the principal point coordinates, f x , f y are the focal lengths, and all come from the first virtual camera intrinsic matrix K v1 .

[0067] In step S62, the first virtual camera is generated by fusing the third camera (A1) and the first camera G1, and its coordinate system is completely consistent with the A1 coordinate system (both intrinsic and extrinsic parameters are inherited from A1), so the first set of three-dimensional point clouds P d has the same coordinates as the second set of three-dimensional point clouds P c , i.e. P c =P d .

[0068] If there is a slight pose deviation between the virtual camera and A1 (such as geometric error in the fusion process), the correction matrix R cal , T cal can be used for fine tuning to ensure the accuracy of coordinate transformation: P c =R cal ×P d +T cal , and the correction matrix is obtained through calibration experiments.

[0069] In step S63, based on the installation pose of the third camera (A1) relative to the sling, the conversion to the sling coordinate system is performed. After the conversion, the origin of the three-dimensional point cloud P h coincides with the center of the sling, and the coordinate axes are consistent with the motion direction of the sling (such as the X-axis being parallel to the length direction of the sling, and the Y-axis being perpendicular upward), directly reflecting the spatial position of the target relative to the sling.

[0070] In step S64, in the sling coordinate system, the vertical direction is the Y-axis direction, so the third set of three-dimensional point clouds P h The Y coordinate component of each point is the vertical distance of the point to the sling. Extracting the Y coordinate components corresponding to all pixels forms a vertical distance matrix aligned with the pixels of the first view image, which is the near-ground ranging result and can be directly output to the crane automatic control system for sling positioning, collision avoidance, and other operations.

[0071] The above scheme converts the abstract dense depth map into a vertical distance in the sling coordinate system, directly matches the input requirements of the crane control system, and does not require additional data conversion, thereby improving operation efficiency. Through a three-level coordinate conversion chain, the camera calibration parameters and installation pose are strictly followed to ensure that the measurement result is the true vertical distance relative to the sling regardless of the height and attitude of the sling, and there is no system deviation.

[0072] In some embodiments, the depth estimation neural network model is trained by the following method: S81: Construct a training sample set, each sample in the training sample set includes a stereo image pair composed of a first training view image and a second training view image, and a corresponding sparse training depth map; wherein the sparse training depth map is generated by randomly sampling or noise simulation on a dense ground truth depth map aligned with the pixels of the first training view image; S82: input the training sample set into the depth estimation neural network model, and the model performs the following operations: perform feature extraction and stereo matching on the stereo image pair to generate an initial depth estimate; use the sparse training depth map as guide information to fuse with the initial depth estimate during the stereo matching process or in a subsequent optimization process; output depth maps at different stages or different resolutions, including a first dense depth map and a second dense depth map; wherein the first dense depth map is an intermediate coarse depth map, and the second dense depth map is a final output fine depth map; S83: for the first dense depth map, calculate the difference between the first dense depth map and a first reference dense depth map to obtain a first loss L coarse , the first reference dense depth map is obtained by spatially downsampling a second reference dense depth map; for the second dense depth map, calculate the difference between the second dense depth map and a second reference dense depth map to obtain a second loss L final , the second reference dense depth map is the dense ground truth depth map; For the sparse pixel points with valid depth values in the sparse training depth map, the difference between the predicted depth value of the second dense depth map at the corresponding pixel point and the sparse ground truth depth value is calculated, and a third loss L sparse ; Based on the first loss L coarse , the second loss L final and the third loss L sparse , a total loss function is constructed, and the formula of the total loss function is as follows: L total =L final +α·L coarse +β·L sparse ; Wherein, L total represents the total loss, and α and β are preset weight coefficients. S84: The total loss L total is used to update the parameters of the depth estimation neural network model through a back propagation algorithm. S85: Steps S82 to S84 are repeated until the model converges, and a pre-trained depth estimation neural network model is obtained.

[0073] The above scheme uses multi-stage supervision and sparse depth prior guidance. The model not only learns the mapping relationship between image features and depth, but also masters the geometric constraint rules. When deployed across scenes, it does not need to be retrained, and only needs to load a new scene calibration file. Triple loss supervision ensures that the model not only guarantees the accuracy of the global geometric structure, but also optimizes the local details. The final fine depth map has a precision close to that of a laser radar, meeting the high-precision requirements of port near-ground operations. The sparse depth prior provides strong geometric constraints for the model. Even in scenes such as weak texture and occlusion, which are difficult for traditional stereo matching, the model can still generate reliable dense depth maps based on sparse anchor points, and the robustness is better than that of a pure data-driven model.

[0074] In this embodiment, the depth estimation neural network model includes five core modules, which work together to achieve high-precision depth estimation. Specifically, it includes: A multi-modal feature extraction network module includes a twin feature network (weight sharing) and a depth prior encoder, which respectively extract image features (low resolution 32 channels, high resolution 16 channels) of a stereo image pair and depth prior features (8 channels) of a sparse depth map, and concatenate and fuse the image features and the depth prior features to obtain 40-channel fusion features.

[0075] A 4D cost space construction module constructs a 4D cost space with a dimension of [B, 48, 120, 160] (B is the batch size, 48 is the disparity range, and 120x160 is the low resolution feature size) for low resolution features (downsampling factor 4), and represents the matching cost by calculating the correlation of left and right features.

[0076] 3D cost space regularization network module, adopts 3D U-Net architecture, through 3D convolution, pooling, deconvolution and skip connection, smoothes and contextually aggregates 4D cost space, highlights low cost value corresponding to correct disparity, and generates regularized cost space.

[0077] Differentiable disparity regression module, through Soft-Argmin operation, converts regularized cost space into probability distribution, calculates weighted average disparity, and obtains low-resolution rough disparity map (120*160), and supports end-to-end training.

[0078] Depth map optimization and upsampling module, through bilinear interpolation, the rough disparity map is upsampled to 480*640, spliced with high-resolution left features, and outputted fine disparity map through lightweight 2D CNN optimization network, and finally converted into dense depth map.

[0079] In a second aspect, the present application further provides a computer readable storage medium, having a computer program stored thereon, wherein the computer program is executed by a processor to realize the method for near-ground ranging based on virtual large-baseline four-vision according to the first aspect of the present application.

[0080] The computer readable storage medium can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories.

[0081] The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD ROM).

[0082] The volatile memory can be a Random Access Memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Sync Link Dynamic Random Access Memory (SLDRAM), Direct Rambus Random Access Memory (DRRAM). The computer-readable storage media described in the embodiments of the present application are intended to include these and any other suitable types of memory.

[0083] As shown in Figure 9 In a third aspect, the present application provides an electronic device 10 comprising a processor 101 and a storage medium 102, wherein the storage medium 102 has stored thereon a computer program which, when executed by the processor 101, implements the method for near-ground ranging based on virtual large-baseline four-eye vision according to the first aspect of the present application.

[0084] In some embodiments, the processor can be implemented by software, hardware, firmware or a combination thereof, and can use at least one of circuit, single or multiple Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), controller, microcontroller, microprocessor, so that the processor can execute part or all of the steps or any combination of the steps in the virtual large baseline four-vision based ground proximity ranging method in various embodiments of the present application.

[0085] Finally, it should be noted that although the above embodiments have been described in the specification and drawings of the present application, the patent protection scope of the present application should not be limited thereby. Any technical solutions obtained by replacing or modifying the equivalent structure or equivalent flow based on the essential concept of the present application, using the content described in the specification and drawings of the present application, and directly or indirectly implementing the technical solutions of the above embodiments in other related technical fields, etc., are all included in the patent protection scope of the present application.

Claims

1. A virtual long baseline four-eye vision based ground proximity ranging method applied to a container crane, the crane comprising a gantry structure and a spreader vertically movable along the gantry structure, characterized in that, The gantry structure is provided with a first camera and a second camera at a first preset height, and the spreader is provided with a third camera and a fourth camera at a second preset height, the first preset height being higher than the second preset height; The method comprises: S1: when the spreader enters a preset near-ground operation range, synchronously triggering and acquiring a first image collected by the first camera, a second image collected by the second camera, a third image collected by the third camera and a fourth image collected by the fourth camera; S2: based on a calibration relationship between the first camera and the third camera, projecting and transforming the first image to the visual angle of the third camera, and fusing with the third image to generate a first virtual camera image, and based on a calibration relationship between the second camera and the fourth camera, projecting and transforming the second image to the visual angle of the fourth camera, and fusing with the fourth image to generate a second virtual camera image; S3: performing stereo rectification on the first virtual camera image and the second virtual camera image to obtain a pair of rectified images in line alignment; cropping a common field of view region from the pair of rectified images to obtain a first visual angle image and a second visual angle image which are in line alignment and have the same size; S4: performing feature point matching and triangulation on the first virtual camera image and the second virtual camera image to obtain coordinates and corresponding depth values of a plurality of feature points; filling the depth values into a two-dimensional matrix with the same size as the first visual angle image according to the coordinate mapping relationship between the plurality of feature points in the first virtual camera image and the first visual angle image to generate a sparse reference depth map in pixel alignment with the first visual angle image, wherein the matrix elements without filled depth values are empty or zero; S5: inputting the first visual angle image, the second visual angle image and the sparse reference depth map into a pre-trained depth estimation neural network model to output a dense depth map in pixel alignment with the first visual angle image; S6: based on the depth values of each pixel in the dense depth map, the intrinsic matrix of the first virtual camera, the geometric relationship between the first virtual camera and the third camera and the installation pose of the third camera relative to the spreader, converting the coordinates of each pixel to a coordinate system with the spreader as the origin, and taking the vertical component of the coordinates as the vertical distance from the pixel to the spreader to obtain the near-ground ranging result.

2. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 1, wherein, The projecting and transforming the first image to the perspective of the third camera based on the calibration relationship between the first camera and the third camera comprises: calculating a first homography matrix between the first camera and the third camera , and performing a first perspective transformation on the first image based on the first homography matrix The calculation formula of the first homography matrix is as follows: ； wherein G1 represents the first camera, A1 represents the third camera, h represents a height when the spreader enters a preset near-ground operation range, K A1 represents an intrinsic matrix of the third camera, represents a rotation matrix from the first camera to the third camera at the height h, represents a translation vector from the origin of the coordinate system of the first camera to the origin of the coordinate system of the third camera at the height h, and n represents a unit normal vector of a preset reference plane, represents a vertical distance from the origin of the world coordinate system to the preset reference plane, represents an inverse matrix of the intrinsic matrix of the first camera; The projecting and transforming the second image to the perspective of the fourth camera based on the calibration relationship between the second camera and the fourth camera comprises: calculating a second homography matrix between the second camera and the fourth camera , and performing a second perspective transformation on the second image based on the second homography matrix The calculation formula of the second homography matrix is as follows: ； wherein G2 represents the second camera, A2 represents the fourth camera, K A2 represents an intrinsic matrix of the fourth camera, represents a rotation matrix from the second camera to the fourth camera at height h, represents a translation vector from the origin of the coordinate system of the second camera to the origin of the coordinate system of the fourth camera at height h, represents an inverse matrix of the intrinsic matrix of the second camera.

3. The near-ground ranging method based on virtual large-baseline four-eye vision according to claim 2, wherein the first virtual camera image is generated in the following manner: aligning the first image after the first perspective transformation with the original image of the third image in the image space to determine the overlapping region between the two images; in the overlapping region, performing pixel-by-pixel fusion on the corresponding pixels of the first image after the first perspective transformation and the original image of the third image using a weight-based fusion algorithm to generate the first virtual camera image with an extended field of view; the second virtual camera image is generated in the following manner: aligning the second image after the second perspective transformation with the original image of the fourth image in the image space to determine the overlapping region between the two images; In the overlapping region, corresponding pixels of the second perspective-transformed second image and the original fourth image are fused pixel by pixel by using a weight-based fusion algorithm to generate the second virtual camera image with an extended field of view.

4. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 3, wherein, The weight-based fusion algorithm includes a multi-band fusion algorithm. The weight-based fusion algorithm includes a multi-band fusion algorithm. The weight-based fusion algorithm includes a multi-band fusion algorithm. The first and third images are respectively subjected to Laplacian pyramid decomposition; At each layer of the Laplacian pyramid, a first fusion weight map is calculated according to the position of the pixel in the overlapping region; The pyramid coefficients of the corresponding layers of the first and third images are weighted and fused using the first fusion weight map; The weighted and fused Laplacian pyramid is reconstructed to generate the first virtual camera image with an extended field of view. The weight-based fusion algorithm includes a multi-band fusion algorithm. The second and fourth images are respectively subjected to Laplacian pyramid decomposition; At each layer of the Laplacian pyramid, a second fusion weight map is calculated according to the position of the pixel in the overlapping region; The pyramid coefficients of the corresponding layers of the second and fourth images are weighted and fused using the second fusion weight map; 5. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 1, wherein, The weighted and fused Laplacian pyramid is reconstructed to generate the second virtual camera image with an extended field of view. The first and second virtual camera images are subjected to stereo rectification to obtain a pair of row-aligned rectified images, including the following steps: S31: based on the camera parameters associated with the first and second virtual camera images, calculate stereo rectification parameters; the camera parameters include the relative pose between the first and second virtual cameras and the respective intrinsic matrices; ； ； wherein, (x, y) represents pixel coordinates of the first virtual camera image before correction, , (x, y) represents pixel coordinates of the first virtual camera image after correction, , (x, y) represents pixel coordinates of the second virtual camera image before correction, , (x, y) represents pixel coordinates of the second virtual camera image after correction, , (x, y) represents pixel coordinates of the second virtual camera image after correction, represents an inverse matrix of the intrinsic matrix of the first virtual camera , represents an inverse matrix of the intrinsic matrix of the second virtual camera , represents a correction rotation matrix of the first virtual camera, represents a correction rotation matrix of the second virtual camera, represents an intrinsic matrix after correction and unification.

6. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 1, wherein, S32: using the calculated stereo rectification parameters, respectively performing geometric transformation on the first and second virtual camera images to obtain a pair of row-aligned rectified images, the calculation formula of the geometric transformation is as follows: The feature point matching and triangulation on the first and second virtual camera images are performed to obtain the coordinates of a plurality of feature points and their corresponding depth values, including: Feature points are extracted and corresponding feature descriptors are generated on the first and second virtual camera images; The feature points in the first and second virtual camera images are matched by using the feature descriptors to obtain an initial matching point pair set; The initial matching point pair set is screened to obtain a high-confidence matching point pair set, wherein any high-confidence matching point in the first virtual camera image is denoted as p1, the coordinates of p1 are denoted as (u1, v1), and the corresponding matching point in the second virtual camera image is denoted as p2, the coordinates of p2 are denoted as (u2, v2); The feature points in the first and second virtual camera images are matched by using the feature descriptors to obtain an initial matching point pair set; The initial matching point pair set is screened to obtain a high-confidence matching point pair set, wherein any high-confidence matching point in the first virtual camera image is denoted as p1, the coordinates of p1 are denoted as (u1, v1), and the corresponding matching point in the second virtual camera image is denoted as p2, the coordinates of p2 are denoted as (u2, v2); For each pair of matching points (p1, p2) in the set of high-confidence matching point pairs, a homogeneous linear equation set is constructed, and the homogeneous linear equation set is solved using a singular value decomposition algorithm to obtain the coordinates P=(X, Y, Z) of the matching point (p1, p2) in the three-dimensional space, and the homogeneous linear equation set is expressed as follows: ； wherein, represents the projection matrix of the first row of the projection matrix v1 and M v2 represent the projection matrix of the first virtual camera and the second virtual camera, respectively, and the representation formulae of M v1 and M v2 are as follows: ； ； wherein, represents an intrinsic matrix of the first virtual camera, represents an intrinsic matrix of the second virtual camera, represents an extrinsic matrix when the first virtual camera coordinate system is the world coordinate system, represents a relative pose matrix transforming a point from the first virtual camera coordinate system to the second virtual camera coordinate system; convert the coordinate P=(X, Y, Z) to the camera coordinate system of the first virtual camera to obtain a coordinate P v1 , and the conversion formula is as follows: , wherein, is the optical center coordinate of the first virtual camera; and taking the Z-axis component of P v1 as a depth value Z depth corresponding to the matching point p1. indicates a rotation matrix for converting the coordinate P to the camera coordinate system of the first virtual camera. Iterate through all high-confidence matching point pairs to get the coordinates (u1, v1) of the multiple feature points in the first virtual camera image and their corresponding depth values Z depth of the set.

7. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 1, wherein, The step S6 specifically includes: S61: based on the depth value of each pixel in the dense depth map and the intrinsic matrix of the first virtual camera, each pixel is back-projected to the coordinate system of the first virtual camera to obtain a first set of three-dimensional point clouds P d ; S62: based on the geometric relationship between the first virtual camera and the third camera, convert the first set of three-dimensional point clouds P d into the coordinate system of the third camera to obtain a second set of three-dimensional point clouds P c ; S63: based on the mounting pose of the third camera relative to the lifting appliance, the second set of three-dimensional point clouds P c converted into the coordinate system with the lifting appliance as the origin, a third set of three-dimensional point clouds P h ; S64: extract the third set of three-dimensional point clouds P h The vertical direction coordinate component of each point in the ground plane as the vertical distance from the corresponding pixel to the sling, and the near-ground ranging result is obtained.

8. The virtual Great Baseline Four-ocular Vision based ground proximity ranging method of claim 1, wherein, The depth estimation neural network model is trained in the following manner: S81: Construct a training sample set, each sample in the training sample set includes a stereo image pair composed of a first training perspective image and a second training perspective image, and a corresponding sparse training depth map; wherein the sparse training depth map is generated by randomly sampling or noise simulation on a dense ground truth depth map aligned with pixels of the first training perspective image; S82: input the training sample set into the depth estimation neural network model, and the model performs the following operations: feature extraction and stereo matching on the stereo image pair to generate an initial depth estimation; The sparse training depth map is used as guide information to fuse with the initial depth estimation during the stereo matching process or a subsequent optimization process; output depth maps at different stages or different resolutions, including a first dense depth map and a second dense depth map; wherein the first dense depth map is an intermediate coarse depth map, and the second dense depth map is a final output fine depth map; S83: For the first dense depth map, calculate the difference between the first dense depth map and a first reference dense depth map to obtain a first loss L coarse , the first reference dense depth map is obtained by spatially down-sampling a second reference dense depth map; For the second dense depth map, a difference between the second dense depth map and a second reference dense depth map is computed, a second loss L final , the second reference dense depth map being the dense ground truth depth map; For the sparse pixel points with valid depth values in the sparse training depth map, the difference between the predicted depth value of the second dense depth map on the corresponding pixel point and the sparse ground truth depth value is calculated, and a third loss L sparse ; based on the first loss L coarse , the second loss L final , and the third loss L sparse The total loss function is constructed, and the total loss function is as follows: L total = L final + a · L coarse + β · L sparse ; wherein L total represents the total loss, and a and β are preset weight coefficients. S84: Utilize total loss L total updating parameters of the depth estimation neural network model by a backpropagation algorithm; S85: repeat steps S82 to S84 until the model converges to obtain a pre-trained depth estimation neural network model.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the virtual large-baseline four-eye vision based near-ground ranging method of any one of claims 1 to 8.

10. An electronic device on which a computer program is stored, characterized by The program is executed by the processor to implement the virtual large-baseline four-eye vision based near-ground ranging method of any one of claims 1 to 8. The program is executed by the processor to implement the virtual large-baseline four-eye vision based near-ground ranging method of any one of claims 1 to 8.

Images