An autonomous method of operation of a deep sea mining vehicle

Abstract

The application discloses a kind of autonomous operation methods of deep-sea mining car, and mining car is equipped with sonar sensor, camera sensor and control unit.Through sonar, three-dimensional point cloud information of seabed operation area is obtained, and synchronous positioning and map construction are carried out in combination with vehicle pose, to form global point cloud map;Based on the map, cubic surface fitting is carried out on seabed topography, to generate continuous three-dimensional curved surface model;Through camera, seabed image is collected and sulfide target is identified, in combination with camera calibration parameter, vehicle real-time pose and curved surface model, two-dimensional target information is mapped to global coordinate system, to obtain target three-dimensional space position;Construct the grid map containing sulfide target distribution and operation state mark, generate target-oriented mining operation path, and control mining car to execute mining operation;After operation is completed, detection and identification are carried out again using sonar and camera, and operation completion state is updated.The application improves target positioning accuracy, path planning efficiency and autonomous operation ability in deep-sea mining.

Description

Technical Field

[0001] This invention relates to the fields of deep-sea mineral resource development and mobile robot technology, specifically to an autonomous operation method for a deep-sea mining vehicle. Background Technology

[0002] With the increasing demand for deep-sea mineral resource development, relying on deep-sea mining vehicles to perform autonomous exploration, mineral identification, path planning, and mineral collection in complex seabed environments has become a key development direction in the field of deep-sea mining equipment. Since satellite positioning signals cannot be received in deep-sea areas, mining vehicles need to combine multiple technologies such as acoustic positioning, sonar mapping, inertial navigation, visual recognition, and path planning to support autonomous operations. Three-dimensional modeling of the seabed topography is a core prerequisite for achieving positioning and path planning for mining vehicles.

[0003] The current mainstream underwater terrain 3D modeling results are mainly divided into two categories: raster elevation maps and discrete point clouds. However, both have inherent defects: raster elevation maps have a fixed resolution and are prone to step distortion and loss of terrain details when faced with complex seabed terrain with drastic undulations; discrete point clouds can maintain high measurement accuracy, but they do not have a continuously differentiable mathematical representation and cannot provide stable geometric support for the analytical projection of two-dimensional visual mineral targets into three-dimensional seabed space.

[0004] Existing related patent technologies also suffer from shortcomings in data fusion. Chinese patent CN113433553B discloses a multi-source acoustic information fusion precision navigation scheme for underwater robots, relying on multi-beam sonar and underwater acoustic beacon communication to collect seabed topographic mapping data and distance observation data, achieving real-time high-precision positioning of the autonomous underwater vehicle (AUV). Chinese patent CN121937472A discloses a multi-modal data fusion method for deep-sea polymetallic nodule image segmentation, fusing light and acoustic image features to automatically segment nodule mineral regions, using only visual data for two-dimensional detection and classification of mineral targets. Both of these schemes treat seabed topographic data and visual image data as two independent processing links, failing to construct a geometric fusion mechanism based on camera projection models and seabed topographic models. This results in the inability to accurately map mineral targets obtained from image recognition to three-dimensional seabed topography, significantly reducing the accuracy of spatial positioning of mineral targets and the precision of mining operation area delineation.

[0005] In addition, the high complexity of existing path planning technologies is insufficiently adapted to the actual working conditions of deep-sea mining. Chinese patent CN117555341B discloses a path planning method and system for deep-sea mining vehicles based on an improved ant colony algorithm. This method relies solely on a grid map containing obstacles, passage areas, and terrain elevation for path optimization. Its planning objectives are limited to target point-driven approaches, obstacle avoidance, and optimal paths, without incorporating mineral spatial distribution information. Furthermore, the algorithm employs complex solution logic to adapt to 3D obstacle avoidance search, resulting in high computational load and insufficient real-time performance. In actual deep-sea mining scenarios, mining vehicles can flexibly adjust their operating height using biomimetic mechanisms or propellers. Terrain undulations in mineral-rich areas generally do not pose significant obstacles. Redundant and complex 3D path search algorithms would increase the computational load on the onboard control system, failing to meet the operational requirements of efficient, real-time, and autonomous mining by deep-sea mining vehicles. Summary of the Invention

[0006] The purpose of this invention is to provide an autonomous operation method for deep-sea mining vehicles, so as to at least solve the technical problems in existing deep-sea mining operations, such as insufficient accuracy of seabed topography modeling, independence of sonar topography modeling and visual target recognition, inaccurate spatial positioning of mineral targets, lack of target guidance in path planning, and high computational complexity of three-dimensional path planning.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: an autonomous operation method for a deep-sea mining vehicle, wherein the deep-sea mining vehicle is equipped with a sonar sensor, a camera sensor and a control unit, and the autonomous operation method includes the following steps: Step S1, using the sonar sensor to detect the seabed operation area and obtain the three-dimensional point cloud information of the seabed operation area.

[0008] Step S2: Combine the pose information of the deep-sea mining vehicle with the three-dimensional point cloud information to perform synchronous positioning and map construction to obtain a global point cloud map.

[0009] Step S3: Based on the global point cloud map, perform cubic surface fitting on the seabed topography of the seabed operation area to generate a continuous three-dimensional surface model for representing the undulation of the seabed topography.

[0010] Step S4: Acquire image information of the seabed operation area through the camera sensor, identify sulfide targets in the image information, and obtain two-dimensional target information of the sulfide targets.

[0011] Step S5: Based on the continuous three-dimensional surface model, the two-dimensional target information, the camera calibration parameters, and the real-time pose of the deep-sea mining vehicle, the sulfide target is mapped from the image coordinate system to the global coordinate system to obtain the three-dimensional spatial position information of the sulfide target.

[0012] Step S6: Fuse the three-dimensional spatial location information of the sulfide target with the continuous three-dimensional surface model to construct an operation map containing the distribution information of the sulfide target, and convert the operation map into a raster map containing sulfide target distribution markers and operation completion status markers.

[0013] Step S7: Generate a target-oriented mining operation path covering the sulfide target area based on the grid map, and control the deep-sea mining vehicle to perform mining operations according to the target-oriented mining operation path.

[0014] Step S8: After the mining operation is completed, the seabed operation area is re-detected and identified by the sonar sensor and the camera sensor, and the operation completion status in the grid map is updated according to the re-detection and identification results.

[0015] As a further improvement of the present invention, in step S1, the step of detecting the seabed operation area by the sonar sensor and obtaining the three-dimensional point cloud information of the seabed operation area specifically includes: step S11, transmitting an acoustic detection signal to the seabed operation area by the sonar sensor and receiving the acoustic echo signal returned by the seabed operation area.

[0016] Step S12: Perform echo processing, distance calculation, and spatial coordinate transformation on the acoustic echo signal to generate three-dimensional point cloud information containing three-dimensional spatial coordinates and distance information of multiple detection points.

[0017] Step S13: Use the three-dimensional point cloud information as the basic environmental data for the deep-sea mining vehicle to perform synchronous positioning and map construction, seabed topography fitting and target spatial projection.

[0018] As a further improvement of the present invention, in step S2, the step of combining the pose information of the deep-sea mining vehicle and the three-dimensional point cloud information to perform synchronous positioning and map construction to obtain a global point cloud map specifically includes: step S21, converting the height information in the three-dimensional point cloud information into a two-dimensional height image.

[0019] Step S22: Extract feature points from the two-dimensional height image to obtain stable feature points for characterizing seabed topography.

[0020] Step S23: Perform nearest neighbor matching on the stable feature points in different data frames to obtain the initial inter-frame matching relationship.

[0021] Step S24: Optimize the initial inter-frame matching relationship using the random sampling consensus algorithm, eliminate abnormal matches, and obtain the optimized inter-frame matching relationship.

[0022] Step S25: Calculate the spatial transformation relationship between different data frames based on the optimized inter-frame matching relationship, and combine the odometer information or motion pose information to perform pose correction and stitching on the three-dimensional point cloud information to construct a globally consistent point cloud map.

[0023] As a further improvement of the present invention, in step S3, the step of performing cubic surface fitting on the seabed topography of the seabed operation area based on the global point cloud map to generate a continuous three-dimensional surface model for characterizing the undulation of the seabed topography specifically includes: step S31, selecting point cloud data for seabed topography fitting from the global point cloud map within a preset range around the deep-sea mining vehicle.

[0024] Step S32: Establish a cubic surface fitting expression based on the point cloud data. The cubic surface fitting expression is used to describe the continuous relationship between the seabed topographic height and the planar coordinates.

[0025] Step S33: Solve the model parameters of the cubic surface fitting expression using the linear least squares method to obtain a continuous three-dimensional surface model.

[0026] Step S34: Calculate the maximum fitting error and the fitting variance of the continuous three-dimensional surface model, and determine whether the continuous three-dimensional surface model meets the preset accuracy requirements based on the maximum fitting error and the fitting variance.

[0027] Step S35: When the continuous three-dimensional surface model meets the preset accuracy requirements, the continuous three-dimensional surface model is used as the seabed topography model of the current seabed operation area.

[0028] As a further improvement of the present invention, step S3 further includes: step S36, when the maximum fitting error or the fitting variance does not meet the preset accuracy requirement, the preset range is narrowed and point cloud data is reselected.

[0029] Step S37: Perform three more surface fittings based on the reselected point cloud data until a continuous three-dimensional surface model that meets the preset accuracy requirements is obtained.

[0030] Step S38: Update the continuous three-dimensional curved surface model based on the three-dimensional point cloud information acquired in real time by the sonar sensor.

[0031] As a further improvement of the present invention, in step S4, the step of acquiring image information of the seabed operation area through the camera sensor and identifying sulfide targets in the image information to obtain two-dimensional target information of the sulfide targets specifically includes: step S41, acquiring visible light images of the seabed operation area through the camera sensor.

[0032] Step S42: Perform image enhancement processing on the visible light image. The image enhancement processing includes homomorphic filtering, automatic white balance, and dark channel prior dehazing.

[0033] Step S43: Input the enhanced image into the pre-trained YOLO target recognition model to detect and recognize sulfide targets in the image.

[0034] Step S44: Output the coordinates, category information, and confidence information of the two-dimensional bounding box of the sulfide target, and extract the pixel coordinates of the four corner points of the two-dimensional bounding box.

[0035] Step S45: Use the two-dimensional bounding box coordinates, category information, confidence information, and the pixel coordinates of the four corner points as the two-dimensional target information of the sulfide target.

[0036] As a further improvement of the present invention, in step S5, the step of mapping the sulfide target from the image coordinate system to the global coordinate system based on the continuous three-dimensional surface model, the two-dimensional target information, the camera calibration parameters and the real-time pose of the deep-sea mining vehicle to obtain the three-dimensional spatial position information of the sulfide target specifically includes: step S51, extracting the pixel coordinates of the four corner points of the two-dimensional bounding box corresponding to each sulfide target in the two-dimensional target information.

[0037] Step S52: Based on the camera intrinsic parameters, camera extrinsic parameters, and the real-time pose of the deep-sea mining vehicle, determine the spatial projection ray corresponding to the pixel coordinates of each corner point.

[0038] Step S53: Solve the system of equations between the spatial projection ray and the continuous three-dimensional surface model to obtain the three-dimensional projection point of each corner pixel coordinate on the continuous three-dimensional surface model.

[0039] Step S54: Connect the four 3D projection points corresponding to the same 2D bounding box to form a 3D mapped quadrilateral.

[0040] Step S55: Determine the three-dimensional spatial distribution range of the sulfide target in the global coordinate system based on the three-dimensional mapping quadrilateral.

[0041] As a further improvement of the present invention, in step S6, the three-dimensional spatial location information of the sulfide target is fused with the continuous three-dimensional surface model to construct an operation map containing the distribution information of the sulfide target, and the operation map is converted into a grid map containing the distribution markers of the sulfide target and the operation completion status markers. Specifically, step S61 is to establish a reference plane for path planning and coordinate calculation.

[0042] Step S62: Project the three-dimensional spatial position information of the sulfide target onto the reference plane to obtain the planar projection area of ​​the sulfide target on the reference plane.

[0043] Step S63: Generate the minimum envelope rectangle based on the planar projection region corresponding to each sulfide target, and define the set of all minimum envelope rectangles as the current global sulfide target distribution region.

[0044] Step S64: Construct a two-dimensional map based on the current global sulfide target distribution area and the continuous three-dimensional surface model.

[0045] Step S65: Convert the two-dimensional map into a raster map, wherein each raster cell contains at least terrain height information, sulfide target distribution marker information, and operation completion status marker information.

[0046] As a further improvement of the present invention, in step S7, a target-oriented mining operation path covering the sulfide target area is generated based on the grid map, and the deep-sea mining vehicle is controlled to perform mining operations according to the target-oriented mining operation path. Specifically, step S71 is to set the current position of the deep-sea mining vehicle in the reference plane coordinate system as the starting point of the path planning.

[0047] Step S72: Establish a preset search area centered on the starting point of the path planning, and search for the corner point of the minimum envelope rectangle corresponding to the sulfide target distribution area within the preset search area.

[0048] Step S73: When the minimum envelope rectangle corner point does not exist within the preset search area, expand the preset search area and search again.

[0049] Step S74: Calculate the planar Euclidean distance between each corner point of the minimum envelope rectangle obtained from the search and the starting point of the path planning, and determine the corner point of the minimum envelope rectangle with the smallest distance as the current path planning endpoint.

[0050] Step S75: Generate a straight path based on the path planning starting point and the current path planning ending point, as the transfer path for the deep-sea mining vehicle to reach the current sulfide target distribution area.

[0051] Step S76: Generate a round-trip coverage path for the current sulfide target distribution area, and enable the deep-sea mining vehicle to traverse the current sulfide target distribution area along the round-trip coverage path.

[0052] Step S77: After completing the operation in the current sulfide target distribution area, take the current path endpoint as the starting point of the next path planning, and continue to plan the path for the remaining sulfide target distribution areas until all grid cells marked with sulfide targets in the grid map have been traversed and covered.

[0053] As a further improvement of the present invention, in step S8, after the mining operation is performed, the seabed operation area is re-detected and identified by the sonar sensor and the camera sensor, and the operation completion status in the grid map is updated according to the re-detection and identification results. Specifically, step S81 is to extract the target point plane coordinates corresponding to the current execution step length according to the preset execution step length during the mining operation performed by the deep-sea mining vehicle.

[0054] Step S82: Generate the three-dimensional coordinates of the target point based on the plane coordinates of the target point and the elevation information of the continuous three-dimensional surface model at the plane coordinates of the target point.

[0055] Step S83: Based on the current three-dimensional coordinates of the deep-sea mining vehicle and the three-dimensional coordinates of the target point, generate control commands for controlling the movement of the deep-sea mining vehicle and mining operations.

[0056] Step S84: After the deep-sea mining vehicle completes the path execution, the sonar sensor and the camera sensor are used to perform secondary detection and image acquisition of the seabed operation area.

[0057] Step S85: Based on the secondary detection and image acquisition results, detect and evaluate the sulfide target processing status, path execution deviation, and area coverage, and generate operation execution result data.

[0058] Step S86: Feed back the job execution result data to the grid map, mark the grid cells of completed jobs, and update the global job completion status.

[0059] Compared with existing technologies, the present invention has the following advantages: 1. The present invention acquires three-dimensional point cloud information of the seabed operating area through sonar sensors, and performs cubic surface fitting based on the three-dimensional point cloud information to generate a continuous three-dimensional surface model. Compared with traditional raster elevation maps or discrete point cloud maps, the present invention can represent the seabed topographic undulations in the form of a continuous surface, reducing the staircase effect and detail loss caused by discretization modeling, improving the accuracy of restoring complex seabed topography, and providing a more accurate topographic geometric benchmark for subsequent spatial projection and path execution of sulfide targets.

[0060] 2. This invention fuses a continuous three-dimensional curved surface model constructed by sonar with two-dimensional sulfide target information obtained by camera recognition. Combining camera calibration parameters and the real-time pose of the mining vehicle, the sulfide target in the image is projected onto a three-dimensional seabed topography model, thereby obtaining the three-dimensional spatial position information of the sulfide target in the global coordinate system. This overcomes the problem of independent sonar topography modeling and visual target recognition in existing technologies, achieving integrated processing of "topography modeling, target recognition, and three-dimensional positioning," thus improving the accuracy and reliability of sulfide target positioning.

[0061] 3. This invention integrates the spatial distribution information of sulfide targets with seabed topographic information to construct an operational map containing sulfide target distribution markers. It further generates a raster map, enabling path planning to move beyond solely relying on obstacle or terrain elevation information and instead perform target-oriented planning based on the actual distribution of sulfide targets. By prioritizing coverage of sulfide target areas, it reduces ineffective travel by mining vehicles in non-target areas, lowers energy consumption, and improves the efficiency of deep-sea sulfide extraction operations.

[0062] 4. This invention leverages the mobility of deep-sea mining vehicles, enabling them to adapt to undulating seabed terrain and adjust their altitude. It employs straight paths as the primary transfer routes between target areas and a round-trip coverage path within a single sulfide target distribution area. This approach avoids the use of complex 3D path search algorithms in unnecessary scenarios, reduces the computational complexity of path planning and the difficulty of implementing control systems, and improves the real-time performance and engineering applicability of path planning.

[0063] 5. After mining operations are completed, this invention uses sonar and camera sensors to conduct secondary detection and image acquisition of the work area. It detects and evaluates the processing of sulfide targets, path execution deviations, and area coverage, and feeds the results back to the raster map to update the operation completion status. This forms a closed-loop control process from environmental perception, positioning and mapping, target identification, path planning, operation execution to result detection and progress updates, facilitating real-time monitoring and management of deep-sea mining operations and improving the continuity and reliability of autonomous operations.

[0064] 6. This invention, through the coordinated work of sonar sensing and visual recognition, can provide deep-sea mining vehicles with stable environmental perception, target positioning, and operation planning capabilities in deep-sea environments with no GPS signals, insufficient light, and complex terrain. It improves the robustness, accuracy, and automation level of sulfide target identification, three-dimensional positioning, and autonomous collection operations in complex deep-sea environments. Attached Figure Description

[0065] Figure 1 This is a flowchart of the autonomous operation method of the deep-sea mining vehicle of the present invention.

[0066] Figure 2 This is a schematic diagram illustrating the visual recognition information projection principle of the present invention.

[0067] Figure labeling: 1. Camera coordinate origin; 2. Camera imaging plane; 3. YOLO recognition bounding box; 4. Cubic surface; 5. 3D mapped quadrilateral; 6. Reference plane; 7. Plane projection point; 8. Minimum envelope rectangle. Detailed Implementation

[0068] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the following description is provided in conjunction with... Figure 1 and Figure 2 The following is a further description of a specific embodiment of the present invention. It should be noted that the deep-sea mining vehicle in this embodiment is only used as a mobile platform carrying sonar sensors, camera sensors, control units, and work execution mechanisms. This application does not limit the specific mechanical structure of the deep-sea mining vehicle, which achieves buoyancy through biomimetic flexible fins or propellers.

[0069] This embodiment takes the autonomous detection, spatial positioning, target-guided path planning, and mining operations of seabed sulfide targets as an example. Figure 1 As shown, the autonomous operation method includes, in sequence, three-dimensional point cloud acquisition, synchronous positioning and map construction, seabed topography cubic surface fitting, sulfide target identification, mapping of two-dimensional targets to three-dimensional topography, target distribution grid map construction, target-oriented path planning and execution, and operation result verification and status update.

[0070] Before executing this embodiment, a global coordinate system, a camera coordinate system, an image coordinate system, and a reference plane coordinate system for path planning are established. The camera's intrinsic parameters and extrinsic parameters relative to the mobile platform are pre-calibrated. The real-time pose of the mobile platform is used to determine the coordinate transformation relationship between the camera coordinate system and the global coordinate system.

[0071] Step S1: Acquire three-dimensional point cloud information of the seabed operating area. Control the sonar sensor to emit acoustic detection signals towards the seabed operating area and receive acoustic echo signals returned by the seabed topography and seabed targets. The control unit performs echo processing, distance calculation, and spatial coordinate transformation on the acoustic echo signals to obtain point cloud data frames that are continuously output over time. Each detection point includes at least the three-dimensional spatial coordinates and distance information of that detection point. The point cloud data frames serve as the basic environmental data for subsequent synchronous positioning, map building, terrain fitting, and visual target spatial projection.

[0072] Step S2 involves synchronous localization and map construction based on 3D point cloud information. For the current point cloud data frame, each detection point is projected onto the horizontal plane, and the height value of the detection point is written to the corresponding pixel according to a preset resolution to generate a 2D height image. Stable feature points are extracted from the 2D height image using an approximate Harris corner detection method, and nearest neighbor matching is performed on the stable feature points in adjacent data frames to obtain an initial inter-frame matching relationship.

[0073] Subsequently, the initial inter-frame matching relationships are filtered using a random sampling consensus algorithm to remove abnormal matching points. The spatial transformation relationship between adjacent data frames is calculated using the retained valid matching points. This spatial transformation relationship is then fused with odometer information or the motion pose information of the mobile platform to obtain the real-time pose of the mobile platform in the global coordinate system. Based on the real-time pose, coordinate transformation, pose correction, and stitching are performed on each point cloud data frame to form a globally consistent point cloud map.

[0074] Step S3: Perform cubic surface fitting on the seabed topography. Using the current position of the mobile platform as the center, select point cloud data within a preset range in the global point cloud map. In this embodiment, point cloud data within a 10m×10m range around the mobile platform can be selected, and a bivariate cubic surface model is established using the planar coordinates x and y of each point in the point cloud as independent variables and the height coordinate z as the dependent variable.

[0075]

[0076] .

[0077] The selected point cloud coordinates are substituted into the bivariate cubic surface model, and the coefficients a0 to a9 are solved using the linear least squares method to obtain a continuous three-dimensional surface model representing the current seabed topographic undulations. The residuals between the measured height and the fitted height at each point are further calculated, and the maximum fitting error and fitting variance are obtained accordingly. When both the maximum fitting error and the fitting variance meet the preset accuracy requirements, the current fitting result is accepted; if any index does not meet the preset accuracy requirements, the point cloud selection range is narrowed and refitting is performed until a continuous three-dimensional surface model that meets the accuracy requirements is obtained. As the sonar sensor acquires new point cloud data, the continuous three-dimensional surface model is updated in the above manner.

[0078] Step S4: Acquire and identify seabed sulfide targets. The camera sensor acquires visible light images of the seabed operating area. The acquired images are encoded in H.264 format and transmitted to the control unit via a local area network. The control unit uses GStreamer to extract the stream and convert the format to obtain RGB format images. The RGB format images are then subjected to homomorphic filtering, automatic white balance, and dark channel prior dehazing to compress the brightness dynamic range, correct color deviations, and improve underwater image contrast.

[0079] The enhanced image is input into a pre-trained YOLO target recognition model; in this embodiment, the YOLO target recognition model can be the YOLOv5 model. The model outputs the category information, confidence information, and two-dimensional bounding box coordinates for each sulfide target. For the bounding box B=( , , , Extract the pixel coordinates of its four corner points p1=( , p2=( , p3=( , ) and p4=( , The above information will be used as the two-dimensional target information for sulfide targets.

[0080] Step S5: Map the sulfide target from the image coordinate system to the global coordinate system. For example... Figure 2 As shown, the camera coordinate origin 1 and the camera imaging plane 2 determine the camera's projection geometry, and the YOLO recognition bounding box 3 is located within the camera imaging plane 2. For any corner point of the YOLO recognition bounding box 3, connecting the camera coordinate origin 1 to that corner point determines the corresponding spatial projection ray.

[0081] The spatial point of the sulfide target in the global coordinate system is =( , , If the pixel coordinates of ᵀ are p=(u, v, 1), then the following projection relationship is satisfied: .

[0082] Let be the intrinsic parameter matrix of the camera, where , It is the ratio of focal length to the length of a single pixel on the x and y axes. , , represents the coordinates of the image center point in the pixel coordinate system, determined through camera calibration; the rotation matrix R and translation vector T represent the transformation relationship from the global coordinate system to the camera coordinate system; For the scalar to be determined, changes in its magnitude cause Move along the aforementioned spatial projection ray.

[0083] Combining the above projection relationship with the cubic surface expression obtained in step 3, we can obtain the intersection point of the spatial projection ray corresponding to the corner point and the cubic surface, that is, the three-dimensional projection point of the corner point on the cubic surface. Performing the above solution on the four corner points of the same YOLO recognition annotation box 3 respectively, we obtain four three-dimensional projection points; connecting the four three-dimensional projection points sequentially forms... Figure 2 The three-dimensional mapping quadrilateral 5 shown represents the spatial distribution range of the corresponding sulfide target on the three-dimensional topography of the seabed.

[0084] Step S6: Construct a raster map containing sulfide target distribution information. First, establish... Figure 2 The reference plane 6 shown is used for unified path planning and two-dimensional coordinate calculation. The four vertices of the three-dimensional mapped quadrilateral 5 are projected along a direction perpendicular to the reference plane 6 to obtain four planar projection points 7. These four projection points 7 form a planar projection region, and the minimum envelope rectangle 8 of this region is calculated. Each minimum envelope rectangle 8 corresponds to a sulfide target distribution area, and the set of all minimum envelope rectangles 8 constitutes the current global sulfide target distribution region.

[0085] The reference plane 6 is discretized according to a preset grid size to form a two-dimensional grid map. For each grid cell, its center plane coordinates are recorded, and the corresponding terrain height information is written according to the continuous three-dimensional surface model f(x, y). When the grid cell intersects with any minimum envelope rectangle 8, its sulfide target distribution mark is set to the target state; otherwise, it is set to the non-target state. At the same time, a work completion status mark is set for each grid cell, with the initial state set to incomplete. Thus, the two-dimensional grid map simultaneously contains three types of information: terrain height, sulfide target distribution, and work completion status.

[0086] Step S7: Generate a target-oriented mining operation path based on the grid map. The current position of the mobile platform in the reference plane coordinate system is set as the starting point for path planning, and a square search area is established centered on this starting point. In this embodiment, the initial side length of the square search area can be set to 5m, and the corner points of the smallest envelope rectangle corresponding to the sulfide target distribution areas where operations have not yet been completed are retrieved within this search area. If no corner point is found within the search area, the side length of the search area is increased by 2m and the search is repeated until at least one candidate corner point is obtained.

[0087] For each candidate corner point obtained from the retrieval, calculate its planar Euclidean distance to the starting point of the current path planning: ;in , The coordinates of the starting point on the two-dimensional reference plane, , The candidate corner point with the smallest distance to the candidate endpoint on the two-dimensional reference plane is determined as the current path planning endpoint. Based on the fact that the mobile platform can adjust its operating height according to the three-dimensional target point, and that there are no significant terrain undulations affecting passage within the current sulfide target distribution area, this embodiment generates a straight path between the path planning start point and the current path planning endpoint, serving as the transfer path for the mobile platform to reach the current sulfide target distribution area.

[0088] After the mobile platform reaches the current sulfide target distribution area, it generates a round-trip coverage path within the corresponding minimum envelope rectangle. Specifically, starting from the initial corner point of the minimum envelope rectangle, it first moves along the positive y-axis of the reference plane coordinate system to the rectangle boundary; then it turns 90°, moves along the x-axis at a preset coverage interval, turns 90° again, and moves along the negative y-axis to the opposite boundary; thereafter, it alternately changes the y-axis movement direction and shifts along the x-axis in the same manner until all grid cells marked with sulfide targets within the minimum envelope rectangle are covered. The preset coverage interval is determined based on the width of a single effective operation to avoid missed sampling between adjacent operation zones.

[0089] Upon completion of each coverage path segment, the completion status of the raster cells actually covered by that path is updated to "completed." After completing the current sulfide target distribution area, the endpoint of the coverage path for that area is used as the starting point for the next path planning, repeating the search, transfer, and round-trip coverage process until all raster cells marked with sulfide targets in the raster map have been traversed. The straight-line transfer paths between target distribution areas and the round-trip coverage paths within each target distribution area are connected sequentially to form... Figure 1 The target-oriented mining operation path described in step S7.

[0090] During path execution, the control unit performs discrete sampling of the two-dimensional planned path according to a preset execution step size. In this embodiment, the execution step size can be set to 0.1m. For the planar coordinates of the current sampling point... Read the height of this location from a continuous 3D surface model. f represents the coordinate relationship of the three-dimensional fitted surface, which constitutes the three-dimensional coordinates of the target point. The control unit generates motion control commands based on the current three-dimensional coordinates of the mobile platform and the three-dimensional coordinates of the target point, and controls the mobile platform and the work execution mechanism to complete the movement and mining operations corresponding to each target point in sequence. At the same time, it updates the operation status in the grid map according to the progress of the path execution.

[0091] Step S8 involves re-detecting, evaluating, and updating the status of the operation results. After completing the target-oriented mining operation path, a sonar sensor is used to conduct a secondary detection of the already operated area, and a camera sensor is used to acquire secondary images. The secondary detection data and secondary images are processed according to the processing methods in steps S1 to S5, including point cloud processing, terrain updating, image enhancement, sulfide target identification, and spatial projection, to obtain the sulfide target distribution information after the operation.

[0092] The post-operation sulfide target distribution information is compared with the pre-operation target distribution information. Combined with the deviation between the actual path and the planned path, as well as the coverage of the target grid cells, operation execution result data is generated. For grid cells that meet the preset completion criteria, their operation completion status is maintained or updated to "completed." For grid cells that still detect sulfide targets or do not meet the preset coverage requirements, their incomplete status is retained for subsequent path planning.

[0093] Finally, the operation results data are fed back to the global grid map, and the global target area, completed area and uncompleted area are displayed through a visual interface, thus forming a closed-loop autonomous operation process of environmental perception, terrain modeling, target positioning, path planning, operation execution and result verification.

[0094] The 10m×10m terrain fitting range, 5m initial search side length, 2m search side length increment, and 0.1m path execution step size mentioned above are only optional parameters in this embodiment and can be adjusted according to the sonar detection range, seabed topography complexity, target distribution density, and operational control accuracy. The target recognition model is not limited to YOLOv5; as long as it can output the two-dimensional bounding box of the sulfide target and its corner pixel coordinates, it can be used to perform the above-mentioned mapping process from two-dimensional targets to three-dimensional surfaces.

[0095] The specific embodiments of the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments. Equivalent substitutions, simple transformations, or combinations made by those skilled in the art without departing from the concept of the present invention should all fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for autonomous operation of a deep-sea mining vehicle, characterized in that, The deep-sea mining vehicle is equipped with sonar sensors, camera sensors, and a control unit. The autonomous operation method includes the following steps: Step S1: Detect the seabed operation area using the sonar sensor and obtain three-dimensional point cloud information of the seabed operation area; Step S2: Combine the pose information of the deep-sea mining vehicle with the three-dimensional point cloud information to perform synchronous positioning and map construction to obtain a global point cloud map; Step S3: Based on the global point cloud map, perform cubic surface fitting on the seabed topography of the seabed operation area to generate a continuous three-dimensional surface model for representing the undulation of the seabed topography. Step S4: Acquire image information of the seabed operation area through the camera sensor, identify sulfide targets in the image information, and obtain two-dimensional target information of the sulfide targets; Step S5: Based on the continuous three-dimensional surface model, the two-dimensional target information, the camera calibration parameters, and the real-time pose of the deep-sea mining vehicle, the sulfide target is mapped from the image coordinate system to the global coordinate system to obtain the three-dimensional spatial position information of the sulfide target. Step S6: Fuse the three-dimensional spatial location information of the sulfide target with the continuous three-dimensional surface model to construct an operation map containing the distribution information of the sulfide target, and convert the operation map into a raster map containing the distribution markers of the sulfide target and the operation completion status markers; Step S7: Generate a target-oriented mining operation path covering the sulfide target area based on the grid map, and control the deep-sea mining vehicle to perform mining operations according to the target-oriented mining operation path; Step S8: After the mining operation is completed, the seabed operation area is re-detected and identified by the sonar sensor and the camera sensor, and the operation completion status in the grid map is updated according to the re-detection and identification results.

2. The autonomous operation method for deep-sea mining vehicles according to claim 1, characterized in that, In step S1, the step of using the sonar sensor to detect the seabed operating area and obtain three-dimensional point cloud information of the seabed operating area specifically includes: Step S11: Transmit acoustic detection signals to the seabed operation area through the sonar sensor, and receive acoustic echo signals returned from the seabed operation area; Step S12: Perform echo processing, distance calculation, and spatial coordinate transformation on the acoustic echo signal to generate three-dimensional point cloud information containing three-dimensional spatial coordinates and distance information of multiple detection points; Step S13: Use the three-dimensional point cloud information as the basic environmental data for the deep-sea mining vehicle to perform synchronous positioning and map construction, seabed topography fitting and target spatial projection.

3. The autonomous operation method for deep-sea mining vehicles according to claim 1, characterized in that, In step S2, the simultaneous localization and map construction, combining the pose information of the deep-sea mining vehicle and the 3D point cloud information to obtain a global point cloud map, specifically includes: Step S21: Convert the height information in the three-dimensional point cloud information into a two-dimensional height image; Step S22: Extract feature points from the two-dimensional height image to obtain stable feature points for characterizing seabed topography features; Step S23: Perform nearest neighbor matching on the stable feature points in different data frames to obtain the initial inter-frame matching relationship; Step S24: Optimize the initial inter-frame matching relationship using the random sampling consensus algorithm, eliminate abnormal matches, and obtain the optimized inter-frame matching relationship; Step S25: Calculate the spatial transformation relationship between different data frames based on the optimized inter-frame matching relationship, and combine the odometer information or motion pose information to perform pose correction and stitching on the three-dimensional point cloud information to construct a globally consistent point cloud map.

4. The autonomous operation method for deep-sea mining vehicles according to claim 1, characterized in that, In step S3, the step of performing cubic surface fitting on the seabed topography of the seabed operation area based on the global point cloud map to generate a continuous three-dimensional surface model for representing the undulations of the seabed topography specifically includes: Step S31: Within a preset range around the deep-sea mining vehicle, select point cloud data from the global point cloud map for seabed topography fitting; Step S32: Establish a cubic surface fitting expression based on the point cloud data. The cubic surface fitting expression is used to describe the continuous relationship between the seabed topographic height and the planar coordinates. Step S33: Solve the model parameters of the cubic surface fitting expression using the linear least squares method to obtain a continuous three-dimensional surface model; Step S34: Calculate the maximum fitting error and the fitting variance of the continuous three-dimensional surface model, and determine whether the continuous three-dimensional surface model meets the preset accuracy requirements based on the maximum fitting error and the fitting variance; Step S35: When the continuous three-dimensional surface model meets the preset accuracy requirements, the continuous three-dimensional surface model is used as the seabed topography model of the current seabed operation area.

5. The autonomous operation method for deep-sea mining vehicles according to claim 4, characterized in that, Step S3 also includes: Step S36: When the maximum fitting error or the fitting variance does not meet the preset accuracy requirement, the preset range is narrowed and point cloud data is reselected; Step S37: Perform three-dimensional surface fitting again based on the reselected point cloud data until a continuous three-dimensional surface model that meets the preset accuracy requirements is obtained. Step S38: Update the continuous three-dimensional curved surface model based on the three-dimensional point cloud information acquired in real time by the sonar sensor.

6. The autonomous operation method for a deep-sea mining vehicle according to claim 1, characterized in that, In step S4, the step of acquiring image information of the seabed operating area through the camera sensor and identifying sulfide targets in the image information to obtain two-dimensional target information of the sulfide targets specifically includes: Step S41: Acquire visible light images of the seabed operating area using the camera sensor; Step S42: Perform image enhancement processing on the visible light image. The image enhancement processing includes homomorphic filtering, automatic white balance, and dark channel prior dehazing. Step S43: Input the enhanced image into the pre-trained YOLO target recognition model to detect and recognize sulfide targets in the image; Step S44: Output the coordinates, category information, and confidence information of the two-dimensional bounding box of the sulfide target, and extract the pixel coordinates of the four corner points of the two-dimensional bounding box; Step S45: Use the two-dimensional bounding box coordinates, category information, confidence information, and the pixel coordinates of the four corner points as the two-dimensional target information of the sulfide target.

7. The autonomous operation method for a deep-sea mining vehicle according to claim 1, characterized in that, In step S5, the step of mapping the sulfide target from the image coordinate system to the global coordinate system based on the continuous three-dimensional surface model, the two-dimensional target information, the camera calibration parameters, and the real-time pose of the deep-sea mining vehicle, to obtain the three-dimensional spatial position information of the sulfide target, specifically includes: Step S51: Extract the pixel coordinates of the four corner points of the two-dimensional bounding box corresponding to each sulfide target in the two-dimensional target information; Step S52: Based on the camera intrinsic parameters, camera extrinsic parameters, and the real-time pose of the deep-sea mining vehicle, determine the spatial projection ray corresponding to the pixel coordinates of each corner point; Step S53: Solve the problem by combining the spatial projection ray with the continuous three-dimensional surface model to obtain the three-dimensional projection point of each corner pixel coordinate on the continuous three-dimensional surface model; Step S54: Connect the four 3D projection points corresponding to the same 2D bounding box to form a 3D mapped quadrilateral; Step S55: Determine the three-dimensional spatial distribution range of the sulfide target in the global coordinate system based on the three-dimensional mapping quadrilateral.

8. The autonomous operation method for a deep-sea mining vehicle according to claim 1, characterized in that, In step S6, fusing the three-dimensional spatial location information of the sulfide target with the continuous three-dimensional surface model to construct a work map containing sulfide target distribution information, and converting the work map into a raster map containing sulfide target distribution markers and work completion status markers, specifically includes: Step S61: Establish a reference plane for path planning and coordinate calculation; Step S62: Project the three-dimensional spatial position information of the sulfide target onto the reference plane to obtain the planar projection area of ​​the sulfide target on the reference plane; Step S63: Generate the minimum envelope rectangle based on the planar projection region corresponding to each sulfide target, and define the set of all minimum envelope rectangles as the current global sulfide target distribution region; Step S64: Construct a two-dimensional map based on the current global sulfide target distribution area and the continuous three-dimensional surface model; Step S65: Convert the two-dimensional map into a raster map, wherein each raster cell contains at least terrain height information, sulfide target distribution marker information, and operation completion status marker information.

9. The autonomous operation method for a deep-sea mining vehicle according to claim 1, characterized in that, In step S7, a target-oriented mining operation path covering the sulfide target area is generated based on the raster map, and the deep-sea mining vehicle is controlled to perform mining operations according to the target-oriented mining operation path, specifically including: Step S71: Set the current position of the deep-sea mining vehicle in the reference plane coordinate system as the starting point for path planning; Step S72: Establish a preset search area centered on the starting point of the path planning, and search for the corner points of the minimum envelope rectangle corresponding to the sulfide target distribution area within the preset search area; Step S73: When the minimum envelope rectangle corner point does not exist within the preset search area, expand the preset search area and search again; Step S74: Calculate the planar Euclidean distance between each corner point of the minimum envelope rectangle obtained from the search and the starting point of the path planning, and determine the corner point of the minimum envelope rectangle with the smallest distance as the current path planning endpoint; Step S75: Generate a straight path based on the path planning starting point and the current path planning ending point, as the transfer path for the deep-sea mining vehicle to reach the current sulfide target distribution area; Step S76: Generate a round-trip coverage path for the current sulfide target distribution area, and enable the deep-sea mining vehicle to traverse the current sulfide target distribution area along the round-trip coverage path; Step S77: After completing the operation in the current sulfide target distribution area, take the current path endpoint as the starting point of the next path planning, and continue to plan the path for the remaining sulfide target distribution areas until all grid cells marked with sulfide targets in the grid map have been traversed and covered.

10. The autonomous operation method for a deep-sea mining vehicle according to claim 1, characterized in that, In step S8, after the mining operation is completed, the seabed operation area is re-detected and identified using the sonar sensor and the camera sensor, and the operation completion status in the grid map is updated based on the re-detection and identification results. Specifically, this includes: Step S81: During the mining operation performed by the deep-sea mining vehicle, extract the planar coordinates of the target point corresponding to the current execution step length according to the preset execution step length; Step S82: Generate the three-dimensional coordinates of the target point based on the plane coordinates of the target point and the elevation information of the continuous three-dimensional surface model at the plane coordinates of the target point; Step S83: Based on the current three-dimensional coordinates of the deep-sea mining vehicle and the three-dimensional coordinates of the target point, generate control commands for controlling the movement of the deep-sea mining vehicle and mining operations; Step S84: After the deep-sea mining vehicle completes the path execution, the sonar sensor and the camera sensor are used to perform secondary detection and image acquisition on the seabed operation area; Step S85: Based on the secondary detection and image acquisition results, detect and evaluate the sulfide target processing status, path execution deviation, and area coverage, and generate operation execution result data; Step S86: Feed back the job execution result data to the grid map, mark the grid cells of completed jobs, and update the global job completion status.

Images