A cooperative control method based on unmanned ship, unmanned aerial vehicle and ROV

Abstract

The application relates to the field of underwater operation, and discloses a cooperative control method based on an unmanned ship, an unmanned aerial vehicle and an ROV, which comprises the following steps: obtaining a cooperative operation instruction issued by a console, controlling the unmanned ship to release an underwater remotely operated vehicle and controlling an unmanned aerial vehicle to fly directly above the unmanned ship; obtaining a water area image containing the unmanned ship and the umbilical cable based on the unmanned aerial vehicle; obtaining a cable connection point of the umbilical cable and the unmanned ship and an umbilical cable water entry point of the umbilical cable entering the water surface based on the water area image; calculating a projection included angle of the umbilical cable water entry point relative to the cable connection point; generating a traction vector representing a swimming trend of the underwater remotely operated vehicle based on the projection included angle; calculating a heading correction amount and a speed matching value of the unmanned ship according to the traction vector; and converting the heading correction amount and the speed matching value into a convoy instruction and issuing the convoy instruction to the unmanned ship to adjust the heading and the speed of the unmanned ship.

Description

Technical Field

[0001] This application relates to the field of underwater engineering, and in particular to a collaborative control method based on unmanned vessels, drones and ROVs. Background Technology

[0002] In underwater engineering inspections or search and rescue operations, unmanned surface vessels (USVs) and remotely operated vehicles (ROVs) typically work together. Due to limitations in wireless communication, ROVs need to establish a connection with the USV via an umbilical cable to obtain power and transmit monitoring data back to the surface.

[0003] One existing collaborative method involves equipping the winch of the unmanned vessel with a tension sensor. When the underwater remotely operated vehicle (ROV) is swimming underwater and the umbilical cable is straightened, the tension sensor detects that the tension exceeds a threshold and controls the ROV to sail in the direction of the force to follow it.

[0004] The inventors discovered that the fundamental flaw of the above-mentioned passive following method is that the unmanned vessel only starts the following action after the umbilical cable is taut and the tension reaches the preset value. There is a time delay in the response. During this time delay, the underwater remotely operated vehicle is dragged backward by the surface unmanned vessel, which disrupts its underwater working posture. In addition, the slack section of the umbilical cable is prone to entanglement or scratching in waters with obstacles. Summary of the Invention

[0005] This application provides a collaborative control method based on unmanned ships, drones, and ROVs to at least partially solve the above-mentioned technical problems.

[0006] To achieve the above objectives, this application provides a collaborative control method based on unmanned surface vessels, unmanned aerial vehicles, and ROVs, comprising: A collaborative control method based on unmanned surface vessels, unmanned aerial vehicles, and ROVs, characterized by comprising: The system receives collaborative operation instructions from the console, controls the unmanned vessel to release the underwater remotely operated vehicle, and controls the drone to fly directly above the unmanned vessel; it also acquires images of the water area containing the unmanned vessel and the umbilical cable based on the drone. Based on the water area image, the cable connection point between the umbilical cable and the unmanned vessel, as well as the umbilical cable entry point into the water surface, are obtained. Calculate the projected angle between the umbilical cable entry point and the cable connection point; A traction vector characterizing the underwater remotely operated vehicle's diving trend is generated based on the projection angle; Based on the traction vector, calculate the heading correction and speed matching value of the unmanned vessel; The heading correction and speed matching value are converted into escort commands and sent to the unmanned vessel to adjust its heading and speed.

[0007] In this embodiment of the application, the above technical solution effectively avoids the reverse dragging of the ROV by the unmanned vessel, ensuring the stability of the ROV's underwater operation posture.

[0008] Other features and advantages of this application will be described in detail in the following detailed description section. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 This is a flowchart illustrating the steps of a collaborative control method based on unmanned ships, drones, and ROVs provided in an exemplary embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the protection scope of this application.

[0012] This application provides a collaborative control method based on unmanned surface vessels, unmanned aerial vehicles, and ROVs. Please refer to [link / reference]. Figure 1 The collaborative control method based on unmanned surface vessels, unmanned aerial vehicles, and ROVs provided in this application includes the following steps: Step S101: Obtain the collaborative operation command issued by the console, control the unmanned vessel to release the underwater remotely operated vehicle and control the drone to fly directly above the unmanned vessel; acquire water area images containing the unmanned vessel and the umbilical cable based on the drone; Step S102: Based on the water area image, obtain the cable connection point between the umbilical cable and the unmanned vessel, as well as the umbilical cable entry point into the water. Step S103: Calculate the projection angle between the umbilical cable entry point and the cable connection point; Step S104: Generate a traction vector representing the diving trend of the underwater remotely operated vehicle based on the projection angle; Step S105: Calculate the heading correction and speed matching value of the unmanned vessel based on the traction vector; Step S106: Convert the heading correction amount and the speed matching value into a navigation command and send it to the unmanned vessel to adjust the heading and speed of the unmanned vessel.

[0013] Specifically, the system receives collaborative operation instructions from the control console, controls the unmanned surface vessel (USV) to release the remotely operated vehicle (ROV) based on these instructions, and directs the drone to fly directly above the USV. The drone's onboard vision equipment captures images of the water area, including the USV and the umbilical cable. The control console refers to a remote control terminal used to issue operation instructions and coordinate the overall operation of the USV, drone, and ROV. The ROV is abbreviated as ROV. The umbilical cable is a composite cable connecting the USV and the ROV, providing power and signal transmission, and also providing traction for the ROV.

[0014] After acquiring complete images of the water area, the images are analyzed to identify the cable connection points where the umbilical cable connects to the unmanned surface vessel (USV) and the umbilical cable entry point where the cable enters the water from the air. The cable connection points refer to the points where the umbilical cable is fixed and connected to the hull of the USV; the umbilical cable entry point refers to the critical point where the umbilical cable extends from above the water surface to underwater.

[0015] Using the cable connection point as a reference, the projected angle between the umbilical cable entry point and the reference position is calculated, and a traction vector is constructed based on this projected angle. The aforementioned projected angle refers to the azimuth angle formed by the entry point relative to the cable connection point when the water surface area is considered as a two-dimensional plane; the traction vector refers to the vector parameter generated with the projected angle as the direction and combined with the spatial position characteristics of the cable, used to characterize the underwater swimming trend of the remotely operated vehicle.

[0016] Based on the obtained traction vector, the required heading correction and speed matching value for the unmanned surface vessel (USV) are calculated. The heading correction refers to the angle difference used to adjust the current navigation direction of the USV to match the motion trend of the remotely operated vehicle (ROV); the speed matching value refers to the target navigation speed that the USV needs to adjust to in order to adapt to the umbilical cable traction state and avoid abnormal cable stress.

[0017] The course correction and speed matching values ​​are converted into escort commands that can be executed by the unmanned surface vessel (USV) and sent to the USV. The USV's navigation direction and speed are adjusted in real time, enabling the three to work together.

[0018] Existing technologies rely on tension sensors onboard unmanned surface vessels (USVs) to detect umbilical cable tension for passive following. The USV only responds after the umbilical cable tauts and the tension reaches a threshold, resulting in significant delays. This not only causes reverse dragging of the ROV and disrupts its underwater operating posture, but also makes the slack umbilical cable highly susceptible to entanglement and scraping against obstacles in complex waters. This new solution uses real-time aerial vision from an unmanned aerial vehicle (UAV) to acquire water images, pre-identify the umbilical cable's connection and entry points, calculate the projected angle, and generate a traction vector reflecting the ROV's underwater movement. It also pre-calculates the USV's heading and speed adjustment parameters and issues escort commands. This proactive following method anticipates the ROV's movement without waiting for the umbilical cable to deform, reducing the response delays of traditional passive following methods. It effectively avoids reverse dragging of the ROV by the USV, ensures the ROV's stable underwater operating posture, reduces the risk of entanglement and scraping caused by umbilical cable slack or swaying, and improves the overall operational stability of the USV, UAV, and ROV in coordinated operations.

[0019] In some embodiments, obtaining the cable connection point between the umbilical cable and the unmanned vessel, and the umbilical cable entry point into the water surface, based on the water area image, includes: The water area image is subjected to color space conversion and filtering for noise reduction to obtain a continuous pixel region with the color of the umbilical cable as the cable outline; Target detection is performed on the water area image to extract the real-time image contour boundary of the unmanned vessel; The intersection operation is performed between the cable outline and the boundary of the real-time image outline, and the pixel coordinates corresponding to the intersection position are used as the cable connection point. The end pixel coordinates of the cable contour on the side away from the cable connection point are obtained and combined with the abrupt boundary of the water surface texture to mark the end pixel coordinates as the water entry point of the umbilical cable.

[0020] Specifically, color space conversion and filtering / denoising are performed on the water area images. From the processed images, continuous pixel regions with unique color characteristics specific to the umbilical cable are selected and defined as the cable outline. The aforementioned water area images refer to on-site images captured by a UAV's visual sensor, including unmanned vessels, the umbilical cable, and the water surface environment. Color space conversion refers to converting the original color representation mode of the image into a color mode suitable for target object recognition, highlighting target features and reducing environmental interference. Filtering / denoising refers to using image processing algorithms to filter out invalid noise such as water splashes, light and shadow, and specks in the image, improving image clarity and feature recognition. The cable outline refers to the continuous pixel outline region in the image corresponding to the physical shape of the umbilical cable.

[0021] The target detection operation is performed on the same water area image to extract the real-time image contour boundary of the unmanned vessel. The target detection mentioned above refers to the technical means of locating a specified target within an image and extracting the target's shape, boundary and other features using image recognition algorithms; the real-time image contour boundary refers to the pixel boundary line in the image that outlines the overall shape of the unmanned vessel.

[0022] An intersection operation is performed on the cable outline and the real-time image outline of the unmanned surface vessel. The pixel coordinates corresponding to the intersection points of the two outlines are then determined as the cable connection points. The aforementioned intersection operation refers to the pixel calculation method used in image processing to determine the overlapping parts of two outline regions.

[0023] Along the cable's outline, locate the coordinates of the terminal pixel on the side furthest from the cable connection point. Simultaneously, combine this with the abrupt boundary of the water surface texture to mark the umbilical cable's entry point into the water. The abrupt boundary of the water surface texture refers to the dividing point where the texture features such as water ripples and light and shadow undergo significant changes; this point corresponds to the interface where the umbilical cable enters the water.

[0024] The above scheme effectively overcomes the image interference problem caused by complex water environment by color space conversion and filtering to remove image noise and enhance umbilical cable features. It extracts the contour of the unmanned vessel based on target detection, uses contour intersection operation to determine the cable connection point, and combines the end pixel of the cable with the abrupt change feature of water surface texture to determine the entry point.

[0025] In some embodiments, obtaining the end pixel coordinates of the cable contour on the side away from the cable connection point and combining them with the abrupt boundary of the water surface texture, and calibrating the end pixel coordinates as the water entry point of the umbilical cable includes: A local search window containing the cable outline is established along the extension direction of the cable outline, and the pixel grayscale change curves along the extension direction within the local search window are extracted. Within the local search window, the texture roughness of adjacent background pixel blocks perpendicular to the extension direction is calculated to generate a texture threshold characterizing the state of water ripples. The pixel grayscale change curve is compared with the texture threshold point by point to locate the pixel grayscale change curve where the absolute value of the attenuation slope reaches a local maximum value and the texture roughness of the surrounding pixels crosses the boundary pixel of the texture threshold. The pixel coordinates corresponding to the boundary pixel points are used as the end pixel coordinates of the cable contour on the side away from the cable connection point and are marked as the umbilical cable entry point.

[0026] Specifically, a local search window is constructed along the extension direction of the cable outline to completely encompass the cable outline, and the pixel grayscale variation curves distributed within this local search window along the extension direction of the cable outline are extracted. The aforementioned local search window refers to a local image region defined with the cable outline as the center, used to narrow the recognition range and reduce the computational load and interference of global image processing; the pixel grayscale variation curve refers to a curve drawn with pixel position as the horizontal axis and pixel grayscale value as the vertical axis.

[0027] Within a local search window, the texture roughness of adjacent background pixel blocks perpendicular to the cable's outline extension direction is calculated, and a texture threshold reflecting the actual state of the current water surface ripples is set based on the calculation results. The aforementioned texture roughness refers to a feature parameter used to measure the density and undulation of texture in a local area of ​​the image, distinguishing between a smooth water surface and areas with ripples or water body boundaries; the texture threshold is the critical value used to distinguish between normal water surface texture and abnormal texture at the interface where the cable enters the water.

[0028] The pixel grayscale change curve is compared point-by-point with the texture threshold to locate boundary pixels that simultaneously meet two conditions: first, the absolute value of the attenuation slope of the pixel grayscale change curve at this location reaches a local maximum; second, the texture roughness value of the surrounding area of ​​this pixel crosses a preset texture threshold. These boundary pixels refer to the pixels corresponding to the critical position where the umbilical cable transitions from above water to below water in the image; the attenuation slope refers to the rate of change of the descending segment of the grayscale change curve, representing the speed of change in pixel brightness.

[0029] The pixel coordinates corresponding to the aforementioned boundary pixel points are used as the end pixel coordinates of the cable profile on the side away from the cable connection point, and this coordinate position is marked as the umbilical cable entry point.

[0030] The above-mentioned method relies on the rough determination of the water entry point by simply using the pixels at the end of the cable outline, which is easily affected by water surface clutter and light and shadow interference, resulting in positioning deviation and making it impossible to distinguish the boundary between the above-water and underwater cables. This method narrows the recognition range by defining a local search window, locates the boundary pixels based on the dual judgment conditions of pixel grayscale change characteristics and water surface texture roughness, and completes the recognition by relying on grayscale attenuation slope and texture threshold, effectively resisting environmental interference such as water surface ripples and light and shadow, and improving the positioning accuracy of the umbilical cable water entry point.

[0031] In some embodiments, calculating the projected angle between the umbilical cable entry point and the cable connection point, and generating a traction vector based on the projected angle to characterize the underwater remotely operated vehicle's (ROV) diving trend, includes: A reference coordinate system is established with the cable connection point as the origin and the current bow orientation of the unmanned vessel as the vertical axis. Calculate the angle of the umbilical cable entry point in the reference coordinate system, and use it as the projection angle; Calculate the pixel distance between the umbilical cable entry point and the cable connection point, and use it as the water surface cable projection spacing. The traction vector is obtained by combining the included projection angle as the direction component and the projection spacing of the water surface cable as the modulus component.

[0032] Specifically, a reference coordinate system is constructed with the cable connection point as the origin and the current bow orientation of the unmanned vessel as the vertical axis. The aforementioned origin refers to the reference point of the planar coordinate system, which is the unified reference for subsequent angle and distance calculations; the bow orientation refers to the direction directly facing the bow of the unmanned vessel, serving as the positive reference direction for the unmanned vessel's navigation attitude.

[0033] Calculate the azimuth angle corresponding to the point where the umbilical cable enters the water within the reference coordinate system, and determine this angle as the projection angle.

[0034] The pixel distance between the umbilical cable entry point and the cable connection point in the image is calculated and recorded as the cable projection spacing on the water surface. The pixel distance refers to the straight-line distance between two pixel coordinate points in the image; the cable projection spacing on the water surface refers to the straight-line length from the cable connection point to the entry point on the water surface projection segment of the umbilical cable, reflecting the cable's extension state on the water surface.

[0035] The traction vector is obtained by combining the projected angle as the directional component and the distance between the projected cable projections as the magnitude component. The directional component determines the direction of the vector and is used to characterize the diving direction of the remotely operated vehicle (ROV); the magnitude component determines the magnitude of the vector and is used to characterize the diving range and traction force of the ROV.

[0036] In some embodiments, before calculating the heading correction and speed matching value of the unmanned surface vessel, the method further includes: Real-time acquisition of the three-dimensional tension of the umbilical cable at the cable connection point; decomposition of the three-dimensional tension of the cable into horizontal tension components and vertical tension components along the longitudinal axis of the hull; Calculate the ratio of the vertical tension component to the horizontal tension component to obtain the tilt angle compensation coefficient characterizing the cable's water entry angle; The projected spacing of the water surface cable is compensated based on the tilt angle compensation coefficient. When the tilt angle compensation coefficient is greater than a set threshold, the projected spacing of the water surface cable is proportionally enlarged as the equivalent force-bearing length. Calculate the difference between the equivalent stress length and the projected spacing of the water surface cable to generate the spacing compensation amount; The spacing compensation is superimposed on the modulus component of the traction vector to obtain the corrected traction vector.

[0037] Specifically, before calculating the heading correction and speed matching value of the unmanned surface vessel (USV), the traction vector needs to be corrected. The specific steps are as follows: Real-time acquisition of the three-dimensional tension of the umbilical cable at the cable connection point. This three-dimensional tension is then decomposed into horizontal and vertical components distributed along the longitudinal axis of the hull. The aforementioned three-dimensional cable tension refers to the comprehensive tension exerted on the USV by the umbilical cable in three spatial directions, including forces in multiple directions such as horizontal and vertical. The longitudinal axis of the hull refers to the central axis extending from the bow to the stern of the USV, serving as the reference axis for force and attitude analysis. The horizontal tension component refers to the horizontal component of the three-dimensional cable tension along the longitudinal axis of the hull; the vertical tension component refers to the vertical component of the three-dimensional cable tension.

[0038] The ratio of the vertical tension component to the horizontal tension component is calculated to obtain the tilt angle compensation coefficient. Based on this coefficient, the projected spacing of the cables on the water surface is compensated. When the tilt angle compensation coefficient exceeds a pre-set threshold, the projected spacing of the cables on the water surface is enlarged according to a predetermined ratio, and the enlarged value is determined as the equivalent stress length. The tilt angle compensation coefficient refers to a characteristic coefficient used to characterize the magnitude of the umbilical cable's water entry tilt angle; the threshold is a manually set critical value used as a criterion for determining whether to activate spacing compensation; the equivalent stress length refers to the length value that reflects the actual stress effect on the cable body after correction of the umbilical cable's water entry tilt angle.

[0039] Calculate the difference between the equivalent force-bearing length and the original projected spacing of the cable on the water surface to generate a spacing compensation amount. Add this spacing compensation amount to the modulus component of the original traction vector to obtain the corrected traction vector.

[0040] The above scheme obtains the tilt angle compensation coefficient by collecting and decomposing the three-dimensional tension of the umbilical cable. Based on this tilt angle compensation coefficient, the projection spacing is compensated and corrected. The magnitude component of the traction vector is updated using the spacing compensation amount. This makes up for the deficiency of ignoring the cable's water inclination angle and spatial force in two-dimensional planar calculation, making the traction vector more consistent with the actual working conditions on site.

[0041] In some embodiments, calculating the heading correction and speed matching value of the unmanned surface vessel includes: Obtain the set deflection angle safety threshold and allowable spacing range; When the absolute value of the projected angle is greater than the deflection angle safety threshold, the heading correction amount of the unmanned vessel is obtained based on the difference between the absolute value and the deflection angle safety threshold. The projected spacing of the surface cable is obtained within multiple consecutive control cycles, the time change rate of the modulus component is calculated, and the rated operating speed is determined based on the current operating mode. When the projected spacing of the surface cables is within the allowable spacing range, the rated operating speed is used as the speed matching value; When the projected spacing of the water surface cables is greater than the upper limit of the allowable spacing range, a positive acceleration compensation amount is calculated based on the time change rate and the positive acceleration compensation amount is superimposed on the rated operating speed as the speed matching value.

[0042] Specifically, the system retrieves the deflection angle safety threshold and allowable spacing range. The aforementioned deflection angle safety threshold refers to the critical angle value used to determine whether the projected angle is within the safe range. If the angle does not exceed this value, there is no need to adjust the course. The allowable spacing range refers to the reasonable range of projected spacing of surface cables that ensures normal stress on the umbilical cable and prevents excessive stretching or slack.

[0043] When the absolute value of the projected angle is greater than the deflection angle safety threshold, the difference between the absolute value and the deflection angle safety threshold is used as the heading correction amount for the unmanned vessel.

[0044] The system collects the projected spacing of the water surface cables over multiple consecutive control cycles, calculates the time rate of change of the traction vector modulus component, and determines the rated operating speed of the unmanned surface vessel (USV) based on the current operating mode. A control cycle refers to a fixed time unit in which the control system performs one data acquisition, calculation, and command issuance operation; the time rate of change refers to how quickly the modulus component changes over time, reflecting the changing trend of the umbilical cable's elongation; and the rated operating speed refers to the standard speed set for the USV in the corresponding operating mode.

[0045] If the projected spacing of the cables on the water surface falls within the allowable spacing range, the rated operating speed will be used directly as the speed matching value for the unmanned vessel.

[0046] If the projected spacing of the cables on the water surface exceeds the upper limit of the allowable spacing range, a positive acceleration compensation is calculated based on the time change rate of the modulus component, and this positive acceleration compensation is added to the rated operating speed to obtain the speed matching value of the unmanned vessel. The aforementioned positive acceleration compensation refers to the speed increment added to reduce the projected spacing of the cables and match the movement trend of the ROV.

[0047] The above scheme enables real-time control of the unmanned vessel's heading and speed, keeping the projection angle and cable spacing within a safe range, avoiding abnormal stress on the umbilical cable, and ensuring the stability of the unmanned vessel, ROV, and drones in coordinated navigation.

[0048] In some embodiments, the method further includes: Calculate the pixel coordinates of the midpoint between the cable connection point and the umbilical cable entry point in the water area image; Calculate the minimum pixel spacing from the point where the umbilical cable enters the water to the edge of the water area image; When the minimum pixel spacing is less than the set blind spot warning threshold, the pixel offset of the midpoint pixel coordinates relative to the geometric center of the water area image is calculated; Based on the intrinsic parameter matrix of the UAV's visual sensor and the UAV's current flight altitude, the pixel offset is converted into a translation vector, and the UAV is driven to perform horizontal translation according to the translation vector; When the minimum pixel spacing continues to decrease to exceed the set extreme value limit, the drone is driven to increase its flight altitude to expand the actual field of view, ensuring that the cable connection point and the umbilical cable entry point are within the effective observation range of the water area image.

[0049] Specifically, the midpoint pixel coordinates of the cable connection point and the umbilical cable entry point in the water area image are calculated; the aforementioned midpoint pixel coordinates refer to the coordinate position of the midpoint of the line connecting the two target pixels in the image pixel coordinate system.

[0050] The minimum pixel spacing from the umbilical cable entry point to the edge of the water area image is calculated. When the minimum pixel spacing is less than the system's preset blind spot warning threshold, the pixel offset of the midpoint pixel coordinates relative to the geometric center of the water area image is further calculated. The minimum pixel spacing mentioned above refers to the minimum distance between the pixel position of the entry point and the four edges of the image, used to determine whether the target is close to the image boundary; the blind spot warning threshold refers to the critical pixel distance at which the target is about to enter the image's blind spot, used to trigger the UAV's position adjustment action; the geometric center of the water area image refers to the center position of the entire image's pixel area, which is the ideal reference point for UAV visual observation; the pixel offset refers to the positional deviation between the target's midpoint and the image center in the pixel dimension.

[0051] Using the intrinsic parameter matrix of the visual sensor onboard the drone and the drone's current flight altitude, the pixel offset is converted into a translation vector, and the drone is controlled to perform horizontal translation movements based on this translation vector. The aforementioned intrinsic parameter matrix of the visual sensor refers to the matrix characterizing the inherent optical parameters such as the camera lens focal length and principal point position; the translation vector refers to the control vector used to control the drone's horizontal movement direction and distance.

[0052] If the minimum pixel spacing continues to decrease and exceeds the preset extreme value limit, the UAV will be controlled to increase its flight altitude to expand the field of view and ensure that the cable connection point and the umbilical cable entry point are always within the effective observation area of ​​the water area image. The aforementioned extreme value limit refers to the critical pixel distance at which the target is about to move out of the effective observation area, which is the highest warning value for field of view warning; the effective observation area refers to the pixel area in the image that is clearly imaged and can be used for target recognition and data calculation normally.

[0053] By monitoring the pixel distance between the water entry point and the image edge in real time, the drone is driven to move horizontally to correct the observation position based on the pixel offset. When the target is about to completely leave the field of view, the flight altitude is increased to expand the field of view, ensuring that the two core points are in the effective observation area and avoiding the problem of loss of field of view.

[0054] In some embodiments, based on the intrinsic parameter matrix of the UAV's visual sensor and the UAV's current flight altitude, the pixel offset is converted into a translation vector, and the UAV is driven to perform horizontal translation according to the translation vector, including: Obtain the focal length parameter from the intrinsic parameter matrix of the vision sensor; Calculate the ratio of the current flight altitude of the UAV to the focal length parameter, and generate a spatial projection scale representing the mapping relationship between pixels and actual size; Multiply the pixel offset by the spatial projection scale to calculate the horizontal offset distance of the midpoint pixel coordinates on the actual water surface reference plane; The current heading angle of the UAV is obtained, and the horizontal offset distance is decomposed by coordinate system rotation based on the heading angle to obtain the longitudinal translation control component along the longitudinal axis of the fuselage and the lateral translation control component along the transverse axis of the fuselage. The translation vector is obtained by vector synthesis of the longitudinal translation control component and the lateral translation control component.

[0055] Specifically, the focal length parameter is read from the intrinsic parameter matrix of the visual sensor. The aforementioned visual sensor refers to a camera device mounted on a drone for capturing images of water and collecting visual data; the focal length parameter refers to the inherent optical parameters of the visual sensor lens, which determines the mapping ratio between the imaging angle, pixels, and actual space.

[0056] The ratio of the UAV's current flight altitude to its focal length parameter is calculated to obtain the spatial projection scale. The aforementioned spatial projection scale refers to the conversion factor used to establish the correspondence between image pixel units and actual water surface length units.

[0057] By multiplying the pixel offset by the spatial projection scale, the horizontal offset distance of the midpoint pixel coordinates on the water surface reference plane is calculated. The aforementioned water surface reference plane refers to a real-world spatial plane constructed with the surface of the operating area as a reference, serving as the reference surface for displacement calculation.

[0058] The current heading angle of the UAV is obtained. Based on this angle, the horizontal offset distance is decomposed into a coordinate system rotation, separating the longitudinal translation control component along the longitudinal axis and the lateral translation control component along the transverse axis. The heading angle refers to the deflection angle of the UAV's central axis relative to the geographic reference direction, used for spatial coordinate system transformation; the longitudinal axis refers to the central axis along the direction from the UAV's nose to its tail; the longitudinal translation control component refers to the displacement control amount of the UAV along the forward and backward direction; the transverse axis refers to the axis perpendicular to the UAV's longitudinal axis and along the left and right direction; the lateral translation control component refers to the displacement control amount of the UAV along the left and right direction.

[0059] A vector synthesis operation is performed on the longitudinal translation control component and the lateral translation control component to obtain the translation vector used to control the movement of the UAV.

[0060] This solution uses the sensor focal length and flight altitude to calculate the conversion scale between pixels and actual distance, and uses the UAV's heading angle to complete coordinate decomposition and vector synthesis, achieving accurate conversion from pixel deviation to actual displacement. This makes the distance and direction of the UAV's horizontal translation more accurate, ensuring that the visual observation center is always aligned with the target area, and improving the accuracy of the UAV's field of view adjustment.

[0061] In some embodiments, when the minimum pixel pitch continues to decrease to exceed a set extreme value limit, driving the drone to increase its flight altitude to expand its actual field of view includes: Obtain the maximum pixel distance between the cable connection point and the umbilical cable entry point in the water area image; Multiply the maximum pixel distance by the spatial projection scale to calculate the target projection distance between the two points; Obtain the set safety margin of the field of view boundary, add the target projection spacing to the safety margin of the field of view boundary, and generate the safe field of view coverage width; Based on the triangular projection relationship between the safe field of view coverage width and the optical field of view angle parameters of the UAV, the target flight altitude that completely covers the safe field of view coverage width is calculated. The difference between the target flight altitude and the current flight altitude is converted into an ascent command, which is input to the flight controller of the UAV to drive the UAV to ascend to the target flight altitude.

[0062] Specifically, the maximum pixel distance between the cable connection point and the umbilical cable entry point in the water area image is extracted. This maximum pixel distance is then multiplied by the spatial projection scale obtained earlier to calculate the target projection distance between the two points on the actual water surface. The maximum pixel distance mentioned above refers to the straight-line pixel length between the pixel coordinates of the cable connection point and the pixel coordinates of the umbilical cable entry point in the image; the target projection distance refers to the straight-line distance between the cable connection point and the umbilical cable entry point on the actual water surface.

[0063] The system's set field-of-view boundary safety margin is retrieved, and the target projection distance is added to the field-of-view boundary safety margin to calculate the safe field-of-view coverage width. The aforementioned field-of-view boundary safety margin refers to the extra observation width reserved to avoid target points from being too close to the edge of the image, thereby improving the safety of visual observation; the safe field-of-view coverage width refers to the minimum observation width that can fully accommodate two key points while reserving a safe area.

[0064] By utilizing the optical field of view parameters of the UAV and based on the triangular projection geometric relationship between the safe field of view coverage width and the optical field of view angle, the target flight altitude that can completely cover the safe field of view coverage width is calculated. The aforementioned optical field of view parameters refer to the angle range that the UAV's visual sensor lens can normally image, and are inherent parameters for optical imaging calculations; the target flight altitude refers to the standard flight altitude of the UAV that ensures all target points are within the effective observation range.

[0065] The difference between the target flight altitude and the drone's current flight altitude is calculated, and this difference is converted into a corresponding ascent command. This ascent command is then input into the drone's flight controller to drive the drone to climb to the target flight altitude. The aforementioned ascent command refers to the flight control command sent to the drone to control its upward climb; the flight controller refers to the drone's core control unit, which receives commands and executes adjustments to flight attitude, altitude, and position.

[0066] This solution uses pixel distance and spatial projection scale to calculate the actual distance, and determines a reasonable field of view width by combining the safety margin of the field of view boundary. Based on the trigonometric relationship of the optical field of view angle, it accurately calculates the target flight altitude, realizes the determination and real-time adjustment of the UAV flight altitude, ensures that the key points of the cable are always within the imaging field of view, and avoids the image quality degradation caused by invalid elevation, thus continuously ensuring stable and reliable visual acquisition and target recognition.

[0067] In some embodiments, the method further includes: The remaining battery power of the drone is monitored in real time, and a first battery power threshold and a second battery power threshold are set, wherein the first battery power threshold is greater than the second battery power threshold; When the remaining battery power is lower than the first battery power threshold, the execution of the command to drive the drone to move is paused, a hovering command is generated to lock the current position and flight altitude of the drone, and the drone enters a static observation mode. In the static observation mode, the pixel coordinates of the cable connection point in the water area image are extracted in real time, and the visual deviation vector of the pixel coordinates relative to the geometric center of the water area image is calculated. The visual deviation vector is converted into a centering guidance command for the unmanned vessel and superimposed on the escort command and sent to the unmanned vessel to drive the unmanned vessel to autonomously adjust its course and speed until the cable connection point returns to the preset range of the geometric center of the water area image. When the remaining power is lower than the second power threshold, the real-time coordinates of the unmanned vessel in the water area image are extracted to generate a return landing trajectory, and the unmanned vessel is controlled to autonomously land along the return landing trajectory to the landing pad configured on the unmanned vessel.

[0068] Specifically, the remaining battery power of the drone is monitored in real time, and a first battery power threshold and a second battery power threshold are preset. The remaining battery power refers to the available electrical energy currently remaining in the drone's onboard battery, which is the main indicator for judging the drone's operational endurance. The first battery power threshold and the second battery power threshold are two different levels of battery power judgment thresholds, and the first battery power threshold is greater than the second battery power threshold, which is used to trigger different emergency strategies in stages.

[0069] When the drone's remaining battery power falls below a first battery threshold, the system suspends all control commands that drive the drone's horizontal movement, generates a hovering command, locks the drone's current position and altitude, and puts the drone into static observation mode. The hovering command refers to a flight command that keeps the drone stationary in its current spatial position, altitude, and attitude; static observation mode refers to a working mode where the drone no longer moves and relies solely on a fixed position to continuously collect images of the water area.

[0070] During static observation mode operation, the pixel coordinates of the cable connection point in the water area image are continuously extracted, and the visual deviation vector of these pixel coordinates relative to the geometric center of the water area image is calculated. This visual deviation vector is converted into a centering guidance command for the unmanned surface vessel (USV), which is then superimposed on the existing escort command and issued to the USV. This command instructs the USV to adjust its navigation state until the cable connection point returns to a preset range corresponding to the geometric center of the water area image. The aforementioned visual deviation vector refers to a vector parameter characterizing the direction and degree of offset of the cable connection point relative to the center position in the image. The USV centering guidance command is a control command that guides the USV to adjust its course and speed to bring the cable connection point back to the center region of the image. The preset range refers to the normal observation area defined with the geometric center of the image as a reference.

[0071] When the remaining battery power of the drone further decreases below the second battery threshold, the real-time physical coordinates of the unmanned vessel in the water area image are extracted. Based on these coordinates, a return landing trajectory is planned, and the drone is controlled to fly along this trajectory and land on the landing pad provided on the unmanned vessel. The aforementioned real-time physical coordinates refer to the actual position coordinates of the unmanned vessel in the operating water area; the return landing trajectory refers to the preset flight path of the drone flying back from its current position and landing at the designated point; the landing pad refers to the dedicated area installed on the surface of the unmanned vessel for drone docking and landing.

[0072] Drones have limited battery life. If they continue to perform mobile observations when their battery is low, they are prone to running out of power, losing control, and crashing. Furthermore, they may lose the target observation point due to position deviation, leading to the interruption of collaborative operations. This solution sets two levels of battery thresholds to achieve hierarchical management. When the battery is low, the drone stops moving and switches to static observation. Instead, it guides the unmanned surface vessel back to center to ensure the observation effect. When the battery is about to run out, it plans a trajectory to return and land, thus avoiding the safety risks caused by drones flying with low battery.

[0073] The above are merely preferred embodiments of this application and are not intended to limit this application in any way. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of this application without departing from the scope of the technical solution of this application shall still fall within the scope of the technical solution of this application.

Claims

1. A collaborative control method based on unmanned surface vessels, unmanned aerial vehicles, and ROVs, characterized in that, include: Receive collaborative operation instructions from the console, control the unmanned vessel to release the underwater remotely operated vehicle and control the drone to fly directly above the unmanned vessel; Images of the water area containing the unmanned vessel and umbilical cable were acquired using drones; Based on the water area image, the cable connection point between the umbilical cable and the unmanned vessel, as well as the umbilical cable entry point into the water surface, are obtained. Calculate the projected angle between the umbilical cable entry point and the cable connection point; A traction vector characterizing the underwater remotely operated vehicle's diving trend is generated based on the projection angle; Based on the traction vector, calculate the heading correction and speed matching value of the unmanned vessel; The heading correction and the speed matching value are converted into escort commands and sent to the unmanned vessel to adjust the heading and speed of the unmanned vessel; Calculating the projected angle between the umbilical cable entry point and the cable connection point, and generating a traction vector based on the projected angle to characterize the underwater remotely operated vehicle's (ROV) diving trend, includes: A reference coordinate system is established with the cable connection point as the origin and the current bow orientation of the unmanned vessel as the vertical axis. Calculate the angle of the umbilical cable entry point in the reference coordinate system, and use it as the projection angle; Calculate the pixel distance between the umbilical cable entry point and the cable connection point, and use it as the water surface cable projection spacing. The traction vector is obtained by combining the included projection angle as the direction component and the projection spacing of the water surface cable as the modulus component. Calculating the heading correction and speed matching value of the unmanned surface vessel includes: Obtain the set deflection angle safety threshold and allowable spacing range; When the absolute value of the projected angle is greater than the deflection angle safety threshold, the heading correction amount of the unmanned vessel is obtained based on the difference between the absolute value and the deflection angle safety threshold. The projected spacing of the surface cable is obtained within multiple consecutive control cycles, the time change rate of the modulus component is calculated, and the rated operating speed is determined based on the current operating mode. When the projected spacing of the surface cables is within the allowable spacing range, the rated operating speed is used as the speed matching value; When the projected spacing of the water surface cables is greater than the upper limit of the allowable spacing range, a positive acceleration compensation amount is calculated based on the time change rate and the positive acceleration compensation amount is superimposed on the rated operating speed as the speed matching value.

2. The method according to claim 1, characterized in that, Based on the water area image, the cable connection point between the umbilical cable and the unmanned vessel, as well as the umbilical cable entry point into the water, are obtained, including: The water area image is subjected to color space conversion and filtering for noise reduction to obtain a continuous pixel region with the color of the umbilical cable as the cable outline; Target detection is performed on the water area image to extract the real-time image contour boundary of the unmanned vessel; The intersection operation is performed between the cable outline and the boundary of the real-time image outline, and the pixel coordinates corresponding to the intersection position are used as the cable connection point. The end pixel coordinates of the cable contour on the side away from the cable connection point are obtained and combined with the abrupt boundary of the water surface texture to mark the end pixel coordinates as the water entry point of the umbilical cable.

3. The method according to claim 2, characterized in that, Obtaining the end pixel coordinates of the cable contour on the side away from the cable connection point and combining them with the abrupt boundary of the water surface texture, and calibrating the end pixel coordinates as the umbilical cable entry point, includes: A local search window containing the cable outline is established along the extension direction of the cable outline, and the pixel grayscale change curves along the extension direction within the local search window are extracted. Within the local search window, the texture roughness of adjacent background pixel blocks perpendicular to the extension direction is calculated to generate a texture threshold characterizing the state of water ripples. The pixel grayscale change curve is compared with the texture threshold point by point to locate the pixel grayscale change curve where the absolute value of the attenuation slope reaches a local maximum value and the texture roughness of the surrounding pixels crosses the boundary pixel of the texture threshold. The pixel coordinates corresponding to the boundary pixel points are used as the end pixel coordinates of the cable contour on the side away from the cable connection point and are marked as the umbilical cable entry point.

4. The method according to claim 3, characterized in that, Before calculating the heading correction and speed matching value of the unmanned vessel, the method further includes: Real-time acquisition of the three-dimensional tension of the umbilical cable at the cable connection point; decomposition of the three-dimensional tension of the cable into horizontal tension components and vertical tension components along the longitudinal axis of the hull; Calculate the ratio of the vertical tension component to the horizontal tension component to obtain the tilt angle compensation coefficient characterizing the cable's water entry angle; The projected spacing of the water surface cable is compensated based on the tilt angle compensation coefficient. When the tilt angle compensation coefficient is greater than a set threshold, the projected spacing of the water surface cable is proportionally enlarged as the equivalent force-bearing length. Calculate the difference between the equivalent stress length and the projected spacing of the water surface cable to generate the spacing compensation amount; The spacing compensation is superimposed on the modulus component of the traction vector to obtain the corrected traction vector.

5. The method according to claim 4, characterized in that, The method further includes: Calculate the pixel coordinates of the midpoint between the cable connection point and the umbilical cable entry point in the water area image; Calculate the minimum pixel spacing from the point where the umbilical cable enters the water to the edge of the water area image; When the minimum pixel spacing is less than the set blind spot warning threshold, the pixel offset of the midpoint pixel coordinates relative to the geometric center of the water area image is calculated; Based on the intrinsic parameter matrix of the UAV's visual sensor and the UAV's current flight altitude, the pixel offset is converted into a translation vector, and the UAV is driven to perform horizontal translation according to the translation vector; When the minimum pixel spacing continues to decrease to exceed the set extreme value limit, the drone is driven to increase its flight altitude to expand the actual field of view, ensuring that the cable connection point and the umbilical cable entry point are within the effective observation range of the water area image.

6. The method according to claim 5, characterized in that, Based on the intrinsic parameter matrix of the UAV's visual sensor and the UAV's current flight altitude, the pixel offset is converted into a translation vector, and the UAV is driven to perform horizontal translation according to the translation vector, including: Obtain the focal length parameter from the intrinsic parameter matrix of the vision sensor; Calculate the ratio of the current flight altitude of the UAV to the focal length parameter, and generate a spatial projection scale representing the mapping relationship between pixels and actual size; Multiply the pixel offset by the spatial projection scale to calculate the horizontal offset distance of the midpoint pixel coordinates on the actual water surface reference plane; The current heading angle of the UAV is obtained, and the horizontal offset distance is decomposed by coordinate system rotation based on the heading angle to obtain the longitudinal translation control component along the longitudinal axis of the fuselage and the lateral translation control component along the transverse axis of the fuselage. The translation vector is obtained by vector synthesis of the longitudinal translation control component and the lateral translation control component.

7. The method according to claim 6, characterized in that, When the minimum pixel pitch continues to decrease to exceed a set extreme value limit, the drone is driven to increase its flight altitude to expand its actual field of view, including: Obtain the maximum pixel distance between the cable connection point and the umbilical cable entry point in the water area image; Multiply the maximum pixel distance by the spatial projection scale to calculate the target projection distance between the two points; Obtain the set safety margin of the field of view boundary, add the target projection spacing to the safety margin of the field of view boundary, and generate the safe field of view coverage width; Based on the triangular projection relationship between the safe field of view coverage width and the optical field of view angle parameters of the UAV, the target flight altitude that completely covers the safe field of view coverage width is calculated. The difference between the target flight altitude and the current flight altitude is converted into an ascent command, which is input to the flight controller of the UAV to drive the UAV to ascend to the target flight altitude.

8. The method according to claim 7, characterized in that, The method further includes: The remaining battery power of the drone is monitored in real time, and a first battery power threshold and a second battery power threshold are set, wherein the first battery power threshold is greater than the second battery power threshold; When the remaining battery power is lower than the first battery power threshold, the execution of the command to drive the drone to move is paused, a hovering command is generated to lock the current position and flight altitude of the drone, and the drone enters a static observation mode. In the static observation mode, the pixel coordinates of the cable connection point in the water area image are extracted in real time, and the visual deviation vector of the pixel coordinates relative to the geometric center of the water area image is calculated. The visual deviation vector is converted into a centering guidance command for the unmanned vessel and superimposed on the escort command and sent to the unmanned vessel to drive the unmanned vessel to autonomously adjust its course and speed until the cable connection point returns to the preset range of the geometric center of the water area image. When the remaining battery power is lower than the second battery power threshold, the real-time coordinates of the unmanned vessel in the water area image are extracted to generate a return landing trajectory, and the unmanned vessel is controlled to autonomously land along the return landing trajectory to the landing pad configured on the unmanned vessel.

Images