Underwater multi-view virtual imaging plane model construction method and device, switching method and device

Abstract

Underwater multi-view virtual imaging plane model construction method and device, switching method and device, relate to the fields of computer vision and robotics. In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail in conjunction with the accompanying drawings, specifically: underwater multi-view virtual imaging plane model construction method, the method includes: collecting the configuration of the underwater robot-manipulator system; according to the configuration of the underwater robot-manipulator system, the various cameras on the underwater robot-manipulator system are associated; according to the associated relationship, a multi-view imaging model is constructed; the homogeneous transformation matrix of the binocular camera of the hull where the underwater robot-manipulator system is located to the monocular camera coordinate system of the underwater robot-manipulator system is obtained; according to the homogeneous transformation matrix and the multi-view imaging model, a field of view sharing model is obtained. It can be applied to underwater operations.

Description

Technical Field

[0001] The invention relates to the fields of computer vision and robotics, and in particular to underwater multi-view fusion. Background Art

[0002] Affected by the underwater environment, the distance of visual image transmission is very limited. When our robot configuration is large, the camera located at the bow of the hull is far away from the UVMS's operating object, and the collected image will become blurred, resulting in a decrease in the underwater operation accuracy of the UVMS. In addition, the complex underwater environment and the configuration of the UVMS itself will also block the observation field of view of the bow camera of the hull during the operation. Compared with single-eye and binocular visual servoing operation control, multi-view visual servoing will provide the robot's operation process with more comprehensive operation target information and richer environmental interaction information.

[0003] The existing related research "Underwater Multi-View Image Fusion for Object Detection and Tracking" proposes an underwater multi-view image fusion method, which improves the accuracy of underwater target detection and tracking by fusing images from multiple underwater cameras. This implementation method uses the complementary information of multi-view images to obtain a clearer and more comprehensive underwater target image through an image fusion algorithm. However, the disadvantage of this implementation method is that it is sensitive to lighting changes and noise interference in the underwater environment, which can easily lead to distortion and errors in the image fusion results. Summary of the invention

[0004] In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically:

[0005] A method for constructing an underwater multi-view virtual imaging plane model, the method comprising:

[0006] Steps for collecting the configuration of the underwater robot-manipulator system;

[0007] According to the configuration of the underwater robot-manipulator system, the steps of associating the cameras on the underwater robot-manipulator system;

[0008] A step of constructing a multi-view imaging model according to the associated relationship;

[0009] The step of obtaining a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the coordinate system of the monocular camera of the underwater robot-manipulator system;

[0010] The step of obtaining a field of view sharing model according to the homogeneous transformation matrix and the multi-view imaging model.

[0011] Furthermore, a preferred embodiment is provided, in which the cameras are associated with each other through the relative posture relationship between the cameras.

[0012] Furthermore, a preferred embodiment is provided, in which a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the monocular camera coordinate system of the underwater robot-manipulator system is obtained through UVMS configuration.

[0013] Furthermore, a preferred implementation is provided, in which respective underlying field of view sharing virtual imaging plane models are established for all cameras to form a field of view sharing system, and then a field of view sharing model is obtained based on the homogeneous transformation matrix and the multi-view imaging model.

[0014] Furthermore, a preferred embodiment is provided, wherein the field of view sharing system includes camera parameters, pixel size and image plane density.

[0015] Based on the same inventive concept, the present invention also provides an underwater multi-view virtual imaging plane model construction device, the device comprising:

[0016] A module for collecting the configuration of an underwater robot-manipulator system;

[0017] A module for associating the cameras on the underwater robot-manipulator system according to the configuration of the underwater robot-manipulator system;

[0018] According to the associated relationship, a module of a multi-view imaging model is constructed;

[0019] A module for obtaining a homogeneous transformation matrix from a binocular camera of the hull where the underwater robot-manipulator system is located to a monocular camera coordinate system of the underwater robot-manipulator system;

[0020] A module of a field of view sharing model is obtained according to the homogeneous transformation matrix and the multi-view imaging model.

[0021] Based on the same inventive concept, the present invention also provides an underwater multi-view virtual imaging plane switching method, which is implemented based on a model constructed by the underwater multi-view virtual imaging plane model construction method, and the method includes:

[0022] When the monocular camera of the underwater robot-manipulator system cannot observe the target, a step of using the lower field of view to share the data feedback of the virtual imaging plane;

[0023] When the monocular camera of the underwater robot-manipulator system observes the target, a step of feeding back the upper layer high-resolution virtual imaging plane data based on a sub-pixel estimation algorithm is adopted.

[0024] Based on the same inventive concept, the present invention also provides an underwater multi-view virtual imaging plane switching device, which is implemented based on a model constructed by the underwater multi-view virtual imaging plane model construction device, and includes:

[0025] When the monocular camera of the underwater robot-manipulator system cannot observe the target, a module for data feedback of the virtual imaging plane shared by the lower field of view is used;

[0026] When the monocular camera of the underwater robot-manipulator system observes a target, a module for upper-layer high-resolution virtual imaging plane data feedback based on a sub-pixel estimation algorithm is used.

[0027] Based on the same inventive concept, the present invention also provides a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the underwater multi-view virtual imaging plane model construction method.

[0028] Based on the same inventive concept, the present invention also provides a computer, including a processor and a storage medium. When the computer program stored in the storage medium is read by the processor, the computer executes the underwater multi-view virtual imaging plane model construction method.

[0029] Compared with the prior art, the technical solution provided by the present invention is beneficial in that:

[0030] The underwater multi-view virtual imaging plane model construction method provided by the present invention uses multi-view vision to perform underwater visual servo operation control, and realizes the observation and control of underwater targets by establishing a visual field sharing virtual imaging plane model and switching strategy. Compared with single-eye and binocular visual servo operation control, multi-view vision provides more comprehensive operation target information and high-precision operation control.

[0031] The underwater multi-view virtual imaging plane model construction method provided by the present invention can provide more comprehensive operation target information and rich environmental interaction information by using multiple cameras to observe the target, thereby enhancing the observation capability of the underwater visual perception system.

[0032] The underwater multi-view virtual imaging plane model construction method provided by the present invention can realize the operation transition from long distance to short distance by switching different virtual imaging plane models, and solve the observation accuracy problem of underwater robots at different distances. This switching strategy can smoothly realize the operation transition from long distance to short distance, and enhance the robustness and stability of the system.

[0033] The underwater multi-view virtual imaging plane model construction method provided by the present invention improves the observation ability and operation accuracy of the underwater robot through the multi-view fusion model and the virtual imaging plane model switching strategy. Traditional underwater visual perception systems often only use a single camera for observation and cannot provide comprehensive operation target information. However, through the multi-view fusion model, the observation results of multiple cameras can be used at the same time to provide more comprehensive operation target information.

[0034] Compared with the traditional underwater visual perception system, the underwater multi-view virtual imaging plane model construction method provided by the present invention often has the problem of reduced observation accuracy when transitioning between long-distance and short-distance operations. However, through the virtual imaging plane model switching strategy, the transition between long-distance and short-distance operations can be smoothly achieved, thereby improving the operation accuracy.

[0035] The underwater multi-view virtual imaging plane model construction method provided by the present invention can be applied to enhance the observation capability of the underwater visual perception system and improve the underwater operation accuracy of the UVMS. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of a multi-view visual model;

[0037] Figure 2 Schematic diagram of binocular stereo vision model;

[0038] Figure 3 It is a schematic diagram of a virtual imaging plane model;

[0039] Figure 4 Schematic diagram of the switching strategy based on the multi-layer underwater multi-view virtual imaging plane model;

[0040] Figure 5 It is a schematic diagram of the homogeneous transformation relationship between the UVMS configuration and the multi-view image plane coordinate system;

[0041] Figure 6 for multi-view visual servoing. DETAILED DESCRIPTION

[0042] In order to make the advantages and benefits of the technical solution provided by the present invention more clearly reflected, the technical solution provided by the present invention is now further described in detail with reference to the accompanying drawings, specifically:

[0043] Embodiment 1: This embodiment provides a method for constructing an underwater multi-view virtual imaging plane model, the method comprising:

[0044] Steps for collecting the configuration of the underwater robot-manipulator system;

[0045] According to the configuration of the underwater robot-manipulator system, the steps of associating the cameras on the underwater robot-manipulator system;

[0046] A step of constructing a multi-view imaging model according to the associated relationship;

[0047] The step of obtaining a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the coordinate system of the monocular camera of the underwater robot-manipulator system;

[0048] The step of obtaining a field of view sharing model according to the homogeneous transformation matrix and the multi-view imaging model.

[0049] Specifically, the methods include:

[0050] Step 1: Determine the known underwater robot-manipulator system configuration.

[0051] Step 2: Obtain the relative position and posture relationships of each phase head from the known underwater robot-manipulator system configuration, and associate these relative position and posture relationships.

[0052] Step 3: Establish the coordinate system of the multi-view camera, and further associate the multi-view imaging model according to the relationship associated in step 2. According to the known UVMS configuration, the homogeneous transformation matrix from the hull binocular camera coordinate system to the end effector monocular camera coordinate system is obtained. First, according to the installation of the binocular camera and the monocular camera, their homogeneous transformation relative to the hull and end effector coordinate systems can be obtained. Then, according to the posture of the hull in the geodetic coordinate system, the homogeneous transformation of the hull coordinate system in the geodetic coordinate system is obtained as follows: Then, according to the known UVMS configuration, the homogeneous transformation from the hull coordinate system to the end effector coordinate system can be obtained: Therefore, the homogeneous transformation from the earth coordinate system to the end effector coordinate system can be obtained: Homogeneous transformation from the end effector monocular camera coordinate system to the hull binocular camera coordinate system

[0053] Step 4: Establish a shared virtual imaging plane model of the underlying field of view for each camera in the multi-view, such as Figure 4 shown.

[0054] Step 5: According to the above steps, the field of view sharing result from the hull binocular camera to the end effector monocular camera can be obtained ( C1 u, C1 v) can be obtained from the following formula:

[0055]

[0056] Where, the focal length of the camera is f, ρx ×ρ y is the pixel size, ρ x and ρ y is the pixel density of the image plane, the angle between the two coordinate axes of the camera image plane is γ, α=ρ x f,β=ρ y f, b is the distance between the optical axes of the two cameras of the hull binocular camera, are the maximum pixel values ​​along the v-axis and u-axis of the image plane respectively, and f[x] is the rounding function.

[0057] In addition to establishing a virtual imaging plane for shared view in each camera of the multi-view, a sub-pixel high-resolution virtual imaging plane is constructed on the virtual imaging plane level of shared view based on the pixel estimation algorithm for high-precision operation in the later stage of the underwater robot-manipulator system task. The characteristics of the sub-pixel estimation algorithm are as follows.

[0058] For a grayscale image, the n pixel information of the target area is expressed as are the integer position coordinates of each pixel point along the u-axis and v-axis directions in the image plane, is the brightness value of each pixel. C3 u, C3 v) is:

[0059]

[0060] in is the segmentation threshold function.

[0061] Implementation method 2: This implementation method further limits the underwater multi-view virtual imaging plane model construction method provided in implementation method 1, and associates the cameras through the relative posture relationship between the cameras.

[0062] Implementation method three: This implementation method is a further limitation of the underwater multi-view virtual imaging plane model construction method provided in implementation method one. Through UVMS configuration, a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the monocular camera coordinate system of the underwater robot-manipulator system is obtained.

[0063] Implementation method 4. This implementation method further limits the underwater multi-view virtual imaging plane model construction method provided in implementation method 1. It establishes respective lower-level field of view shared virtual imaging plane models for all cameras to form a field of view sharing system, and then obtains the field of view sharing model based on the homogeneous transformation matrix and the multi-view imaging model.

[0064] Implementation method 5: This implementation method further limits the underwater multi-view virtual imaging plane model construction method provided in implementation method 4. The field of view sharing system includes camera parameters, pixel size and image plane density.

[0065] Embodiment 6: This embodiment provides an underwater multi-view virtual imaging plane model construction device, the device comprising:

[0066] A module for collecting the configuration of an underwater robot-manipulator system;

[0067] A module for associating the cameras on the underwater robot-manipulator system according to the configuration of the underwater robot-manipulator system;

[0068] According to the associated relationship, a module of a multi-view imaging model is constructed;

[0069] A module for obtaining a homogeneous transformation matrix from a binocular camera of the hull where the underwater robot-manipulator system is located to a monocular camera coordinate system of the underwater robot-manipulator system;

[0070] A module of a field of view sharing model is obtained according to the homogeneous transformation matrix and the multi-view imaging model.

[0071] Specifically, the methods include:

[0072] When the underwater robot-manipulator system approaches the target, the image acquisition resolution is low at a long distance. The data feedback of the virtual imaging plane shared by the lower field of view is adopted. This feedback is based on the discrete integer pixels, with high stability but low precision. It can efficiently guide the underwater robot-manipulator system to approach the target. After the underwater robot-manipulator system approaches the target, at a closer distance, the multi-view camera can acquire a higher resolution image, but due to the principle of the camera's photosensitive element to acquire images, its feedback accuracy has an upper limit of the integer discretization, which limits the operation accuracy. At this stage, the upper layer high-resolution virtual imaging plane data feedback based on the sub-pixel estimation algorithm is adopted to improve the observation accuracy of the multi-view camera to the target, thereby improving the operation accuracy of the underwater robot-manipulator system.

[0073] Embodiment 7: This embodiment provides an underwater multi-view virtual imaging plane switching method, which is implemented based on a model constructed by the underwater multi-view virtual imaging plane model construction method provided in embodiment 1, and the method includes:

[0074] When the monocular camera of the underwater robot-manipulator system cannot observe the target, a step of using the lower field of view to share the data feedback of the virtual imaging plane;

[0075] When the monocular camera of the underwater robot-manipulator system observes the target, a step of feeding back the upper layer high-resolution virtual imaging plane data based on a sub-pixel estimation algorithm is adopted.

[0076] Embodiment 8: This embodiment provides an underwater multi-view virtual imaging plane switching device, which is implemented based on the model constructed by the underwater multi-view virtual imaging plane model construction device provided in Embodiment 6, and the device includes:

[0077] When the monocular camera of the underwater robot-manipulator system cannot observe the target, a module for data feedback of the virtual imaging plane shared by the lower field of view is used;

[0078] When the monocular camera of the underwater robot-manipulator system observes a target, a module for upper-layer high-resolution virtual imaging plane data feedback based on a sub-pixel estimation algorithm is used.

[0079] Embodiment 9: This embodiment provides a computer storage medium for storing a computer program. When the computer program is read by a computer, the computer executes the underwater multi-view virtual imaging plane model construction method provided in embodiment 1.

[0080] Embodiment 10: This embodiment provides a computer, including a processor and a storage medium. When the computer program stored in the storage medium is read by the processor, the computer executes the underwater multi-view virtual imaging plane model construction method provided in embodiment 1.

[0081] Implementation eleven: This implementation further describes the technical solutions provided in implementations one to eight in detail, specifically:

[0082] Affected by the underwater environment, the distance of visual image transmission is very limited. When our robot configuration is large, the camera located at the bow of the hull is far away from the UVMS operation object, and the collected image will become blurred, resulting in a decrease in the underwater operation accuracy of the UVMS. In addition, the complex underwater environment and the configuration of the UVMS itself will also block the observation field of view of the bow camera of the hull during the operation.

[0083] like Figure 6 As shown in the figure, compared with the single-eye and binocular visual servoing operation control, multi-view visual servoing will provide the robot with more comprehensive operation target information and richer environmental interaction information during the operation process. In addition, based on the proposed virtual imaging plane, the image errors of multiple cameras of UVMS are unified in the generalized degrees of freedom of UVMS. Compared with the visual servoing control method of switching between far and near cameras, the problem of transition from long-distance to short-distance operations can be smoothly realized, which greatly enhances the robustness and stability of the system.

[0084] Based on the above content, this embodiment relates to an underwater multi-view fusion model method and a virtual imaging plane model switching strategy. Mainly aimed at the difficult underwater perception environment of the underwater robot-manipulator system, the underwater visual servo is limited by the underwater visual perception space of the camera. An underwater multi-view fusion model is proposed to enhance the observation ability of the underwater visual perception system, thereby improving the underwater operation accuracy of the UVMS. In addition, based on the proposed virtual imaging plane, the image errors of multiple cameras of the UVMS are unified in the generalized degrees of freedom of the UVMS. Compared with the visual servo control method of switching between far and near cameras, the problem of transitioning from long distance to short distance operation can be smoothly realized, which greatly enhances the robustness and stability of the system.

[0085] In view of the different observation fields and observation accuracies of different cameras, this implementation method, based on the known UVMS model, constructs a virtual imaging plane model based on a large field of view camera to achieve simultaneous observation of multiple cameras of UVMS on the target. Based on the high-precision observation capability of the monocular camera on the UVMS end effector and the sub-pixel position estimation method of the image feature points, a high-precision information sharing virtual imaging plane is constructed to achieve high-precision operation control of the UVMS on the target. Combining the proposed field of view sharing virtual imaging plane and the high-precision information sharing virtual imaging plane, a virtual imaging plane model switching strategy is designed to achieve the stability and robustness of UVMS during the operation process.

[0086] The implementation is as follows:

[0087] An underwater multi-view fusion model method and a virtual imaging plane model switching strategy, the method steps are as follows:

[0088] Step 1: Establish binocular stereo vision for the UVMS hull binocular camera head, such as Figure 1 As shown. Define the position coordinates of target point 1 in the earth coordinate system {I} in the camera coordinate system {C1} as 1 P1=( 1 x1, 1 y1, 1 z1), it is calculated using binocular stereo vision.

[0089] Step 2: For the UVMS end-effector monocular camera, establish a virtual imaging plane and a virtual imaging plane coordinate system. The virtual imaging plane coincides with the imaging plane of the monocular camera. The x-axis of the virtual imaging plane coincides with the u-axis of the monocular camera imaging plane and has the same direction; the y-axis of the virtual imaging plane coincides with the v-axis of the monocular camera imaging plane and has the same direction; the z-axis of the virtual imaging plane coincides with the z-axis of the monocular camera optical axis and has the same direction, as shown in Figure 2. Figure 2 shown.

[0090] Step 3: Based on the known UVMS configuration, obtain the homogeneous transformation matrix from the hull binocular camera coordinate system to the end effector monocular camera coordinate system. First, based on the installation of the binocular camera and the monocular camera, their homogeneous transformations relative to the hull and end effector coordinate systems can be obtained. Then, according to the posture of the hull in the geodetic coordinate system, the homogeneous transformation of the hull coordinate system in the geodetic coordinate system is obtained as follows: Then, according to the known UVMS configuration, the homogeneous transformation from the hull coordinate system to the end effector coordinate system can be obtained: Therefore, the homogeneous transformation from the earth coordinate system to the end effector coordinate system can be obtained: like Figure 5 shown.

[0091] Step 4: When the UVMS multi-camera system's hull double-sided camera observes the target, the projection position result (u, v) of the target observed by the hull binocular camera on the virtual imaging plane established in step 2 can be obtained from the following formula according to the relationship between the hull double-sided coordinate system mentioned in step 3 and the end-effector monocular camera coordinate system:

[0092]

[0093] Where, the focal length of the camera is f, ρ x ×ρ y is the pixel size, ρ x and ρ y is the pixel density of the image plane, the angle between the two coordinate axes of the camera image plane is γ, α=ρ x f,β=ρ y f, b is the distance between the optical axes of the two cameras of the hull binocular camera. are the maximum pixel values ​​along the v-axis and u-axis of the image plane respectively, and f[x] is the rounding function.

[0094] Step 5: According to the observation requirements of the UVMS end-effector, the expected position η of the UVMS end-effector in the earth coordinate system {I} can be obtained eed Then, a closed-loop controller is designed to use the multi-degree-of-freedom motion adjustment of UVMS to achieve the actual position η of the end effector ee Towards the desired pose η eed Convergence means that the UVMS multi-camera system can observe the target simultaneously. Due to the deviation of the known configuration of UVMS, the relationship between the homogeneous transformation matrices in step 3 will be inaccurate. In addition, since the hull binocular camera is far away from the target, the observation accuracy of the target is low, resulting in deviation in the observation result in step 1. Therefore, the above steps can realize a low-precision lower-level field of view sharing virtual imaging plane model based on underwater multi-view.

[0095] Step 6: When the UVMS multi-camera system observes the target at the same time, the monocular camera with the end effector is closer to the target, so a higher-precision observation result can be obtained. Combined with the sub-pixel estimation method of the image plane feature point, the sub-pixel observation result of the target can be obtained on the virtual imaging plane of the monocular camera of the end effector.

[0096] Step 7: Based on the high-precision observation results obtained by the end-effector monocular camera in step 6, the high-precision fusion information of the hull binocular camera can also be obtained based on the homogeneous change relationship between the hull double-sided camera coordinate system and the end-effector monocular camera coordinate system obtained in step 3. At this time, although the hull binocular camera has low observation accuracy due to its long distance from the target, the monocular camera on the end-effector close to the target completes the high-precision observation. Then, the sharing and fusion of multi-view high-precision perception information is realized through the upper high-resolution virtual imaging plane model based on the sub-pixel estimation algorithm and the underwater multi-view virtual imaging plane model, thereby realizing the high-precision observation of the target by the UVMS multi-camera system.

[0097] Step 8: During the process of UVMS approaching the target, when the monocular camera on the end effector cannot observe the target, UVMS switches to the underwater multi-view low-precision lower-level field of view sharing virtual imaging plane model method to share the results of the hull binocular wide field of view observation to the monocular camera of the end effector, and realizes the simultaneous observation of the target by UVMS multiple cameras according to the method described in step 5. When UVMS multiple cameras observe the target at the same time, UVMS switches to the upper-level high-resolution virtual imaging plane model method of the underwater multi-view virtual imaging plane model to share the monocular close-range high-precision observation results to UVMS to realize its overall high-precision control.

[0098] The implementation steps of this embodiment also include some specific implementation principles, as follows:

[0099] a) Virtual imaging plane model based on underwater multi-view field of view sharing

[0100] According to the camera perspective projection imaging model, for a specific camera j, its position in the geodetic coordinate system {I} is uniquely determined. Due to the size limit of the camera's photosensitive element, the position of camera j in its coordinate system {C j The range of the imaging plane under} can be expressed as:

[0101]

[0102] Then the camera's projection mapping matrix is ​​in the camera coordinate system {C j} is expressed as:

[0103]

[0104] Therefore, the projection of camera j on the edge of the imaging plane is consistent with the camera j coordinate system {C j}Target feature points under j P=( j x, j y, j z) is:

[0105]

[0106] Then, the field of view space that camera j can observe is in its coordinate system {C j} is expressed as:

[0107]

[0108] Therefore, for a given camera j, its field of view is limited by the size of the photosensitive element (|u lim |,|v lim |), the focal length of the camera, and the depth of the camera relative to the target j z. And it can be concluded that the larger the size of the camera's photosensitive element (|u lim |,|v lim |), the smaller the focal length f of the camera, the closer the depth of the camera relative to the target j The larger the z is, the larger the field of view is. However, in actual operation, firstly, the propagation of light underwater decays very quickly as the propagation distance increases, so when images with higher imaging accuracy are required, it is necessary to use j z is as small as possible without threatening the safety of the camera and UVMS. For a camera j that has been manufactured, the size of its photosensitive element and the focal length f of the camera will not change in subsequent use. Therefore, in order to solve the contradiction of small field of view of close-range imaging and poor image quality of long-range imaging when UVMS is working underwater, this embodiment proposes to use multi-view vision to control the underwater visual servoing operation of UVMS. In this way, the camera j in different positions of the geodetic coordinate system {I} can be used to servo the target at the same position in the geodetic coordinate system {I}. I P=( I x, I y, I z) to perform observation operations, which can satisfy the target of the UVMS system containing each camera j under the geodetic coordinate system {I} I P=( I x, I y, I z), not only in different j Under z, observations with a large field of view and high-precision imaging are realized. Figure 1 shown.

[0109] In actual UVMS underwater operations, one of the cameras is often the first to observe the target in the geodetic coordinate system {I} within its field of view. I P=( I x, I y, I z), and then under the control of each degree of freedom of UVMS, the target can be observed within the field of view of multiple cameras. I P=( I x, I y, I z). The usual method of controlling UVMS in this process is to use two cameras on the hull to form binocular stereo vision, so that the two cameras on the bow of the hull can focus on the same target. I P=( I x, I y, I z) Imaging feature points in their respective imaging planes and The parallax formed by binocular stereo vision is used to calculate the target I P=( I x, I y, I z) is the position in the camera coordinate system {C1} or {C2}. The binocular stereo vision model described by the camera projection perspective model is as follows: Figure 2 shown.

[0110] Let, in the geodetic coordinate system {I} I P=( I x, I y, I z) in the camera coordinate system {C1} is 1 P1=( 1 x1, 1 y1, 1 z1), then binocular stereo vision is used to calculate:

[0111]

[0112] in, is the component of the projection point of target point 1 on the image plane of camera j on the u-axis of the image plane coordinate system, is the component of the projection point of target point 1 on the image plane of camera j on the v-axis of the image plane coordinate system, b is the distance between the two camera optical axes of the binocular camera, ρ x , ρ y are the pixel densities along the u-axis and v-axis of the camera image plane, respectively, and f is the focal length of the camera.

[0113] Virtual imaging plane model Figure 3 As shown. Corresponding to the definition of the imaging plane of the camera, the camera j is in its coordinate system {C j The range of the virtual imaging plane under} can be expressed as:

[0114]

[0115] Then, from equation (4), we can get the field of view space observed by camera j based on the virtual imaging plane in its coordinate system {C j} is expressed as:

[0116]

[0117] Therefore, in the virtual imaging plane, camera j can realize the target in any geodetic coordinate system {I} I P=( I x, I y, I z) to perform "observation". In practice, the position of the target in the geodetic coordinate system {I} I P=( I x, I y, I z) can be obtained by calculating the disparity of binocular stereo vision or monocular continuous frames. Therefore, in the multi-view visual servoing process of the virtual imaging plane, when one of the multiple cameras carried by the UVMS observes the target, it is equivalent to all cameras observing the target.

[0118] b) A switching strategy based on a multi-layer underwater multi-view virtual imaging plane model

[0119] When UVMS only observes the target through the hull binocular, UVMS performs motion control based on the feedback information from the hull binocular. Then the binocular information obtained is used to adjust the end effector of UVMS in a low-precision lower-level field of view shared virtual imaging plane model, so that the monocular camera on the end effector can achieve close-range observation of the target. When the monocular camera on the end effector of UVMS observes the target, the monocular camera digitizes the high-precision observation results obtained at close range through a sub-pixel estimation algorithm, and then the shared virtual imaging plane model is switched to the upper-level high-resolution virtual imaging plane model for high-precision observation information sharing, thereby achieving high-precision operation control of UVMS. In this process, when the target is lost from the field of view of the monocular camera on the end effector of UVMS, the shared virtual imaging plane model is switched again to the low-precision lower-level field of view shared virtual imaging plane model to adjust the end effector of UVMS to achieve simultaneous observation of the target by each camera of UVMS. The switching strategy based on the multi-layer underwater multi-view virtual imaging plane model is as follows: Figure 4 shown.

[0120] Embodiment 12: This embodiment is a preferred variation of the technical solution provided in Embodiment 11, specifically:

[0121] Here are the steps:

[0122] Step 1: Establish binocular stereo vision for the UVMS hull binocular camera head, such as Figure 1 As shown. Define the position coordinates of target point 1 in the earth coordinate system {I} in the camera coordinate system {C1} as 1 P1=( 1 x1, 1 y1, 1 z1), then binocular stereo vision is used to calculate:

[0123]

[0124] in, is the component of the projection point of target point 1 on the image plane of camera j on the u-axis of the image plane coordinate system, is the component of the projection point of target point 1 on the image plane of camera j on the v-axis of the image plane coordinate system, b is the distance between the two camera optical axes of the binocular camera, ρ x , ρ y are the pixel densities along the u-axis and v-axis of the camera image plane, respectively, and f is the focal length of the camera.

[0125] Step 2: For the UVMS end-effector monocular camera, establish a virtual imaging plane and a virtual imaging plane coordinate system. The virtual imaging plane coincides with the imaging plane of the monocular camera. The x-axis of the virtual imaging plane coincides with the u-axis of the monocular camera imaging plane and has the same direction; the y-axis of the virtual imaging plane coincides with the v-axis of the monocular camera imaging plane and has the same direction; the z-axis of the virtual imaging plane coincides with the z-axis of the monocular camera optical axis and has the same direction, as shown in Figure 2. Figure 2 shown.

[0126] Step 3: Based on the known UVMS configuration, obtain the homogeneous transformation matrix from the hull binocular camera coordinate system to the end effector monocular camera coordinate system. First, based on the installation of the binocular camera and the monocular camera, their homogeneous transformations relative to the hull and end effector coordinate systems can be obtained. Then, according to the posture of the hull in the geodetic coordinate system, the homogeneous transformation of the hull coordinate system in the geodetic coordinate system is obtained as follows: Then, according to the known UVMS configuration, the homogeneous transformation from the hull coordinate system to the end effector coordinate system can be obtained: Therefore, the homogeneous transformation from the earth coordinate system to the end effector coordinate system can be obtained: The homogeneous transformation relationship between UVMS configuration and multi-view image plane coordinate system is as follows: Figure 5 shown.

[0127] Step 4: When the UVMS multi-camera system's hull double-sided camera observes the target, the projection position result (u, v) of the target observed by the hull binocular camera on the virtual imaging plane established in step 2 can be obtained from the following formula according to the relationship between the hull double-sided coordinate system mentioned in step 3 and the end-effector monocular camera coordinate system:

[0128]

[0129] Where, the focal length of the camera is f, ρ x ×ρ y is the pixel size, ρ x and ρ y is the pixel density of the image plane, the angle between the two coordinate axes of the camera image plane is γ, α=ρ x f,β=ρ y f, b is the distance between the optical axes of the two cameras of the hull binocular camera. are the maximum pixel values ​​along the v-axis and u-axis of the image plane respectively, and f[x] is the rounding function.

[0130] Step 5: According to the observation requirements of the UVMS end-effector, the expected position η of the UVMS end-effector in the earth coordinate system {I} can be obtained eed Then, a closed-loop controller is designed to use the multi-degree-of-freedom motion adjustment of UVMS to achieve the actual position η of the end effector ee Towards the desired pose η eed Once convergence, the UVMS multi-camera system can observe the target simultaneously.

[0131] The multi-view feature point information fusion strategy can be but is not the only one:

[0132] The expected position of each feature point in each camera image plane is defined as:

[0133]

[0134] Then, the error of each feature point in each camera image plane is:

[0135]

[0136] in, is the actual position of feature point i on the image plane of camera j.

[0137] According to the velocity Jacobian relationship between UVMS and each camera, the relationship between the error of each image plane feature point and the UVMS multi-degree-of-freedom adjustment error can be further obtained:

[0138]

[0139] Among them, J UVMS,cj is the velocity Jacobian relationship between UVMS and camera j, is the image Jacobian matrix of camera j. The self-adjusting weighting matrix is ​​used to adjust the hull freedom for long-distance targets and the manipulator freedom strategy for short-distance targets. The fusion weighting matrix is ​​defined as follows:

[0140]

[0141] in is the manually set fusion weight matrix parameter. Then the image error of camera j and the multi-degree-of-freedom redundancy allocation expectation of UVMS are:

[0142]

[0143] In the multi-view fusion visual servo process, since each camera has different degrees of observation of the target, it is necessary to weight the image errors transmitted by each camera and the UVMS multi-degree-of-freedom allocation according to the confidence level. The multi-camera image error to UVMS multi-degree-of-freedom redundant allocation is weighted fused according to the confidence level to obtain the fusion expectation strategy:

[0144]

[0145] Among them, s j is the real-time confidence of camera j, and s0 is the set confidence benchmark parameter.

[0146] The expected value ζ of multi-degree-of-freedom redundant allocation of UVMS is obtained d After that, the closed-loop controller is designed for each degree of freedom of UVMS to complete the convergence of each degree of freedom of UVMS. Because the expected position of each camera feature point in its image plane is in its actual imaging plane, when the position of the feature point converges to the expected position, the UVMS multi-camera system can observe the target simultaneously.

[0147] Step 6: When the UVMS multi-camera system observes the target at the same time, the monocular camera with the end effector is closer to the target, so a higher-precision observation result can be obtained. Combined with the sub-pixel estimation method of the image plane feature point, the sub-pixel observation result of the target can be obtained on the virtual imaging plane of the monocular camera of the end effector. The characteristics of the sub-pixel estimation algorithm are as follows.

[0148] For a grayscale image, the n pixels in the target area are represented as are the integer position coordinates of each pixel point along the u-axis and v-axis directions in the image plane, is the brightness value of each pixel. C3 u, C3 v) is:

[0149]

[0150] in is the segmentation threshold function.

[0151] Step 7: Based on the high-precision observation results obtained by the end-effector monocular camera in step 6, the high-precision fusion information of the hull binocular camera can also be obtained based on the homogeneous change relationship between the hull double-sided camera coordinate system and the end-effector monocular camera coordinate system obtained in step 3. At this time, although the hull binocular camera has low observation accuracy due to its long distance from the target, the monocular camera on the end-effector close to the target completes the high-precision observation. Then, the sharing and fusion of multi-view high-precision perception information is realized by the upper-layer high-resolution virtual imaging plane model based on the sub-pixel estimation algorithm and the underwater multi-view virtual imaging plane model, thereby realizing the high-precision observation of the target by the UVMS multi-camera system. The strategy for fusing the high-precision perception information of the monocular camera with the UVMS binocular camera information in this step can, but not exclusively, adopt the fusion strategy in step 5.

[0152] Step 8: During the process of UVMS approaching the target, when the monocular camera on the end effector cannot observe the target, UVMS switches to the underwater multi-view low-precision lower-level field of view sharing virtual imaging plane model method to share the results of the hull binocular wide field of view observation to the monocular camera of the end effector, and realizes the simultaneous observation of the target by UVMS multiple cameras according to the method described in step 5. When UVMS multiple cameras observe the target at the same time, UVMS switches to the upper-level high-resolution virtual imaging plane model method of the underwater multi-view virtual imaging plane model to share the monocular close-range high-precision observation results to UVMS to realize its overall high-precision control.

[0153] The technical solution provided by the present invention is further described in detail above through several specific implementation modes in order to highlight the advantages and benefits of the technical solution provided by the present invention. However, the several specific implementation modes described above are not intended to be used as limitations on the present invention. Any reasonable modification and improvement of the present invention, combination of implementation modes and equivalent substitution within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0154] In the description of this specification, only the preferred embodiments of the present invention are provided, and the scope of the rights of the present invention cannot be limited by this. In addition, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, in the absence of mutual contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. In addition, the terms "first" and "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "N" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. Any process or method description in the flowchart or otherwise described herein can be understood to represent a module, fragment or portion of code including one or more executable instructions for implementing the steps of a custom logic function or process, and the scope of the preferred embodiment of the present invention includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved, which should be understood by a person skilled in the art of the art to which the embodiments of the present invention belong. The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logic function, can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, device or device and execute instructions), or in combination with these instruction execution systems, devices or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit programs for use by an instruction execution system, device or device or in combination with these instruction execution systems, devices or devices. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection having one or N wirings (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM).In addition, the computer readable medium can even be paper or other suitable medium on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, then editing, interpreting or processing in other suitable ways as necessary, and then storing it in a computer memory. It should be understood that the various parts of the present invention can be implemented with hardware, software, firmware, or a combination thereof. In the above-mentioned embodiment, N steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented with hardware, as in another embodiment, any one of the following technologies known in the art or their combination can be used to implement: a discrete logic circuit with a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit with a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0155] A person of ordinary skill in the art can understand that all or part of the steps carried by the method of the above-mentioned embodiment can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the program, when executed, includes one or a combination of the steps of the method embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

Claims

1. A method for constructing an underwater multi-view virtual imaging plane model, characterized in that: The method comprises: Steps for collecting the configuration of the underwater robot-manipulator system; According to the configuration of the underwater robot-manipulator system, the steps of associating the cameras on the underwater robot-manipulator system; Specifically, the cameras are associated with each other through the relative position and posture relationship of the cameras; A step of constructing a multi-view imaging model according to the associated relationship; Specifically, the coordinate system of the multi-view camera is established. First, according to the installation of the binocular camera and the monocular camera, the homogeneous transformation relative to the hull and the end effector coordinate system is obtained. Then, according to the posture of the hull in the geodetic coordinate system, the homogeneous transformation of the hull coordinate system in the geodetic coordinate system is obtained: Then, according to the known UVMS configuration, the homogeneous transformation from the hull coordinate system to the end effector coordinate system is obtained: The homogeneous transformation from the earth coordinate system to the end effector coordinate system can be obtained Homogeneous transformation from the end effector monocular camera coordinate system to the hull binocular camera coordinate system To build a multi-view imaging model; The step of obtaining a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the coordinate system of the monocular camera of the underwater robot-manipulator system; A step of obtaining a view sharing model according to the homogeneous transformation matrix and the multi-view imaging model; Specifically, the vision sharing result ( C1 u, C1 v) is obtained from the following formula: Where, the focal length of the camera is f, ρ x ×ρ y is the pixel size, ρ x and ρ y is the pixel density of the image plane, the angle between the two coordinate axes of the camera image plane is γ, α=ρ x f,β=ρ y f, b is the distance between the optical axes of the two cameras of the hull binocular camera, are the maximum pixel values ​​along the v-axis and u-axis directions of the image plane respectively, f[x] is the rounding function toward zero; I on the left side of T represents the camera, and l represents the earth inertial coordinate system.

2. The underwater multi-view virtual imaging plane model construction method according to claim 1, characterized in that: The cameras are associated with each other through the relative position and posture relationship between the cameras.

3. The underwater multi-view virtual imaging plane model construction method according to claim 1, characterized in that: Through UVMS configuration, a homogeneous transformation matrix from the binocular camera of the hull where the underwater robot-manipulator system is located to the monocular camera coordinate system of the underwater robot-manipulator system is obtained.

4. The underwater multi-view virtual imaging plane model construction method according to claim 1, characterized in that: A respective lower-level field of view sharing virtual imaging plane model is established for all cameras to form a field of view sharing system, and then a field of view sharing model is obtained according to the homogeneous transformation matrix and the multi-view imaging model.

5. The underwater multi-view virtual imaging plane model construction method according to claim 4, characterized in that: The field of view sharing system includes camera parameters, pixel size and image plane density.

6. An underwater multi-view virtual imaging plane model construction device, characterized in that: The device comprises: A module for collecting the configuration of an underwater robot-manipulator system; A module for associating the cameras on the underwater robot-manipulator system according to the configuration of the underwater robot-manipulator system; Specifically, the cameras are associated with each other through the relative position and posture relationship of the cameras; According to the associated relationship, a module of a multi-view imaging model is constructed; Specifically, the coordinate system of the multi-view camera is established. First, according to the installation of the binocular camera and the monocular camera, the homogeneous transformation relative to the hull and the end effector coordinate system is obtained. Then, according to the posture of the hull in the geodetic coordinate system, the homogeneous transformation of the hull coordinate system in the geodetic coordinate system is obtained: Then, according to the known UVMS configuration, the homogeneous transformation from the hull coordinate system to the end effector coordinate system is obtained: The homogeneous transformation from the earth coordinate system to the end effector coordinate system can be obtained Homogeneous transformation from the end effector monocular camera coordinate system to the hull binocular camera coordinate system To build a multi-view imaging model; A module for obtaining a homogeneous transformation matrix from a binocular camera of the hull where the underwater robot-manipulator system is located to a monocular camera coordinate system of the underwater robot-manipulator system; A module for obtaining a view sharing model according to the homogeneous transformation matrix and the multi-view imaging model; Specifically, the vision sharing result ( C1 u, C1 v) is obtained from the following formula: Where, the focal length of the camera is f, ρ x ×ρ y is the pixel size, ρ x and ρ y is the pixel density of the image plane, the angle between the two coordinate axes of the camera image plane is γ, α=ρ x f,β=ρ y f, b is the distance between the optical axes of the two cameras of the hull binocular camera, are the maximum pixel values ​​along the v-axis and u-axis directions of the image plane respectively, f[x] is the rounding function toward zero; I on the left side of T represents the camera, and l represents the earth inertial coordinate system.

7. An underwater multi-view virtual imaging plane switching method, characterized in that: The method is implemented based on a model constructed by the underwater multi-view virtual imaging plane model construction method according to claim 1, and the method comprises: When the monocular camera of the underwater robot-manipulator system cannot observe the target, a step of using the lower field of view to share the data feedback of the virtual imaging plane; When the monocular camera of the underwater robot-manipulator system observes the target, a step of feeding back the upper layer high-resolution virtual imaging plane data based on a sub-pixel estimation algorithm is adopted.

8. An underwater multi-view virtual imaging plane switching device, characterized in that: The device is implemented based on the model constructed by the underwater multi-view virtual imaging plane model construction device according to claim 6, and the device includes: When the monocular camera of the underwater robot-manipulator system cannot observe the target, a module for data feedback of the virtual imaging plane shared by the lower field of view is used; When the monocular camera of the underwater robot-manipulator system observes a target, a module for upper-layer high-resolution virtual imaging plane data feedback based on a sub-pixel estimation algorithm is used.

9. A computer storage medium for storing a computer program, characterized in that: When the computer program is read by a computer, the computer executes the underwater multi-view virtual imaging plane model construction method according to claim 1.

10. A computer, comprising a processor and a storage medium, characterized in that: When the computer program stored in the storage medium is read by the processor, the computer executes the underwater multi-view virtual imaging plane model construction method according to claim 1.

Images