A Method for Estimating the Inner Wall Depth of a Facility-Based Crab Cultivation Cage Based on Machine Vision

Abstract

The present application discloses a method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision, which relates to the technical field of facility-based breeding. The present application combines a monocular depth estimation model with the geometric structure constraints of the crab breeding cage. Through the monocular depth estimation model, the depth of objects in the image is preliminarily predicted. On this basis, relying on prior information such as the structure and physical size of the crab breeding cage, by constructing an analytic geometry calculation model, the depth values of each point on the inner wall of the crab breeding cage in the camera coordinate system are restored, which can significantly alleviate the scale drift and deviation problems existing in the existing monocular depth estimation methods and improve the accuracy of the depth estimation results. The present application does not rely on deep learning technology and does not require the construction and annotation of a large-scale training data set.

Description

Technical Field

[0001] This application relates to the technical field of facility aquaculture, and particularly to a method for estimating the inner wall depth of a facility crab culture cage based on machine vision. Background Art

[0002] Facility aquaculture is an important direction for the future development of smart agriculture. To accurately monitor the growth status of aquaculture objects, it is necessary to restore the true distances of various objects in the scene in the camera coordinate system based on image information. Existing depth estimation models based on monocular vision usually have different degrees of scale drift and deviation in different regions of the image, resulting in a large difference between the predicted depth and the true depth, and it is difficult to meet the usage requirements of growth status monitoring. In addition, image depth estimation based on deep learning often requires collecting a large amount of depth image data to train or fine-tune the model, which requires a large data collection and training cost. Summary of the Invention

[0003] The purpose of this application is to provide a method for estimating the inner wall depth of a facility crab culture cage based on machine vision to improve the accuracy of estimating the inner wall depth of the facility crab culture cage.

[0004] To achieve the above purpose, the following solutions are provided in this application.

[0005] In the first aspect, this application provides a method for estimating the inner wall depth of a facility crab culture cage based on machine vision, including the following steps.

[0006] Obtain a top view image of the crab culture cage.

[0007] Based on a monocular depth estimation model, obtain the depth estimation value of each pixel point in the top view image.

[0008] According to the depth estimation values of each pixel point, convert the pixel coordinates of each pixel point on the top view image into the three-dimensional point cloud data of each pixel point; the three-dimensional point cloud data of the pixel point is the three-dimensional coordinate of the pixel point in the camera coordinate system.

[0009] Calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point.

[0010] Based on the normal vector of each pixel point, determine the plane of the crab culture cage to which each pixel point belongs; the crab culture cage includes 5 planes, and the 5 planes are the front, back, left, right, and bottom of the crab culture cage respectively.

[0011] Respectively, according to the pixel coordinates of the pixel points on each plane and the physical size of the crab culture cage, determine the inner wall depth at different pixel points of each plane.

[0012] According to the specific embodiments provided in the present application, the present application has the following technical effects.

[0013] The present application provides a method for estimating the depth of the inner wall of a facility-based crab breeding cage based on machine vision. The present application combines a monocular depth estimation model with the geometric structure constraints of the crab breeding cage. Through the monocular depth estimation model, the depth of objects in the image is initially predicted. On this basis, relying on prior information such as the structure and physical dimensions of the crab breeding cage, by constructing an analytic geometry calculation model, the depth values of each point on the inner wall of the crab breeding cage in the camera coordinate system are restored, which can significantly alleviate the scale drift and deviation problems existing in the existing monocular depth estimation methods and improve the accuracy of the depth estimation results. The present application does not rely on deep learning technology and does not require constructing and annotating a large-scale training dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0015] Figure 1 It is a schematic flow chart of a method for estimating the depth of the inner wall of a facility-based crab breeding cage based on machine vision provided by an embodiment of the present application.

[0016] Figure 2 It is a schematic diagram of the principle of a method for estimating the depth of the inner wall of a facility-based crab breeding cage based on machine vision provided by an embodiment of the present application.

[0017] Figure 3 It is a structural layout diagram of a camera provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0018] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0019] To make the above objects, features, and advantages of the present application more obvious and understandable, the present application will be further described in detail below with reference to the drawings and specific embodiments.

[0020] Existing monocular depth estimation models are generally constructed based on deep learning methods, which focus more on using image pixel values and local neighborhood information. However, in facility-based three-dimensional aquaculture, crab cages are usually used, which have regular geometric structures, and such information has not been effectively utilized in existing monocular depth estimation models. Therefore, further improvement is needed.

[0021] This application combines a monocular depth estimation model with the geometric structure constraints of crab cages. Through the monocular depth estimation model, the depth of objects in the image is initially predicted. On this basis, relying on prior information such as the camera imaging model and the size of crab cages, by constructing an analytic geometry calculation model, the depth values of each point on the inner wall of the crab cage in the camera coordinate system are restored, which can alleviate the scale drift and deviation problems existing in existing monocular depth estimation methods and improve the accuracy of depth estimation results. This application does not rely on deep learning technology and does not require the construction and annotation of large-scale training datasets.

[0022] In an exemplary embodiment, a method for estimating the depth of the inner wall of a facility-based crab cage based on machine vision is provided. This method can use the relative depth output by the depth estimation model, combined with the camera internal parameters and the physical size of the crab cage, to estimate the depth values of each part of the inner wall of the crab cage in the image in the camera coordinate system. Specifically, it includes: image depth estimation and point cloud generation, surface normal vector calculation, inner wall segmentation of the crab cage, and depth calculation of the inner wall of the crab cage, etc. This method can be deployed in the actual environment to provide an intelligent analysis tool for facility-based crab aquaculture.

[0023] As Figure 1 and Figure 2 shown, the method for estimating the depth of the inner wall of a facility-based crab cage based on machine vision of this application includes the following steps 101 - step 106.

[0024] Step 101, obtain a top-down image of the crab cage.

[0025] Step 102, based on the monocular depth estimation model, obtain the depth estimation value of each pixel point in the top-down image.

[0026] Step 103, convert the pixel coordinates of each pixel point on the top-down image into the three-dimensional point cloud data of each pixel point according to the depth estimation value of each pixel point; the three-dimensional point cloud data of the pixel point is the three-dimensional coordinates of the pixel point in the camera coordinate system.

[0027] Step 104, calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point.

[0028] Step 105, determine the plane of the crab cage to which each pixel point belongs based on the normal vector of each pixel point; the crab cage includes 5 planes, and the 5 planes are the front, back, left, right, and bottom of the crab cage respectively.

[0029] Step 106: Determine the inner wall depth at different pixel points on each plane according to the pixel coordinates of the pixel points on each plane and the physical dimensions of the crab cultivation cage respectively.

[0030] Implementing the above Step 101 - Step 106 can alleviate the scale drift and deviation problems existing in the existing monocular depth estimation method and improve the accuracy of the depth estimation result.

[0031] In another exemplary embodiment, in the above Step 101, a regular cuboid crab cultivation cage is used. As Figure 3 shown, its internal physical dimensions (length, width, height) are known quantities. The camera is placed above the center point of the crab cultivation cage for a top-down shot. The camera imaging plane A'B'C'D' is parallel to the bottom surface ABCD of the crab cultivation cage, and the camera optical center O' is aligned with the geometric center O of the bottom surface of the crab cultivation cage; the camera internal parameter matrix is a known quantity and can be obtained through an offline calibration method.

[0032] In another exemplary embodiment, in the above Step 102, a monocular depth estimation model (such as Depth Pro, etc.) is used to perform depth estimation on the top-down image captured by the camera to obtain the depth estimation value of each pixel point in the top-down image , that is, the z-axis coordinate of the pixel point in the camera coordinate system .

[0033] In another exemplary embodiment, in the above Step 103, based on the depth estimation value of the pixel point and the camera internal parameter matrix , the pixel coordinates of each pixel point in the top-down image are converted into the three-dimensional point cloud data of each pixel point, forming a three-dimensional description of the camera's field of view. The conversion formula is as follows.

[0034] , , .

[0035] Among them, is the three-dimensional coordinate of the pixel point in the camera coordinate system, that is, the three-dimensional point cloud data, , and are the x-axis, y-axis, and z-axis coordinates of the pixel point in the camera coordinate system respectively; is the depth estimation value of the pixel point; is the pixel coordinate of the pixel point on the top-down image, and are the horizontal coordinate and vertical coordinate of the pixel point on the top-down image respectively; is the camera principal point coordinate, and are the horizontal coordinate and the vertical coordinate of the principal point of the camera, respectively; and are the focal lengths of the camera in the x-axis direction and the y-axis direction of the camera coordinate system, respectively.

[0036] In this application, the pixel coordinates of a pixel point are the coordinates of the pixel point in the image coordinate system, and the horizontal coordinate and the vertical coordinate of the pixel point are the x-axis coordinate and the y-axis coordinate of the pixel point in the image coordinate system.

[0037] In another exemplary embodiment, based on the three-dimensional point cloud data, the normal vectors corresponding to each point on the inner wall of the crab breeding cage (i.e., each pixel point in the top view image) are calculated. Let the neighborhood search radius be r and the maximum number of nearest neighbors be max_nn. Under the limitation of the maximum number of nearest neighbors max_nn and the neighborhood search radius r, find all the nearest neighbor points centered on the pixel point Based on each pixel point and its neighborhood points, a local plane can be fitted, and the normal vector of this local plane is the normal vector of the pixel point . In the embodiment of this application, step 104 above can be replaced by the following steps 201 - 203.

[0038] Step 201: Obtain a plurality of pixel points within the neighborhood range of the pixel point .

[0039] Step 202: Perform plane fitting on the pixel point and a plurality of pixel points within the neighborhood range of the pixel point to obtain the local plane at the pixel point .

[0040] Step 203: Solve the normal vector of the local plane as the normal vector of the pixel point .

[0041] During the solution process, it is necessary to first determine the centroid of the local plane: .

[0042] where is the centroid of the local plane, is the number of pixel points on the local plane, is the th three-dimensional point cloud data of the pixel point on the local plane.

[0043] Then, decentralize the local plane to obtain the decentralized three-dimensional point cloud data of each pixel point on the local plane: .

[0044] where is the The three-dimensional point cloud data after decenter of each pixel point.

[0045] Then, use the three-dimensional point cloud data after decenter of each pixel point on the local plane to construct a covariance matrix : ; where the superscript T represents transpose.

[0046] Then, perform eigenvalue decomposition on the covariance matrix : .

[0047] Among them, is the eigenvalue matrix of the covariance matrix , , and are all eigenvalues of the covariance matrix , is the eigenvector matrix of the covariance matrix , , , are all eigenvectors of the covariance matrix . The eigenvector corresponding to the minimum eigenvalue is the normal vector . Perform normalization on the normal vector to obtain the normalized normal vector , where the normalization formula is: .

[0048] In another exemplary embodiment, in step 105 above, each plane of the crab breeding cage is segmented by analyzing the normal vector. With the three-dimensional point cloud taking the camera optical center as the observation point, when the camera imaging plane is parallel to the bottom surface of the crab breeding cage and the camera optical center is aligned with the geometric center of the bottom surface of the crab breeding cage, it can be considered that the normal vectors of the front, back, left, right, and bottom five side surfaces are respectively , and calculate the cosine similarity between the normal vector of each pixel point and the normal vectors of these five planes.

[0049] .

[0050] Among them, is the similarity between the normal vector of the pixel point and the normal vector of the th plane of the crab breeding cage, is the normal vector of the pixel point , is the normal vector of the th plane of the crab breeding cage.

[0051] In another exemplary embodiment, the above step 106 may be replaced by the following steps 301 - 304 .

[0052] Step 301, constructing plane equations of various sides of the crab breeding cage according to the physical dimensions of the crab breeding cage; the various sides of the crab breeding cage include: the front, the back, the left and the right of the crab breeding cage.

[0053] Step 302: Perform a reverse projection transformation on each pixel point to obtain the three-dimensional coordinates of each pixel point in the world coordinate system.

[0054] Step 303: convert the three-dimensional coordinates of the pixel point in the world coordinate system The inner wall depth at different pixel points on each side is calculated by combining the plane equation of the side to which the pixel point belongs; wherein, , and They are the x-axis, y-axis, and z-axis coordinates of the pixel point in the world coordinate system.

[0055] According to the camera imaging model, there is the following correspondence between points in the image coordinate system and points in the world coordinate system: .

[0056] Right now, .

[0057] in, is the pixel coordinate of the pixel point on the overhead image; is the z-axis coordinate of the pixel in the camera coordinate system, ; and are the rotation matrix and displacement vector of the camera extrinsic matrix, which represent the rotation and displacement parameters between the camera coordinate system and the world coordinate system respectively; to simplify the calculation, the present embodiment assumes that the world coordinate system coincides with the camera coordinate system (both are ), so we have: , .

[0058] is the camera intrinsic parameter matrix, which can be obtained through camera calibration and is defined as: .

[0059] remember ， , , , and Respectively The horizontal position index and vertical position index of the element in the , , , is the x-axis coordinate transformation coefficient, , , , is the y-axis coordinate transformation coefficient, , , , is the z-axis coordinate transformation coefficient, then each pixel point on the top view image Corresponding The coordinate transformation equation is:

[0060] .

[0061] in, is the three-dimensional coordinate of the pixel point in the world coordinate system. This point is located at the same pixel point on the top view image as the optical center of the camera. On the line of connection. Due to the inner length of the crab breeding cage ( )、Inner length( ) is known, the four inner walls, front, back, left and right, can be regarded as four known planes in the world coordinate system, and their equations are shown in the following formula.

[0062] .

[0063] .

[0064] .

[0065] .

[0066] By combining the four plane equations with the above coordinate transformation equations, we can get the intersection coordinates: ; At this time, the point in the world coordinate system obtained by transformation is just located on the inner wall plane of the crab breeding cage, so The value is the real depth of the corresponding point on the inner wall in the camera coordinate system. The four simultaneous equations are shown below.

[0067] .

[0068] .

[0069] .

[0070] .

[0071] in, , , and Respectively represent the front, back, left and right, and are the horizontal and vertical coordinates of the pixel points on the front surface respectively, and are the horizontal and vertical coordinates of the pixel points on the back surface respectively, and are the horizontal and vertical coordinates of the pixel points on the left surface respectively, and are the horizontal and vertical coordinates of the pixel points on the right surface respectively, is the pixel coordinate of the pixel point on the front surface at the depth estimation value, is the pixel coordinate of the pixel point on the back surface at the depth estimation value, is the pixel coordinate of the pixel point on the left surface at the depth estimation value, is the pixel coordinate of the pixel point on the right surface at the depth estimation value, is the inner width of the crab breeding cage, is the inner length of the crab breeding cage.

[0072] Based on the plane equation of the side surface of the crab breeding cage, the formula for calculating the true depth corresponding to each point on each side surface is shown as follows.

[0073] .

[0074] .

[0075] .

[0076] .

[0077] Wherein, is the inner wall depth at the pixel coordinate of the pixel point on the left surface , is the inner wall depth at the pixel coordinate of the pixel point on the right surface , is the inner wall depth at the pixel coordinate of the pixel point on the front surface , is the inner wall depth at the pixel coordinate of the pixel point on the back surface .

[0078] Step 304, determine the inner wall depth of the bottom surface of the crab breeding cage according to the distance between the bottom surface of the crab breeding cage and the camera imaging plane as: ; wherein, is the inner wall depth of the bottom surface of the crab breeding cage, It is the distance between the bottom surface of the crab breeding cage and the camera imaging plane.

[0079] Based on the results of the existing monocular depth estimation model, this application combines the camera imaging model (internal / external parameters) with the geometric structure information of the crab breeding cage through steps such as 3D point cloud generation, surface normal vector calculation, inner wall segmentation of the crab breeding cage, and inner wall depth calculation of the crab breeding cage. By adopting the method of constructing a spatial analytic geometry model, the effective recovery of the inner wall depth of the crab breeding cage is realized, which can significantly eliminate the scale drift and scale deviation existing in the existing monocular depth estimation results and improve the monocular depth estimation accuracy in the facility aquaculture scenario.

[0080] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope recorded in this specification.

[0081] Specific examples are used in this article to elaborate on the principle and implementation manner of this application. The description of the above embodiments is only used to help understand the method and its core idea of this application; at the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A method for estimating the depth of the inner wall of a facility-based crab breeding cage based on machine vision, characterized in that, Including: Obtain a top-down image of the crab breeding cage; Based on a monocular depth estimation model, obtain the depth estimation value of each pixel point in the top-down image; According to the depth estimation values of each pixel point, convert the pixel coordinates of each pixel point on the top-down image into the three-dimensional point cloud data of each pixel point; The three-dimensional point cloud data of the pixel point is the three-dimensional coordinates of the pixel point in the camera coordinate system; Calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point; Based on the normal vector of each pixel point, determine the plane of the crab breeding cage to which each pixel point belongs; the crab breeding cage includes 5 planes, and the 5 planes are respectively the front, back, left, right and bottom of the crab breeding cage; Respectively determine the inner wall depth at different pixel points of each plane according to the pixel coordinates of the pixel points on each plane and the physical size of the crab breeding cage, specifically including: According to the physical size of the crab breeding cage, construct the plane equations of each side of the crab breeding cage; each side of the crab breeding cage includes: the front, back, left and right sides of the crab breeding cage; Perform inverse projection transformation on each pixel point to obtain the three-dimensional coordinates of each pixel point in the world coordinate system; Simultaneously solve the three-dimensional coordinates of the pixel point in the world coordinate system and the plane equation of the side to which the pixel point belongs, and calculate the inner wall depth at different pixel points of each side; Determine the inner wall depth of the bottom surface of the crab cultivation cage according to the distance between the bottom surface of the crab cultivation cage and the camera imaging plane as follows: ; where is the distance between the bottom surface of the crab cultivation cage and the camera imaging plane, is the inner wall depth of the bottom surface of the crab cultivation cage; Calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point, specifically including: Obtain pixel points and multiple pixel points within the neighborhood range of For a pixel point and multiple pixel points within the neighborhood range of the pixel point perform plane fitting to obtain the local plane at the pixel point ; Solve for the normal vector of the local plane as the normal vector of the pixel point of the pixel point 2. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 1, wherein The formula for converting the pixel coordinates of each pixel point on the top-down image into the three-dimensional point cloud data of each pixel point according to the depth estimation values of each pixel point is: ； ； ； Among them, is the three-dimensional coordinate of the pixel point in the camera coordinate system, , and are the x-axis, y-axis and z-axis coordinates of the pixel point in the camera coordinate system respectively, is the depth estimation value of the pixel point, is the pixel coordinate of the pixel point on the top-down image, and are the horizontal coordinate and vertical coordinate of the pixel point on the top-down image respectively, is the camera principal point coordinate, and are the horizontal coordinate and vertical coordinate of the camera principal point respectively; and are the focal lengths of the camera in the x-axis direction and y-axis direction of the camera coordinate system respectively.

3. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 1, characterized in that Solve for the normal vector of the local plane as the normal vector of the pixel point , which specifically includes: Use the following formula to calculate the centroid of the local plane: ； Among them, is the centroid of the local plane, is the number of pixel points on the local plane, is the th three-dimensional point cloud data of the pixel points on the local plane; According to the centroid of the local plane, use the following formula to decentralize the three-dimensional point cloud data of each pixel point on the local plane to obtain the decentralized three-dimensional point cloud data of each pixel point on the local plane: ； Among them, is the three-dimensional point cloud data after decenterization of the th pixel point on the local plane; Construct a covariance matrix using the three-dimensional point cloud data of each pixel point on the local plane after decenterization : ; where the superscript T represents transpose; Perform eigenvalue decomposition on the covariance matrix to obtain the eigenvalues of the covariance matrix and the eigenvectors corresponding to each eigenvalue; Determine that the eigenvector corresponding to the minimum eigenvalue is the normal vector of the local plane, that is, the normal vector of the pixel point of the normal vector.

4. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 1, wherein Solving for the normal vector of the local plane as the normal vector of the pixel point and then further including: Normalize the normal vector of the pixel point .

5. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 1, wherein Based on the normal vector of each pixel point, determine the plane of the crab breeding cage to which each pixel point belongs, specifically including: Calculate the similarity between the normal vector of the pixel point and the normal vector of each plane of the crab farming cage respectively by using the following formula; ； Among them, is the similarity between the normal vector of the pixel point and the normal vector of the th plane of the crab breeding cage, is the normal vector of the pixel point , is the normal vector of the th plane of the crab breeding cage; Determine the plane with the greatest similarity as the plane to which the pixel point belongs. belongs.

6. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 5, characterized in that When the camera imaging plane is parallel to the bottom surface of the crab breeding cage and the optical center of the camera is aligned with the geometric center of the bottom surface of the crab breeding cage, the normal vectors of the front, rear, left, right, and bottom surfaces of the crab breeding cage are respectively .

7. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 1, wherein The plane equations of each side are respectively: ； ； ； ； Among them, , , and respectively represent the front, the back, the left side and the right side. , , , are the x-axis coordinate transformation coefficients. , , , are the y-axis coordinate transformation coefficients. and are respectively the horizontal coordinate and the vertical coordinate of the pixel point on the front surface. and are respectively the horizontal coordinate and the vertical coordinate of the pixel point on the back surface. and are respectively the horizontal coordinate and the vertical coordinate of the pixel point on the left surface. and are respectively the horizontal coordinate and the vertical coordinate of the pixel point on the right surface. is the pixel coordinate of the pixel point on the front surface at the depth estimation value. is the pixel coordinate of the pixel point on the back surface at the depth estimation value. is the pixel coordinate of the pixel point on the left surface at the depth estimation value. is the pixel coordinate of the pixel point on the right surface at the depth estimation value. is the inner width of the crab breeding cage. is the inner length of the crab breeding cage.

8. The method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision according to claim 7, wherein The formula for calculating the inner wall depth at different pixel points of each side is: ； ； ； ； Among them, is the pixel coordinate of the pixel point on the left surface at the inner wall depth, is the pixel coordinate of the pixel point on the right surface at the inner wall depth, is the pixel coordinate of the pixel point on the front surface at the inner wall depth, is the pixel coordinate of the pixel point on the back surface at the inner wall depth.

Images