Machine vision-based facility crab breeding cage inner wall depth estimation method

Abstract

The invention discloses a facility crab breeding cage inner wall depth estimation method based on machine vision, and relates to the technical field of facility breeding. According to the method, the monocular depth estimation model is combined with the geometric structure constraint of the crab breeding cage, the depth of the object in the image is preliminarily predicted through the monocular depth estimation model, and on this basis, the depth of the object in the image is calculated by constructing the analytic geometry calculation model based on the prior information such as the structure and the physical size of the crab breeding cage. The depth value of each point of the inner wall of the crab breeding cage under the camera coordinate system is recovered, the problems of scale drift and deviation existing in an existing monocular depth estimation method can be remarkably relieved, and the precision of a depth estimation result is improved. The method does not depend on a deep learning technology, and a large-scale training data set does not need to be constructed and labeled.

Description

Technical Field

[0001] The present application relates to the technical field of facility aquaculture, and in particular to a method for estimating the inner wall depth of a facility crab aquaculture cage based on machine vision. Background Art

[0002] Facility farming is an important direction for the development of smart agriculture in the future. In order to accurately monitor the growth status of farming objects, it is necessary to restore the real distance of each object in the scene in the camera coordinate system based on image information. The output results of the existing monocular vision-based depth estimation model usually have different degrees of scale drift and deviation in different areas of the image, resulting in a large difference between the predicted depth and the real depth, which is difficult to meet the needs of growth status monitoring. In addition, image depth estimation based on deep learning often requires the collection of a large amount of depth image data to train or fine-tune the model, which requires a large data collection and training cost. Summary of the invention

[0003] The purpose of this application is to provide a method for estimating the inner wall depth of a facility crab breeding cage based on machine vision, so as to improve the accuracy of estimating the inner wall depth of a facility crab breeding cage.

[0004] To achieve the above objectives, this application provides the following solutions.

[0005] In a first aspect, the present application provides a method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision, comprising the following steps.

[0006] Get a bird's-eye view of the crab cages.

[0007] The depth estimation value of each pixel in the top-view image is obtained based on the monocular depth estimation model.

[0008] The pixel coordinates of each pixel point on the overhead image are converted into three-dimensional point cloud data of each pixel point according to the depth estimation value of each pixel point; the three-dimensional point cloud data of the pixel point is the three-dimensional coordinate of the pixel point in the camera coordinate system.

[0009] The normal vector of each pixel is calculated based on the 3D point cloud data of each pixel.

[0010] The plane of the crab breeding cage to which each pixel belongs is determined based on the normal vector of each pixel; the crab breeding cage includes 5 planes, which are the front, back, left, right and bottom of the crab breeding cage.

[0011] The inner wall depths at different pixel points on each plane are determined according to the pixel coordinates of the pixel points on each plane and the physical size of the crab breeding cage.

[0012] According to the specific embodiments provided in this application, this application has the following technical effects.

[0013] This application provides a method for estimating the inner wall depth of facility crab breeding cages based on machine vision. This application combines a monocular depth estimation model with the geometric structure constraints of crab breeding cages, and uses a monocular depth estimation model to make a preliminary prediction of the depth of objects in the image. On this basis, relying on prior information such as the structure and physical size of the crab breeding cage, by constructing an analytical geometry calculation model, the depth value of each point on the inner wall of the crab breeding cage in the camera coordinate system is restored, which can significantly alleviate the scale drift and deviation problems existing in the existing monocular depth estimation method and improve the accuracy of the depth estimation result. This application does not rely on deep learning technology and does not require the construction and annotation of large-scale training data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0015] Figure 1 A flow chart of a method for estimating the inner wall depth of a facility-based crab breeding cage based on machine vision provided in one embodiment of the present application.

[0016] Figure 2 A schematic diagram of a method for estimating the inner wall depth of a crab breeding cage in a facility based on machine vision provided in one embodiment of the present application.

[0017] Figure 3 A structural diagram of the camera layout provided in accordance with an embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0020] Existing monocular depth estimation models are generally built based on deep learning methods, which focus more on utilizing image pixel values ​​and local neighborhood information. However, facility-based three-dimensional aquaculture usually uses crab farming cages, which have regular geometric structures. This type of information has not been effectively utilized in existing monocular depth estimation models, so it needs further improvement.

[0021] This application combines a monocular depth estimation model with the geometric structure constraints of crab breeding cages. The monocular depth estimation model is used to make a preliminary prediction of the depth of objects in the image. On this basis, relying on the camera imaging model and prior information such as the size of the crab breeding cage, the depth value of each point on the inner wall of the crab breeding cage in the camera coordinate system is restored by constructing an analytical geometry calculation model. This can alleviate the scale drift and deviation problems existing in the existing monocular depth estimation method and improve the accuracy of the depth estimation result. This application does not rely on deep learning technology and does not require the construction and annotation of large-scale training data sets.

[0022] In an exemplary embodiment, a method for estimating the inner wall depth of crab breeding cages in facilities based on machine vision is provided. The method can use the relative depth output by the depth estimation model, combined with the camera intrinsic parameters and the physical size of the crab breeding cage, to estimate the depth values ​​of each part of the inner wall of the crab breeding cage in the image in the camera coordinate system. Specifically, the method includes: image depth estimation and point cloud generation, surface normal vector calculation, crab breeding cage inner wall segmentation and crab breeding cage inner wall depth calculation. The method can be deployed in an actual environment to provide an intelligent analysis tool for facility crab breeding.

[0023] like Figure 1 and Figure 2 As shown, the method for estimating the inner wall depth of a facility crab breeding cage based on machine vision of the present application includes the following steps 101 to 106.

[0024] Step 101, obtaining a bird's-eye view image of the crab breeding cage.

[0025] Step 102: Obtain a depth estimation value for each pixel in the overhead image based on a monocular depth estimation model.

[0026] Step 103 , converting the pixel coordinates of each pixel point on the overhead image into three-dimensional point cloud data of each pixel point according to the depth estimation value of each pixel point; the three-dimensional point cloud data of the pixel point is the three-dimensional coordinate of the pixel point in the camera coordinate system.

[0027] Step 104: Calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point.

[0028] Step 105, determining the plane of the crab breeding cage to which each pixel belongs based on the normal vector of each pixel; the crab breeding cage includes 5 planes, which are the front, back, left, right and bottom of the crab breeding cage.

[0029] Step 106, determining the inner wall depth at different pixel points on each plane according to the pixel coordinates of the pixel points on each plane and the physical size of the crab breeding cage.

[0030] Implementing the above steps 101 to 106 can alleviate the scale drift and deviation problems existing in the existing monocular depth estimation methods and improve the accuracy of the depth estimation results.

[0031] In another exemplary embodiment, in the above step 101, the crab breeding cage adopts a regular rectangular crab breeding cage, such as Figure 3 As shown, its internal physical dimensions (length, width, height) are known. The camera is placed above the center of the crab breeding cage and looks down to shoot. The camera imaging plane A'B'C'D' is parallel to the bottom surface ABCD of the crab breeding cage. The camera optical center O' is aligned with the geometric center O of the bottom surface of the crab breeding cage; the camera internal parameter matrix is a known quantity and can be obtained through offline calibration.

[0032] In another exemplary embodiment, the above step 102 uses a monocular depth estimation model (such as Depth Pro, etc.) to perform depth estimation on the top view image taken by the camera to obtain a depth estimation value of each pixel in the top view image. , that is, the z-axis coordinate of the pixel point in the camera coordinate system .

[0033] In another exemplary embodiment, the above step 103 is based on the depth estimation value of the pixel point. and the camera intrinsic matrix , the pixel coordinates of each pixel in the overhead image are converted into the three-dimensional point cloud data of each pixel, forming a three-dimensional description of the camera's field of view. The conversion formula is as follows.

[0034] , , .

[0035] in, It is the three-dimensional coordinate of the pixel point in the camera coordinate system, that is, the three-dimensional point cloud data. , and The x-axis, y-axis and z-axis coordinates of the pixel point in the camera coordinate system respectively; is the depth estimate of the pixel; is the pixel coordinate of the pixel point on the top view image, and are the horizontal and vertical coordinates of the pixel points on the overhead image respectively; is the camera principal point coordinate, and are the horizontal and vertical coordinates of the camera principal point respectively; and are the focal lengths of the camera in the x-axis and y-axis directions of the camera coordinate system, respectively.

[0036] In the present application, the pixel coordinates of a pixel point are the coordinates of the pixel point in the image coordinate system, and the horizontal coordinates and vertical coordinates of the pixel point are the x-axis coordinates and y-axis coordinates of the pixel point in the image coordinate system.

[0037] In another exemplary embodiment, based on the three-dimensional point cloud data, the normal vector corresponding to each point on the inner wall of the crab breeding cage (i.e., each pixel point in the top view) is calculated. Assume that the neighborhood search radius is r, the maximum number of neighbors is max_nn, and the neighborhood search radius r is limited to the maximum number of neighbors being max_nn and the pixel point being searched. All nearest neighbor points centered on each pixel and its neighborhood points can fit a local plane, and the normal vector of the local plane is the pixel point In the embodiment of the present application, the above step 104 can be replaced by the following steps 201 to 203.

[0038] Step 201: Get pixel points Multiple pixels within the neighborhood range.

[0039] Step 202: Pixels and pixels Plane fitting is performed on multiple pixel points within the neighborhood range to obtain pixel points The local plane at .

[0040] Step 203: Calculate the normal vector of the local plane as a pixel point The normal vector of .

[0041] During the solution process, it is necessary to first determine the centroid of the local plane: .

[0042] in, is the centroid of the local plane, is the number of pixels on the local plane, is the first 3D point cloud data of pixels.

[0043] Then the local plane is decentralized to obtain the decentralized 3D point cloud data of each pixel on the local plane: .

[0044] in, The local plane The 3D point cloud data after the pixels are decentralized.

[0045] Then, the covariance matrix is ​​constructed using the decentralized 3D point cloud data of each pixel on the local plane. : ; Wherein, the superscript T represents transpose.

[0046] Then the covariance matrix Perform eigenvalue decomposition: .

[0047] in, is the covariance matrix The eigenvalue matrix of , and The covariance matrix The characteristic value of is the covariance matrix The eigenvector matrix of , , The covariance matrix The eigenvector of . The smallest eigenvalue The corresponding eigenvector is the normal vector . Normal vector Perform normalization to obtain the normalized normal vector , where the normalized formula is: .

[0048] In another exemplary embodiment, the above step 105 segments each plane of the crab breeding cage by analyzing the normal vector. The three-dimensional point cloud uses the optical center of the camera as the observation point. When the camera imaging plane is parallel to the bottom of the crab breeding cage and the optical center of the camera is aligned with the geometric center of the bottom of the crab breeding cage, it can be considered that the normal vectors of the five sides, front, back, left, right and bottom, are , calculate the cosine similarity between the normal vector of each pixel and the normal vectors of these five planes.

[0049] .

[0050] in, Pixel The normal vector of the crab cage is The similarity of the normal vectors of the planes, Pixel The normal vector of For crab breeding cage The normal vector of a plane.

[0051] In another exemplary embodiment, the above step 106 may be replaced by the following steps 301 - 304 .

[0052] Step 301, constructing plane equations of various sides of the crab breeding cage according to the physical dimensions of the crab breeding cage; the various sides of the crab breeding cage include: the front, the back, the left and the right of the crab breeding cage.

[0053] Step 302: Perform a reverse projection transformation on each pixel point to obtain the three-dimensional coordinates of each pixel point in the world coordinate system.

[0054] Step 303: convert the three-dimensional coordinates of the pixel point in the world coordinate system The inner wall depth at different pixel points on each side is calculated by combining the plane equation of the side to which the pixel point belongs; wherein, , and They are the x-axis, y-axis, and z-axis coordinates of the pixel point in the world coordinate system.

[0055] According to the camera imaging model, there is the following correspondence between points in the image coordinate system and points in the world coordinate system: .

[0056] Right now, .

[0057] in, is the pixel coordinate of the pixel point on the overhead image; is the z-axis coordinate of the pixel in the camera coordinate system, ; and are the rotation matrix and displacement vector of the camera extrinsic matrix, which represent the rotation and displacement parameters between the camera coordinate system and the world coordinate system respectively; to simplify the calculation, the present embodiment assumes that the world coordinate system coincides with the camera coordinate system (both are ), so we have: , .

[0058] is the camera intrinsic parameter matrix, which can be obtained through camera calibration and is defined as: .

[0059] remember ， , , , and Respectively The horizontal position index and vertical position index of the element in the , , , is the x-axis coordinate transformation coefficient, , , , is the y-axis coordinate transformation coefficient, , , , is the z-axis coordinate transformation coefficient, then each pixel point on the top view image Corresponding The coordinate transformation equation is: .

[0060] in, is the three-dimensional coordinate of the pixel point in the world coordinate system. This point is located at the same pixel point on the top view image as the optical center of the camera. On the line of connection. Due to the inner length of the crab breeding cage ( )、Inner length( ) is known, the four inner walls, front, back, left and right, can be regarded as four known planes in the world coordinate system, and their equations are shown in the following formula.

[0061] .

[0062] .

[0063] .

[0064] .

[0065] By combining the four plane equations with the above coordinate transformation equations, we can get the intersection coordinates: ; At this time, the point in the world coordinate system obtained by transformation is just located on the inner wall plane of the crab breeding cage, so The value is the real depth of the corresponding point on the inner wall in the camera coordinate system. The four simultaneous equations are shown below.

[0066] .

[0067] .

[0068] .

[0069] .

[0070] in, , , and Respectively represent the front, back, left and right, and are the horizontal and vertical coordinates of the pixel point in front, respectively. and They are the horizontal and vertical coordinates of the pixel points on the back, and are the horizontal and vertical coordinates of the pixel on the left, and are the horizontal and vertical coordinates of the pixel on the right, The pixel coordinates of the pixel point on the front The depth estimate at The pixel coordinates of the pixel point on the back The depth estimate at The pixel coordinates of the pixel on the left The depth estimate at is the pixel coordinate of the pixel on the right The depth estimate at is the inner width of the crab breeding cage, It is the inner length of the crab breeding cage.

[0071] Based on the plane equation of the side of the crab breeding cage, the formula for calculating the true depth corresponding to each point on each side is shown as follows.

[0072] .

[0073] .

[0074] .

[0075] .

[0076] in, The pixel coordinates of the pixel on the left The inner wall depth at is the pixel coordinate of the pixel on the right The inner wall depth at The pixel coordinates of the pixel point on the front The inner wall depth at The pixel coordinates of the pixel point on the back The inner wall depth at .

[0077] Step 304, according to the distance between the bottom surface of the crab breeding cage and the camera imaging plane, the inner wall depth of the bottom surface of the crab breeding cage is determined as: ;in, is the inner wall depth of the bottom of the crab breeding cage, is the distance between the bottom of the crab breeding cage and the camera imaging plane.

[0078] Based on the results of the existing monocular depth estimation model, this application combines the camera imaging model (internal / external parameters) and the geometric structure information of the crab breeding cage through the steps of three-dimensional point cloud generation, surface normal vector calculation, crab breeding cage inner wall segmentation, crab breeding cage inner wall depth calculation, etc., and realizes the effective recovery of the inner wall depth of the crab breeding cage by constructing a spatial analytical geometric model. It can significantly eliminate the scale drift and scale deviation existing in the existing monocular depth estimation results, and improve the monocular depth estimation accuracy in facility breeding scenarios.

[0079] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0080] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A method for estimating the inner wall depth of a facility crab breeding cage based on machine vision, characterized in that: include: Obtain a bird's-eye view of the crab cages; Obtain the depth estimation value of each pixel in the top view image based on the monocular depth estimation model; The pixel coordinates of each pixel point on the overhead image are converted into three-dimensional point cloud data of each pixel point according to the depth estimation value of each pixel point; The three-dimensional point cloud data of a pixel point is the three-dimensional coordinate of the pixel point in the camera coordinate system; Calculate the normal vector of each pixel point according to the three-dimensional point cloud data of each pixel point; The plane of the crab breeding cage to which each pixel belongs is determined based on the normal vector of each pixel; the crab breeding cage includes 5 planes, which are the front, back, left, right and bottom of the crab breeding cage; The inner wall depths at different pixel points on each plane are determined according to the pixel coordinates of the pixel points on each plane and the physical size of the crab breeding cage.

2. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 1, characterized in that: The formula for converting the pixel coordinates of each pixel point on the overhead image into the three-dimensional point cloud data of each pixel point according to the depth estimation value of each pixel point is: ； ； ； in, is the three-dimensional coordinate of the pixel point in the camera coordinate system, , and The x-axis, y-axis, and z-axis coordinates of the pixel points in the camera coordinate system, is the depth estimate of the pixel, is the pixel coordinate of the pixel point on the top view image, and are the horizontal and vertical coordinates of the pixel points on the overhead image, respectively. are the camera principal point coordinates, and are the horizontal and vertical coordinates of the camera principal point respectively; and are the focal lengths of the camera in the x-axis and y-axis directions of the camera coordinate system, respectively.

3. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 1, characterized in that: Calculate the normal vector of each pixel point according to the 3D point cloud data of each pixel point, specifically including: Get pixel points Multiple pixel points within the neighborhood range; Pixel and pixels Perform plane fitting on multiple pixel points in the neighborhood of to obtain pixel points The local plane at Solve the normal vector of the local plane as the pixel point The normal vector of .

4. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 3 is characterized in that: Solve the normal vector of the local plane as the pixel point The normal vector of , specifically including: The centroid of the local plane is calculated using the following formula: ； in, is the centroid of the local plane, is the number of pixels on the local plane, is the first 3D point cloud data of pixels; According to the centroid of the local plane, the three-dimensional point cloud data of each pixel point on the local plane is decentralized using the following formula to obtain the decentralized three-dimensional point cloud data of each pixel point on the local plane: ； in, The local plane The 3D point cloud data after the pixels are decentralized; Construct the covariance matrix using the decentralized 3D point cloud data of each pixel on the local plane : ; Wherein, the superscript T represents transposition; Performing eigenvalue decomposition on the covariance matrix to obtain eigenvalues ​​of the covariance matrix and eigenvectors corresponding to each eigenvalue; Determine the eigenvector corresponding to the minimum eigenvalue as the normal vector of the local plane, that is, the pixel point The normal vector of .

5. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 3 is characterized in that: Solve the normal vector of the local plane as the pixel point The normal vector of , followed by: Pixel Normalize the normal vector.

6. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 1, characterized in that: The plane of the crab breeding cage to which each pixel belongs is determined based on the normal vector of each pixel, specifically including: Use the following formula to calculate the pixel points respectively The similarity between the normal vector of and the normal vector of each plane of the crab breeding cage; ； in, Pixel The normal vector of the crab cage is The similarity of the normal vectors of the planes, Pixel The normal vector of For crab breeding cage The normal vector of the plane; Determine the plane with the greatest similarity as the pixel point The plane to which it belongs.

7. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 6 is characterized in that: When the camera imaging plane is parallel to the bottom of the crab breeding cage, and the camera optical center is aligned with the geometric center of the bottom of the crab breeding cage, the normal vectors of the front, back, left, right, and bottom of the crab breeding cage are .

8. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 1, characterized in that: According to the pixel coordinates of the pixel points on each plane and the physical size of the crab breeding cage, the inner wall depth at different pixel points on each plane is determined, specifically including: According to the physical dimensions of the crab breeding cage, the plane equations of the various sides of the crab breeding cage are constructed; the various sides of the crab breeding cage include: the front, back, left and right sides of the crab breeding cage; Perform inverse projection transformation on each pixel point to obtain the three-dimensional coordinates of each pixel point in the world coordinate system; The three-dimensional coordinates of the pixel point in the world coordinate system are combined with the plane equation of the side surface to which the pixel point belongs, and the inner wall depth at different pixel points of each side surface is calculated; According to the distance between the bottom surface of the crab breeding cage and the camera imaging plane, the inner wall depth of the bottom surface of the crab breeding cage is determined as: ;in, is the distance between the bottom of the crab cage and the camera imaging plane, It is the inner wall depth of the bottom of the crab breeding cage.

9. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 8, characterized in that: The plane equations for each side are: ； ； ； ； in, , , and Respectively represent the front, back, left and right, , , , is the x-axis coordinate transformation coefficient, , , , is the y-axis coordinate transformation coefficient, and are the horizontal and vertical coordinates of the pixel point in front, respectively. and They are the horizontal and vertical coordinates of the pixel points on the back, and are the horizontal and vertical coordinates of the pixel on the left, and are the horizontal and vertical coordinates of the pixel on the right, The pixel coordinates of the pixel point on the front The depth estimate at The pixel coordinates of the pixel point on the back The depth estimate at The pixel coordinates of the pixel on the left The depth estimate at is the pixel coordinate of the pixel on the right The depth estimate at is the inner width of the crab breeding cage, It is the inner length of the crab breeding cage.

10. The method for estimating the inner wall depth of a facility crab breeding cage based on machine vision according to claim 9, characterized in that: The formula for calculating the inner wall depth at different pixel points on each side is: ； ； ； ； in, The pixel coordinates of the pixel on the left The inner wall depth at is the pixel coordinate of the pixel on the right The inner wall depth at The pixel coordinates of the pixel point on the front The inner wall depth at The pixel coordinates of the pixel point on the back The inner wall depth at .

Images