Cotton phenotype detection robot

Abstract

The invention provides a cotton phenotype detection robot, and relates to the field of intelligent agriculture, the cotton phenotype detection robot comprises a box body (2), a lifting mechanism (3), a shading mechanism (4), a steering mechanism (5), a driving mechanism (6) and a control system, the control system comprises a system control box (71), an acquisition module (72), an interaction screen (73) and a radar and power supply system (1), the system control box (71) is arranged in an inner cavity of the box body (2), and a lifting holder is arranged in the box body (2); the acquisition module (72) is arranged on the lifting holder, the radar is arranged at the front end of the outer side of the box body (2), an interaction screen (73) is arranged on the outer wall of the box body (2), and the power supply system (1) is arranged on the top surface of the box body (2). The robot is high in data acquisition efficiency, the analytic model is high in robustness, efficient, comprehensive, rapid and accurate cotton three-dimensional reconstruction can be achieved, external interference is avoided, and data support is provided for cotton planting.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent agriculture, and in particular to a cotton phenotype detection robot. Background Art

[0002] In the process of planting cotton, cotton phenotypic detection is an important means to comprehensively evaluate the external characteristics and some physiological characteristics of cotton. It has key value for many aspects such as cotton variety selection, quality assessment and planting management. It can screen out high-quality varieties that are suitable for machine picking, early maturity, high yield, reliable and stress-resistant from massive cotton samples, so as to meet the requirements of modern cotton breeding. At present, the phenotypic detection of cotton is mainly carried out manually. However, the manual detection method is not only time-consuming and inefficient, but also easily affected by human subjective factors, with low detection accuracy and large errors. At the same time, manual cotton phenotypic detection has very strict requirements on light and is greatly affected by external environmental factors, resulting in poor collection stability and large detection errors. That is, the existing technology cannot achieve efficient, comprehensive, accurate and highly stable cotton phenotypic detection, resulting in the inability to provide strong support for cotton variety improvement, high-quality and high-yield cultivation and quality control. Summary of the invention

[0003] In view of the problems existing in the above-mentioned prior art, the purpose of the present invention is to provide a cotton phenotype detection robot, which has high data acquisition efficiency and strong analytical model robustness. It can realize efficient, comprehensive, fast and accurate reconstruction of cotton three-dimensional model, avoid interference from human subjective factors and external environmental factors, and provide data support for cotton variety selection, quality assessment and planting management.

[0004] The purpose of the present invention is achieved through the following technical solutions: A cotton phenotype detection robot comprises a box, a lifting mechanism, a shading mechanism, a steering mechanism, a driving mechanism and a control system. The box is a rectangular parallelepiped structure as a whole, and the lifting mechanisms are respectively arranged at the four corners of the bottom thereof; the shading mechanisms are respectively arranged at the bottom of the box and between two groups of lifting mechanisms (that is, the shading mechanisms are respectively arranged at the bottom of the box and corresponding to the four sides thereof); the steering mechanisms are respectively arranged at the ends of the lifting mechanisms away from the box; the driving mechanism is connected to the lower ends of the steering mechanisms through steering brackets; the control system comprises a system control box, a collection module, an interactive screen, a radar and a power supply system; the system control box is fixedly arranged in the inner cavity of the box and a lifting platform is arranged in the box; the collection module is arranged on the lifting platform; the radar is arranged at the front end of the outer side of the box; the interactive screen is arranged at the top of the outer wall of one side of the box and the power supply system is arranged on the top surface of the box.

[0005] Based on the further optimization of the above scheme, the lifting mechanism includes a fixed column, a lifting column, a driving motor, a lifting worm, a transmission wheel, a lifting screw and a limiter. The top of the fixed column is fixedly connected to the four corners of the bottom of the box body respectively, and the lifting column is slidably arranged in the inner cavity of the fixed column. The front and rear sides of the lifting column are hollow and the bottom end thereof passes through the bottom surface of the fixed column; a driving motor is fixedly arranged on one side of the top of the fixed column and the output end of the driving motor is connected to the lifting worm arranged in the inner cavity of the fixed column, a transmission wheel is coaxially arranged in the inner cavity of the top of the fixed column and the bottom of the transmission wheel is fixedly connected to the lifting screw, the lifting screw is rotatably connected to the end face of the positioning block arranged on the bottom surface of the inner cavity of the fixed column, the transmission wheel is meshed with the lifting worm for transmission, the lifting screw passes through the end face of the lifting column and is threadedly connected; the fixed column is located at the lower side of the transmission wheel and the inner cavity of the bottom surface of the fixed column respectively has limiters.

[0006] Based on the further optimization of the above scheme, the shading mechanism includes a telescopic box, a shading cloth, a positioning rod and a folding rod. The telescopic boxes are respectively fixedly arranged at the bottom of the four sides of the box body and a rotating rod is rotatably arranged in the inner cavity of the telescopic box. The upper end of the shading cloth is fixedly arranged on the outer wall of the rotating rod in the telescopic box through a groove and the lower end of the shading cloth passes through the telescopic box and is fixedly connected to the outer wall of the positioning rod. The two ends of the positioning rod are respectively slidably connected to the corresponding outer walls of the fixed column; two groups of folding rods are symmetrically arranged between the telescopic box and the positioning rod and on the back side of the shading cloth, and the two ends of the folding rod are respectively rotatably connected to the bottom surface of the telescopic box and the end surface of the positioning rod.

[0007] Based on the further optimization of the above scheme, the steering mechanism includes a steering box, a steering motor and a steering transmission. The steering box is fixedly arranged at the bottom of the lifting column and a steering motor is arranged on one side of the steering box. The steering transmission is arranged in the inner cavity of the steering box and the steering transmission is connected to the output end of the steering motor; the bottom end of the steering transmission passes through the bottom surface of the steering box and is connected to the steering bracket arranged on the lower side of the steering box; the driving mechanism includes a protective cover, an axle, a brake device, a hub motor and a running wheel. The protective cover is fixedly arranged at the bottom end of the steering bracket and an axle is arranged at the bottom thereof, a running wheel is arranged on the outer wall of the middle part of the axle and a hub motor is installed on the running wheel, the hub motor is connected to the steering bracket by bolts through the axle, and the brake device is arranged on the hub motor.

[0008] Based on further optimization of the above scheme, the lifting pan-tilt includes a fixed screw slide, a movable screw slide, a movable slider, a lifting slide rail and a camera pan-tilt, and the fixed screw slide, the movable screw slide, the movable slider and the lifting slide rail are all two groups; wherein, the two groups of fixed screw slides are arranged in parallel at the front and rear ends of the inner cavity of the box body, and two groups of movable screw slides are slidably arranged between the two groups of fixed screw slides, the movable sliders are respectively slidably arranged on the two groups of movable screw slides, and the lifting slide rails are respectively fixedly arranged on the bottom surfaces of the movable sliders, and multiple groups of camera pan-tilts are respectively slidably arranged along the vertical direction on the two lifting slide rails for installing acquisition modules, and the acquisition modules include high-definition industrial cameras, depth cameras, spectrometers, etc.

[0009] A method for automatically detecting cotton phenotypes, using the above-mentioned detection robot, comprising: Step S1, obtaining a navigation path, and using a lifting mechanism to drive the box to lift according to the height of the cotton plant, and using a steering mechanism and a driving mechanism to control the detection robot to move forward; during the forward movement, the radar scans to establish the forward movement of the detection robot, and realizes real-time correction of the navigation path; Step S2: During the movement of the detection robot, the shading mechanism is adjusted according to the lighting conditions, and the position of the collection module is adjusted by lifting the pan / tilt to obtain the best shooting angle, and the collection module collects data on the cotton in real time; Step S3, using the information collected by the acquisition module, and using feature point extraction and matching, sparse point cloud reconstruction, sparse point cloud densification, and spectral data collection and processing, the cotton is three-dimensionally reconstructed, and a four-dimensional information data set of cotton point cloud, image, and spectrum based on time series is constructed to complete the establishment of the analysis data set.

[0010] Based on the further optimization of the above solution, the step S1 is specifically as follows: First, before the detection robot enters the cotton field, the boundary of the cotton field to be detected is measured by the high-precision Beidou satellite positioning system, the initial map area of ​​the cotton field is constructed, and the coordinates of the four vertices of the initial map area are marked ( x 1 ,y 1) ( x 2 ,y 2) ( x 3 ,y 3) (x 4 ,y 4); Then, based on the shape and size of the initial map area, determine the starting point for data collection S With the end point E (Generally, the starting point S Select the midpoint and end point of one end of the cotton field E Select the midpoint of the other end boundary opposite to it); divide the initial map area into uniform grids and the size of each grid is: ; Then the number of grids in the initial map area is :

[0011] Where: L x Indicates that the initial map area is x Length in direction; L y Indicates that the initial map area is y Length in direction; represents the floor function; Afterwards, an improved method based on the A* algorithm is used to plan the collection path line by line, where the evaluation function of the A* algorithm is:

[0012] Where: g(n) Indicates starting point S To Node n The actual cost, h(n) Represents a slave node n The estimated cost to reach the end point E; In the cotton field, row by row picking scene, g(n) , h(n) Defined as: a. Assume that the current node n The coordinates of are ( x n ,y n ), the length of the path traveled is d ,but g(n)=d ; b. If you are currently in i Target behavior j OK( i <j ),but:

[0013] Where: y n-r Indicates the collection point corresponding to the next row y coordinate; Finally, the robot moves forward according to the planned collection path and uses radar to scan the road conditions ahead in real time. If there are obstacles, the robot is controlled by the steering mechanism to bypass the obstacles and then continues to move forward according to the principle of line-by-line collection.

[0014] Based on the further optimization of the above solution, the feature point extraction and matching in step S3 are specifically as follows: First, the original image I(x,y) Perform Gaussian convolution of different scales to obtain Gaussian pyramids:

[0015] Where: represents the scale parameter; Subtract the height images of adjacent scales and construct the Gaussian difference (DOG) scale space:

[0016] Where: k represents the scale interval factor; In the Gaussian difference scale space, if a point is the extreme point of all points compared with its 8 adjacent points of the same size and the points of its upper and lower adjacent sizes (i.e., 9x2, a total of 18 points), then the point is a potential feature point; Then, the potential feature points initially obtained are accurately located: Assume that the function of the Gaussian difference scale space is At the key point The quadratic Taylor expansion nearby is:

[0017] Where: Indicates the offset of key point position and scale; Solution D(X) The partial derivative of is zero X values ​​to precisely locate key points:

[0018] After that, the gradient amplitude of the pixels in its neighborhood is calculated with the key point as the center With direction :

[0019]

[0020] Where: L(x,y) Represents the image of the corresponding scale in the Gaussian pyramid; Construct a histogram and count the gradient direction distribution of pixels in the neighborhood; the peak direction of the histogram is used as the main direction of the key point, and other directions that are 80% greater than the peak of the main direction are used as auxiliary directions of the key point; Then, taking the key point as the center, a local coordinate system is constructed in its neighborhood according to the selected main direction; this neighborhood is divided into multiple sub-regions, and the gradient direction histogram is statistically calculated in each sub-region. Each histogram has a direction interval, so that a one-dimensional feature vector, i.e., a feature descriptor, is obtained; Finally, for each feature point in one image, the nearest neighbor and next nearest neighbor feature point in the feature point set of another image are searched through the K-dimensional space binary tree model, and the Euclidean distance between the nearest neighbor feature point, the next nearest neighbor feature point and the feature point in the current image is calculated respectively. d 1 and d 2; Set the distance ratio threshold T m ,like , the two feature points are considered as matching points, the matching points are associated and the corresponding relationship of the associated views is established, otherwise, the two feature points do not match.

[0021] Based on the further optimization of the above scheme, in order to remove low-contrast key points and uncertain edge response points, the contrast threshold is preset before obtaining the key points using the Gaussian difference scale space function T c Ratio threshold to principal curvature T r , if the contrast of the obtained key point is lower than the contrast threshold T c Or the principal curvature ratio is greater than the principal curvature ratio threshold T r , the corresponding key points are discarded.

[0022] Based on the further optimization of the above scheme, the sparse point cloud reconstruction in step S3 is specifically as follows: based on the extracted key matching points and their corresponding relationships, the camera space pose information corresponding to each view image is restored, and the sparse point cloud reconstruction is realized by using the inverse projection method, wherein the projection error formula is:

[0023] Where: M represents the camera parameters, P represents the key matching point set; N Indicates the total number of images acquired, Z Indicates the total number of key matching points calculated; w i,j Indicates matching point j Is it in the image i If so, then w i,j =1, otherwise, w i,j =0; q ij and P ij Respectively represent the real image point and its corresponding 3D point; Represents the back-projected image point obtained by projecting the 3D point back to the image. The projection error quantifies the error between the spatial pose information and the estimated 3D point coordinates through the error between the real and back-projected image points.

[0024] Based on the further optimization of the above solution, the densification of the sparse point cloud in step S3 is specifically as follows: First, cluster the image set to reduce the amount of data in the reconstruction process, improve the calculation speed and reconstruction accuracy: Normalize the input images so that all images have the same size, that is, H x W ; If the original image I i The size is H i xW i , then each pixel in the image (x,y) After scaling, the image Pixels in for:

[0025]

[0026] Use convolutional neural network to extract features from normalized images. After convolution layer and pooling layer, image features are obtained. F i , whose dimensions are H f x W f x C ,in, H f and W f Represents the height and width of the feature map, C Indicates the number of feature channels; For different image features F i and F j Between the feature maps F i Each feature point in p i , in the feature map F j Find similar feature points in p j , and use Euclidean distance as the similarity measure, that is:

[0027] Where: V pi , V pj Respectively p i , p j The eigenvector of Randomly select B images from the image set as the initial cluster centers, that is, C 1 ,C 2 ,…,C B , for each image I m , the above method is used to calculate the similarity score S( I m ，C I), I=1,2,…,B ; Calculate at the same time I m The number of matching points or the average similarity score between all initial cluster centers; I m Assigned to the cluster with the highest similarity score C k* middle:

[0028] Calculate the average feature of the images in each cluster again and use it as the new cluster center. In Location (x,y) The eigenvalues ​​at are:

[0029] Where: Represents clustering C B The number of images in Repeat the above process until the clustering result no longer changes or the preset maximum number of iterations is reached, and the clustering of the image set is completed; Then, the dense point cloud is reconstructed under the constraints of local photometric consistency and global visibility: Based on the sparse point cloud, representative seed points are selected in the image. The seed set is: ,in, s i =( x i ,y i ), represents the coordinates of the seed point in the image; For each seed point s i As the center, construct a w x w Patch, match the patch between different images, and measure the similarity of the two patches:

[0030]

[0031] Where: PP i , PP j Represent the patches in image A and image B respectively; , Respectively PP i , PP j The mean of Preset Threshold T NCC ,like ,but PP i and PP j Match between them, otherwise do not match; For each matched seed point pair, the corresponding three-dimensional space point Q is obtained by using the camera's pose information and triangulation principle:

[0032] Where: , denote the scale factors respectively; P A , P B Represents the projection matrix of the two cameras; p A , p B Respectively represent the matching image points:

[0033] For each calculated 3D point Q Centered at radius r Find other possible matching points in the neighborhood of ; in the image, according to the camera projection model, the three-dimensional point Q Projection points are obtained by projecting onto different images, and then patches are constructed around these projection points and matched. If a new matching point pair is found, its corresponding 3D point is added to the point cloud collection. For each 3D point Q , and obtain the photometric variance of its projection points on all visible images:

[0034] Where: p i express Q In the image I i The projection point on the I i (p i ) ; n express Q The number of images; Preset photometric variance threshold T L ,like Var L (Q) Not greater than the photometric variance thresholdT L , then the photometric consistency of the point is considered to be good; otherwise, the photometric consistency of the point is considered to be poor and it is deleted from the point cloud set; At the same time, preset the global visibility threshold T V ,like , then the global visibility of the point is considered to be poor and it is deleted from the point cloud set; otherwise, the point is retained; where: V obs (Q) The actual observed Q A collection of camera perspectives, V (Q) Representing 3D points Q Theoretical set of visibility under all camera views.

[0035] Based on the further optimization of the above scheme, the collection and processing of spectral data in step S3 is specifically as follows: While collecting image data, the spectrometer simultaneously acquires cotton spectral information and corrects the wavelength of the collected spectral data:

[0036] Where: represents the corrected wavelength; Indicates the measured wavelength; represents the fitting coefficient; Afterwards, feature extraction is performed on the corrected spectral data; Then, the spectral data is fused with the point cloud data and the image data in the spatial dimension, and the corresponding spectral data is assigned to each point according to the spatial coordinates of each point in the point cloud data (for example: for point O ( x,y,z ), when collecting the point cloud, the spectrometer is aimed at the cotton area near the point to form an association between the spectral information and the point cloud data); then, through the conversion relationship between the image coordinates and the point cloud coordinates, the spectral information is mapped to the image to form an image with spectral information annotation; Finally, using the data acquisition time as the index, the point cloud, image, and spectral data collected at different times are integrated, and the data are arranged in chronological order to form a time series-based data set.

[0037] The following are the technical effects of the solution of the present invention: The present invention is a detection robot composed of a box, a lifting mechanism, a shading mechanism, a steering mechanism, a driving mechanism and a control system. The box can be freely lifted up and down by adjusting the lifting mechanism, so as to adapt to cotton plants of different heights. The shading mechanism can effectively adjust the lighting conditions inside the box, reduce the influence of external ambient light on the data collection process, and improve the accuracy of data collection. The steering mechanism and the driving mechanism are combined to realize the operation and obstacle avoidance of the entire detection robot in the cotton field, and improve the flexibility of data collection. At the same time, the present invention uses a collection module composed of a high-definition industrial camera, a depth camera, and a spectrometer through the setting of the control system. The collaborative collection of multi-source data such as images, spectra, point clouds, and positions is realized, which not only effectively reduces the time cost of data collection and improves the fusion efficiency of multi-source and multi-dimensional data, but also has high robustness and effectively enhances the confidence of the collected data, thereby realizing rapid, accurate, efficient, and comprehensive three-dimensional reconstruction of cotton, providing data support for phenotypic analysis and growth monitoring of cotton, and having small human subjective influence factors and small errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 It is a schematic diagram of the overall structure of the detection robot of the present invention.

[0039] Figure 2 This is a schematic diagram of the internal structure of the box of the detection robot of the present invention.

[0040] Figure 3 It is a schematic structural diagram of the lifting mechanism of the detection robot of the present invention.

[0041] Figure 4 It is a structural schematic diagram of the shading mechanism of the detection robot of the present invention.

[0042] Figure 5 The figure is a schematic diagram of the structure of the steering mechanism of the detection robot of the present invention.

[0043] Figure 6 It is a schematic structural diagram of the driving mechanism of the detection robot of the present invention.

[0044] Among them, 1. power supply system; 2. box; 21. fixed screw slide; 22. mobile screw slide; 23. mobile slider; 24. lifting slide rail; 25. camera gimbal; 3. lifting mechanism; 31. fixed column; 32. lifting column; 33. drive motor; 34. lifting worm; 35. transmission wheel; 36. lifting screw; 37. limiter; 4. shading mechanism; 41. telescopic box; 42. shading cloth; 43. positioning rod; 44. folding rod; 5. steering mechanism; 51. steering box; 52. steering motor; 53. steering transmission; 54. steering bracket; 6. driving mechanism; 61. protective cover; 62. axle; 63. brake device; 64. hub motor; 65. walking wheel; 71. system control box; 72. acquisition module; 73. interactive screen. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0046] Embodiment 1: A cotton phenotype detection robot comprises a box 2, a lifting mechanism 3, a shading mechanism 4, a steering mechanism 5, a driving mechanism 6 and a control system. The box 2 is a rectangular parallelepiped structure (such as Figure 1 As shown in the figure, the four corners of the bottom are respectively provided with a lifting mechanism 3, the lifting mechanism 3 comprises a fixed column 31, a lifting column 32, a driving motor 33, a lifting worm 34, a transmission wheel 35, a lifting screw 36 and a limiter 37, the top of the fixed column 31 is respectively fixedly connected to the four corners of the bottom of the box body 1 and the lifting column 32 is slidably arranged in the inner cavity of the fixed column 31, and the front and rear sides of the lifting column 32 are hollow (that is, the lifting column 32 is as shown in the figure). Figure 3 The lower left side and the upper right side as shown in the figure are not provided with side walls) and the bottom end thereof passes through the bottom surface of the fixed column 31; a driving motor 33 is fixedly arranged on one side of the top of the fixed column 31, and the output end of the driving motor 33 is connected to the lifting worm 34 arranged in the inner cavity of the fixed column 31 (so that the lifting worm 34 is driven to rotate by the driving motor 33), a transmission wheel 35 is coaxially arranged in the inner cavity of the top of the fixed column 31, and the bottom of the transmission wheel 35 is fixedly connected to the lifting screw 36, and the lifting screw 36 is rotatably connected to the end face of the positioning block arranged on the bottom surface of the inner cavity of the fixed column 31 (such as Figure 3 As shown in FIG. 1 , the transmission wheel 35 is meshed with the lifting worm 34 for transmission, and the lifting screw 36 penetrates the end surface of the lifting column 32 and is threadedly connected; the fixed column 31 is located at the lower side of the transmission wheel 35 and the inner cavity of the bottom surface of the fixed column 31 is respectively provided with a limiter 37 (as shown in FIG. 1 ). Figure 3 As shown, it is used to limit the upward and downward movement of the lifting column 32).

[0047] The bottom of the box body 2 and between the two sets of lifting mechanisms 3 are provided with shading mechanisms 4 (ie, the bottom of the box body 1 and the corresponding four sides thereof are provided with shading mechanisms 4, such as Figure 1 As shown), the shading mechanism 4 includes a telescopic box 41, a shading cloth 42, a positioning rod 43 and a folding rod 44. The telescopic box 41 is fixedly arranged at the bottom of the four sides of the box body 1, and a rotating rod is rotatably arranged in the inner cavity of the telescopic box 41. The upper end of the shading cloth 42 is fixedly arranged on the outer wall of the rotating rod in the telescopic box 41 through a groove, and the lower end of the shading cloth 42 passes through the telescopic box 41 and is fixedly connected to the outer wall of the positioning rod 43. The two ends of the positioning rod 43 are respectively slidably connected to the outer walls of the corresponding fixed columns 31; two groups of folding rods 44 are symmetrically arranged between the telescopic box 41 and the positioning rod 42 and on the back of the shading cloth 42, and the two ends of the folding rod 44 are respectively rotatably connected to the bottom surface of the telescopic box 41 and the end surface of the positioning rod 42 (as shown in FIG. Figure 4 At the same time, a spring may be provided inside the folding rod 44 to provide tension for the folding rod 44).

[0048] A steering mechanism 5 is provided at one end of the lifting mechanism 3 away from the box body 2. The steering mechanism 5 includes a steering box 51, a steering motor 52 and a steering transmission 53. The steering box 51 is fixedly arranged at the bottom of the lifting column 32 and a steering motor 52 is arranged on one side of the steering box 51 (such as Figure 5 As shown in the figure, a steering transmission device 53 is arranged in the inner cavity of the steering box 51 and the steering transmission device 53 is connected to the output end of the steering motor 52; the bottom end of the steering transmission device 53 passes through the bottom surface of the steering box 51 and is connected to the steering bracket 54 arranged at the lower side of the steering box 51. The driving mechanism 6 is connected to the lower end of the steering mechanism 5 through the steering bracket 54; the driving mechanism 6 includes a protective cover 61, an axle 62, a brake device 63, a hub motor 64 and a running wheel 65. The protective cover 61 is fixedly arranged at the bottom end of the steering bracket 54 and the axle 62 is arranged at the bottom thereof (combined with Figure 5 and Figure 6 As shown in the figure, a running wheel 65 is provided on the outer wall of the middle part of the axle 62 and a hub motor 64 is installed on the running wheel 65. The hub motor 64 is connected to the steering bracket 54 by bolts through the axle 62, and a brake device 63 is provided on the hub motor 64.

[0049] The control system includes a system control box 71, an acquisition module 72, an interactive screen 73, a radar and power supply system 1, the system control box 71 is fixedly arranged in the inner cavity of the box body 2 and a lifting platform is arranged in the box body 2, the acquisition module 72 is arranged on the lifting platform, the lifting platform includes a fixed lead screw slide 21, a movable lead screw slide 22, a movable slider 23, a lifting slide rail 24 and a camera platform 25, and the fixed lead screw slide 21, the movable lead screw slide 22, the movable slider 23 and the lifting slide rail 24 are all two groups; wherein, two groups of fixed lead screw slides 21 are arranged in parallel at the front and rear ends of the inner cavity of the box body 1, and two groups of movable lead screw slides 22 are slidably arranged between the two groups of fixed lead screw slides 21, the movable sliders 23 are respectively slidably arranged on the two groups of movable lead screw slides 22, and the lifting slide rails 24 are respectively fixedly arranged on the bottom surfaces of the movable sliders 23, and multiple groups of camera platforms 25 are respectively slidably arranged on the two lifting slide rails 24 in the vertical direction, and the acquisition module 72 (such as Figure 2 As shown, in this embodiment, three camera gimbals 25 are arranged on a lifting rail 24, and the acquisition module 72 includes a high-definition industrial camera, a depth camera, a spectrometer, etc. The radar is arranged at the front end of the outer side of the box 2, an interactive screen 73 is arranged at the top of the outer wall of one side of the box 2, and a power supply system 1 is arranged on the top surface of the box 2 (the power supply system 1 can use a conventional photovoltaic panel or a battery pack).

[0050] Embodiment 2: As a preferred embodiment of the present invention, a method for automatically detecting cotton phenotypes, using the detection robot described in the above embodiment 1, comprises: Step S1: Get the navigation path, that is: First, before the detection robot enters the cotton field, the boundary of the cotton field to be detected is measured by the high-precision Beidou satellite positioning system, the initial map area of ​​the cotton field is constructed, and the coordinates of the four vertices of the initial map area are marked ( x 1 ,y 1) ( x 2 ,y 2) ( x 3 ,y 3) (x 4 ,y 4); Then, based on the shape and size of the initial map area, determine the starting point for data collection S With the end point E (Generally, the starting point S Select the midpoint and end point of one end of the cotton field E Select the midpoint of the other end boundary opposite to it); divide the initial map area into uniform grids and the size of each grid is: ; Then the number of grids in the initial map area is :

[0051] Where: L x Indicates that the initial map area is x Length in direction; L y Indicates that the initial map area is y Length in direction; represents the floor function; Afterwards, an improved method based on the A* algorithm is used to plan the collection path line by line, where the evaluation function of the A* algorithm is:

[0052] Where: g(n) Indicates starting point S To Node n The actual cost, h(n) Represents a slave node n The estimated cost to reach the end point E; In the cotton field, row by row picking scene, g(n) , h(n) Defined as: a. Assume that the current node n The coordinates of are ( x n ,y n ), the length of the path traveled is d ,but g(n)=d ; b. If you are currently in i Target behavior j OK( i <j ),but:

[0053] Where: y n-r Indicates the collection point corresponding to the next row y coordinate; The specific search process is: Initialize an open list "Open" and a closed list "Closed", add the starting point S to the list "Open"; when the list "Open" is not empty, select f(n)The node n with the smallest value is taken as the current node; the current node n is removed from the list "Open" and added to the list "Closed"; check whether the current node n is the end point E. If so, find a path and generate a complete path by backtracking the nodes in the list "Closed"; if not, expand the adjacent nodes m of the current node n (i.e., the four directions of up, down, left, and right, and limit the direction according to the actual situation at the boundary of the cotton field); for each adjacent node m, calculate its g(m) and h(m) If m is not in the list "Open" and the list "Closed", add it to the list "Open" and record its parent node as n; if m is in the list "Open" and the newly calculated g(m) The value is smaller than the original g (n) value, then update m's g(m) The value and parent node is n.

[0054] According to the height of the cotton plant, the lifting mechanism is used to drive the box to rise, and the steering mechanism and the driving mechanism are used to control the detection robot to move; During the forward movement, the radar scans to establish the road condition ahead of the detection robot and realizes real-time correction of the navigation path, that is, the robot moves forward according to the planned collection path and scans the road condition information ahead in real time through the radar. If there is an obstacle, the detection robot is controlled by the steering mechanism to bypass the obstacle and then continues to move forward according to the principle of line-by-line collection.

[0055] Step S2: During the movement of the detection robot, the shading mechanism is adjusted according to the lighting conditions, and the position of the collection module is adjusted by lifting the pan / tilt to obtain the best shooting angle, and the collection module collects data on the cotton in real time; Step S3, using the information collected by the collection module, and using feature point extraction and matching, sparse point cloud reconstruction, sparse point cloud densification, and spectral data collection and processing, the cotton is three-dimensionally reconstructed, and a four-dimensional information data set of cotton point cloud, image, and spectrum based on time series is constructed to complete the establishment of the analysis data set; Among them, feature point extraction and matching are specifically as follows: First, the original image I(x,y) Perform Gaussian convolution of different scales to obtain Gaussian pyramids:

[0056] Where: represents the scale parameter; Subtract the height images of adjacent scales and construct the Gaussian difference (DOG) scale space:

[0057] Where:k represents the scale interval factor; In the Gaussian difference scale space, if a point is the extreme point of all points compared with its 8 adjacent points of the same size and the points of its upper and lower adjacent sizes (i.e., 9x2, a total of 18 points), then the point is a potential feature point; Then, the potential feature points initially obtained are accurately located: Assume that the function of the Gaussian difference scale space is At the key point The quadratic Taylor expansion nearby is:

[0058] Where: Indicates the offset of key point position and scale; Solution D(X) The partial derivative of is zero X values ​​to precisely locate key points:

[0059] At the same time, before obtaining the key points using the Gaussian difference scale space function, the contrast threshold is preset T c Ratio threshold to principal curvature T r , if the contrast of the obtained key point is lower than the contrast threshold T c Or the principal curvature ratio is greater than the principal curvature ratio threshold T r , then discard the corresponding key points; After that, the gradient amplitude of the pixels in its neighborhood is calculated with the key point as the center With direction :

[0060]

[0061] Where: L(x,y) Represents the image of the corresponding scale in the Gaussian pyramid; Construct a histogram and count the gradient direction distribution of pixels in the neighborhood; the peak direction of the histogram is used as the main direction of the key point, and other directions that are 80% greater than the peak of the main direction are used as auxiliary directions of the key point; Then, with the key point as the center, a local coordinate system is constructed in its neighborhood according to the selected main direction; the neighborhood is divided into multiple sub-regions, and the gradient direction histogram is counted in each sub-region. Each histogram has a direction interval, so that a one-dimensional feature vector, i.e., a feature descriptor, is obtained (for example: remove the 16x16 neighborhood window and divide it into 4x4 windows. For each sub-window, the gradient information in 8 directions is counted to form an 8-dimensional vector. Each key point can be described by a 4x4x8=128-dimensional vector); Finally, for each feature point in one image, the nearest neighbor and next nearest neighbor feature point in the feature point set of another image are searched through the K-dimensional space binary tree model, and the Euclidean distance between the nearest neighbor feature point, the next nearest neighbor feature point and the feature point in the current image is calculated respectively. d 1 and d 2; Set the distance ratio threshold T m (Distance ratio threshold T m is 0.8), if , the two feature points are considered as matching points, the matching points are associated and the corresponding relationship of the associated views is established, otherwise, the two feature points do not match.

[0062] The specific steps of sparse point cloud reconstruction are as follows: based on the extracted key matching points and their corresponding relationships, the camera space pose information corresponding to each view image is restored, and the sparse point cloud reconstruction is realized by the inverse projection method. The projection error formula is:

[0063] Where: M represents the camera parameters, P represents the key matching point set; N Indicates the total number of images acquired, Z Indicates the total number of key matching points calculated; w i,j Indicates matching point j Is it in the image i If so, then w i,j =1, otherwise, w i,j =0; q ij and P ij Respectively represent the real image point and its corresponding 3D point; Represents the back-projected image point obtained by projecting the 3D point back to the image. The projection error quantifies the error between the spatial pose information and the estimated 3D point coordinates through the error between the real and back-projected image points.

[0064] The densification of sparse point clouds is as follows: First, cluster the image set to reduce the amount of data in the reconstruction process, improve the calculation speed and reconstruction accuracy: Normalize the input images so that all images have the same size, that is, H x W ; If the original image I i The size is H i x W i , then each pixel in the image (x,y) After scaling, the image Pixels in for:

[0065]

[0066] Use convolutional neural network to extract features from normalized images. After convolution layer and pooling layer, image features are obtained. F i , whose dimensions are H f x W f x C ,in, H f and W f Represents the height and width of the feature map, C Indicates the number of feature channels; For different image features F i and F j Between the feature maps F i Each feature point in p i , in the feature map F j Find similar feature points in p j , and use Euclidean distance as the similarity measure, that is:

[0067] Where: V pi , V pj Respectively p i , p j The eigenvector of Randomly select B images from the image set as the initial cluster centers, that is, C 1 ,C 2 ,…,C B , for each image I m , the above method is used to calculate the similarity score S( I m ，C I ), I=1,2,…,B ; Calculate at the same time I m The number of matching points or the average similarity score between all initial cluster centers; I m Assigned to the cluster with the highest similarity score C k* middle:

[0068] Calculate the average feature of the images in each cluster again and use it as the new cluster center. In Location (x,y) The eigenvalues ​​at are:

[0069] Where: Represents clustering C B The number of images in Repeat the above process until the clustering result no longer changes or the preset maximum number of iterations is reached, and the clustering of the image set is completed; Then, the dense point cloud is reconstructed under the constraints of local photometric consistency and global visibility: Based on the sparse point cloud, representative seed points are selected in the image. The seed set is: ,in, s i =( x i ,y i ), represents the coordinates of the seed point in the image; For each seed point s i As the center, construct a w x w Patch, match the patch between different images, and measure the similarity of the two patches:

[0070]

[0071] Where: PP i , PP j Represent the patches in image A and image B respectively; , Respectively PP i , PP j The mean of Preset Threshold T NCC ,like ,but PP i and PP j Match between them, otherwise do not match; For each matched seed point pair, the corresponding three-dimensional space point Q is obtained by using the camera's pose information and triangulation principle:

[0072] Where: , denote the scale factors respectively; P A , P B Represents the projection matrix of the two cameras; p A , p B Respectively represent the matching image points:

[0073] For each calculated 3D point Q Centered at radius r Find other possible matching points in the neighborhood of ; in the image, according to the camera projection model, the three-dimensional point Q Projection points are obtained by projecting onto different images, and then patches are constructed around these projection points and matched. If a new matching point pair is found, its corresponding 3D point is added to the point cloud collection. For each 3D point Q , and obtain the photometric variance of its projection points on all visible images:

[0074] Where: p i express Q In the image I i The projection point on theI i (p i ) ; n express Q The number of images; Preset photometric variance threshold T L ,like Var L (Q) Not greater than the photometric variance threshold T L , then the photometric consistency of the point is considered to be good; otherwise, the photometric consistency of the point is considered to be poor and it is deleted from the point cloud set; At the same time, preset the global visibility threshold T V ,like , then the global visibility of the point is considered to be poor and it is deleted from the point cloud set; otherwise, the point is retained; where: V obs (Q) The actual observed Q A collection of camera perspectives, V (Q) Representing 3D points Q Theoretical set of visibility under all camera views.

[0075] The collection and processing of spectral data are as follows: While collecting image data, the spectrometer simultaneously acquires cotton spectral information and corrects the wavelength of the collected spectral data:

[0076] Where: represents the corrected wavelength; Indicates the measured wavelength; represents the fitting coefficient; Afterwards, feature extraction is performed on the corrected spectral data; Then, the spectral data is fused with the point cloud data and the image data in the spatial dimension, and the corresponding spectral data is assigned to each point according to the spatial coordinates of each point in the point cloud data (for example: for point O ( x,y,z ), when collecting the point cloud, the spectrometer is aimed at the cotton area near the point to form an association between the spectral information and the point cloud data); then, through the conversion relationship between the image coordinates and the point cloud coordinates, the spectral information is mapped to the image to form an image with spectral information annotation; Finally, using the data acquisition time as the index, the point cloud, image, and spectral data collected at different times are integrated, and the data are arranged in chronological order to form a time series-based data set.

[0077] Embodiment 3: As a preferred embodiment of the present invention, based on the scheme of Example 2, when encountering an obstacle in step S1, a collision-free path is searched with the position of the obstacle as the center through a local variant of the Dijkstra algorithm, wherein the radar transmits the detected obstacle information to the control system in the form of point cloud data; thereby regaining the collision-free path and realizing obstacle avoidance of the detection robot.

[0078] Embodiment 4: As a preferred embodiment of the present invention, based on the solution of embodiment 2, in order to evaluate the accuracy of the three-dimensional reconstruction in step S3, the root mean square error is used. RMSE , relative error RE and coefficient of determination R 2 Quantitative description of the degree of agreement between the reconstructed value and the actual value:

[0079]

[0080]

[0081] Where: D i , d i Respectively represent i The estimated value and the actual value; D avg , d avg Respectively represent i The average value of the values ​​to be evaluated and the average value of the actual value; n Indicates the number of values ​​to be evaluated.

Claims

1. A cotton phenotype detection robot, characterized in that: It includes a box body, a lifting mechanism, a shading mechanism, a steering mechanism, a driving mechanism and a control system. The box body is a rectangular structure as a whole, and the lifting mechanisms are respectively arranged at the four corners of the bottom. Shading mechanisms are respectively arranged at the bottom of the box body and between the two groups of lifting mechanisms. Steering mechanisms are respectively arranged at the end of the lifting mechanism away from the box body, and the driving mechanism is connected to the lower end of the steering mechanism through a steering bracket; the control system includes a system control box, an acquisition module, an interactive screen, a radar and a power supply system. The system control box is fixedly arranged in the inner cavity of the box body and a lifting pan-tilt platform is arranged in the box body, the acquisition module is arranged on the lifting pan-tilt platform, the radar is arranged at the front end of the outer side of the box body, an interactive screen is arranged at the top of the outer wall on one side of the box body, and a power supply system is arranged on the top surface of the box body.

2. A cotton phenotype detection robot according to claim 1, characterized in that: The lifting mechanism comprises a fixed column, a lifting column, a driving motor, a lifting worm, a transmission wheel, a lifting screw and a limiter. The top of the fixed column is fixedly connected to the four corners of the bottom of the box body respectively, and the lifting column is slidably arranged in the inner cavity of the fixed column. The front and rear sides of the lifting column are hollow and the bottom end thereof passes through the bottom surface of the fixed column; a driving motor is fixedly arranged on one side of the top of the fixed column and the output end of the driving motor is connected to the lifting worm arranged in the inner cavity of the fixed column; a transmission wheel is coaxially arranged in the inner cavity of the top of the fixed column and the bottom of the transmission wheel is fixedly connected to the lifting screw, the lifting screw is rotatably connected to the end face of the positioning block arranged on the bottom surface of the inner cavity of the fixed column, the transmission wheel is meshed with the lifting worm for transmission, the lifting screw passes through the end face of the lifting column and is threadedly connected; the fixed column is located at the lower side of the transmission wheel and the inner cavity of the bottom surface of the fixed column respectively has limiters.

3. A cotton phenotype detection robot according to claim 1 or 2, characterized in that: The shading mechanism includes a telescopic box, a shading cloth, a positioning rod and a folding rod. The telescopic boxes are respectively fixedly arranged at the bottom of the four side surfaces of the box body and a rotating rod is rotatably arranged in the inner cavity of the telescopic box. The upper end of the shading cloth is fixedly arranged on the outer wall of the rotating rod in the telescopic box through a groove, and the lower end of the shading cloth passes through the telescopic box and is fixedly connected to the outer wall of the positioning rod, and the two ends of the positioning rod are respectively slidably connected to the outer walls of the corresponding fixed columns; two groups of folding rods are symmetrically arranged between the telescopic box and the positioning rod and on the back side of the shading cloth, and the two ends of the folding rods are respectively rotatably connected to the bottom surface of the telescopic box and the end surface of the positioning rod.

4. A cotton phenotype detection robot according to claim 3, characterized in that: The steering mechanism includes a steering box, a steering motor and a steering transmission. The steering box is fixedly arranged at the bottom of the lifting column and a steering motor is arranged on one side of the steering box. The steering transmission is arranged in the inner cavity of the steering box and is connected to the output end of the steering motor. The bottom end of the steering transmission passes through the bottom surface of the steering box and is connected to the steering bracket arranged on the lower side of the steering box. The driving mechanism includes a protective cover, an axle, a brake device, a hub motor and a running wheel. The protective cover is fixedly arranged at the bottom end of the steering bracket and an axle is arranged at the bottom thereof. A running wheel is arranged on the outer wall of the middle part of the axle and a hub motor is installed on the running wheel. The hub motor is connected to the steering bracket by bolts via the axle, and the brake device is arranged on the hub motor.

5. A cotton phenotype detection robot according to claim 4, characterized in that: The lifting pan-tilt platform includes a fixed screw slide, a movable screw slide, a movable slider, a lifting slide rail and a camera pan-tilt platform. The fixed screw slide, the movable screw slide, the movable slider and the lifting slide rail are all two groups; wherein, the two groups of fixed screw slides are arranged in parallel at the front and rear ends of the inner cavity of the box body, and two groups of movable screw slides are slidably arranged between the two groups of fixed screw slides, the movable sliders are respectively slidably arranged on the two groups of movable screw slides, and the lifting slide rails are respectively fixedly arranged on the bottom surfaces of the movable sliders, and multiple groups of camera pan-tilt platforms are respectively slidably arranged along the vertical direction on the two lifting slide rails for installing acquisition modules, and the acquisition modules include high-definition industrial cameras, depth cameras, spectrometers, etc.

6. An automated detection method using a cotton phenotype detection robot as claimed in claim 1, characterized in that: include: Step S1, obtaining a navigation path, and using a lifting mechanism to drive the box to lift according to the height of the cotton plant, and using a steering mechanism and a driving mechanism to control the detection robot to move forward; during the forward movement, the radar scans to establish the forward movement of the detection robot, and realizes real-time correction of the navigation path; Step S2: During the movement of the detection robot, the shading mechanism is adjusted according to the lighting conditions, and the position of the collection module is adjusted by lifting the pan / tilt to obtain the best shooting angle, and the collection module collects data on the cotton in real time; Step S3, using the information collected by the acquisition module, and using feature point extraction and matching, sparse point cloud reconstruction, sparse point cloud densification, and spectral data collection and processing, the cotton is three-dimensionally reconstructed, and a four-dimensional information data set of cotton point cloud, image, and spectrum based on time series is constructed to complete the establishment of the analysis data set.

7. The automated detection method of a cotton phenotype detection robot according to claim 6, characterized in that: The step S1 is specifically as follows: First, before the detection robot enters the cotton field, the boundary of the cotton field to be detected is measured by the high-precision Beidou satellite positioning system, the initial map area of ​​the cotton field is constructed, and the coordinates of the four vertices of the initial map area are marked ( x 1 ,y 1) ( x 2 ,y 2) ( x 3 ,y 3) (x 4 ,y 4); Then, based on the shape and size of the initial map area, determine the starting point for data collection S With the end point E ; Divide the initial map area into uniform grids and the size of each grid is: ; Then the number of grids in the initial map area is : Where: L x Indicates that the initial map area is x Length in direction; L y Indicates that the initial map area is y Length in direction; represents the floor function; Afterwards, an improved method based on the A* algorithm is used to plan the collection path line by line, where the evaluation function of the A* algorithm is: Where: g(n) Indicates starting point S To Node n The actual cost of h(n) Represents a slave node n The estimated cost to reach the end point E; In the cotton field, row by row picking scene, g(n) , h(n) Defined as: a. Assume that the current node n The coordinates of are ( x n ,y n ), the length of the path traveled is d ,but g(n)=d ; b. If you are currently in i Target behavior j OK( i <j ),but: Where: y n-r Indicates the collection point corresponding to the next row y coordinate; Finally, the robot moves forward according to the planned collection path and uses radar to scan the road conditions ahead in real time. If there are obstacles, the robot is controlled by the steering mechanism to bypass the obstacles and then continues to move forward according to the principle of line-by-line collection.

8. The automated detection method of a cotton phenotype detection robot according to claim 6, characterized in that: The feature point extraction and matching in step S3 are specifically as follows: First, the original image I(x,y) Perform Gaussian convolution of different scales to obtain Gaussian pyramids: Where: represents the scale parameter; Subtract the height images of adjacent scales and construct the Gaussian difference (DOG) scale space: Where: k represents the scale interval factor; In the Gaussian difference scale space, if a point is an extreme point of all points compared with its eight adjacent points of the same size and the points of its upper and lower adjacent sizes, then the point is a potential feature point; Then, the potential feature points initially obtained are accurately located: Assume that the function of the Gaussian difference scale space is At the key point The quadratic Taylor expansion nearby is: Where: Indicates the offset of key point position and scale; Solution D(X) The partial derivative of is zero X values ​​to precisely locate key points: After that, the gradient amplitude of the pixels in its neighborhood is calculated with the key point as the center With direction : Where: L(x,y) Represents the image of the corresponding scale in the Gaussian pyramid; Construct a histogram and count the gradient direction distribution of pixels in the neighborhood; the peak direction of the histogram is used as the main direction of the key point, and other directions that are 80% greater than the peak of the main direction are used as auxiliary directions of the key point; Then, taking the key point as the center, a local coordinate system is constructed in its neighborhood according to the selected main direction; this neighborhood is divided into multiple sub-regions, and the gradient direction histogram is statistically calculated in each sub-region. Each histogram has a direction interval, so that a one-dimensional feature vector, i.e., a feature descriptor, is obtained; Finally, for each feature point in one image, the nearest neighbor and next nearest neighbor feature point in the feature point set of another image are searched through the K-dimensional space binary tree model, and the Euclidean distance between the nearest neighbor feature point, the next nearest neighbor feature point and the feature point in the current image is calculated respectively. d 1 and d 2; Set the distance ratio threshold T m ,like , the two feature points are considered as matching points, the matching points are associated and the corresponding relationship of the associated views is established, otherwise, the two feature points do not match.

9. The automated detection method of a cotton phenotype detection robot according to claim 8, characterized in that: The sparse point cloud reconstruction in step S3 is specifically as follows: based on the extracted key matching points and their corresponding relationships, the camera space pose information corresponding to each view image is restored, and the sparse point cloud reconstruction is realized by using the inverse projection method, wherein the projection error formula is: Where: M represents the camera parameters, P represents the key matching point set; N Indicates the total number of images acquired, Z Indicates the total number of key matching points calculated; w i,j Indicates matching point j Is it in the image i If so, then w i,j =1, otherwise, w i,j =0; q ij and P ij Respectively represent the real image point and its corresponding 3D point; Represents the back-projected image point obtained by projecting the 3D point back to the image. The projection error quantifies the error between the spatial pose information and the estimated 3D point coordinates through the error between the real and back-projected image points.

10. The automated detection method of a cotton phenotype detection robot according to claim 9, characterized in that: The densification of the sparse point cloud in step S3 is specifically as follows: First, cluster the image set to reduce the amount of data in the reconstruction process, improve the calculation speed and reconstruction accuracy: Normalize the input images so that all images have the same size, that is, H x W ; If the original image I i The size is H i x W i , then each pixel in the image (x,y) After scaling, the image Pixels in for: Use convolutional neural network to extract features from normalized images. After convolution layer and pooling layer, image features are obtained. F i , whose dimensions are H f x W f x C ,in, H f and W f Represents the height and width of the feature map, C Indicates the number of feature channels; For different image features F i and F j Between the feature maps F i Each feature point in p i , in the feature map F j Find similar feature points in p j , and use Euclidean distance as the similarity measure, that is: Where: V pi , V pj Respectively p i , p j The eigenvector of Randomly select B images from the image set as the initial cluster centers, that is, C 1 ,C 2 ,…,C B , for each image I m , the above method is used to calculate the similarity score S( I m ，C I ), I=1,2,…,B ; Calculate at the same time I m The number of matching points or the average similarity score between all initial cluster centers; I m Assigned to the cluster with the highest similarity score C k* middle: Calculate the average feature of the images in each cluster again and use it as the new cluster center. In Location (x,y) The eigenvalues ​​at are: Where: Represents clustering C B The number of images in Repeat the above process until the clustering result no longer changes or the preset maximum number of iterations is reached, and the clustering of the image set is completed; Then, the dense point cloud is reconstructed under the constraints of local photometric consistency and global visibility: Based on the sparse point cloud, representative seed points are selected in the image. The seed set is: ,in, s i =( x i ,y i ), represents the coordinates of the seed point in the image; For each seed point s i As the center, construct a w x w Patch, match the patch between different images, and measure the similarity of the two patches: Where: PP i , PP j Represent the patches in image A and image B respectively; , Respectively PP i , PP j The mean of Preset Threshold T NCC ,like ,but PP i and PP j Match between them, otherwise do not match; For each matched seed point pair, the corresponding three-dimensional space point Q is obtained by using the camera's pose information and triangulation principle: Where: , denote the scale factors respectively; P A , P B Represents the projection matrix of the two cameras; p A , p B Respectively represent the matching image points: For each calculated 3D point Q Centered at radius r Find other possible matching points in the neighborhood of ; in the image, according to the camera projection model, the three-dimensional point Q Projection points are obtained by projecting onto different images, and then patches are constructed around these projection points and matched. If a new matching point pair is found, its corresponding 3D point is added to the point cloud collection. For each 3D point Q , and obtain the photometric variance of its projection points on all visible images: Where: p i express Q In the image I i The projection point on the I i (p i ) ; n express Q The number of images; Preset photometric variance threshold T L ,like Var L (Q) Not greater than the photometric variance threshold T L , then the photometric consistency of the point is considered to be good; otherwise, the photometric consistency of the point is considered to be poor and it is deleted from the point cloud set; At the same time, preset the global visibility threshold T V ,like , then the global visibility of the point is considered to be poor and it is deleted from the point cloud set; otherwise, the point is retained; where: V obs (Q) The actual observed Q A collection of camera perspectives, V(Q) Representing 3D points Q Theoretical set of visibility under all camera views.

Images