Crop posture collection and detection equipment and key point detection method

Abstract

The present application provides a crop posture acquisition detection device and a key point detection method. The detection device of the present application uses a vehicle-mounted chassis to realize automatic driving and tracking walking, and the image acquisition platform thereon provides support and two steering degrees of freedom for the camera, thereby realizing multi-angle crop phenotypic image acquisition. The present application adopts a heat map regression detection algorithm to realize key point detection, and then adopts UNet as the optimal network model for crop posture key point detection, and preferably adopts an improved 11 key point detection method for wheat crop phenotypic characteristics to obtain a model detection effect that is better than the original hypothesis key point. The present application can effectively identify the angles of the top one, two, and three leaves in the crop posture and the length parameters of the top one, two, and three leaves, and more intuitively display the changing trends of wheat leaf length and leaf angle during drought stress to meet the detection needs of more parameters. The present application provides a powerful tool for studying the diverse changes in wheat posture, and for identification and classification, and has high practical value.

Description

Technical Field

[0001] The present application relates to crop phenotyping technology, and more specifically to a crop posture collection and detection device and a key point detection method. Background Art

[0002] As an important food crop in the world, wheat has important phenotypic traits such as plant height, stem width and leaf inclination, which can effectively reflect the growth, stress and nutritional status of crops. While deep learning is booming, posture estimation has achieved success in the human body and has also made new progress in other deformable joint objects (such as animals).

[0003] However, in terms of plant detection, research progress in the direction of posture estimation and key point detection has been slow.

[0004] The morphological structure and parameters of plants have important biological significance in agricultural production and research. It can detect in real time the impact of different environmental conditions and different genotypes of plants on the changes in plant morphological parameters. Therefore, accurately capturing and analyzing important characteristics of plants based on machine vision, image segmentation, and deep learning technologies plays a vital role in biological breeding and agricultural production. Summary of the invention

[0005] In response to the deficiencies of the prior art, the present application provides a crop posture collection and detection device and a key point detection method, which is based on deep learning and machine vision technology, and uses a crop phenotype image collection robot to automatically collect side-view posture images of potted crops and produce a crop posture evaluation data set. The coordinate regression network and the heat map regression network are used to discuss the applicability of crop posture key point detection, and the optimal network model UNet is selected to quantify the morphological structure of the crop, and then analyze the physiological state of the crop. The present application not only provides a new idea for non-invasive analysis of the physiological state of crops, but also has important significance for crop drought disaster prevention and mitigation. The present application specifically adopts the following technical solutions.

[0006] Firstly, to achieve the above-mentioned purpose, a crop posture acquisition and detection device is proposed, whose main structure includes: a vehicle-mounted chassis and an image acquisition platform installed on the vehicle-mounted chassis; wherein, the vehicle-mounted chassis adopts a wheeled chassis, and the main structure of the wheeled chassis includes two steel plates, which are arranged as a control layer and a guiding power layer from top to bottom, wherein a driving system is installed on the steel plate of the guiding power layer, a tracking system and a sensor are installed on the bottom of the steel plate of the guiding power layer, and a control unit is installed on the steel plate of the control layer for controlling the operation of the driving system and transmitting and processing the crop phenotype images acquired by the image acquisition platform; the image acquisition platform is electrically connected to the control unit in the control layer, and a USB camera is arranged in the image acquisition platform for image acquisition, and the image A camera bracket is also provided in the acquisition platform, and the camera bracket has a first deflection freedom turntable with an axis perpendicular to the plane of the wheeled chassis steel plate as the rotation center, and a second pitch freedom turntable installed on the first deflection freedom turntable and arranged to have an axis parallel to the plane of the wheeled chassis steel plate as the rotation center. The USB camera is installed on the second pitch freedom turntable. The first deflection freedom turntable and the second pitch freedom turntable are respectively provided with steering power in two degrees of freedom directions by two servos to realize multi-angle crop phenotypic image acquisition; the control unit also realizes the transmission and automatic storage of image data after receiving the crop phenotypic image, so as to trigger the screening and analysis of the image data to obtain the morphological parameters of ear length, leaf length and leaf angle.

[0007] Optionally, in a crop posture collection and detection device as described in any of the above, the control unit further triggers the following screening and analysis steps on the image data after receiving the crop phenotypic image, so as to obtain a crop posture key point connection image and crop posture image morphological parameters: using the PyTorch deep learning framework to build a model, using a deep neural network to identify and predict the key points of a single crop plant through a heat map regression detection algorithm, finding a nonlinear mapping relationship with the smallest loss function, and obtaining the position coordinates of the maximum value of the heat map in each crop posture key point connection image, converting the position coordinates of the maximum value of the heat map into key point coordinates through the heatmap_to_numpy function, and then according to the crop morphological parameter calculation formula, calculating the morphological parameters of the ear length, leaf length, and leaf angle according to the coordinates of each key point.

[0008] Optionally, a crop posture acquisition and detection device as described in any of the above, wherein the sensor includes an infrared tracking sensor fixed at the front end below the chassis, for detecting thermal radiation absorbed by a black guide line laid on the operating surface of the crop posture acquisition and detection device, and thereby determining position information between the installation position of the infrared tracking sensor in the center of the vehicle body and the black guide line, so that the driving system of the crop posture acquisition and detection device tracks along the black guide line pre-laid on the foam board.

[0009] At the same time, in order to achieve the above-mentioned purpose, the present application also provides a crop posture key point detection method, which is used to perform the following screening and analysis steps on the image data collected by any of the crop posture collection and detection devices as described above, so as to obtain the crop posture key point connection image and the crop posture image morphological parameters: use the PyTorch deep learning framework to build a model, use the deep neural network to identify and predict the key points of individual crops through the heat map regression detection algorithm, find the nonlinear mapping relationship with the minimum loss function, and obtain the position coordinates of the maximum value of the heat map in each crop posture key point connection image, convert the position coordinates of the maximum value of the heat map into key point coordinates through the heatmap_to_numpy function, and then according to the crop morphological parameter calculation formula, calculate the morphological parameters of the ear length, leaf length, and leaf angle according to the coordinates of each key point.

[0010] Optionally, a crop posture key point detection method as described in any of the above, wherein a model obtained by training through the following steps is used to identify and extract crop morphological parameters: the key points of each crop side view posture image are calibrated using Landmark_annotation software, and each image is classified into a training data set, a validation data set and a test data set according to a fixed ratio; then, on a PC configured with NVIDIA RTX3060, a heat map network model is constructed using the PyTorch deep learning framework, the crop side view posture images and the corresponding key point label information in the training data set are fused, and the encoder-decoder is input for model training to obtain the key point detection model and heat map information, the validation data set is used to verify the fitting degree of the key point detection model and the network parameters are adjusted accordingly, and finally, the accuracy of the key point detection model is tested using the test data set to obtain the nonlinear mapping relationship with the minimum loss function to complete the model training.

[0011] Optionally, in any of the key point detection methods described above, after calibrating the key points of each crop side view posture image using Landmark_annotation software, any of the following enhancement operations are performed on each image: flipping, rotation, brightness enhancement, contrast enhancement, and the images obtained by the enhancement operations are classified into the training data set, the verification data set and the test data set according to the same fixed ratio.

[0012] Optionally, in any of the key point detection methods described above, the key points of the side view posture image of the potted crop include: endpoint coordinates and / or central coordinates of the ear of wheat, the first top leaf, the second top leaf, and the third top leaf.

[0013] Optionally, a key point detection method as described in any of the above, wherein the morphological parameters of the crop posture image include: the angle between the top one, second and third leaves and / or the length of the top one, second and third leaves; landmark[0] represents the key point coordinates of the top of the wheat ear, landmark[1] represents the key point coordinates of the bottom of the wheat ear, landmark[2] represents the key point coordinates of the top of the top leaf, landmark[3] represents the key point coordinates of the middle of the top leaf, landmark[4] represents the key point coordinates of the bottom of the top leaf, landmark[5] represents the key point coordinates of the top of the second top leaf, l andmark[6] represents the key point coordinates of the middle of the second apical leaf, landmark[7] represents the key point coordinates of the bottom of the second apical leaf, landmark[8] represents the key point coordinates of the top of the third apical leaf, landmark[9] represents the key point coordinates of the middle of the third apical leaf, and landmark

[10] represents the key point coordinates of the bottom of the third apical leaf. Then, the calculation formula for the morphological parameters of the spike length is Line(landmark[0], landmark[1]); the calculation formula for the morphological parameters of the length of the first apical leaf is Line(landmark[2], landmark[ 3])+Line(landmark[3],landmark[4]); the morphological parameter calculation formula for the length of the second top leaf is Line(landmark[5],landmark[6])+Line(landmark[6],landmark[7]); the morphological parameter calculation formula for the length of the third top leaf is Line(landmark[8],landmark[9])+Line(landmark[9],landmark

[10] ); the morphological parameter calculation formula for the angle of the first top leaf is ng(landmark[1],landmark[4],landmark[3]); the morphological parameter calculation formula for the angle of the second top leaf is Ang(landmark[4],landmark[7],landmark[6]); the morphological parameter calculation formula for the angle of the third top leaf is Ang(landmark[7],landmark

[10] ,landmark[9]); where Line is the Euclidean distance calculation formula, which is used to calculate the distance between the pixel coordinates of the key points in the image; Ang is the angle calculation formula, which is used to calculate the angle between the pixel coordinates of the three key points.

[0014] Optionally, in any of the key point detection methods described above, the step of fusing the crop side view posture image and the corresponding key point label information in the training data set to input the encoder-decoder for model training also includes the following preprocessing step of the key point coordinates, thereby converting the coordinates of the key points into a three-dimensional tensor to obtain the heat map labels corresponding to the key points: first, generating a two-dimensional matrix for each marked key point, the size of the two-dimensional matrix being the same as the resolution of the crop side view posture image; then, mapping the coordinates of the key points marked on the crop side view posture image to the two-dimensional matrix having the same size as the resolution of the crop side view posture image; finally, using Gaussian coding, determining the value of the key point in the heat map in, σ is the standard deviation of the Gaussian function, (xi, yi) represents the coordinate value of the i-th key point in the heat map, and (x, y) represents the label coordinate value of the key point in the original image.

[0015] Optionally, the key point detection method as described in any of the above, wherein the PyTorch deep learning framework builds models including: Res-Net18, UNet, HRNet48 and HRNet32 models, which use Adam and SGD optimizers, and the batch size is set to 16, 8, 6, 4, 2 according to the complexity of the model, the number of training times is set to 100, 150, 200 iterations, and the learning rate is set from 1×10 -4 Gradually decrease to 1×10 -5 ; The image size input to the model is set to 1280×1024. The heat map size is set to 1 / 2 of the input size; The decoding process of the heat map in the model specifically uses the argmax function to calculate the coordinates of the maximum value point of the predicted heat map, scale it back to the original image space, and output it; In the heat map regression process, the mean square error (MSE) loss function or the binary cross entropy loss function (BCE) is used to carry out wheat key point detection based on the heat map regression network.

[0016] Beneficial Effects

[0017] The crop posture acquisition detection device and key point detection method provided in this application, the detection device uses the vehicle chassis to realize automatic driving and tracking walking, and the image acquisition platform thereon provides support and two steering degrees of freedom for the camera, so as to realize multi-angle crop phenotypic image acquisition. This application proposes an evaluation index for wheat key point detection based on the human key point detection evaluation index, adopts a heat map regression detection algorithm to realize key point detection, and then constructs the optimal UNet for wheat posture key point detection through the PyTorch deep learning framework as the optimal network model for crop posture key point detection, and preferably adopts an improved 11 key point detection method for wheat crop phenotypic characteristics to obtain a model detection effect that is better than the original hypothesis key point. This application can effectively identify the angle of the top one, two, and three leaves in the crop posture and the length parameters of the top one, two, and three leaves, and more intuitively display the length of wheat leaves and the changing trend of leaf angles during drought stress to meet the detection needs of more parameters. This application provides a powerful tool for studying the diversified changes in wheat posture, and for identification and classification, and has high practical value.

[0018] Other features and advantages of the present application will be set forth in the following description, and in part will become apparent from the description, or may be understood by practicing the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings are used to provide a further understanding of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings:

[0020] Figure 1 is a schematic diagram of a crop posture image acquisition robot for automatically acquiring crop posture images provided in the present application;

[0021] Figure 2 The intention of this application is to remotely log in to the internal operating system of the Raspberry Pi through VNC software, remotely control and monitor the robot to obtain crop posture images, and after the image acquisition is completed, realize the transmission and automatic storage of image data, screen and analyze image data, obtain crop phenotypic parameters and perform error calculation, and establish a series of potted crop posture test databases;

[0022] Figure 3 This is a test image of crop posture image acquisition

[0023] Figure 4 This is the UNet model training process for crop posture key point detection

[0024] Figure 5 This is the fitting diagram of plant height and measured value

[0025] Figure 6 This is the fitting diagram of stem width and measured value

[0026] Figure 7 Changes in morphological parameters of each crop genotype in the control group

[0027] Figure 8 Changes of morphological parameters of each crop genotype under drought stress group

[0028] Fig. 9 Crop posture change process diagram

[0029] Fig.10 Fusion diagram of crop posture change process

[0030] Fig.11 Crop posture image acquisition robot tracking logic diagram

[0031] Fig.12 Crop height and stem width extraction process

[0032] Fig.13 Plant height extraction process

[0033] Fig.14 Improved styles for crop keypoints.

[0034] Fig.15 Reverse the dehazing algorithm process

[0035] Fig.16 Dataset expansion effect diagram

[0036] Fig.17 Visual prediction results of crop key points based on coordinate regression network

[0037] Fig.18 Crop keypoint and pose visualization prediction results based on heatmap regression network

[0038] Fig.19 Loss rate curve of H-Net model training process

[0039] Fig. 20 Crop height extraction process and parameter display interface

[0040] Fig.21 Crop stem width extraction process and parameter display interface

[0041] Fig. 22 UNet model prediction results after key point improvement DETAILED DESCRIPTION

[0042] In order to make the purpose and technical solution of the embodiment of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described in conjunction with the drawings of the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, not all of the embodiments. Based on the described embodiment of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art to which this application belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0044] The meaning of "and / or" described in this application means that the situations where each exists alone or both exist at the same time are included.

[0045] The meaning of "left and right" mentioned in this application refers to that when the user is facing the forward direction of the crop posture collection and detection equipment, the user's left is left and the user's right is right, rather than a specific limitation on the device mechanism of this application.

[0046] The term “connection” as used in this application may mean a direct connection between components or an indirect connection between components via other components.

[0047] The meaning of "up and down" mentioned in this application refers to that when the user is facing the forward direction of the crop posture collection and detection device, the direction from the ground to the top of the crop posture collection and detection device is up, and the opposite is down, rather than a specific limitation on the device structure of this application.

[0048] Posture collection and key point detection are widely used in human and animal phenotyping studies. However, research on key point detection of plant posture is limited due to the lack of labeled datasets. Traditional crop posture phenotyping studies mostly rely on manual collection and recognition, which has problems such as low efficiency and low accuracy.

[0049] The recognition and detection of crop posture can be evolved from human posture detection technology. Human posture refers to the fixed posture formed by the joints, limbs and torso of the human body at a certain moment. Human posture key point detection is posture estimation, which consists of two parts: key point detection and skeleton connection between key points. Its main purpose is to accurately and quickly locate and identify the key points of the human skeleton and connect the skeleton between key points, so as to depict the shape of the human body. For crops, the movement of their nutritional, reproductive and other organs is the result of the integration of complex traits reflected by endogenous growth and development, and is controlled by multiple influencing factors such as the biological clock and the external environment. Over a period of time, different environmental conditions cause crops to undergo corresponding morphological changes. Similar to human posture estimation, the purpose of crop posture analysis is to represent the current morphology of the crop by the detected key points of each crop organ, so as to obtain the corresponding morphological parameters. Therefore, crop posture analysis based on machine vision is of great significance. The crop posture acquisition and detection equipment provided in this application has a main structure mainly composed of... Figure 3 The upper right side is composed of a vehicle chassis and an image acquisition platform installed on the vehicle chassis. Among them, the vehicle chassis adopts a wheeled chassis, which is divided into a control layer and a guide power layer from top to bottom. The main structure of the wheeled chassis includes two steel plates, which are arranged as a control layer and a guide power layer from top to bottom. Among them, the steel plate of the guide power layer is equipped with a drive system, and the bottom of the steel plate of the guide power layer is equipped with a tracking system and sensors are arranged at the bottom of the chassis; the steel plate of the control layer is equipped with a control unit for controlling the operation of the drive system and transmitting and processing the crop phenotype images acquired by the image acquisition platform. Each sensor is placed under the chassis. Among them, the infrared tracking sensor is fixed at the front end below the chassis to detect the thermal radiation absorbed by the black guide line laid on the road surface where the crop posture acquisition detection equipment runs, and thereby determine the position information between the installation position of the infrared tracking sensor in the center of the vehicle body and the black guide line, so that the robot can smoothly complete the tracking movement of approximate straight lines and turns along the black tape guide magnetic strip on the pre-laid foam board.

[0050] In the image acquisition platform, a USB camera performs image acquisition, a camera bracket provides support for the camera and reserves two steering degrees of freedom, and two servos are connected to the bracket to provide steering power in two degrees of freedom directions, so as to realize multi-angle crop phenotyping image acquisition. Specifically, the camera bracket can be configured as a first deflection degree of freedom turntable with an axis perpendicular to the plane of the wheeled chassis steel plate as the rotation center, and a second pitch freedom turntable installed on the first deflection degree of freedom turntable and configured to have an axis parallel to the plane of the wheeled chassis steel plate as the rotation center, the USB camera is installed on the second pitch freedom turntable, and the first deflection degree of freedom turntable and the second pitch freedom turntable are provided with steering power in two degrees of freedom directions by two servos, respectively, to realize multi-angle crop phenotyping image acquisition.

[0051] After receiving the crop phenotype image, the control unit in the crop posture acquisition and detection device first transmits and automatically stores the image data, and then triggers the screening and analysis of the image data in the PC or edge computing processing module according to the system settings to obtain the morphological parameters of ear length, leaf length, and leaf angle. The specific processing process can use the PyTorch deep learning framework to build a model, use the deep neural network to identify and predict the key points of individual crops through the heat map regression detection algorithm, find the nonlinear mapping relationship with the minimum loss function, and obtain the position coordinates of the maximum value of the heat map in each crop posture key point connection image. The position coordinates of the maximum value of the heat map are converted into key point coordinates through the heatmap_to_numpy function, and then according to the crop morphological parameter calculation formula, the morphological parameters of ear length, leaf length, and leaf angle are calculated according to the coordinates of each key point. The form of the heat map network can make network learning easier and prediction more accurate, and provide a more effective solution to the problem of crop phenotype image occlusion affecting detection. Similar to the semantic segmentation network, the heatmap regression network is a fully convolutional neural network consisting of an input layer, an encoder, a decoder, and an output layer, where the encoder is the downsampling part of the network and the decoder is the upsampling part of the network.

[0052] Since the output of the heat map regression network is a three-dimensional tensor with a shape of N×H×W, where N is the number of channels of the heat map, that is, the number of key points, and H and W are the height and width of the heat map, respectively. The original label is a wheat key point coordinate vector with a shape of 2N, which makes it impossible to directly input the original label into the heat map regression network. Therefore, the heat map regression network needs to first pre-process the key point coordinates before training the model based on the marked key points. By converting the key points from the form of a one-dimensional coordinate vector into the form of a three-dimensional tensor, the heat map label is obtained, so that the heat map regression network model of this application can be correctly trained. The preprocessing process includes the following three steps:

[0053] (1) Generate a two-dimensional matrix for each wheat label key point. The matrix size is generally the same as the resolution of the input wheat image.

[0054] (2) Map the horizontal and vertical coordinates of the key points on the input wheat image to the matrix prepared in the first step. Since the corresponding scale is adjusted before the input wheat image, the output key point coordinates are usually floating point numbers. Before establishing the mapping, the output floating point numbers need to be approximated and rounded, which is also the main source of prediction error of the heat map regression method.

[0055] (3) There are many forms of heat map encoding, which are usually expressed as center attenuation, that is, the heat value is the highest at the key point, and the farther away from the key point, the lower the heat value. The heat map of wheat key point detection in this paper adopts Gaussian encoding. For a wheat image containing N key points, let (xi, yi) represent the coordinate value of the i-th key point in the heat map, and (x, y) represent the label coordinate value on the original image. Then the expression of Gaussian distribution R(x, y) is shown in Formula 1:

[0056]

[0057] Among them, σ is the standard deviation of the Gaussian function, which represents the amplitude of the Gaussian distribution. The value Hi of the key point in the heat map is expressed as shown in Formula 2:

[0058]

[0059] The heat map in this paper has the same size as the original image. As can be seen from Formula 1, when (x, y) = (xi, yi), the key point heat map takes the maximum value on its corresponding original image, and finally transforms the two-dimensional posture image into the corresponding coordinate vector. The commonly used conversion method is to take the position of the maximum value of each heat map as the coordinate vector of the corresponding key point. The core part of heat map decoding is to use the argmax function to find the coordinates of the predicted heat map maximum value point, scale it back to the original image space, and get the final task output.

[0060] In the heatmap regression model used in this application, the loss function can be the mean square error (MSE) loss function and the binary cross entropy loss function (BCE). In the field of face key point detection and human and animal posture estimation, the MSE loss function performs better than the BCE loss function in the key point detection algorithm of the heatmap regression network. Therefore, this paper uses the MSE loss function to carry out wheat key point detection based on the heatmap regression network.

[0061] In specific practice, this application can meet the needs of crop posture collection and key point detection by allowing the robot to track and walk to collect side-view posture images of potted crops during the walking process, thereby achieving high-throughput acquisition of side-view posture images of crops from the flowering stage to the maturity stage.

[0062] In practice, we compared the Res-Net18 network model for coordinate regression and the UNet, HRNet32, and HRNet48 network models for heat map regression, and found that UNet is the optimal network model for crop posture key point detection. Therefore, this application can further expand and optimize the UNet key point detection model in the following way to analyze the change rules of morphological parameters of three crop genotypes under drought and non-drought conditions.

[0063] Analysis shows that compared with the Res-Net18 network model of coordinate regression and the UNet, HRNet32 and HRNet48 network models of heat map regression, the optimal model mAP of crop posture key point detection based on the coordinate regression network and the heat map regression network can reach 38.15% and 90.65% respectively; the UNet model mAP after expanding the key points reaches 94.72%, proving that the improved model can more effectively detect crop posture key points. By identifying crop posture through key point detection, it is found that drought stress has a greater impact on the leaf angle of crops, but has no obvious effect on ear length and leaf length. Provides a new method for non-invasive analysis of crop physiological status.

[0064] Specific reference Figure 1 The robot shown in the figure is used for automatically collecting crop posture images, and its overall structure is mainly composed of a vehicle-mounted chassis and an image collection platform located at the upper end.

[0065] The vehicle chassis adopts a wheeled chassis, which is divided into a control layer and a guiding power layer from top to bottom. The chassis body is made of two steel plates. The drive system and tracking system are arranged at the bottom of the chassis, and the sensors are placed under the chassis. Among them, the infrared tracking sensor is fixed at the front end below the chassis, so that the robot can smoothly complete the tracking movement of approximate straight lines and turns along the black tape guide magnetic strip on the pre-laid foam board.

[0066] Among them, in the image acquisition platform, a USB camera is used for image acquisition, the camera bracket provides support for the camera and reserves two steering degrees of freedom, and two servos are connected to the bracket to provide steering power in the directions of two degrees of freedom respectively, so as to realize multi-angle crop phenotype image acquisition.

[0067] In one implementation, the present application can write crop posture image acquisition robot control software based on Python's Tkinter library function, and use the robot's control interface as Figure 1 As shown in the middle, it includes the login interface and the control interface. The operation process of the software system is as follows:

[0068] First, open the control software and enter the login interface. Enter the control interface by entering the correct account and password. If the account or login password is entered incorrectly, a dialog box will pop up with the message "The account or login password is entered incorrectly!" Click "OK" and re-enter the account and password. Figure 1 shown.

[0069] After successfully logging in, you will enter the control interface, which is mainly divided into two parts: the left side is for robot motion control, and the right side is for camera control. The motion control part mainly controls the robot's forward, backward, and stop movements, as well as the rotation of the camera gimbal's servo. Fig.11In the servo drive mode shown, the operator can click "forward" according to the needs of image acquisition, and the program calls the corresponding function of the button to drive the servo at the corresponding position of the body to control the robot to move forward; click "backward" to make the robot move backward, and the program calls the corresponding function of the button to drive the servo at the corresponding position of the body to control the robot to move backward; click "turn left" or "turn right", and the robot will be driven by the corresponding servo to turn left or right on the spot. Click "stop" to stop the motor and the robot stops moving. In the servo control, the four buttons of "lift", "lower", "turn left" and "turn right" control the corresponding servo to rotate 5° in the specified direction each time the corresponding button is clicked with the mouse. After clicking the "reset" button, the two servos return to the initial position specified by the system.

[0070] The right side of the control interface is the camera control. The exposure and brightness parameters of the camera can be selected by sliding the slider in the upper right corner of the interface; the image save location is entered through the storage path; click the "Open" button to open the camera, click the "Save" button to save the photo, and click the "Close" button to close the camera. Since this control software uses multi-threaded programming, the robot movement can be controlled while the camera is turned on without conflict.

[0071] There are "Auto" and "Quit" buttons at the bottom of the control interface. After clicking the "Auto" button, the robot enters the automatic tracking and photography mode, and the robot cannot be manually controlled at this time; after clicking the "Quit" button, the automatic tracking and photography mode is exited, and the robot can be manually controlled at this time.

[0072] After the software control system is designed, the crop posture image acquisition robot control system can automatically realize the function of collecting crop posture images in the following way:

[0073] (1) Control the robot to smoothly follow the pre-laid black guide line to complete the tracking motion of approximate straight lines and turns;

[0074] (2) Control the robot to accurately complete the functions of parking at a fixed point, turning the servo, and photographing and locating individual crops;

[0075] (3) The image acquisition unit can be controlled to adjust the camera parameters and successfully obtain crop posture images under different environments;

[0076] (4) Use a PC or laptop to connect to the Raspberry Pi WiFi and remotely log in to the Raspberry Pi's internal operating system through VNC software to remotely control and monitor the robot.

[0077] The robot remotely logs into the Raspberry Pi's internal operating system through VNC software. Figure 2The method shown is to remotely control and monitor the robot to obtain crop posture images. After the image acquisition is completed, the image data is transmitted and automatically stored, the image data is screened and analyzed, the crop phenotypic parameters are obtained, and the error is calculated. Finally, a series of potted crop posture test images are established.

[0078] The above robot is used in a drought stress test to detect the accuracy of the system in extracting morphological parameters of crop posture images before and after the test.

[0079] Three wheat genotypes (Ruihuamai 523, Xumai 33, Jimai 22) were planted in the experimental base. Drought stress was applied from April 21, 2021. Eight pots of each genotype were selected, including 4 pots of drought stress (DS) and 4 pots of control group (CK). Because wheat is a multi-tillering plant, generally a single wheat seed can produce 1-5 effective tillers and 2-6 wheat ears. Therefore, in this experiment, the excess tillers were cut off after the wheat entered the flowering period on April 21, 2021, and only one wheat plant was retained in each pot to ensure that the collected experimental data set was a single wheat plant image set. The test wheat was potted on November 17, 2020, and wheat posture images were collected by robot on April 21, 22, 23, 26, 27, 28, and May 1, 4, 6, and 7, 2021.

[0080] Figure 3 This is the test scene for wheat posture image collection. The robot moves forward along the black tape fixed on the runway through the tracking module. When the infrared sensor recognizes the tracking landmark, the robot turns off the power unit, stops moving forward, and turns on the camera system to collect wheat side posture images from multiple angles. Figure 3 (a) and (b) show the site construction and equipment debugging process when wheat was in the middle seedling stage (December 25, 2020) and the jointing stage (March 19, 2021), respectively. Figure 3 (c) On April 21, 2021, the crop posture image acquisition robot was controlled to conduct a wheat posture image acquisition experiment after the potted wheat entered the flowering period. Figure 3 In (d), the robot steering angle γ is controlled to achieve multi-angle collection of wheat side-view posture images. In order to increase the complexity of the wheat posture image dataset, a black background cloth is set as the shooting background of some potted wheat, such as Figure 3 (e) as shown.

[0081] In the experiment, based on the wheat images directly collected, the inversion defogging algorithm was further used to improve the detection effect of low-light wheat posture images. The enhancement principle is to perform inversion, defogging and enhancement on the original image to obtain an enhanced image of the low-light image, which is used to strengthen the training of the model. The display effect of the enhancement processing is as follows: Fig.15 shown.

[0082] After image enhancement, some images that cannot fully display the wheat ears and the top one, two, and three leaves are removed, and 1200 images that can be used for wheat posture analysis are selected. These 1200 images are used to further expand the image data set to increase the complexity of the data set and the robustness of the deep learning model, effectively alleviating the overfitting of the model. Other expansion methods of the data set also include geometric transformation and color space transformation, which can be used for the images of this application. Among them, geometric transformation includes flipping, scaling, shifting, cropping, rotating, etc., and color space transformation includes channel separation, brightness, grayscale, saturation adjustment, etc.

[0083] This study used three external libraries, cv2, Numpy, and PIL, to implement image reading, matrix calculation, and image processing. Img.rotate, Image.FLIP_LEFT_RIGHT, ImageEnhance.Contrast, ImageEnhance.Brightness, and other functions were used to implement image rotation, flipping, brightness enhancement, and contrast enhancement. Flipping means flipping the image horizontally along the central axis, rotating means randomly rotating the image 5° to the left or right, adjusting the brightness to 1.3 times the original image, and adjusting the contrast to 1.5 times the original image. After the dataset was expanded, there were 2,400 images in total. The effect of the dataset expansion is as follows: Fig.16 shown.

[0084] After obtaining the wheat side view posture image, the present application can use the wheat posture key point detection to automatically extract the crop posture side view posture image parameters in the following manner:

[0085] 1 Label dataset construction

[0086] In this experiment, the Landmark_annotation software was used to calibrate the key points of each potted wheat side-view posture image. Wheat key points are distributed on wheat organs with clear semantic information, such as the ends of wheat ears, top leaves, top leaves, and top leaves. The extended key points can also include the middle parts of wheat ears, top leaves, top leaves, and top leaves. Wheat key point detection refers to the automatic identification and positioning of several specified wheat key points on the wheat side-view posture image through an algorithm, which can roughly reflect the geometric shape information of wheat. The processing process includes: data processing of the collected images, data enhancement operations, including flipping, rotation (±5°), brightness enhancement (+20%) and contrast enhancement (+20%), and finally a total of 2400 images for annotation are obtained. The original collected images and all the images obtained after enhancement processing are divided into training data set, verification data set and test data set according to the ratio of 8:1:1.

[0087] 2. Wheat key point detection model parameter setting

[0088] This posture evaluation test was conducted on a PC configured with NVIDIARTX3060, and the model was built using the PyTorch deep learning framework. The experiment selected Res-Net18, UNet, HRNet48 and HRNet32 models, and the Adam and SGD optimizers were used respectively. The batch size was set to 16, 8, 6, 4, and 2 according to the complexity of the model. The number of training times was set to 100, 150, and 200 iterations, and the learning rate gradually decreased from 1×10-4 to 1×10-5. The image size input to the model can be adjusted to 1280×1024. The heat map size can be set to 1 / 2 of the pixel size of the input image. (The training parameters of the four network models Res-Net, UNet, HRNet32 and HRNet48 are shown in Tables 1, 2, 3, and 4)

[0089] Table 1 ResNet18 model network training parameters

[0090]

[0091] Table 2 HRNet model network training parameters

[0092]

[0093] Table 3 UNet model network training parameters

[0094]

[0095]

[0096] Table 4 UNet model structure parameters

[0097]

[0098] 3 Model training process

[0099] This application intends to use deep neural networks to identify and predict the key points of individual wheat plants, and to find the nonlinear mapping relationship with the minimum loss function. In specific applications, the key point detection algorithm can be mainly implemented through the coordinate regression detection algorithm and the heat map regression detection algorithm. In theory, due to the discreteness of the coordinates of wheat key points, there is a certain nonlinearity when the coordinate regression network directly regresses the coordinates, which leads to problems such as model non-convergence or falling into local optimality. The heat map regression network transforms wheat key point detection into a heat map estimation problem, which can overcome the influence of nonlinearity to a certain extent and make the model more robust. Therefore, this study selected four classic representative network models, Res-Net18 for coordinate regression and UNet, HRNet32 and HRNet48 for heat map regression. Among them, the specific training process of the UNet network model, which generally has a higher accuracy in key point detection, is as follows Figure 4 As shown:

[0100] Figure 4 The wheat posture key point detection UNet model training process shown can fuse the original image of the training set and the corresponding key point label information, input the encoder-decoder for model training, obtain the wheat key point detection model and heat map information, use the validation set to verify the model fitting degree, and finally use the test set to test the model accuracy, and obtain the wheat posture key point connection image and wheat posture side view posture image parameters.

[0101] Due to the physiological phenomenon of wheat leaf curling, this application preferably uses UNet as the optimal network model for crop posture key point detection, and preferably uses an improved 11 key point detection method for wheat crop phenotypic characteristics, that is, on the basis of the ends of the wheat ear, the top leaf, the top two leaves, and the top three leaves as key points, the middle positions of the top one leaf, the top two leaves, and the top three leaves are further added as additional key points to obtain a model detection effect that is better than the original hypothesis key points. The morphological parameters of the improved model are extracted to first obtain a single wheat posture Fig.11 The position coordinates of the maximum values ​​of the heatmap are converted into the key point coordinates landmark[0]-landmark

[10] through the heatmap_to_numpy function. Among them, landmark[0] represents the key point coordinates (a, b) of the top of the ear, and landmark[1] represents the key point coordinates (c, d) of the bottom of the ear, etc. The morphological parameters such as ear length, leaf length, and leaf angle are obtained through the predicted key point coordinates and length and angle calculation formulas. Table 5 shows the calculation formulas for some morphological parameters. Line is the Euclidean distance calculation formula, which is used to calculate the distance between pixels in the image; Ang is the angle calculation formula, which is used to calculate the angle between three pixels.

[0102] Table 5 Calculation formulas for wheat morphological parameters

[0103]

[0104] In addition, this application can also be Fig.12 , Fig.13 The method is to extract the plant height and stem width of crops: first, identify the connected area formed by the overall contour of the crop phenotype image, then detect the pixel values ​​of the two connected areas, and calculate the predicted data of the plant height according to the proportional coefficient K in the following way;

[0105] The crop stem width can be measured by manually selecting the stem width area to be measured, triggering the system to identify the pixel connected area in the stem width area image, detecting the stem width pixel value corresponding to the connected area, and calculating the predicted value of the stem width according to the proportional coefficient K.

[0106] Fig.12This is the image preprocessing and crop height and stem width extraction process, including image preprocessing, image calibration separation and morphological parameter extraction. In order to distinguish the wheat posture image from the background, the crop image is first preprocessed. The grayscale of the image is processed according to the input RGB picture, and the three color components of the image RGB are converted to HSV. The wheat plant part is extracted through the HSV color component; the holes after color extraction are filled by the expansion operation, and the noise after expansion is eliminated to obtain a new binary image. Then the plant is identified, the connected domain is drawn and the number of pixels is calculated. The ratio of the actual diameter of the calibration to the number of pixels is calculated to obtain the proportional coefficient K1, the connected domain rectangle is identified, and the sum of the length of the rectangle is obtained. The number of pixels N1, the height of the wheat plant is K1×N1. When processing the stem width, it is necessary to manually select the connected domain rectangle where the stem width is to be calculated. The sum of the wide pixels in the new rectangle is N2, and the length of the wheat stem width is K1×N2. In this article, the measured height of a single wheat plant is the vertical distance from the top of the wheat ear to the top of the pot, and the actual measured value of the wheat stem width is the average of the distance from the wheat ear to the top leaf and from the top leaf to the top two leaves.

[0107] For the wheat posture key point detection model defined by the above parameters, this application can evaluate the accuracy of the model through the following steps:

[0108] Based on the human body posture evaluation method, the wheat key point similarity evaluation index OKSw (OKS For Wheat) is defined, and the expression is shown in Formula 1:

[0109]

[0110] Where: i represents the key point with serial number i; d represents the Euclidean distance between the predicted value and the true value of the wheat key point currently detected. Sw represents the scale factor of wheat, and its value is the square root of the area of ​​the wheat detection box, that is, w, h are the width and height of the detection box; σ i It is a normalization factor, which reflects the standard deviation of the current key point annotation. The larger the σ, the more difficult it is to annotate the key point.

[0111]

[0112] Through the above steps, the OKSw of the first key point is calculated, and the OKSw of the other 7 key points can be obtained in turn. The average of the 8 key points is obtained to get AvgOKSw. The threshold T is set artificially. When AvgOKSw is greater than T, it is considered to be TruePositive, otherwise it is FalsePositive. The GroundTruth key point without a match is considered to be FalseNegative, that is, AP can be obtained. The expression is shown in Formula 2, where Len represents the total number of images in the test set.

[0113] T∈{0.55:0.95:0.05}(3)

[0114] mAP is a commonly used detection indicator. Specifically, different values ​​of the artificial threshold T in the AP indicator are set. The range of the threshold change is 0.55-0.95, and multiple AP indicators are obtained. Finally, the mAP is obtained by averaging multiple AP indicators. The value range and span expression of T are shown in Formula 3. The calculation expression of mAP is shown in Formula 4, where AP55 represents the value of AP when T=0.55.

[0115]

[0116] In addition, in order to detect the posture key points of wheat, this application also proposed the hypothetical positions of 8 posture key points of a single wheat plant based on the side view morphological distribution characteristics of the physiological structure of wheat, and compared the applicability of the coordinate regression network and the heat map regression network for the wheat posture key point detection algorithm, and trained the ResNet18, UNet, HRNet32 and HRNet48 network models respectively, so as to predict and identify the key points of each posture of a single wheat plant through the network model.

[0117] The experiment found that the prediction effect of the wheat key point detection model based on the coordinate regression method is not ideal. The key point regression error of ResNet18 is large. Fig.17 shown. Fig.18 It can be seen from the figure that the error of wheat key points predicted by the UNet model is small, and it can accurately regress wheat key points and predict wheat posture under complex backgrounds with occlusion. Because the HRNet32 and HRNet48 models are more complex and require more calculations, and the regression task of the eight key points of a single wheat plant is relatively simple, the loss curve of this model is overfitting. In addition, due to Fig.19 (d) The refined curve shows that the validation set loss rate curve fluctuates greatly, and the fitting effect is worse than that of the UNet model.

[0118] Therefore, this application can draw the following conclusions based on the above identification analysis:

[0119] S.1 Analysis of the test results of the navigation accuracy of the crop phenotype image acquisition robot

[0120] The indoor image acquisition robot detects the position information between the center of the vehicle body and the black guide line through four tracking infrared sensors, and can accurately control the motion trajectory of the system on hard ground. The two different motion state tests of straight-moving and turning carried out in the embodiments of this application respectively test the motion speed and running posture control effect of the platform to verify its operation reliability. The motion trajectory of the robot at three speeds of 0.2m / s, 0.4m / s, and 0.6m / s is recorded with a marker pen, and a measurement is taken every 10cm to record the distance between it and the black guide line. 20 measurements are made at each speed. The measurement results are shown in Table 6. The deviation can meet the requirements of crop phenotypic image acquisition and detection.

[0121] Table 6 Test results of tracking robot navigation accuracy

[0122]

[0123] S.2 Analysis of image acquisition accuracy test results

[0124] This application draws a scatter plot of image acquisition accuracy based on the predicted values ​​of potted wheat plant height and stem width and the manually measured values ​​(such as Fig. 20 , 21), the linear regression correlation coefficients of the extracted values ​​of wheat plant height and stem width and the artificial measured values ​​were analyzed, and the correlation fitting images of the predicted values ​​and the true values ​​of plant height and stem width were obtained. Figure 5 , Figure 6 As shown in the figure, the linear regression correlation coefficients (R) of plant height and stem width reached 84.6% and 79.2%, respectively, indicating that the predicted values ​​of plant height and stem width of the three genotypes were close to the true values, the fitting function was significantly positively correlated, and the accuracy was high, indicating that the method of using the crop posture image acquisition robot to obtain the posture images and parameters of potted wheat in a certain area is feasible.

[0125] S.3 Analysis of the detection results of wheat key point detection model

[0126] The evaluation criteria for the quality of the model include main factors such as accuracy parameter mAP, complexity and running time. Through two regression methods, coordinate regression and heat map regression, the input size, calculation amount, parameter amount and mAP of four models, Res-Net, UNet, HRNet32 and HRNet48, are compared, as shown in Table 7. It is slightly difficult for Res-Net to directly learn the absolute values ​​of the coordinates of wheat key points using the coordinate regression method, with an accuracy of 38.15%, which is significantly behind the heat map regression method in performance. For the heat map regression method, the mAP of the HRNet model is as high as 84.54%, while the accuracy of the UNet model is as high as 90.65%, indicating that the HRNet model structure is more complex but is not suitable for the key point regression model of potted single wheat plants. Based on the above analysis, the heat map regression network is more suitable for wheat posture evaluation than the coordinate regression network. In this application, the UNet model can be used to predict wheat key points.

[0127] Table 7 Comparison of performance of various models for wheat key point detection

[0128]

[0129] Due to the physiological phenomena of wheat leaf curling, this application can further change the 8 key point model (UNet) to an 11 key point model (UNet*), and perform OKSw accuracy tests on the improved trained models. The test results are shown in Table 3. Compared with the original key points under the same parameters, the model accuracy of the improved key points is increased from 90.65% to 94.72%, which is better than the model detection effect of the original hypothesis key points.

[0130] S.4 Analysis of wheat morphological parameter extraction results

[0131] The 2400 wheat posture image datasets were divided into six categories: JI22DS, JI22CK, RH523DS, RH523CK, XU33DS, and XU33CK. The images in each category were divided into 1, 2, 3, 6, 7, 8, 9, 12, 14, and 15 days in the order of acquisition time. The optimal network model after key point improvement was used to predict the posture of wheat and extract the angle and length of the top one, two, and three leaves. The box plots and line graphs of the changes in morphological parameters of each wheat genotype in the control group and drought stress group were drawn, as shown in the figure. Figure 7 , 8 As shown. The horizontal axis of the figure is the treatment time (1-15 days), the vertical axis of the box plot is the leaf angle, and the vertical axis of the line graph is the leaf length. The length of the box plot and the shaded area of ​​the line graph represent the range of changes in the leaf angle and leaf length, respectively. The unit of the leaf angle in the figure is degree, and the unit of the leaf length is pixel.

[0132] Reference Figure 7 , Figure 8, with the increase of time, there was no obvious change trend in the leaf length of the three genotypes of potted wheat under non-drought stress environment; under drought stress environment, the length of the top leaf of the three genotypes of potted wheat changed significantly, and there was no obvious change trend in the top two and three leaves. As for the angle, the angle of the top one and three leaves of Xumai 33 drought stress group increased significantly, and there was no obvious change trend in the leaf angle of the control group; the angle of the top one and two leaves of Ruihuamai 523 drought stress group increased significantly compared with the angle of the top one leaf of the control group, and there was no obvious change trend in the others; the angle of the top two leaves of Jimai 22 drought stress group increased significantly, and there was no obvious change trend in the others. Based on the above analysis, compared with the control group, the change trend of the length of the top leaf and the angle of most leaves of the three genotypes of wheat under drought stress environment was more obvious.

[0133] Furthermore, in order to visualize the posture changes of a single potted wheat plant, the present application can further select two stress states of Xumai 33 genotype, DS (drought stress) and CK (control group), and plant them together, and collect their posture changes when the drought lasts for 1-15 days to obtain Fig. 9 Image shown. Fig. 9 In the drought stress group, the angles of the top one and three leaves increased significantly with the increase of time, while the angle of the top two leaves did not change significantly; in the control group, the angles of the top one, two, and three leaves did not change significantly. In order to more intuitively see the changing trend of wheat leaf angles during drought stress, Fig. 9 The wheat posture changes of the two categories of XU33DS and XU33CK are fused, and different RGB colors are used to represent the acquisition time sequence of the wheat posture images. In order to avoid occlusion, the radius of the original key points and the width of the connecting line can be further reduced to obtain Fig.10 The fusion diagram of the posture change process is shown. Fig.10 The results showed that as time went by, the ears and the top one, two, and three leaves all shifted to a certain extent. The angles of the top one and three leaves of wheat in the drought stress group increased to a large extent. After the 12th day of drought stress, the increase in the angles of the top one and three leaves accelerated; the angles of wheat leaves in the control group increased slowly in chronological order, but there was no significant change overall.

[0134] In the wheat posture image dataset, there are certain shooting angles where the three leaves of wheat are not in the ideal straight state, but are bent in the middle, such as Fig.14 (a). Therefore, the line connecting the top and tail of the leaf cannot accurately represent the posture of this wheat plant. Therefore, the present application can also select 1200 single wheat plant data sets in the training image during the training process and mark key points at the tip of the spike, the tail of the spike, the tip of the top leaf, the middle of the top leaf, the tail of the top leaf, the tip of the top second leaf, the middle of the top second leaf, the tail of the top second leaf, the tip of the top third leaf, the middle of the top third leaf, and the tail of the top third leaf, and produce a label data set with improved key points for model training. The label point marking style is as follows: Fig.14 (b) as shown.

[0135] Table 8 UNet network parameter settings after key point improvement

[0136]

[0137] Load the training model after key point improvement, the improved model network parameters are shown in Table 8, and the key point prediction visualization image is obtained as follows Fig. 22 As shown. The model trained with the improved key points was tested for OKSw accuracy, and the results are shown in Table 9. Compared with the original key points under the same parameters, the model accuracy of the improved key points increased from 90.65% to 94.72%, and the accuracy of the three parameter adjustment models reached more than 93%, which is significantly better than the model detection effect of the original hypothesis key points. Therefore, the subsequent posture analysis and parameter extraction in this paper are based on the improved key point detection model.

[0138] Table 9 Model accuracy test results after key point improvement

[0139]

[0140] In summary, this application can accurately detect the posture of wheat by transplanting the application of face key point detection to the detection of plant posture key points, and improving the input, output and loading model in the algorithm structure. After the parameter extraction algorithm provided by this application was tested, it was found that the correlation between the predicted value of wheat plant height and stem width and the true value of manual collection reached 84.6% and 79.2% respectively. The results of SPSS main effect variance analysis of wheat plant height and stem width showed that the actual measured value of plant height and stem width was significantly correlated with the predicted value. Therefore, the crop posture image acquisition robot provided by this application is feasible for the collection box detection method of the posture image of potted wheat. This application realizes the defogging enhancement of low-light wheat posture images based on the preprocessing algorithm, and expands the wheat data set through flipping, rotation, brightness enhancement and contrast enhancement algorithms. The obtained data set can be used for wheat posture key point detection and semantic segmentation model, and obtains more accurate detection results, which can be applied to large-scale field phenotype collection scenes, and can effectively improve the work efficiency of crop phenotype collection.

[0141] The crop posture image acquisition robot provided in this application can automatically acquire wheat side posture images through trajectory recognition technology, and detect wheat posture key points by improving the optimal network model UNet to analyze the changing trend of wheat posture under drought stress. The above-mentioned crop posture acquisition detection equipment and key point detection method provided in this application can:

[0142] (1) Sufficient automatic navigation and image acquisition accuracy are provided by the crop posture image acquisition robot control software. The robot of the present application preferably has a straight-line tracking deviation of less than 3 mm and an image acquisition efficiency of about 18 images per minute when traveling in a straight line at a speed of 0.4 m / s in the test site. In the single-plant wheat posture image acquisition test, after detection by the parameter extraction algorithm, it was found that the correlation between the predicted values ​​of wheat plant height and stem width and the actual values ​​collected manually reached 84.6% and 79.2%, respectively. Therefore, it is feasible to use the crop posture image acquisition robot to collect posture images of potted wheat.

[0143] (2) In order to realize the key point detection of wheat posture, this application also proposes the hypothetical positions of 8 key points of the posture of a single wheat plant based on the side view morphological distribution characteristics of the physiological structure of wheat. It quotes the human body key point evaluation index OKSp and proposes the wheat key point evaluation index OKSw. This application compares the accuracy of the Res-Net18 network model of coordinate regression and the UNet, HRNet32 and HRNet48 network models of heat map regression. The results show that the coordinate regression network has lower accuracy than the heat map regression network. The optimal model mAP based on the coordinate regression network reaches 38.15%, while the optimal model mAP for the heat map regression network reaches 90.65%. The optimal network model UNet for wheat posture key point detection is selected.

[0144] (3) Due to the physiological phenomena of wheat such as leaf curling, the present application can further improve the 8 key points in the most preferred network model UNet to 11 key points, and adjust its 3 hyperparameters. Thus, after the key point expansion, the mAP of the optimal model of the present application reached 94.72%. Based on the improved key point detection optimal model, the present application draws the box plot and line graph of the morphological parameters of each wheat genotype in the control group and the drought stress group over time, and analyzes the effect of drought stress on the leaf length and leaf angle of different genotypes of wheat. A fusion diagram of the change of wheat posture over time is drawn, which more intuitively shows the changing trend of wheat leaf length and leaf angle during drought stress.

[0145] The crop posture image acquisition robot developed by the present application for collecting wheat posture images can meet the collection needs of different users through the detection and parameter extraction of key points of wheat posture, and can meet the detection of more parameters through further development and optimization. This application provides a powerful tool for studying the diverse changes in wheat postures, and for identification and classification, and has high practical value.

[0146] The above is only an implementation method of the present application, and its description is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present application. It should be pointed out that for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application.

Claims

1. A crop posture collection and detection device, characterized in that: The main structure of the equipment includes: a vehicle-mounted chassis and an image acquisition platform installed on the vehicle-mounted chassis; The vehicle chassis adopts a wheeled chassis, and the main structure of the wheeled chassis includes two steel plates, which are arranged as a control layer and a guide power layer from top to bottom, wherein the steel plate of the guide power layer is equipped with a driving system, the bottom of the steel plate of the guide power layer is equipped with a tracking system and a sensor, and the steel plate of the control layer is equipped with a control unit for controlling the operation of the driving system and transmitting and processing the crop phenotype images acquired by the image acquisition platform; The image acquisition platform is electrically connected to the control unit in the control layer, a USB camera is provided in the image acquisition platform for image acquisition, and a camera bracket is also provided in the image acquisition platform, the camera bracket has a first deflection degree of freedom turntable with an axis perpendicular to the plane of the wheeled chassis steel plate as a rotation center, and a second pitch freedom degree turntable installed on the first deflection degree of freedom turntable and arranged to have an axis parallel to the plane of the wheeled chassis steel plate as a rotation center, the USB camera is installed on the second pitch freedom turntable, and the first deflection degree of freedom turntable and the second pitch freedom turntable are respectively provided with steering power in two degrees of freedom directions by two steering gears to realize multi-angle crop phenotype image acquisition; The control unit also transmits and automatically stores the image data after receiving the crop phenotype image, so as to trigger screening and analysis of the image data to obtain morphological parameters of ear length, leaf length, and leaf angle; After receiving the crop phenotype image, the control unit triggers the following screening and analysis steps on the image data to obtain the crop posture key point connection image and the crop posture image morphological parameters: The model was built using the PyTorch deep learning framework. The key points of individual crops were identified and predicted through the heat map regression detection algorithm using a deep neural network. The nonlinear mapping relationship with the smallest loss function was found to obtain the position coordinates of the maximum value of the heat map in each crop posture key point connection image. The heatmap_to_numpy function was used to convert the position coordinates of the maximum value of the heat map into key point coordinates. Then, according to the crop morphological parameter calculation formula, the morphological parameters of ear length, leaf length, and leaf angle were calculated based on the coordinates of each key point.

2. The crop posture collection and detection device according to claim 1, characterized in that: The sensor includes an infrared tracking sensor, which is fixed at the front end below the chassis and is used to detect the thermal radiation absorbed by the black guide line laid on the road surface where the crop posture collection and detection equipment runs, and thereby determine the position information between the installation position of the infrared tracking sensor in the center of the vehicle body and the black guide line, so that the driving system of the crop posture collection and detection equipment can track and run along the black guide line pre-laid on the foam board.

3. A crop posture key point detection method, characterized in that: The image data collected by the crop posture collection and detection device according to claim 1 is subjected to the following screening and analysis steps to obtain a crop posture key point connection image and crop posture image morphological parameters: The model was built using the PyTorch deep learning framework. The key points of individual crops were identified and predicted through the heat map regression detection algorithm using a deep neural network. The nonlinear mapping relationship with the smallest loss function was found to obtain the position coordinates of the maximum value of the heat map in each crop posture key point connection image. The heatmap_to_numpy function was used to convert the position coordinates of the maximum value of the heat map into key point coordinates. Then, according to the crop morphological parameter calculation formula, the morphological parameters of ear length, leaf length, and leaf angle were calculated based on the coordinates of each key point.

4. The crop posture key point detection method according to claim 3, characterized in that: The model trained by the following steps is used to identify and extract crop morphological parameters: Landmark_annotation software was used to calibrate the key points of each crop side-view posture image, and each image was classified into training data set, validation data set and test data set according to a fixed ratio; Then, on a computer configured with NVIDIA RTX3060, the PyTorch deep learning framework was used to build a heat map network model, the crop side view posture images and the corresponding key point label information in the training dataset were fused, and the encoder-decoder was input for model training to obtain the key point detection model and heat map information. The validation dataset was used to verify the fit of the key point detection model and adjust the network parameters accordingly. Finally, the test dataset was used to test the accuracy of the key point detection model, and the nonlinear mapping relationship with the minimum loss function was obtained to complete the model training.

5. The key point detection method as claimed in claim 4, characterized in that: After calibrating the key points of each crop side-view posture image using the Landmark_annotation software, each image is subjected to any of the following enhancement operations: flipping, rotation, brightness enhancement, and contrast enhancement. The images obtained by the enhancement operations are classified into the training data set, the validation data set, and the test data set in the same fixed ratio.

6. The key point detection method according to claim 4, characterized in that: The key points of the side-view posture image of the potted crop include: the endpoint coordinates and / or the central coordinates of the ear, the top leaf, the top two leaves, and the top three leaves.

7. The key point detection method according to claim 6, characterized in that: The crop posture image morphological parameters include: the angle of the top one, second and third leaves and / or the length of the top one, second and third leaves; Let landmark[0] represent the key point coordinates of the top of the ear, landmark[1] represent the key point coordinates of the bottom of the ear, landmark[2] represent the key point coordinates of the top of the first lobe, landmark[3] represent the key point coordinates of the middle of the first lobe, landmark[4] represent the key point coordinates of the bottom of the first lobe, landmark[5] represent the key point coordinates of the top of the second lobe, landmark[6] represent the key point coordinates of the middle of the second lobe, landmark[7] represent the key point coordinates of the bottom of the second lobe, landmark[8] represent the key point coordinates of the top of the third lobe, and landmark[9] represent the key point coordinates of the middle of the third lobe. Landmark[10] represents the coordinates of the key points at the bottom of the top trefoil; Then, the calculation formula of the morphological parameters of ear length is Line(landmark[0],landmark[1]); The formula for calculating the morphological parameters of the top lobe length is Line(landmark[2],landmark[3])+Line(landmark[3],landmark[4]); The formula for calculating the morphological parameters of the length of the second apical lobe is Line(landmark[5],landmark[6])+Line(landmark[6],landmark[7]); The morphological parameter calculation formula for the length of the top three leaves is Line(landmark[8],landmark[9])+Line(landmark[9],landmark[10]); The formula for calculating the morphological parameters of the angle between the top and the first leaf is ng(landmark[1], landmark[4], landmark[3]); The formula for calculating the morphological parameters of the angle between the two top leaves is Ang(landmark[4],landmark[7],landmark[6]); The formula for calculating the morphological parameters of the angle of the apical trefoil is Ang(landmark[7],landmark[10],landmark[9]); Among them, Line is the Euclidean distance calculation formula, which is used to calculate the distance between the pixel coordinates of the key points in the image; Ang is the angle calculation formula, which is used to calculate the angle between the pixel coordinates of the three key points.

8. The key point detection method according to claim 7, characterized in that: The step of fusing the crop side view posture image and the corresponding key point label information in the training data set to input the encoder-decoder for model training also includes the following preprocessing step of the key point coordinates, thereby converting the coordinates of the key points into a three-dimensional tensor to obtain the heat map labels corresponding to the key points: First, a two-dimensional matrix is ​​generated for each marked key point, and the size of the two-dimensional matrix is ​​the same as the resolution of the crop side view posture image; Then, mapping the coordinates of the key points marked on the crop side view posture image to the two-dimensional matrix having the same size as the resolution of the crop side view posture image; Finally, Gaussian coding is used to determine the value of the key point in the heat map in, σ is the standard deviation of the Gaussian function, (xi, yi) represents the i-th The coordinate value of a key point in the heat map, (x, y) represents the label coordinate value of the key point in the original image.

9. The key point detection method according to claim 8, characterized in that: The PyTorch deep learning framework builds models including Res-Net18, UNet, HRNet48, and HRNet32 models. The Adam and SGD optimizers are used. The batch size is set to 16, 8, 6, 4, and 2 according to the complexity of the model. The number of training times is set to 100, 150, and 200 iterations, and the learning rate is set from 1×10 -4 Gradually decrease to 1×10 -5 ; The image size input to the model is set to 1280×1024; the heat map size is set to 1 / 2 of the input size; the decoding process of the heat map in the model specifically uses the argmax function to calculate the coordinates of the maximum value point of the predicted heat map, scale it back to the original image space, and output it; In the heat map regression process, the mean square error (MSE) loss function or the binary cross entropy loss function (BCE) is used to carry out wheat key point detection based on the heat map regression network.

Images