Method, device and agricultural machinery for seedling line recognition based on deep learning

Abstract

The embodiment of the present application provides a method, device and agricultural machinery for seedling line recognition based on deep learning, the method comprising: obtaining a first picture, and generating a second picture based on the first picture, the first picture being a picture taken by a camera of the agricultural machinery; processing the second picture using a pre-trained neural network model to generate seedling lines; calculating the spatial relationship of each seedling line in the first picture based on the coordinates of the seedling lines to obtain the first seedling line and the second seedling line, the first seedling line and the second seedling line being the two most central seedling lines in the seedling lines in the first picture respectively; generating a navigation line based on the first seedling line and the second seedling line, the horizontal coordinate of the navigation line being the average of the horizontal coordinates of the first seedling line and the horizontal coordinates of the second seedling line. The solution provided by the present application can take into account both the speed and accuracy of seedling line recognition.

Description

Technical Field

[0001] The embodiments of the present application relate to the agricultural field, and more specifically, to a method, device and agricultural machinery for seedling line identification based on deep learning. Background Art

[0002] With the development of science and technology, agricultural mechanization is becoming more and more intelligent. Automatic navigation technology of agricultural machinery is a key technology in precision agriculture. Especially when agricultural machinery is operating in the field, the identification of seedling lines is crucial.

[0003] In the prior art, the seedling line identification method is relatively complex and time-consuming, thereby reducing the speed of seedling line identification. Therefore, how to provide a seedling line identification method that can balance speed and accuracy is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The present application provides a method, device and agricultural machinery for seedling line identification based on deep learning, which can take into account both the speed and accuracy of seedling line identification.

[0005] In a first aspect, a method for seedling line recognition based on deep learning is provided, comprising: obtaining a first picture, and generating a second picture based on the first picture, the first picture being a picture taken by a camera of agricultural machinery; processing the second picture using a pre-trained neural network model to generate seedling lines; calculating the spatial relationship of each seedling line in the first picture based on the coordinates of the seedling lines to obtain a first seedling line and a second seedling line, the first seedling line and the second seedling line being the two centermost seedling lines in the first picture respectively; generating a navigation line based on the first seedling line and the second seedling line, the horizontal coordinate of the navigation line being the average of the horizontal coordinates of the first seedling line and the second seedling line.

[0006] The solution provided in this application is based on a pre-trained neural network model, which can correctly and quickly identify the seedling lines during the driving process of agricultural machinery, and output navigation lines according to the identified seedling lines, so that the agricultural machinery can travel according to the output navigation lines, thereby avoiding damage to crops.

[0007] In combination with the first aspect, in some possible implementations, generating a navigation line based on the first seedling line and the second seedling line includes: when the number of crop strips between the two wheels of the agricultural machine is an odd number, generating the navigation line located in the middle of the crop strip, and the first seedling line and the second seedling line are two seedling lines belonging to the same crop strip; when the number of crop strips between the two wheels of the agricultural machine is an even number, generating the navigation line located in the middle of the ridges of two adjacent crop strips, and the first seedling line and the second seedling line are seedling lines adjacent to the ridges of the two adjacent crop strips. By setting the position of the generated navigation line according to the number of crop strips between the two wheels of the agricultural machine, it is possible to avoid generating an erroneous navigation line due to the offset of the agricultural machine's camera or the deviation of the agricultural machine.

[0008] In conjunction with the first aspect, in some possible implementations, generating the second image based on the first image includes: reducing the first image to the second image according to a first ratio; calculating the spatial relationship of each seedling line in the first image according to the coordinates of the seedling line includes: after generating the seedling line, restoring the coordinates of the seedling line in the second image to the coordinates of the seedling line in the first image according to a second ratio, the first ratio corresponding to the second ratio. In this way, the reduced image is close to the original image in ratio, and restoring it according to the second ratio corresponding to the first ratio can reduce the deviation of the seedling line recognition result; at the same time, using the reduced second image for calculation can reduce the amount of calculation in the seedling line recognition process and save computing resources.

[0009] In conjunction with the first aspect, in some possible implementations, before training the neural network model, the method further includes: obtaining a third image; adding a label to the third image, and converting the labeled third image into a fourth image, wherein the fourth image is a grayscale image of the labeled third image, wherein the label corresponds to the spatial position of the seedling line in the third image. In this way, the coordinate values of the seedling line in the fourth image can be used as a reference for the coordinate values of the seedling line output by the neural network model during training.

[0010] In combination with the first aspect, in some possible implementations, before using a pre-trained neural network model to process the second picture to generate the seedling line, the method also includes: training the neural network model to obtain the pre-trained neural network model; wherein, training the neural network model includes: obtaining a third picture, reducing the third picture to a fifth picture according to a first ratio; using a backbone network to extract features of the fifth picture to obtain a first feature map; converting the first feature map into the coordinates of the seedling line; and correcting the neural network model using a loss function.

[0011] In combination with the first aspect, in some possible implementations, the backbone network includes: a first input layer, the first input layer being configured to process the fifth image using a convolution module to obtain a second feature map; a first intermediate layer, the first intermediate layer being configured to process the second feature map using a depthwise separable convolution module to obtain a third feature map; a first output layer, the first output layer being configured to process the third feature map using a convolution module to obtain a first feature map; a second intermediate layer, the second intermediate layer being configured to process the fourth feature map using a depthwise separable convolution module to obtain a fifth feature map; a second output layer, the second output layer being configured to process the fifth feature map using a convolution module to obtain a sixth feature map; a third intermediate layer, the third intermediate layer being configured to process the seventh feature map using a depthwise separable convolution module to obtain an eighth feature map; and a third output layer, the third output layer being configured to process the eighth feature map using a convolution module to obtain a ninth feature map. This backbone network structure can speed up the operation of the neural network model and can be deployed on embedded devices and is compatible with embedded devices.

[0012] In combination with the first aspect, in some possible implementations, the input layer, the first intermediate layer and the first output layer are configured to infer the seedling line; the second intermediate layer, the second output layer, the third intermediate layer and the third output layer are configured to participate in the auxiliary training of the seedling line.

[0013] In conjunction with the first aspect, in some possible implementations, the backbone network includes five downsampling steps, thereby further accelerating the operation speed of the neural network model.

[0014] In combination with the first aspect, in some possible implementations, the first intermediate layer includes three depth-separable convolution modules, and the second intermediate layer and the third intermediate layer each include two depth-separable convolution modules.

[0015] In conjunction with the first aspect, in some possible implementations, the depthwise separable convolution module includes: a feature expansion layer, configured to use a 1*1 convolution kernel to expand the number of feature maps to 6 times the original number; a depthwise convolution layer, configured to perform convolution on a single channel; a pointwise convolution layer, configured to traverse all channels using a 1*1 convolution kernel; and a direct connection layer, configured to connect input and output using a residual structure. Through the depthwise separable convolution module, it is possible to reduce the amount of computation while obtaining more details and improve the operating speed of the neural network model.

[0016] In combination with the first aspect, in some possible implementations, the conversion of the first feature map into the coordinates of the seedling lines includes: using a 1*1 convolution kernel to reduce the dimension of the first feature map to obtain a tenth feature map; using the first route, the second route, and the third route to extract three groups of feature points from the tenth feature map respectively; splicing the three groups of feature points with n fully connected layers to obtain n intermediate layers; connecting the n intermediate layers to the n output layers in a one-to-one correspondence to obtain an output matrix of n*h*(m+1), wherein n corresponds to the number of the seedling lines, h corresponds to the vertical coordinate of the seedling line, and m corresponds to the horizontal coordinate of the seedling line. Reducing the dimension of the first feature map to the tenth feature map can improve the inference speed of the neural network model without significantly losing accuracy; using three different routes to extract features can obtain richer features.

[0017] With reference to the first aspect, in some possible implementations, n=6.

[0018] In combination with the first aspect, in some possible implementations, the use of the first route, the second route, and the third route to extract three groups of feature points from the ninth feature map respectively includes: based on the first route, first extracting features from the vertical coordinate direction, and then extracting features from the horizontal coordinate direction; based on the second route, first extracting features from the horizontal coordinate direction, and then extracting features from the vertical coordinate direction; based on the third route, using full convolution to extract features.

[0019] In conjunction with the first aspect, in some possible implementations, the loss function satisfies:

[0020] L total =L cls +αL str +βL seg ,

[0021] in,

[0022]

[0023] L str =L sim +L shp ,

[0024]

[0025]

[0026] L cls is the classification loss function, L total is the loss function, L str is the structural loss function, L seg is the segmentation loss function, L sim is the adjacent loss function, L shpis the shape loss function, α is L str The scaling parameter, β is L seg The scaling parameter, P i,j The predicted value of the corresponding seedling line coordinate, T i,j The actual value of the corresponding seedling line coordinate, L CE is the cross entropy loss function, Loc i,j is the coordinate of the seedling point in the seedling line. Classification loss function L cls Represents the classification loss of the horizontal coordinate of the seedling point on the seedling line with the vertical coordinate j of the i-th seedling line; the adjacent loss function L sim It can be used to constrain the positions of adjacent seedling points on the same seedling line so that the coordinates of the seedling points remain continuous; the shape loss function L shp Can be used to constrain the shape of the seedling line; segmentation loss function L seg It can be used to assist in training neural network models.

[0027] In a second aspect, a device for seedling line identification is provided, comprising a module for executing the method in the first aspect or any possible implementation of the first aspect.

[0028] In a third aspect, an agricultural machine is provided, comprising: a camera for capturing images of a working field during the driving process of the agricultural machine; and a processor for controlling the camera and executing the method in the first aspect or any possible implementation of the first aspect.

[0029] In a fourth aspect, a computer-readable storage medium is provided, comprising a computer program, which, when executed on a computer device, causes a processing unit in the computer device to execute instructions of the method in the first aspect or any possible implementation of the first aspect.

[0030] In a fifth aspect, a computer program product is provided, comprising computer program instructions, which enable a computer to execute the method in the above-mentioned first aspect or each implementation of the first aspect.

[0031] In a sixth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is an example diagram of an application scenario of an embodiment of the present application;

[0033] Figure 2 This is a schematic flow chart of a method for identifying seedling lines according to an embodiment of the present application;

[0034] Figure 3 A schematic diagram of an identified seedling line according to an embodiment of the present application;

[0035] Figure 4 A schematic diagram of an output navigation line according to an embodiment of the present application;

[0036] Figure 5 is a schematic diagram of an output navigation line according to an embodiment of the present application;

[0037] Figure 6 is a schematic diagram of a navigation line offset according to an embodiment of the present application;

[0038] Figure 7 is a schematic diagram of an output navigation line according to an embodiment of the present application;

[0039] Figure 8 is a schematic diagram of a picture after adding labels according to an embodiment of the present application;

[0040] Figure 9 A schematic diagram of training a neural network model according to an embodiment of the present application;

[0041] Figure 10 is a schematic diagram of a depthwise separable convolution module according to an embodiment of the present application;

[0042] Figure 11 This is a schematic diagram of converting a characteristic map into seedling line coordinates according to an embodiment of the present application;

[0043] Figure 12 This is a flow chart of converting a first feature map into seedling line coordinates according to an embodiment of the present application;

[0044] Figure 13 Schematic diagram of a route for extracting feature points according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings.

[0046] The embodiments of the present application can be applied in field operations to identify the position of the seedling line of the operation plot and output a navigation line, so that the agricultural machinery can travel according to the navigation line.

[0047] Figure 1 This is an example diagram of an application scenario of the technical solution provided in the embodiment of this application. Figure 1 As shown, the application scenario may include at least one agricultural machine 110 and a camera 120. While the agricultural machine 110 is operating, the camera 120 captures images of the work plot, and identifies the position of the seedling line of the work plot based on the captured image of the work plot. The work plot may include one or more crop strips, and one crop strip may correspond to two seedling lines.

[0048] While the agricultural machine 110 is in motion, it typically captures several images using its camera 120. The machine then uses image recognition technology, such as neural networks, to identify the locations of seedling lines in the images. Conventional seedling line recognition methods are slow and require significant computing resources, making it difficult to achieve a balanced accuracy and speed.

[0049] The seedling line recognition algorithm in this application is based on a pre-trained neural network model. It can correctly identify the seedling lines during the driving process of agricultural machinery based on the obtained pictures, and output navigation lines based on the seedling lines. The seedling line recognition algorithm can take into account the accuracy and speed of seedling line recognition.

[0050] The following will be combined Figure 2 The method for seedling line identification of the present application is described in detail.

[0051] Figure 2 This is a schematic flow chart of a method for seedling line identification based on deep learning provided in an embodiment of the present application. Figure 2 As shown, the method 200 includes:

[0052] S210 , obtaining a first picture, and generating a second picture based on the first picture, where the first picture is a picture taken by the camera 120 of the agricultural machine 110 .

[0053] The first image may be an image of the working field captured by camera 120 of agricultural machine 110, and the second image may be an image obtained by processing the first image. For example, the second image may be an image obtained by scaling down the first image at a certain scale. Both the first image and the second image may include one or more crop strips. The second image may be an image obtained by processing the first image using an image processing tool, such as OpenCV.

[0054] S220: Process the second image using a pre-trained neural network model to generate seedling lines.

[0055] The second image is input into a pre-trained neural network model to obtain target pixels on the second image, and the seedling lines during the travel of the agricultural machine 110 are correctly identified and generated based on the target pixels. The generated seedling lines may include the following information: the number of seedling lines, and the coordinates of the seedling points in the seedling lines, such as the vertical and horizontal coordinates.

[0056] Optionally, the second image can also be input into a pre-trained neural network model in the form of data. For example, the data is data obtained after the first image is processed by an OpenCV tool, and the data includes information in the compressed first image.

[0057] S230: Calculate the spatial relationship between the seedling lines in the first picture based on the coordinates of the seedling lines to obtain a first seedling line and a second seedling line. The first seedling line and the second seedling line are respectively the two centermost seedling lines in the first picture.

[0058] Figure 3 Schematic diagram of the seedling line identified in one embodiment of the present application. Figure 3 As shown, 6 seedling lines are shown, and the 6 seedling lines are arranged in sequence along the horizontal axis direction, that is, along the x direction, and are numbered 1, 2, 3, 4, 5, and 6. The first seedling line and the second seedling line are Figure 3 The two most central seedling lines among the six seedling lines shown in FIG, i.e., the seedling lines numbered 3 and 4. The number of crop strips included in the first picture is different, and the number of seedling lines output by the pre-trained neural network model during the operation of the agricultural machine 110 may also be different, but the first seedling line and the second seedling line are both the two most central seedling lines among the identified and output seedling lines.

[0059] S240, generating a navigation line according to the first seedling line and the second seedling line, wherein the horizontal coordinate of the navigation line is the average of the horizontal coordinate of the first seedling line and the horizontal coordinate of the second seedling line.

[0060] like Figure 3 As shown, the first seedling line and the second seedling line can be the seedling lines numbered 3 and 4 in the figure respectively, and the navigation line is located between the first seedling line and the second seedling line. The agricultural machinery 110 can travel according to the generated navigation line.

[0061] Optionally, in one embodiment of the present application, generating the navigation line based on the first seedling line and the second seedling line includes: when the number of crop strips between the two wheels of the agricultural machine is an odd number, generating the navigation line located in the middle of the crop strips, the first seedling line and the second seedling line being two seedling lines belonging to the same crop strip;

[0062] When the number of crop strips between the two wheels of the agricultural machinery is an even number, the navigation line is generated in the middle of the ridges of two adjacent crop strips, and the first seedling line and the second seedling line are respectively the seedling lines adjacent to the ridges of the two adjacent crop strips.

[0063] Figure 4 FIG. 1 is a schematic diagram of an output navigation line in an embodiment of the present application. Figure 4 As shown, a crop belt is included between the two wheels of the agricultural machine 110, that is, two seedling lines, which can be a first seedling line and a second seedling line. At this time, the generated navigation line is located in the middle of the crop belt.

[0064] Figure 5 FIG. 1 is a schematic diagram of an output navigation line in an embodiment of the present application. Figure 5 As shown, there are three crop belts between the two wheels of the agricultural machine 110, that is, six seedling lines. In this case, the first seedling line and the second seedling line can be respectively numbered 3 and 4. In this case, the generated navigation line is located in the middle of the crop belt.

[0065] Figure 6 FIG. 1 is a schematic diagram of a guide line offset according to an embodiment of the present application. Figure 6 As shown, when the camera 120 is offset, the identified seedling line may be offset to the left. That is, at this time, the identified seedling line is Figure 5 The shown becomes Figure 6 As shown, the original Figure 5 The seedling line with the number 1 is Figure 6 The seedling line with the bid number of 2' is Figure 5 Unidentified seedling lines in Figure 6 In this case, the first and second identified seedling lines do not belong to the same crop strip, and the generated guidance line is located in the middle of the ridges of the two adjacent crop strips, which is one ridge away from the correct guidance line. If the agricultural machine 110 follows the incorrect guidance line, it will deviate and damage the crop strip.

[0066] By setting the generated navigation line to be located in the middle of the crop strip when the number of crop strips between the two wheels of the agricultural machinery is an odd number, the deviation of the navigation line caused by the offset of the camera, etc. can be avoided, thereby preventing the agricultural machinery from deviation.

[0067] Figure 7 FIG. 1 is a schematic diagram of an output navigation line in an embodiment of the present application. Figure 7 As shown, there are two crop belts between the two wheels of the agricultural machine 110, that is, four seedling lines. In this case, the first seedling line and the second seedling line are respectively Figure 7 For the seedling lines numbered 2 and 3, the first and second seedling lines belong to different crop strips and are both close to ridges. In this case, the generated navigation line is located on the ridge between the two adjacent crop strips.

[0068] Optionally, in one embodiment of the present application, generating the second picture based on the first picture includes: reducing the first picture to the second picture according to a first ratio; calculating the spatial relationship of each seedling line in the first picture according to the coordinates of the seedling line includes: after generating the seedling line, restoring the coordinates of the seedling line in the second picture to the coordinates of the seedling line in the first picture according to a second ratio, and the first ratio corresponds to the second ratio.

[0069] The size of the first image can be 720*1280 (in pixels), and the first image can be reduced to a second image of 288*400 according to the first ratio. The reduced second image has a ratio close to that of the first image. In the subsequent calculation process, the amount of calculation can be reduced to about 1 / 9 of the original amount of calculation. After generating the coordinates of the seedling points of the seedling line, the coordinates of the seedling points on each seedling line can be restored to the coordinates in the first image according to the second ratio, so that the coordinates of the seedling points on the seedling line in the first image can be obtained. In this way, the coordinates of the seedling points on the seedling line can be closer to the true value. Afterwards, the navigation line can be generated and output according to the coordinate values of the seedling points in the first seedling line and the second seedling line in the first image.

[0070] Optionally, in one embodiment of the present application, before training the neural network model, it also includes: obtaining a third picture; adding a label to the third picture, and converting the labeled third picture into a fourth picture, wherein the fourth picture is a grayscale image of the labeled third picture, wherein the label corresponds to the spatial position of the seedling line in the third picture.

[0071] The third picture can be the original picture used for training through the neural network model. For example, the size of the third picture is 720*1280*3. The Labelme tool is used to add a label to the third picture, and the labeled third picture is converted into a fourth picture. The fourth picture is a grayscale image with a size of 720*1280*1.

[0072] Figure 8 Schematic diagram of a picture after adding labels in one embodiment of the present application. Figure 8 As shown, along the x direction, from left to right, the label values of the seedling lines are 1, 2, 3, 4, 5, 6, and the black background value in the figure is 0. That is to say, the fourth picture, Figure 8 In the , if there is a seedling line, the label value will be marked on the seedling line. If there is no seedling line, the value here is 0.

[0073] The neural network model in the embodiment of the present application represents the seedling line identification as an n*h*m classification problem, where n represents the number of seedling lines, h represents the vertical coordinate of the seedling point on the seedling line, that is, the position in the y direction, and m represents the horizontal coordinate of the seedling point on the seedling line, that is, the position in the x direction. For example, in the seedling line identification process, 36 points can be selected at equal intervals in the vertical coordinate direction from a picture of size 720*1280, and there are 100 possible coordinate values in the horizontal coordinate direction. When the seedling point does not exist here, the horizontal coordinate value is 0. Based on this, the neural network model can identify up to 6 seedling lines, and its output result is a 6*36*101 matrix. The classification result of the model can be obtained according to the following formula (1):

[0074] P i,j =fij (X),i∈[1,n],j∈[1,h]

[0075] Among them, the output result of formula (1) corresponding to an (i, j) is a 1*101 matrix, and the horizontal coordinate corresponding to the largest value among the 1*101 values is the final classification result.

[0076] The spatial relationship of each seedling line is judged according to the value of i, so that the two most central seedling lines among the identified seedling lines can be found, the horizontal coordinates of the two most central seedling lines are averaged, and the coordinate values of 36 points are obtained to generate a navigation line.

[0077] Optionally, in the embodiment of the present application, more points can be selected in the vertical coordinate direction. The specific settings can be set according to actual needs, and the embodiment of the present application does not limit this.

[0078] Optionally, in one embodiment of the present application, before using a pre-trained neural network model to process the second image to generate seedling lines, the method for seedling line recognition further includes: training the neural network model to obtain a pre-trained neural network model.

[0079] Figure 9 This is a schematic diagram of training a neural network model according to an embodiment of the present application, as shown in FIG. Figure 9 As shown, the following steps may be included.

[0080] S310: Acquire a third picture, and reduce the third picture to a fifth picture according to a first ratio.

[0081] The third picture with a size of 720*1280 can be reduced to a fifth picture with a size of 288*400 according to the first ratio. The ratio of the fifth picture is close to that of the third picture, and compared with the third picture, the fifth picture can reduce the calculation amount to about 1 / 9 of the original.

[0082] The third image may be a pre-stored image containing the work plot, which may be used in the training process of the neural network model.

[0083] S320: Use the backbone network to extract features of the fifth image to obtain a first feature map.

[0084] The backbone network, which may also be referred to as a backbone network, is used for feature extraction, specifically for extracting features of the fifth image.

[0085] Optionally, the backbone network may include: a first input layer input1, a first intermediate layer stage1, a first output layer out1, a second intermediate layer stage2, a second output layer out2, a third intermediate layer stage3, and a third output layer out3.

[0086] The first input layer, input1, is configured to process the fifth image using a convolutional block (ConV-Block) to generate a second feature map. For example, if the size of the fifth image is 288*400*3 and the input of input1 is 288*400*3, after the convolution operation, the second feature map is obtained, and the size of the second feature map is 144*200*16.

[0087] The first intermediate layer stage 1 is configured to process the second feature map using a depthwise separable convolution module Deepwise-Block to obtain a third feature map. For example, the size of the second feature map is 144*200*16, and the size of the third feature map is 36*100*8. The first intermediate layer stage 1 may include three depthwise separable convolution modules Deepwise-Block.

[0088] The first output layer out1 is configured to process the third feature map using a convolutional block ConV-Block to obtain a first feature map. For example, the size of the third feature map is 36*100*8, and the size of the first feature map is 36*52*44.

[0089] Optionally, the first input layer input1, the first intermediate layer stage1, and the first output layer out1 are used for seedling line reasoning, that is, the second intermediate layer stage2, the second output layer out2, the third intermediate layer stage3 and the third output layer out3 are not used for seedling line reasoning, but the second intermediate layer stage2, the second output layer out2, the third intermediate layer stage3 and the third output layer out3 participate in the auxiliary training of the seedling line recognition process.

[0090] The second intermediate layer stage2 is configured to process the fourth feature map using a depthwise separable convolution module Deepwise-Block to obtain a fifth feature map. For example, the size of the fourth feature map is 36*100*12, and the size of the fifth feature map is 18*50*36. The second intermediate layer stage2 may include two depthwise separable convolution modules Deepwise-Block.

[0091] The second output layer out2 is configured to process the fifth feature map using a convolutional block ConV-Block to obtain a sixth feature map. For example, the size of the fifth feature map is 18*50*36, and the size of the sixth feature map is 18*26*89.

[0092] The third intermediate layer stage3 is configured to process the seventh feature map using a depthwise separable convolution module Deepwise-Block to obtain an eighth feature map. For example, the size of the seventh feature map is 9*25*56, and the size of the eighth feature map is 9*25*112. The third intermediate layer stage3 may include two depthwise separable convolution modules Deepwise-Block.

[0093] The third output layer out3 is configured to process the eighth feature map using a convolutional module ConV-Block to obtain a ninth feature map. For example, the size of the eighth feature map is 9*25*112, and the size of the ninth feature map is 9*13*448.

[0094] Optionally, the backbone network structure includes 5-times downsampling, that is, by adopting 5-times downsampling, the running speed of the neural network model is further accelerated.

[0095] By using the output of the first output layer (out1) for seedling line inference and the outputs of the second and third output layers (out2 and out3) for auxiliary training, we can improve the model's accuracy and speed. Table 1 shows the specific parameters for feature extraction using the backbone network structure.

[0096] Table 1 Parameters of backbone network

[0097]

[0098] In Table 1, Input represents the size of the input feature map, Operator represents the name of the convolution module used, t represents the number of operations performed by the corresponding convolution module, c is the number of channels, n is the size of the convolution kernel, and s is the initial sliding distance. Starting from the second time, the sliding distance is 1.

[0099] Figure 10 Schematic diagram of a depth-wise separable convolution module according to an embodiment of the present application. Figure 10 As shown in the figure, the depthwise separable convolution module, also known as Deepwise-Block, includes feature expansion layer, depthwise convolution layer, pointwise convolution layer, and direct connection layer.

[0100] The feature expansion layer is configured to use a 1*1 convolution kernel to expand the number of feature maps to 6 times the original to obtain more details.

[0101] The depth-wise convolution layer is also called the Deepwise convolution layer. The depth-wise convolution layer is configured to perform convolution on a single channel. It can reduce the amount of computation to 1 / c times of the ordinary convolution operation while extracting features, where c is the number of channels.

[0102] The point-by-point convolution layer can also be called the Pointwise convolution layer. The point-by-point convolution layer is configured to traverse all channels through a 1*1 convolution kernel, which can connect the global information of the feature map and make up for the loss caused by convolution in a single channel.

[0103] The direct connection layer can also be called the ShortCut layer. The direct connection layer is configured to use a residual structure to connect the input and output to achieve a deeper network structure, so that the neural network model has stronger representation capabilities.

[0104] S330, converting the first feature map into the coordinates of the seedling line.

[0105] Figure 11 This is a schematic diagram of converting a feature map into seedling line coordinates according to an embodiment of the present application. Figure 12 This is a flow chart of converting the first feature map into seedling line coordinates in one embodiment of the present application. Figure 12 As shown, converting the first feature map into the coordinates of the seedling line includes the following steps.

[0106] S610: Use a 1*1 convolution kernel to reduce the dimension of the first feature map to obtain a tenth feature map.

[0107] After passing through the first output layer out1, the size of the first feature map is, for example, 36*52*44. This first feature map can be reduced to obtain the tenth feature map, which has a size of 36*52*8. Reducing the number of feature maps can improve the model's inference speed, while reducing the number of feature maps from 44 to 8 without significantly reducing the model's accuracy.

[0108] S620 , extracting three groups of feature points from the tenth feature map using the first route, the second route, and the third route respectively.

[0109] Figure 13 FIG. 1 is a schematic diagram of a route for extracting feature points according to an embodiment of the present application. Figure 13 As shown, feature points can be extracted through three different routes.

[0110] In the first Conv1 pass, features are extracted first along the ordinate, then along the abscissa. The convolution kernel in this pass is: kernel_size = (36, 1), strides = (36, 1), filters = 32. Adding kernel_size = (1, 52), strides = (1, 52), filters = 32, this gives 32 feature points from a 36*52*8 feature map.

[0111] In the second Conv2 pass, features are extracted first from the horizontal axis, then from the vertical axis. In the second Conv2 pass, the convolution kernel is: kernel_size = (1, 52), strides = (1, 52), filters = 32, plus kernel_size = (36, 1), strides = (36, 1), filters = 32. Through the second Conv2 pass, 32 feature points are obtained from the 36*52*8 feature map.

[0112] In the third Conv3 pass, full convolution is used to extract features. Specifically, the convolution kernel in this pass is: kernel_size = (36, 52), strides = (36, 52), filters = 32. Through this pass, 32 feature points are obtained from a 36*52*8 feature map.

[0113] The feature points obtained from the three routes are connected to obtain 96 feature points. This method of extracting feature points can obtain richer features and achieve faster model running speed.

[0114] S630: Concatenate the three groups of feature points with n fully connected layers to obtain n intermediate layers.

[0115] Optionally, n is 6, corresponding to the number of seedling lines. Alternatively, the 96 feature points are concatenated with 6 fully connected layers, each containing 12 points, resulting in a 96*12*6 matrix. Each intermediate layer is responsible for the inference prediction of a single seedling line.

[0116] S640: Connect the n intermediate layers to the n output layers in a one-to-one correspondence to obtain an output matrix of n*h*(m+1).

[0117] Optionally, n is 6, h is 36, and m is 100. The 6 intermediate layers are connected to the 6 output layers. Each output layer includes 36*101 points. The 6 output layers are connected together to obtain a 6*36*101 matrix.

[0118] S340, using the loss function L_total to correct the neural network model.

[0119] Optionally, in one embodiment of the present application, the loss function L total satisfy:

[0120] L total =L cls +αL str +βL seg ,

[0121]

[0122] L str =L sim +L shp ,

[0123]

[0124]

[0125] Among them, L cls is the classification loss function, L str is the structural loss function, L seg is the segmentation loss function, L sim is the adjacent loss function, L shp is the shape loss function, α is L str The scaling parameter, β is L seg The scaling parameter, P i,j The predicted value of the corresponding seedling line coordinate, T i,j The actual value of the corresponding seedling line coordinate, L CE is the cross entropy loss function, Loc i,j is the coordinate of the seedling point in the seedling line.

[0126] Classification loss function L cls P represents the horizontal coordinate classification loss of the j-th coordinate point of the i-th line. i,j Corresponding to the predicted value of the seedling line coordinates, the coordinates of the predicted seedling point are compared with the coordinates of the actual seedling point. If the two match, then T i,j 1, does not match, T i,j The coordinates of the actual seedling point can be obtained from the fourth image with the label, that is, the fourth image can be used as a basis for whether the seedling point prediction is correct.

[0127] Neighborhood loss function L sim The positions of adjacent seedling points can be constrained to keep the seedling line relatively continuous and avoid the occurrence of scattered points. i,j and P i,j+1 They correspond to two seedling points with adjacent vertical coordinates on the same seedling line.

[0128] Shape loss function L shp The shape of the seedling line can be constrained, so that most of the seedling line can be guaranteed to be straight, avoiding or reducing the situation where the seedling line turns.

[0129] Segmentation loss function L seg It is used to assist in training the model and can aggregate global and local information.

[0130] After the neural network model is trained using the above method, a pre-trained neural network model can be obtained and the seedling line can be identified during the operation. The above process of converting the first feature map into the coordinates of the seedling line can be collectively referred to as post-processing. The model used in the above-mentioned post-processing process is the Dense layer. In other embodiments, a Conv layer, a Deepwise Conv layer, etc. can also be used.

[0131] To obtain the optimal neural network model, different models were used in the post-processing process within the neural network model, namely, the Dense layer, Conv layer, and Deepwise Conv layer. This test combined the backbone network with different post-processing models and ran it on an embedded device. The test results are shown in Table 2. The embedded device can be an HD400B or LPC540.

[0132] Table 2 Test results of backbone network and different post-processing models on embedded devices

[0133]

[0134] Table 2 shows that as the network depth increases, that is, as the model progresses from stage 1 to stage 2 and then to stage 3, the model's Top 1 accuracy gradually increases. In the Conv layer in stage 3, the model's Top 1 accuracy is 78.3%. However, this model consumes more computer resources, and the computing speed of the stage 3 model is significantly lower than that of stage 1.

[0135] Taking into account the overall speed and accuracy of the model, stage1-Dense was selected as the final model for this neural network. Stage1-Dense achieved an inference speed of 7.5ms on a PC, 67.8ms on a 701 device CPU, and 72.8ms on a 701 device GPU, only about 1-2ms slower than a standalone backbone network structure. It also achieved a Top-1 accuracy of 75.5%. Compared to the most accurate Stage3-Conv, stage1-Dense only lost 2.8% of accuracy, and ran approximately 100% and 300% faster on the CPU and GPU, respectively.

[0136] The neural network model of the embodiment of the present application can be run on an embedded device, such as a low-computing power platform, or run in real time, while ordinary models are difficult to achieve embedded operation.

[0137] An embodiment of the present application also provides a device for seedling line identification, including a module for executing the technical solution or part of the technical solution of the present application.

[0138] An embodiment of the present application also provides an agricultural machine, which includes at least: a camera for capturing images of a working field during the driving process of the agricultural machine; and a processor for controlling the camera and executing the technical solution or part of the technical solution of the present application.

[0139] An embodiment of the present application also provides a computer-readable storage medium for storing a computer program.

[0140] It should be understood that the processor of the embodiment of the present application may be an integrated circuit image processing system with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by hardware integrated logic circuits in the processor or software instructions. The above processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0141] During implementation, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor. The software module can be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it will not be described in detail here.

[0142] It should also be understood that the above description of the embodiments of the present application focuses on emphasizing the differences between the various embodiments. The same or similar points that are not mentioned can be referenced with each other. For the sake of brevity, they will not be repeated here.

[0143] It should be understood that in the embodiments of this application, the term "and / or" is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0144] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, or can be electrical, mechanical or other forms of connection.

[0147] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0148] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0150] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A method for seedling line identification based on deep learning, characterized in that: include: Obtaining a first picture, and generating a second picture based on the first picture, where the first picture is a picture taken by a camera of the agricultural machinery; Processing the second image using a pre-trained neural network model to generate seedling lines; Calculating the spatial relationship of each seedling line in the first picture according to the coordinates of the seedling lines to obtain a first seedling line and a second seedling line, wherein the first seedling line and the second seedling line are respectively the two centermost seedling lines among the seedling lines in the first picture; Generate a navigation line according to the first seedling line and the second seedling line, wherein the horizontal coordinate of the navigation line is the average of the horizontal coordinates of the first seedling line and the horizontal coordinates of the second seedling line; Before processing the second image using the pre-trained neural network model to generate the seedling line, the method further includes: training the neural network model to obtain the pre-trained neural network model; The training of the neural network model includes: Acquire a third image, and reduce the third image to a fifth image according to a first ratio; Extract features of the fifth image using a backbone network to obtain a first feature map; Converting the first characteristic map into the coordinates of the seedling line; Modifying the neural network model using a loss function; The loss function satisfies: L total =L cls +αL str +βL seg , in, L str =L sim +L shp , L total is the loss function, L cls is the classification loss function, L str is the structural loss function, L seg is a segmentation loss function, which is used to assist in training the neural network model. sim is the adjacent loss function, L shp is the shape loss function, α is L str The scaling parameter, β is L seg The scaling parameter, P i,j The predicted probability value corresponding to the seedling line coordinate, T i,j The label value corresponding to the seedling line coordinate, L CE is the cross entropy loss function, Loc i,j is the coordinate of the seedling point in the seedling line, n corresponds to the number of the seedling lines, and h corresponds to the vertical coordinate of the seedling line.

2. The method according to claim 1, characterized in that Generating a navigation line according to the first seedling line and the second seedling line includes: When the number of crop strips between the two wheels of the agricultural machine is an odd number, the navigation line is generated in the middle of the crop strips, and the first seedling line and the second seedling line are two seedling lines belonging to the same crop strip; When the number of crop strips between the two wheels of the agricultural machinery is an even number, the navigation line is generated in the middle of the ridges of two adjacent crop strips, and the first seedling line and the second seedling line are respectively the seedling lines adjacent to the ridges of the two adjacent crop strips.

3. The method according to claim 1, characterized in that Generating the second picture according to the first picture includes: reducing the first picture to the second picture according to a first ratio; The calculation of the spatial relationship of each seedling line in the first picture based on the coordinates of the seedling line includes: after generating the seedling line, restoring the coordinates of the seedling line in the second picture to the coordinates of the seedling line in the first picture according to a second ratio, and the first ratio corresponds to the second ratio.

4. The method according to claim 1, wherein Before training the neural network model, the method further includes: Get the third picture; Add a label to the third image, and convert the labeled third image into a fourth image, wherein the fourth image is a grayscale image of the labeled third image, wherein the label corresponds to the spatial position of the seedling line in the third image.

5. The method according to claim 1, wherein The backbone network includes: a first input layer, wherein the first input layer is configured to process the fifth image using a convolution module to obtain a second feature map; a first intermediate layer, wherein the first intermediate layer is configured to process the second feature map using a depthwise separable convolution module to obtain a third feature map; a first output layer, wherein the first output layer is configured to process the third feature map using a convolution module to obtain the first feature map; a second intermediate layer, wherein the second intermediate layer is configured to process the fourth feature map using a depthwise separable convolution module to obtain a fifth feature map; a second output layer, wherein the second output layer is configured to process the fifth feature map using a convolution module to obtain a sixth feature map; a third intermediate layer, wherein the third intermediate layer is configured to process the seventh feature map using a depthwise separable convolution module to obtain an eighth feature map; The third output layer is configured to process the eighth feature map using a convolution module to obtain a ninth feature map.

6. The method according to claim 5, characterized in that The first input layer, the first intermediate layer and the first output layer are configured to infer the seedling line; the second intermediate layer, the second output layer, the third intermediate layer and the third output layer are configured to participate in the auxiliary training of the seedling line.

7. The method according to claim 5, characterized in that The backbone network includes 5 downsampling steps.

8. The method according to claim 6, characterized in that The first intermediate layer includes three depth-separable convolution modules, and the second intermediate layer and the third intermediate layer each include two depth-separable convolution modules.

9. The method according to claim 5, characterized in that The depth-wise separable convolution module includes: A feature expansion layer, wherein the feature expansion layer is configured to use a 1*1 convolution kernel to expand the number of feature maps to 6 times the original number; A depthwise convolutional layer, wherein the depthwise convolutional layer is configured to perform convolution on a single channel; A point-by-point convolution layer, which is configured to traverse all channels through a 1*1 convolution kernel; A directly connected layer is configured to connect the input and output using a residual structure.

10. The method according to claim 1, characterized in that The converting the first characteristic map into the coordinates of the seedling line includes: Use a 1*1 convolution kernel to reduce the dimension of the first feature map to obtain the tenth feature map; Use the first route, the second route, and the third route to extract three groups of feature points from the tenth feature map respectively; Concatenate the three sets of feature points with n fully connected layers to obtain n intermediate layers; Connect the n intermediate layers to the n output layers in a one-to-one correspondence to obtain an output matrix of n*h*(m+1), where n corresponds to the number of seedling lines, h corresponds to the vertical coordinate of the seedling line, and m corresponds to the horizontal coordinate of the seedling line.

11. The method according to claim 10, characterized in that n＝6。 12. The method according to claim 10, characterized in that The extracting three groups of feature points from the tenth feature map using the first route, the second route, and the third route respectively includes: Based on the first route, features are first extracted from the vertical coordinate direction, and then from the horizontal coordinate direction; Based on the second route, features are first extracted from the horizontal coordinate direction, and then from the vertical coordinate direction; Based on the third approach, features are extracted using a full convolution approach.

13. A device for identifying seedling lines, characterized in that: include: Module for performing the method according to any one of claims 1 to 12.

14. An agricultural machine, characterized in that: include: A camera is used to capture images of the working field while the agricultural machinery is traveling; A processor, configured to control the camera and execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that The invention comprises a computer program which, when executed on a computer device, causes a processing unit in the computer device to execute the method according to any one of claims 1 to 12.

Images