Crop identification model training method, crop identification method, device and system
By training a crop recognition model with detection label information and initial extraction results, the method addresses the issue of inaccurate weed identification in farmland, enhancing the model's ability to accurately distinguish weeds from crops.
Patent Information
- Application Number
- CN202310668014.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-06
- Publication Date
- 2024-12-06
AI Technical Summary
Existing methods for identifying weeds in farmland using neural network models often misjudge weeds as crops or miss them, leading to inaccurate recognition results due to challenges in collecting complete weed data and learning crop characteristics.
A crop recognition model training method that utilizes the regular planting shape of crop rows as constraints during training, incorporating detection label information and initial extraction results to iteratively refine the model, reducing the likelihood of misjudging or missing weeds.
The method significantly improves the accuracy of weed identification by using parallel crop rows as constraints, thereby reducing the probability of misidentification and improving recognition results.
Smart Images

Figure CN119091241A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a crop recognition model training method, a crop recognition method, a device and a system. Background Art
[0002] In the field of smart agriculture, weeds in farmland can be identified by analyzing and processing farmland images, thereby helping relevant personnel to manage farmland efficiently.
[0003] In the prior art, a neural network model can be trained using training samples labeled with various weed information or crop information, so that the neural network model learns the characteristics of the weeds or crops, and then the weeds in the farmland can be identified by the neural network model.
[0004] However, the existing methods may misjudge weeds as crops or miss weeds, making it difficult to ensure the accuracy of the recognition results. Summary of the invention
[0005] The purpose of the present application is to provide a crop identification model training method, a crop identification method, a device and a system to address the problem in the prior art that it is difficult to ensure the accuracy of weed identification results.
[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:
[0007] In a first aspect, an embodiment of the present application provides a crop recognition model training method, comprising:
[0008] The initial recognition model is iteratively trained using the training sample set and the initial extraction result until a preset iteration stop condition is met; wherein the training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of a first straight line detection result of the original image; the crop row label information is used to characterize feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image; the initial extraction result is obtained by the initial extraction model predicting the second straight line detection result, and the second straight line detection result is obtained by performing Hough line detection on the features extracted by the initial recognition model;
[0009] When the initial recognition model meets a preset iteration stop condition, the current initial recognition model is determined as a crop recognition model.
[0010] As a possible implementation method, the iterative training of the initial recognition model using the training sample set and the initial extraction result includes:
[0011] Inputting the original image into the current initial recognition model to obtain an initial recognition result output by the current initial recognition model based on the original image, wherein the initial recognition result is used to indicate crop row information obtained by the initial recognition model;
[0012] Constructing a current recognition loss function according to the crop row label information and the initial recognition result;
[0013] Constructing a current extraction loss function according to the detection label information and the initial extraction result;
[0014] Based on the current recognition loss function and the current extraction loss function, the current loss function is constructed;
[0015] The current loss function is used to back-propagate the current initial recognition model to update the model parameters of the current initial recognition model.
[0016] As a possible implementation method, the current loss function is constructed based on the current recognition loss function and the current extraction loss function, including:
[0017] Based on the first weight of the initial recognition model and the second weight of the initial extraction model, the current recognition loss function and the current extraction loss function are weighted to construct a current loss function.
[0018] As a possible implementation method, a current extraction loss function is constructed based on the detection label information and the initial extraction result, including:
[0019] Determine the intersection and union ratio of the detection label information and the initial extraction result;
[0020] Using the preset sample weight as the influencing factor, determine the cross entropy between the detection label information and the initial extraction result;
[0021] Based on the intersection-over-union ratio and the cross entropy, a current extraction loss function is constructed.
[0022] As a possible implementation method, after constructing the current extraction loss function, it also includes:
[0023] The initial extraction model is back-propagated using the current extraction loss function to update the model parameters of the initial extraction model.
[0024] As a possible implementation manner, the detection label information of the first straight line detection result is obtained by the following steps:
[0025] Get multiple original images;
[0026] Performing image transformation on the multiple original images respectively to obtain multiple transformed images, where the transformed images are binary images or mask images of the original images;
[0027] Performing Hough line detection on the multiple transformed images respectively to obtain multiple first line detection results; wherein the first line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line;
[0028] According to the element values of the multiple rows of elements, key result information for characterizing the real position of the crop row in the original image is determined, and the key result information is used as the detection label information.
[0029] As a possible implementation manner, determining key result information for characterizing the real position of the crop row in the original image according to the element values of the multiple row elements includes:
[0030] Determining a plurality of key factor information according to the element values of the plurality of rows of elements, the key factor information comprising: gap density and an average value of crop row elements;
[0031] The key result information is determined according to the plurality of key factor information.
[0032] As a possible implementation, the multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines passing through the point corresponding to the position in the two-dimensional array in the Hough space;
[0033] The determining of multiple key factor information according to the element values of the multiple rows of elements includes:
[0034] Determining crop row elements and non-crop row elements in each row according to element values of elements in each row of the two-dimensional array;
[0035] The ratio of the number of non-crop row elements in each row to the sum of the number of crop row elements and non-crop row elements in each row is calculated to obtain the gap density of each row;
[0036] The average values of the element values of the crop row elements in each row are calculated respectively to obtain the average value of the crop row elements in each row.
[0037] As a possible implementation manner, determining the key result information according to the multiple key factor information includes:
[0038] Calculate the product of the gap density of each row and the average value of the crop row elements to obtain the representation degree value of each row;
[0039] Each element of the row with the largest representation degree value is used as the key result information.
[0040] As a possible implementation manner, after determining the key result information, the method further includes:
[0041] Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located;
[0042] Selecting a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence;
[0043] Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
[0044] As a possible implementation manner, performing Hough line detection on the plurality of transformed images respectively to obtain a plurality of first line detection results includes:
[0045] Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the first line detection result.
[0046] As a possible implementation manner, in the initial extraction model, there is at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
[0047] As a possible implementation manner, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0048] In a second aspect, an embodiment of the present application provides a crop identification method, comprising:
[0049] Acquire an image of a target area, wherein the image of the target area includes crop rows and weeds;
[0050] The image of the target area is input into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model, wherein the crop recognition model is trained based on the method described in the first aspect above.
[0051] In a third aspect, an embodiment of the present application provides a crop recognition model training device, comprising:
[0052] A training module, used for iteratively training an initial recognition model using a training sample set and an initial extraction result until a preset iteration stop condition is met; wherein the training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of a first straight line detection result of the original image; the crop row label information is used to characterize feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough straight line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image; the initial extraction result is obtained by predicting a second straight line detection result by the initial extraction model, and the second straight line detection result is obtained by performing Hough straight line detection on the feature extracted by the initial recognition model;
[0053] The determination module is used to determine the current initial recognition model as the crop recognition model when the initial recognition model meets a preset iteration stop condition.
[0054] In a fourth aspect, an embodiment of the present application provides a crop identification device, including:
[0055] An acquisition module, used for acquiring an image of a target area, wherein the image of the target area includes crop rows and weeds;
[0056] The recognition module is used to input the image of the target area into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model, wherein the crop recognition model is trained based on the method described in the first aspect above.
[0057] In a fifth aspect, an embodiment of the present application provides a smart agricultural system, including:
[0058] An acquisition device, used for acquiring an image of a target area;
[0059] a processing device, configured to process the image of the target area obtained by the acquisition device according to the crop recognition method described in the second aspect above, so as to obtain crop row information in the image of the target area;
[0060] A graphic interface device is used to generate corresponding display information according to the crop row information.
[0061] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to execute the steps of the crop recognition model training method described in the first aspect or the steps of the crop recognition method described in the second aspect.
[0062] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the crop recognition model training method described in the first aspect or the steps of the crop recognition method described in the second aspect are executed.
[0063] The crop recognition model training method, crop recognition method, device and system provided in the embodiments of the present application, in addition to using the crop row label information that characterizes the characteristics of the crop row itself, also introduces detection label information and initial extraction results during the iterative training of the initial recognition model, wherein the detection label information can characterize the positions of parallel strips formed by the crop rows in the original image, and the initial extraction results are information obtained by performing straight line detection and parallel strip information extraction based on the features extracted by the initial recognition model. Since the detection label information and the initial extraction results are both information describing parallel strips, they can be used as constraint information in the initial recognition model training process to avoid misjudging plants outside the parallel strips as crops, thereby greatly reducing the probability of missed and misjudged weeds, thereby greatly improving the accuracy of the recognition results. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0065] Figure 1 A schematic diagram of a process flow of a crop recognition model training method provided in an embodiment of the present application;
[0066] Figure 2 A schematic diagram of an iterative training process of a crop recognition model training method provided in an embodiment of the present application;
[0067] Figure 3 A schematic diagram of feature extraction and recognition analysis based on the initial recognition model;
[0068] Figure 4A schematic diagram of the process of constructing the crop recognition model training method provided in the embodiment of the present application and extracting the loss function;
[0069] Figure 5 A schematic diagram of the process of obtaining detection label information in the crop recognition model training method provided in an embodiment of the present application;
[0070] Figure 6 is a schematic diagram of a transformed image obtained after the original image is binarized;
[0071] Figure 7 is a visualized schematic diagram of a first straight line detection result obtained through the Hough straight line detection of this embodiment;
[0072] Figure 8 A visualization diagram of the line detection result obtained when the origin of the polar coordinate system is set at a vertex of the image;
[0073] Fig. 9 A schematic diagram of the process to determine key outcome information;
[0074] Fig.10 Schematic diagram of the process for determining the gap density and the average value of crop row elements;
[0075] Fig.11 A schematic diagram of the process of filtering key result information;
[0076] Fig.12 A schematic diagram of a process flow of a crop identification method provided in an embodiment of the present application;
[0077] Fig.13 A module structure diagram of a crop recognition model training device provided in an embodiment of the present application;
[0078] Fig.14 A module structure diagram of a crop identification device provided in an embodiment of the present application;
[0079] Fig.15 A schematic diagram of the structure of an electronic device 150 provided in an embodiment of the present application. DETAILED DESCRIPTION
[0080] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.
[0081] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.
[0082] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.
[0083] In general, neural network models are trained using training samples labeled with various weed information or crop information, so that the neural network model learns the characteristics of weeds or crops. Weeds in farmland can then be identified through the neural network model. For example, in a scenario where a neural network model is used to identify weeds in a cotton field, the neural network model can be trained based on a target detection algorithm using training samples labeled with weed information, and the neural network can be used to identify weeds in the cotton field. Alternatively, a neural network model can be trained based on a semantic segmentation algorithm using training samples labeled with cotton information. After the neural network model is trained, cotton in a cotton field can be identified, and plants other than cotton are weeds in the cotton field.
[0084] However, using existing technical methods, there are problems such as difficulty in collecting complete weed data, difficulty in directly detecting weeds, and difficulty in learning crop characteristics, which may lead to weeds being misidentified as crops or weeds being missed, making it difficult to ensure the accuracy of the recognition results.
[0085] Based on the above problems, the embodiments of the present application propose a crop recognition model training method and a method for crop recognition using a crop recognition model. The method utilizes the characteristic that the planting shape of crop rows is regular parallel strips, and uses parallel strips as constraints during the crop recognition model training process, so that the trained crop recognition model has a greatly reduced probability of misjudging or missing weeds, thereby greatly improving the accuracy of the recognition results.
[0086] The embodiments of the present application can be applied to the scene of weed identification in crop rows. The crop rows in the embodiments of the present application can be, for example, cotton rows, soybean rows, potato rows, etc. in farmland. The embodiments of the present application do not limit the specific types of crop rows. It should be understood that the weed identification in crop rows described in the present application refers to the identification of weeds in the farmland where the crop rows are located, including the identification of weeds appearing in the crop rows and outside the crop rows.
[0087] Figure 1 This is a flow chart of the crop recognition model training method provided in the embodiment of the present application. The execution subject of the method can be any electronic device with computing and processing capabilities, such as a desktop computer, a server, etc. Figure 1 As shown, the method includes:
[0088] S101. Iteratively train the initial recognition model using the training sample set and the initial extraction result until the preset iteration stop condition is met. The training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of the first straight line detection result of the original image. The crop row label information is used to characterize the feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image. The initial extraction result is obtained by the initial extraction model predicting the second straight line detection result, and the second straight line detection result is obtained by performing Hough line detection on the features extracted by the initial recognition model.
[0089] Optionally, in this embodiment, the initial recognition model is trained based on the training sample set and the initial extraction result. The training sample set includes the above-mentioned original image, crop row label information, and detection label information. It should be understood that the training sample set may include multiple groups of training samples, each group of training samples includes a frame of original image, crop row label information corresponding to the frame of original image, and detection label information. The following is an explanation of each.
[0090] The original image may be, for example, an image obtained by photographing the area where the crop rows are located by a device having an image acquisition function such as a drone, and the original image includes one or more crop rows. For example, when it is necessary to identify weeds in a cotton field, the drone may be controlled to collect the original image of the cotton field over the cotton field, so that the original image includes complete cotton rows and non-cotton rows.
[0091] The crop row label information may be information obtained by manually or automatically marking the crop rows of the original image, and the crop row label information may be used to characterize the characteristic information of the crop row in the original image. In one example, the position of the cotton in the original image may be marked, and the characteristics of all the pixels at the position may characterize the characteristic information of the cotton row. In another example, the position of the cotton and the position of the weeds in the original image may be marked, and these two positions may jointly characterize the characteristic information of the cotton row. For this example, specifically, if the characteristics of a certain pixel or certain pixels meet the characteristics of the marked position of the cotton, and at the same time, do not meet the characteristics of the marked position of the weeds, it may be determined that the pixel belongs to the cotton. It is worth noting that, since the present application adds the detection label information and the initial extraction results in the training process of the initial recognition model to constrain the parallel strips, it is not necessary to mark the crop rows for all areas in the original image as in the prior art, but only part of the original image may be marked for the crop rows. For example, crop rows can be annotated only for the upper left quarter image, the upper right quarter image, the left half image, the right half image, etc. of the original image, thereby greatly reducing the time consumption of sample annotation and improving the efficiency of sample annotation while ensuring the training effect.
[0092] The detection label information may be information obtained by manually or automatically marking parallel strips of the original image, and the detection label information may be used to characterize the true position of the crop row in the original image. The specific form of the detection label information may be the key result information described below, or the angle and width described below. It is worth noting that, in the specific implementation process, the crop row label information and the detection label information may both be realized by marking the position in the original image, but the role of the crop row label information is to characterize the characteristics of the crop itself in the crop row, while the role of the detection label information is to characterize the parallel strips formed by the crop row. When marking the detection label information, the original image may first be subjected to Hough line detection to obtain the above-mentioned first line detection result, and the above-mentioned detection label information may be automatically marked based on the first line detection result. The process of automatically marking the detection label information will be described in detail in the following embodiments.
[0093] The original image, crop row label information and detection label information introduced above can be pre-constructed as a training sample set before starting iterative training. In addition, during the iterative training process, the initial recognition model can also use the initial extraction result output by the initial extraction model for iterative training. Optionally, the initial extraction model can be a model of the same type or a different type as the initial recognition model. The initial extraction model is used to extract parallel stripe information. Specifically, the input parameter of the initial extraction model is the second straight line detection result, and the output result is the position information obtained by predicting the second straight line detection result, that is, the initial extraction result. The position information can characterize the position of the crop row in the original image. In an optional manner, the position information can be the key result information of the position of the crop row in the original image. The key result information can be the part of the second straight line detection result that best characterizes the parallel stripes. Using the key result information, the position of the crop row in the original image can be calculated. In another optional manner, the position information can include the angle and width of the crop row in the original image. Based on the angle and width, the position of the crop row in the original image can be known.
[0094] It is worth noting that since the training of the initial recognition model depends on the initial extraction result output by the initial extraction model, and at the same time, the input parameter of the initial extraction model is the second straight line detection result obtained by performing Hough straight line detection on the features extracted by the initial recognition model, that is, the processing of the initial extraction model depends on the initial recognition model, therefore, in the specific implementation process, the initial extraction model may not be introduced in the first iteration, and a fixed value may be manually configured for the initial extraction result to complete the first iteration of the initial recognition model, and then, starting from the second iteration, the initial extraction model is introduced. Specifically, the features extracted by the initial recognition model in this iteration are input into the initial extraction model of this iteration to obtain the initial extraction result required for this iteration, and then this iteration is completed based on the training sample set and the initial extraction result, and so on, to ensure the iterative training of the initial recognition model. It should be understood that during the iterative training of the initial recognition model, the initial extraction model can also be iteratively trained at the same time.
[0095] S102: When the initial recognition model satisfies a preset iteration stop condition, the current initial recognition model is determined as a crop recognition model.
[0096] Optionally, in each iteration of the iterative training of the initial recognition model, each group of training samples is input into the initial recognition model, and the initial recognition model predicts the actual result. By comparing the actual result with the label information of the learning sample, it is determined whether the iteration stopping condition is met. When the iteration stopping condition is met, the initial recognition model at the last iteration is used as the above-mentioned crop recognition model.
[0097] After the crop recognition model is obtained, the crop recognition model can be used to recognize weeds in the crop row. This recognition process will be described in detail in the following embodiments.
[0098] In this embodiment, in the process of iteratively training the initial recognition model, in addition to using the crop row label information that characterizes the characteristics of the crop row itself, detection label information and initial extraction results are also introduced, wherein the detection label information can characterize the positions of parallel strips formed by the crop rows in the original image, and the initial extraction results are information obtained by performing straight line detection and parallel stripe information extraction based on the features extracted by the initial recognition model. Since the detection label information and the initial extraction results are both information describing parallel strips, they can be used as constraint information in the initial recognition model training process to avoid misjudging plants outside the parallel strips as crops, thereby greatly reducing the probability of missed and misjudged weeds, thereby greatly improving the accuracy of the recognition results.
[0099] The following is an explanation of the process of iterative training of the initial recognition model. Figure 2 The processing flow shown is an iterative process in the iterative training of the initial recognition model. Before the initial recognition model meets the iteration stop condition, it can go through multiple iterations. Each iteration can be performed as follows Figure 2 Process shown.
[0100] Figure 2 A schematic diagram of an iterative training process of a crop recognition model training method provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, an optional manner of the above step S101 includes:
[0101] S201. Input the original image to the current initial recognition model to obtain an initial recognition result output by the current initial recognition model based on the original image, wherein the initial recognition result is used to indicate crop row information obtained by the initial recognition model.
[0102] Optionally, the crop row information obtained by the initial recognition model can be used to characterize the characteristic information of the crop row in the original image, so that the weed information can be obtained based on the crop row information. In one example, the crop row information can include the location of the crop row and the location of the weed in the original image.
[0103] S202: construct a current recognition loss function based on the crop row label information and the initial recognition result.
[0104] Optionally, the crop row label information is pre-labeled to characterize the real features of the crop row in the original image. The initial recognition result is the recognition result actually output by the initial recognition model in this iteration. By comparing the crop row label information and the initial recognition result, a current recognition loss function is constructed. It should be understood that the current recognition loss function is a loss function that measures the quality of the current initial recognition model.
[0105] S203: construct a current extraction loss function based on the detection label information and the initial extraction result.
[0106] Optionally, the detection label information is pre-labeled and can characterize the true position of the crop rows as parallel strips in the original image. The initial extraction result is the extraction result actually output by the initial extraction model in this iteration. By comparing the detection label information and the initial extraction result, the current extraction loss function is constructed. It should be understood that the current extraction loss function can measure the quality of the current initial extraction model on the one hand, and on the other hand, since the initial extraction result is obtained by the initial extraction model predicting the second straight line detection result, and the second straight line detection result is obtained by performing Hough line detection on the features extracted by the initial recognition model, the initial extraction result is related to the initial recognition model. Accordingly, the current extraction loss function constructed based on the initial extraction result can be used as a constraint to measure the quality of the initial recognition model.
[0107] S204: construct a current loss function based on the current recognition loss function and the current extraction loss function.
[0108] As a possible implementation method, the same weights can be configured for the initial recognition model and the initial extraction model, so as to directly add the current recognition loss function and the current extraction loss function to obtain the current loss function.
[0109] As another possible implementation, different weights can be configured for the initial recognition model and the initial extraction model. The weight of the initial recognition model is the first weight, and the weight of the initial extraction model is the second weight. Based on the first weight of the initial recognition model and the second weight of the initial extraction model, the current recognition loss function and the current extraction loss function can be weighted to construct the current loss function. Specifically, the current recognition loss function is multiplied by the first weight to obtain a first multiplication result, the current extraction loss function is multiplied by the second weight to obtain a second multiplication result, and the first multiplication result is added to the second multiplication result to obtain the current loss function.
[0110] S205 , back-propagating the current initial recognition model using the current loss function to update the model parameters of the current initial recognition model.
[0111] Optionally, the current loss function can be used to determine whether the current initial recognition model meets the above-mentioned iteration stop condition. If so, the aforementioned step S102 can be entered to obtain the crop recognition model. Otherwise, the model parameters of the current initial recognition model can be updated according to the current loss function, and the updated current initial recognition model can be used for the next iteration.
[0112] In this embodiment, when constructing the loss function of the initial recognition model, in addition to introducing the loss of the initial recognition model on the crop row recognition results, the loss of the parallel strips extracted based on the features extracted by the initial recognition model is also introduced, so that the parallel strips are used as constraint information in the training process of the initial recognition model, ensuring that the trained crop recognition model uses parallel strips as constraints to more accurately identify weeds in crop rows.
[0113] The initial recognition model and the initial extraction model described in the embodiment of the present application can be various types of neural network models, and the present application does not limit their types. As an optional implementation, the initial recognition model can include an encoder and a decoder.
[0114] Based on the above structure of the initial recognition model, the above step S201 can be specifically implemented through the following process.
[0115] The original image is input to the current initial recognition model, and the encoder extracts high-level features and low-level features of the original image. Furthermore, the decoder performs feature analysis on the high-level features and low-level features extracted by the encoder and outputs the initial recognition result.
[0116] At the same time, as described above, the Hough line detection based on the features extracted by the initial recognition model can obtain the above-mentioned second line detection result, and the initial extraction model predicts the second line detection result to obtain the initial extraction result. In this embodiment, the high-level features extracted by the encoder can be subjected to Hough line detection to obtain the above-mentioned second line detection result.
[0117] Figure 3 This is a schematic diagram of feature extraction and recognition analysis based on the initial recognition model, such as Figure 3As shown, the initial recognition model includes an encoder and a decoder. After the original image is input into the initial recognition model, the encoder first extracts low-level features, and the low-level features are then subjected to a series of feature processing to extract high-level features. At the same time, the low-level features are input into the decoder. The high-level features extracted by the encoder are input into the decoder. At the same time, the high-level features are output and used as input parameters of the Hough line detection algorithm to perform Hough line detection to obtain a second line detection result, which is then input into the initial extraction model for prediction processing, and the initial extraction model outputs the initial extraction result. In the decoder, the decoder uses the low-level features of the high-level features input by the encoder to perform feature analysis, thereby outputting the initial recognition result.
[0118] The following describes the process of constructing the current extraction loss function in the above step S203.
[0119] Figure 4 A schematic diagram of the process of extracting the loss function for constructing the crop recognition model training method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, the above step S203 may include:
[0120] S401: Determine the intersection-over-union ratio of the detection label information and the initial extraction result.
[0121] Optionally, taking the detection label information and the initial extraction results as the aforementioned key result information as an example, the key result information can be presented in the form of a matrix, and the intersection and union of the matrix of the detection label information and the matrix of the initial extraction results can be calculated respectively, and the ratio of the intersection and the union can be calculated to obtain the above-mentioned intersection-union ratio.
[0122] Exemplarily, the intersection over union (IOU) of the detection label information and the initial extraction result may be calculated based on an intersection over union (IOU) function.
[0123] S402: Using a preset sample weight as an influencing factor, determine the cross entropy between the detection label information and the initial extraction result.
[0124] Optionally, continuing with the example in which the detection label information and the initial extraction results are the aforementioned key result information, and the key result information is presented in the form of a matrix, the cross entropy between the matrix of the detection label information and the matrix of the initial extraction results can be calculated using the above sample weights as coefficients.
[0125] S403: Based on the above intersection-over-union ratio and the above cross entropy, a current extraction loss function is constructed.
[0126] As an example, the sum of the above-mentioned intersection-over-union ratio and cross entropy can be calculated, and the sum of the intersection-over-union ratio and the cross entropy can be used as the result of the current extraction loss function.
[0127] In this embodiment, by determining the intersection-over-union ratio and cross entropy of the detection label information and the initial extraction results, the extraction of positive samples in the learning samples can be guaranteed, so that the initial recognition model and the initial extraction model can better adapt to the unbalanced sample distribution, thereby greatly improving the convergence speed, performance and stability of the initial recognition model and the initial extraction model.
[0128] As an optional implementation, after the current extraction loss function is constructed through the above step S203, the current extraction loss function can also be used to back-propagate the initial extraction model to update the model parameters of the initial extraction model.
[0129] Specifically, using the current extraction loss function, it can be determined whether the current initial extraction model meets the preset iteration stop condition. If so, the training of the initial extraction model can be stopped. Otherwise, the model parameters of the current initial extraction model can be updated according to the current extraction loss function, and the updated current initial extraction model can be used for the next iteration.
[0130] It is worth noting that, in the specific implementation process, the iterative training of the initial recognition model and the iterative training of the initial extraction model can be performed simultaneously, or they can also be performed separately. For example, when the initial recognition model is updated after each iterative training, the initial extraction model will be updated accordingly in the manner of this embodiment. Alternatively, the initial extraction model can also be updated once after the initial recognition model is iteratively updated multiple times.
[0131] The following describes the process of automatically marking the detected label information mentioned in the aforementioned step S101.
[0132] Figure 5 A schematic diagram of the process of obtaining detection label information for the crop recognition model training method provided in the embodiment of the present application, such as Figure 5 As shown, the detection label information of the aforementioned first straight line detection result is obtained by the following steps:
[0133] S501, obtaining multiple original images.
[0134] S502 , performing image transformation on the plurality of original images respectively to obtain a plurality of transformed images, where the transformed images are binary images or mask images of the original images.
[0135] In an optional manner, the original image can be subjected to a binary transformation, and accordingly, the transformed image obtained is the above-mentioned binary image. Specifically, in the process of binary transformation, the grayscale value of each pixel point can be transformed to 0 or 255 according to the original pixel value of each pixel point on the original image, so that the transformed image presents only black and white effects, which is convenient for subsequent straight line detection. In another optional manner, the original image can be subjected to mask processing, and accordingly, the transformed image obtained is the above-mentioned mask image. Specifically, before the mask processing, an image mask can be generated by specifying data values, data ranges, regions of interest, etc. The image mask can be a two-dimensional matrix array or a multi-valued image, etc. Furthermore, during the mask processing, the image mask is multiplied by the original image to obtain a transformed image, so that part of the region in the transformed image is shielded, and only the region of interest is retained for subsequent straight line detection.
[0136] S503, performing Hough line detection on the plurality of transformed images respectively, and obtaining a plurality of the first line detection results. The first line detection results include a plurality of rows of elements, each element corresponding to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line.
[0137] Optionally, the coordinate system where the transformed image is located has the following mapping relationship with the Hough coordinates of the Hough space: each point in the original coordinate system can be mapped to a straight line in the Hough coordinate system. Similarly, each point in the Hough coordinate system can also be mapped to a straight line in the original coordinate system. Accordingly, a straight line in the original coordinate system can be evaluated by the number of lines that intersect at a point in the Hough coordinate system. Specifically, in the Hough coordinate system, the more straight lines passing through a point, it indicates that the line mapped to the point in the original coordinate system is composed of more points, and the greater the probability that the line mapped to the point in the original coordinate system is a straight line.
[0138] Optionally, the original coordinate system of the original image may be a polar coordinate system or a Cartesian coordinate system.
[0139] Based on the above mapping relationship, when performing Hough line detection on the transformed image, all points in the original coordinate system can be first mapped to a straight line in the Hough coordinate system using the mapping relationship, and then the points where the straight lines intersect can be judged. If the number of straight lines passing through a certain point reaches a threshold, the point can be mapped to a line in the original coordinate system as a straight line in the original image. After the Hough line detection is completed, the above-mentioned first straight line detection result can be obtained. Among them, the above-mentioned first straight line detection result includes the above-mentioned multiple rows of elements. Exemplarily, the first straight line detection result can be presented in the form of a two-dimensional array, which includes multiple rows of elements, and the number of elements in each row of the two-dimensional array can be one or more. For each element of each row in the two-dimensional array, its position in the matrix can represent the position of a straight line corresponding to the element in the original image. Among them, the position of the element in the matrix can refer to the row where the element is located and the column where the element is located. At the same time, the element value of the element can represent the reliability of the straight line corresponding to the element. Exemplarily, the larger the element value of the element, the higher the reliability of the straight line corresponding to the element. It should be understood that the higher the reliability, the greater the possibility of belonging to the straight line, and the more reliable the result.
[0140] S504: Determine key result information for characterizing the real position of the crop row in the original image according to the element values of the multiple rows of elements, and use the key result information as the detection label information.
[0141] Since each element in the above-mentioned multiple rows of elements corresponds to a straight line in the original image, and the element value of each element can characterize the reliability of the straight line, the above-mentioned key result information can be obtained by analyzing and comparing the element values of the multiple rows of elements based on the element values of the multiple rows of elements. Optionally, the above-mentioned key result information can include some elements determined from the above-mentioned multiple rows of elements, and these elements are sufficient to identify the parallel strips formed by the crop rows in the original image. Therefore, the above-mentioned key result information can be used as detection label information.
[0142] As another possible implementation, after obtaining the above key result information, the angle and width of the parallel strips can be extracted based on the key result information and the position of at least one row of elements in the multiple rows of elements, and the angle and width can be used as detection label information.
[0143] As an optional implementation, when performing Hough line detection on the transformed image in the above step S503, the process may be as follows.
[0144] Taking the center point of the transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the above-mentioned first line detection result.
[0145] Optionally, when performing Hough line detection on the transformed image, if the origin of the polar coordinate system is set at other positions of the image, such as the left vertex of the image, the line parameters in the line detection results will all fall into the same quadrant, resulting in interlaced and mixed line detection results, which will make it difficult to extract subsequent key result information. Considering this problem, in this embodiment, the center point of the transformed image is used as the origin of the polar coordinate system, so that the line parameters can be evenly distributed, and the line parameters corresponding to the lines at the same angle can be concentrated in a certain area, so that the key result information can be obtained more conveniently.
[0146] Figure 6 is a schematic diagram of the transformed image obtained after the original image is binarized, as shown in Figure 6 As shown, each white strip is a crop row parallel strip, and the crop row parallel strips can be, for example, cotton rows, etc. Figure 7 is a visualization diagram of the first straight line detection result obtained by the Hough straight line detection of this embodiment, as shown in Figure 7 As shown, the white highlighted portion is the location of the key result information, and the key result information is sufficient to identify the parallel strips formed by the crop rows in the original image. Figure 8 This is a visualization diagram of the line detection results obtained when the origin of the polar coordinate system is set at a vertex of the image. Figure 8 It can be seen that this method makes the white highlight part tilted, which makes it difficult to extract the subsequent key result information. Figure 7 and Figure 8 ,Obviously, the first straight line detection result obtained by taking the center point of the transformed ,image as the origin of the polar coordinate system is more conducive to the subsequent ,key result information extraction.
[0147] The process of determining the key result information in the above step S504 is described below.
[0148] Fig. 9 A flow chart to determine key outcome information, such as Fig. 9 As shown, an optional manner of the above step S504 includes:
[0149] S901. Determine multiple key factor information according to the element values of the multiple rows of elements. The key factor information includes: gap density and average value of crop row elements.
[0150] S902: Determine the key result information according to the multiple key factor information.
[0151] As described above, the key result information is sufficient to identify the parallel strips formed by the crop rows in the original image, and the key result information is the partial element that best represents the parallel strips among the multiple row elements. In this embodiment, the key result information is found using the key factor information.
[0152] Optionally, the key factor information includes the above-mentioned gap density and crop row element average value. Among them, the gap density can refer to the proportion of non-crop row elements in multiple elements, and the non-crop row elements can refer to elements with element values of zero or less than a threshold. These elements cannot correspond to a real straight line in the original coordinate system due to their small element values. Corresponding to the non-crop row elements are crop row elements, and crop row elements can refer to elements with element values greater than or equal to the threshold. For multiple elements in the first straight line detection result, the gap density can be used to measure the interval between parallel strips. Based on this interval, some elements that can best characterize parallel strips can be identified without being affected by some interfering non-crop row elements between parallel strips. In addition to the gap density, the key factor information also includes the crop row element average value. The crop row element average value is obtained by statistics of the element values of the crop row elements. The crop row element average value can be used to measure the possibility of elements forming parallel strips, and then this possibility can also be used to identify some elements that can best characterize parallel strips.
[0153] Based on the above void density and crop row element averages, multiple elements can be analyzed and measured to determine key results information.
[0154] In this embodiment, multiple key factor information can be determined based on the element values of multiple rows of elements. The key factor information includes gap density and the average value of crop row elements. The gap density can be used to measure the interval between parallel strips, and the average value of crop row elements can be used to measure the possibility of elements forming parallel strips. Therefore, based on the gap density and the average value of crop row elements, key result information with high accuracy can be determined, and the parallel stripe information extracted based on the key result information has high accuracy.
[0155] As a possible implementation method, the above-mentioned multiple rows of elements can be stored in a two-dimensional array, and the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines in the Hough space passing through the point corresponding to the position in the two-dimensional array.
[0156] Specifically, each element in the two-dimensional array can correspond to a point in the Hough space, and of course, it also corresponds to a straight line in the original image. Therefore, the position of the element in the two-dimensional array can represent the position of the corresponding straight line in the original image. At the same time, each element has an element value in the two-dimensional array, and the element value can indicate the number of straight lines passing through the point corresponding to the element in the Hough space. The following is an example. Assuming that the coordinate system of the original image is a polar coordinate system, there is an element in the two-dimensional array, and the element is located in the a row and b column of the two-dimensional array, which indicates that the element corresponds to the point (a, b) in the Hough space, and corresponds to the straight line with a polar angle of a and a polar radius of b in the polar coordinate system. Assuming that the element value of this element in the a row and b column is c, it indicates that there are c straight lines passing through the point (a, b) in the Hough space.
[0157] Based on the above storage method of multiple rows of elements, an optional method of the above step S901 includes:
[0158] According to the element values of the elements in each row of the two-dimensional array, the gap density of each row and the average value of the crop row elements are determined.
[0159] Optionally, when multiple rows of elements are stored in a two-dimensional array, the gap density of each row and the average value of the crop row elements can be determined on a row-by-row basis.
[0160] Taking rows as units and determining the gap density and the average value of crop row elements for each row can make the determined gap density and the average value of crop row elements highly matched with the parallel strips, thereby making the subsequently determined key result information and parallel strip information highly accurate.
[0161] Below, the process of determining the gap density of each row and the average value of the crop row elements is explained.
[0162] Fig.10 Schematic diagram of the process for determining the gap density and the average value of crop row elements, such as Fig.10 As shown, an optional method for determining the gap density of each row and the average value of the crop row elements according to the element values of the elements in each row includes:
[0163] S1001. Determine crop row elements and non-crop row elements in each row according to element values of elements in each row of a two-dimensional array.
[0164] Optionally, as mentioned above, non-crop row elements may refer to elements whose element values are zero or less than a threshold value. Since these elements have small element values, they cannot correspond to a real straight line in the original coordinate system. Correspondingly, crop row elements may refer to elements whose element values are greater than or equal to the threshold value. Crop row elements may correspond to a real straight line in the original coordinate system. Based on this, in this step, the elements in each row of the two-dimensional array may be judged one by one. If the element value of the element is greater than or equal to the threshold value, the element is regarded as a crop row element. If the element value of the element is 0 or less than the threshold value, the element is regarded as a non-crop row element.
[0165] It is worth noting that in the original image, there are generally no crop rows on the left and right edges of the image. At the same time, there are no plants between the two crop rows or there are only sporadic interference objects. Therefore, by determining the crop row elements and non-crop row elements through this step, the left and right edges of the original image and the middle part that does not belong to the crop row can be identified.
[0166] S1002, respectively calculating the ratio of the number of non-crop row elements in each row to the sum of the number of crop row elements and non-crop row elements in each row, to obtain the gap density of each row.
[0167] As mentioned above, the gap density may refer to the proportion of non-crop row elements in multiple elements. Specifically in this step, since each row is used as a unit, the proportion of the number of non-crop row elements in each row to all elements can be calculated to obtain the gap density of each row. Among them, the larger the proportion of the number of non-crop row elements, the greater the gap density. Conversely, the smaller the proportion of the number of non-crop row elements, the smaller the gap density.
[0168] S1003, respectively calculating the average values of the element values of the crop row elements in each row to obtain the average value of the crop row elements in each row.
[0169] Exemplarily, the element values of all crop row elements in each row may be added together, and the sum may be divided by the number of crop row elements to obtain the average value of the crop row elements in each row.
[0170] In this embodiment, the gap density and the average value of the crop row elements of each row are calculated respectively based on each row, so that the determined gap density and the average value of the crop row elements can have a high matching degree with the parallel strips.
[0171] Based on the above determination of the gap density of each row and the average value of the crop row elements in row units, the key result information can be determined in the following way.
[0172] Optionally, an optional manner of the above step S902 includes:
[0173] The product of the gap density of each row and the average value of the crop row elements is calculated respectively to obtain the representation degree value of each row; each element of the row with the largest representation degree value is used as the above key result information.
[0174] Optionally, in the present application, the gap density and the average value of the crop row elements determined by the above-mentioned embodiments are both positively correlated with the key result information. Specifically, for the original image, there is inevitably some interference information between the two real crop rows, such as weeds between two cotton rows, which is interference information. Obviously, the more interference information there is, the smaller the gap density between the crop rows will be. Conversely, the less interference information there is, the greater the gap density between the crop rows will be. Based on this, rows with larger gap density can be found in each row for subsequent calculations, thereby minimizing the impact of interference information on subsequent calculations. If the average value of the crop row elements of a row is larger, it means that there are more elements in the row that can represent the straight line, and the possibility of forming parallel strips is greater. Therefore, by combining the two positively correlated factors of gap density and crop row element average value, the gap density of each row and the crop row element average value are multiplied to obtain the representation degree value of each row. The larger the representation degree value of a row is, the more the row can be used to accurately identify the parallel strips formed by the crop rows in the original image. On this basis, each element in the row with the largest representation degree value can be used as the key result information. That is, the key result information includes a row of elements in the two-dimensional array.
[0175] In this embodiment, by multiplying the gap density and the average value of the crop row elements that are positively correlated with the key result information, the representation degree value of each row can be obtained, and then the elements of the row with the largest representation degree value can be used as the key result information, so that the key result information can be used to accurately identify parallel strips in the original image.
[0176] As an optional implementation, after obtaining the key result information, the key result information may be filtered in the following manner to remove interference information and improve the availability of learning samples.
[0177] Fig.11 A flowchart for filtering key result information is shown below. Fig.11 As shown, after determining the key result information, it also includes:
[0178] S1101. Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located.
[0179] Optionally, at least one continuous element sequence can be obtained based on the element value of each element of the row with the largest representation degree value. The continuous element sequence is a sequence composed of continuous crop row elements, and the elements in the continuous element sequence are all crop row elements. Specifically, the elements in the row with the largest representation degree value can be traversed, and whenever the element value of a traversed element is 0 or less than a threshold, that is, a non-crop row element is traversed, the crop row element before the traversed element and the element between the first crop row element after the previous continuous element sequence are taken as a continuous element sequence, and so on, until the row with the largest representation degree value is traversed.
[0180] For example, assuming that the elements of the row with the largest representation degree value are arranged in order: 35600653405326, where the value of 0 indicates that the element at that location is a non-crop row element, and the non-0 value indicates that the element at that location is a crop row element, then three consecutive element sequences can be obtained through the above process, namely: 356, 6534, and 5326.
[0181] S1102 . Select a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence.
[0182] By analyzing the element values of each element in the continuous element sequence, a suitable initial denoising threshold can be selected. For example, if the element values of most elements in the continuous element sequence are large, a large threshold can be selected as the initial denoising threshold.
[0183] S1103: Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filter the elements in the continuous element sequence to obtain denoised key result information.
[0184] In the first way, when filtering elements in a continuous element sequence based on an initial denoising threshold, the element value of the element whose element value is less than the initial denoising threshold can be adjusted, for example, the element value can be adjusted to 0 to make the element a non-crop row element.
[0185] In the second method, the initial denoising threshold can be multiplied by a preset coefficient to obtain a target denoising threshold. The target denoising threshold is used to filter the continuous element sequence. Specifically, the element value of the element whose element value is less than the target denoising threshold can be adjusted, for example, the element value is adjusted to 0, so that the element becomes a non-crop row element. Since the preset coefficient is integrated into the initial denoising threshold, the filtering process of the continuous element sequence can be made more flexible and controllable.
[0186] Optionally, before iteratively training the above initial extraction model, the initial extraction model may be first built and initialized.
[0187] As an optional implementation, the constructed initial extraction model may include at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameters of the last layer are preset values.
[0188] When initializing the initial extraction model, the initialization convolution parameters of the last layer can be specified. In the present application, by specifying the initialization convolution parameter value of the last layer as a preset value rather than a random value, the convergence speed of the initial extraction model can be faster.
[0189] As an example, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0190] By specifying the initial value of the bias as a negative number and the initial value of the weight as 0, the gradient explosion caused by too few positive samples in the training sample set can be avoided, making the initial extraction model converge faster.
[0191] The present application embodiment further provides a crop identification method, Fig.12 A schematic diagram of a crop identification method provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the method includes:
[0192] S1201. Acquire an image of a target area, where the image of the target area includes crop rows and weeds.
[0193] For example, the target area may be imaged by a drone, thereby obtaining an image including crop rows and weeds. The electronic device may obtain the image of the target area from the drone based on a communication connection with the drone.
[0194] S1202: Input the above-mentioned image of the target area into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model.
[0195] The crop recognition model is trained based on the aforementioned embodiment.
[0196] Optionally, the crop row information may include the location of the crop row and the location of the weeds in the image of the target area. Exemplarily, after the aforementioned iterative training process, the initial recognition model may have the ability to recognize the crop rows in the image. Then, for the crop recognition model obtained after the training, on the basis of recognizing the crop rows in the image, the plants that do not belong to the crop rows in the image may be recognized as weeds, thereby accurately identifying the weeds in the farmland.
[0197] In this embodiment, after the image of the target area is input into the crop recognition model, the crop row information output by the crop recognition model can be obtained. Since the crop rows can form parallel strips, and the parallel strip information is used as a constraint result during the model training process, the crop recognition model can accurately identify the weeds and greatly reduce the probability of missed judgment and wrong judgment.
[0198] Based on the same inventive concept, the embodiment of the present application also provides a crop recognition model training device corresponding to the crop recognition model training method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned crop recognition model training method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0199] Fig.13 The module structure diagram of the crop recognition model training device provided in the embodiment of the present application is as follows: Fig.13 As shown, the device comprises:
[0200] The training module 1301 is used to iteratively train the initial recognition model using the training sample set and the initial extraction result until a preset iteration stop condition is met; wherein the training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of a first straight line detection result of the original image; the crop row label information is used to characterize feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image; the initial extraction result is obtained by the initial extraction model predicting the second straight line detection result, and the second straight line detection result is obtained by performing Hough line detection on the features extracted by the initial recognition model.
[0201] The determination module 1302 is used to determine the current initial recognition model as a crop recognition model when the initial recognition model meets a preset iteration stop condition.
[0202] As an optional implementation, the training module 1301 is specifically used for:
[0203] The original image is input into the current initial recognition model to obtain an initial recognition result output by the current initial recognition model based on the original image, wherein the initial recognition result is used to indicate the crop row information obtained by the initial recognition model.
[0204] A current recognition loss function is constructed based on the crop row label information and the initial recognition result.
[0205] According to the detection label information and the initial extraction result, a current extraction loss function is constructed.
[0206] Based on the current recognition loss function and the current extraction loss function, the current loss function is constructed.
[0207] The current loss function is used to back-propagate the current initial recognition model to update the model parameters of the current initial recognition model.
[0208] As an optional implementation, the training module 1301 is specifically used for:
[0209] Based on the first weight of the initial recognition model and the second weight of the initial extraction model, the current recognition loss function and the current extraction loss function are weighted to construct a current loss function.
[0210] As an optional implementation, the training module 1301 is specifically used for:
[0211] Determine the intersection-over-union ratio of the detection label information and the initial extraction results.
[0212] The preset sample weight is used as the influencing factor to determine the cross entropy between the detection label information and the initial extraction result.
[0213] Based on the intersection-over-union ratio and the cross entropy, a current extraction loss function is constructed.
[0214] As an optional implementation, the training module 1301 is specifically used for:
[0215] The initial extraction model is back-propagated using the current extraction loss function to update the model parameters of the initial extraction model.
[0216] As an optional implementation manner, the detection label information of the first straight line detection result is obtained by the following steps:
[0217] Get multiple original images.
[0218] The plurality of original images are respectively subjected to image transformation to obtain a plurality of transformed images, wherein the transformed images are binary images or mask images of the original images.
[0219] Performing Hough line detection on the multiple transformed images respectively to obtain multiple first straight line detection results; wherein the first straight line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line.
[0220] According to the element values of the multiple rows of elements, key result information for characterizing the real position of the crop row in the original image is determined, and the key result information is used as the detection label information.
[0221] As an optional implementation, the training module 1301 is specifically used for:
[0222] A plurality of key factor information is determined according to the element values of the plurality of rows of elements, wherein the key factor information includes: gap density and an average value of crop row elements.
[0223] The key result information is determined according to the plurality of key factor information.
[0224] As an optional implementation, the multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines in the Hough space passing through the point corresponding to the position in the two-dimensional array.
[0225] The training module 1301 is specifically used for:
[0226] The crop row elements and non-crop row elements in each row are determined according to the element values of the elements in each row of the two-dimensional array.
[0227] The ratio of the number of non-crop row elements in each row to the sum of the number of crop row elements and non-crop row elements in each row is calculated to obtain the gap density of each row.
[0228] The average values of the element values of the crop row elements in each row are calculated respectively to obtain the average value of the crop row elements in each row.
[0229] As an optional implementation, the training module 1301 is specifically used for:
[0230] The product of the gap density of each row and the average value of the crop row elements is calculated to obtain the representation degree value of each row.
[0231] Each element of the row with the largest representation degree value is used as the key result information.
[0232] As an optional implementation, the training module 1301 is specifically used for:
[0233] At least one continuous element sequence is obtained according to the element value of each element in the row where the key result information is located.
[0234] A threshold is selected according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold of the continuous element sequence.
[0235] Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
[0236] As an optional implementation, the training module 1301 is specifically used for:
[0237] Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the first line detection result.
[0238] As an optional implementation, there is at least one hole convolution layer in the initial extraction model; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
[0239] As an optional implementation, the convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
[0240] Based on the same inventive concept, a crop identification device corresponding to the crop identification method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned crop identification method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0241] Fig.14 The module structure diagram of the crop identification device provided in the embodiment of the present application is as follows: Fig.14 As shown, the device comprises:
[0242] The acquisition module 1401 is used to acquire an image of a target area, where the image of the target area includes crop rows and weeds.
[0243] The recognition module 1402 is used to input the image of the target area into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model, wherein the crop recognition model is trained based on the method described in the aforementioned method embodiment.
[0244] The present application also provides a smart agriculture system, which includes:
[0245] The acquisition device is used to acquire an image of the target area.
[0246] The processing device is used for processing the image obtained by the acquisition device in the crop identification method described in the above method embodiment to obtain the crop row information in the image.
[0247] A graphic interface device is used to generate corresponding display information according to the crop row information.
[0248] It should be noted that the smart agriculture system can execute the method steps in the aforementioned method embodiment. In addition to the technical effects that can be achieved by the aforementioned method, it can facilitate users to view crop row information in the target area, and it is also convenient for the system to control the operating equipment to perform adaptive operations based on the crop row information, or send the crop row information to the operating equipment to enable the operating equipment to operate autonomously. For example, the operating equipment can accurately perform weeding operations along the location of the weeds.
[0249] The present application embodiment also provides an electronic device 150, such as Fig.15 As shown, it is a schematic diagram of the structure of the electronic device 150 provided in the embodiment of the present application, including: a processor 1501, a memory 1502, and optionally, a bus 1503. The memory 1502 stores machine-readable instructions executable by the processor 1501 (for example, Fig.13 The training module 1301 and the determination module 1302 in the device are executed in accordance with the instructions, or, Fig.14 The device acquires the execution instructions corresponding to the module 1401 and the identification module 1402, etc.), when the electronic device 150 is running, the processor 1501 communicates with the memory 1502 through the bus 1503, and when the machine-readable instructions are executed by the processor 1501, the method steps in the aforementioned method embodiment are executed.
[0250] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned object information extraction method are executed.
[0251] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0252] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.
[0253] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.
Claims
1. A crop recognition model training method, characterized in that: include: The initial recognition model is iteratively trained using the training sample set and the initial extraction result until a preset iteration stop condition is met; wherein the training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of a first straight line detection result of the original image; the crop row label information is used to characterize feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image; the initial extraction result is obtained by the initial extraction model predicting the second straight line detection result, and the second straight line detection result is obtained by performing Hough line detection on the features extracted by the initial recognition model; When the initial recognition model meets a preset iteration stop condition, the current initial recognition model is determined as a crop recognition model.
2. The method according to claim 1, characterized in that The iterative training of the initial recognition model using the training sample set and the initial extraction result includes: Inputting the original image into the current initial recognition model to obtain an initial recognition result output by the current initial recognition model based on the original image, wherein the initial recognition result is used to indicate crop row information obtained by the initial recognition model; Constructing a current recognition loss function according to the crop row label information and the initial recognition result; Constructing a current extraction loss function according to the detection label information and the initial extraction result; Based on the current recognition loss function and the current extraction loss function, the current loss function is constructed; The current loss function is used to back-propagate the current initial recognition model to update the model parameters of the current initial recognition model.
3. The method according to claim 2, characterized in that The current loss function is constructed based on the current recognition loss function and the current extraction loss function, including: Based on the first weight of the initial recognition model and the second weight of the initial extraction model, the current recognition loss function and the current extraction loss function are weighted to construct a current loss function.
4. The method according to claim 2, characterized in that: According to the detection label information and the initial extraction result, a current extraction loss function is constructed, including: Determine the intersection and union ratio of the detection label information and the initial extraction result; Using the preset sample weight as the influencing factor, determine the cross entropy between the detection label information and the initial extraction result; Based on the intersection-over-union ratio and the cross entropy, a current extraction loss function is constructed.
5. The method according to claim 2, characterized in that: After constructing the current extraction loss function, it also includes: The initial extraction model is back-propagated using the current extraction loss function to update the model parameters of the initial extraction model.
6. The method according to claim 4, characterized in that The detection label information of the first straight line detection result is obtained by the following steps: Get multiple original images; Performing image transformation on the multiple original images respectively to obtain multiple transformed images, where the transformed images are binary images or mask images of the original images; Performing Hough line detection on the multiple transformed images respectively to obtain multiple first line detection results; wherein the first line detection results include multiple rows of elements, each element corresponds to a straight line in the original image, the position of each element is used to represent the position of the corresponding straight line in the original image, and the element value of each element is used to represent the reliability of the straight line; According to the element values of the multiple rows of elements, key result information for characterizing the real position of the crop row in the original image is determined, and the key result information is used as the detection label information.
7. The method according to claim 6, characterized in that Determining key result information for characterizing the real position of the crop row in the original image according to the element values of the multiple row elements, including: Determining a plurality of key factor information according to the element values of the plurality of rows of elements, the key factor information comprising: gap density and an average value of crop row elements; The key result information is determined according to the plurality of key factor information.
8. The method according to claim 7, characterized in that The multiple rows of elements are stored in a two-dimensional array, the position of each element in the two-dimensional array is used to represent the position of a corresponding straight line in the original image, and the element value of each element in the two-dimensional array is used to indicate the number of straight lines passing through the point corresponding to the position in the two-dimensional array in the Hough space; The determining of multiple key factor information according to the element values of the multiple rows of elements includes: Determining crop row elements and non-crop row elements in each row according to element values of elements in each row of the two-dimensional array; The ratio of the number of non-crop row elements in each row to the sum of the number of crop row elements and non-crop row elements in each row is calculated to obtain the gap density of each row; The average values of the element values of the crop row elements in each row are calculated respectively to obtain the average value of the crop row elements in each row.
9. The method according to claim 7, characterized in that: The determining the key result information according to the plurality of key factor information includes: Calculate the product of the gap density of each row and the average value of the crop row elements to obtain the representation degree value of each row; Each element of the row with the largest representation degree value is used as the key result information.
10. The method according to claim 9, characterized in that After determining the key result information, the method further includes: Obtain at least one continuous element sequence according to the element value of each element in the row where the key result information is located; Selecting a threshold value according to the element value of each element in the continuous element sequence to obtain an initial denoising threshold value of the continuous element sequence; Based on the initial denoising threshold, or based on the initial denoising threshold and a preset coefficient, filtering processing is performed on the elements in the continuous element sequence to obtain denoised key result information.
11. The method according to claim 6, characterized in that The performing Hough line detection on the plurality of transformed images respectively to obtain a plurality of first line detection results includes: Taking the center point of each transformed image as the origin of the polar coordinate system, Hough line detection is performed on the transformed image to obtain the first line detection result.
12. The method according to claim 4, characterized in that In the initial extraction model, there is at least one hole convolution layer; and / or, in the initial extraction model, the initialization convolution parameter of the last layer is a preset value.
13. The method according to claim 12, characterized in that The convolution parameters of the last layer in the initial extraction model include bias and weight, the initial value of the bias is a negative number, and the initial value of the weight is 0.
14. A crop identification method, characterized in that: include: Acquire an image of a target area, wherein the image of the target area includes crop rows and weeds; The image of the target area is input into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model, wherein the crop recognition model is trained based on the method described in any one of claims 1-13.
15. A crop recognition model training device, characterized in that: include: A training module, used for iteratively training an initial recognition model using a training sample set and an initial extraction result until a preset iteration stop condition is met; wherein the training sample set includes: an original image containing crop rows, crop row label information in the original image, and detection label information of a first straight line detection result of the original image; the crop row label information is used to characterize feature information of the crop row in the original image, the first straight line detection result is obtained by performing Hough straight line detection on the original image, and the detection label information is used to characterize the true position of the crop row in the original image; the initial extraction result is obtained by predicting a second straight line detection result by the initial extraction model, and the second straight line detection result is obtained by performing Hough straight line detection on the feature extracted by the initial recognition model; The determination module is used to determine the current initial recognition model as the crop recognition model when the initial recognition model meets a preset iteration stop condition.
16. A crop identification device, characterized in that: include: An acquisition module, used for acquiring an image of a target area, wherein the image of the target area includes crop rows and weeds; A recognition module is used to input the image of the target area into a pre-trained crop recognition model to obtain crop row information in the image of the target area output by the crop recognition model, wherein the crop recognition model is trained based on the method described in any one of claims 1 to 13.
17. A smart agricultural system, characterized in that: include: An acquisition device, used for acquiring an image of a target area; a processing device, configured to process the image of the target area obtained by the acquisition device according to the crop identification method of claim 14, so as to obtain crop row information in the image of the target area; A graphic interface device is used to generate corresponding display information according to the crop row information.
18. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine-readable instructions to execute the steps of the crop recognition model training method as described in any one of claims 1 to 13 or the steps of the crop recognition method as described in claim 14.
19. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the crop identification model training method according to any one of claims 1 to 13 or the steps of the crop identification method according to claim 14.
Citation Information
Patent Citations
Weed area recognition method and device
CN112541383A
Method and device for identifying center line of crop planting row, agricultural machine and medium
CN113807137A
System and method for field pattern analysis
US20190108631A1
