An unmanned harvester wheat lodging recognition method and device and a storage medium

Abstract

The application relates to a wheat lodging recognition method and device of an unmanned harvester and a storage medium, the method comprising the following steps: acquiring a scene original image in front of the unmanned harvester; pre-processing the scene original image; detecting wheat lodging of the pre-processed image; judging a wheat lodging area according to a detection result; generating an information code according to a judgment result; and sending the information code. The wheat lodging recognition method and device of the unmanned harvester and the storage medium can solve the harvesting scheme of the unmanned harvester in the process of unmanned driving, can solve the problem that the shape and features of the wheat lodging are difficult to extract, and can solve the problem that other areas of the picture are too dark due to the direct sunlight of the camera.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of unmanned harvester wheat lodging identification, and in particular to an unmanned harvester wheat lodging identification method, device and storage medium. BACKGROUND

[0002] Wheat lodging is a phenomenon caused by wind, rain and other natural factors or abnormal growth of crops. During the harvesting process of the harvester, the lodged wheat has great randomness and instability, so the speed of harvesting the wheat and the depth of the cutter wheel need to be adjusted manually. However, in the current unmanned driving technology of the harvester, it is difficult to achieve good harvesting of the lodged wheat, resulting in missed harvesting or less harvesting. Therefore, timely identification of wheat lodging is crucial in unmanned harvesting technology.

[0003] Real-time identification of the wheat lodging area in the unmanned driving process through the camera vision can feed back the vehicle control end to realize a series of operation schemes for the lodged wheat, such as header adjustment and speed adjustment, thereby ensuring the harvesting of the lodged wheat. The application of computer vision and deep learning technology provides strong support for the field of wheat lodging identification. Using computer vision and image processing methods, the automatic identification of the lodged image can be realized through pre-processing, feature extraction and feature matching. On this basis, the fusion of deep learning technology and computer vision technology can greatly improve the accuracy and robustness of the lodged image recognition.

[0004] With the rapid development of machine vision, image processing and artificial intelligence technologies, the automatic identification of wheat lodging has made great progress. The commonly used detection methods include remote sensing and field detection. Remote sensing technology mainly obtains satellite images and aerial unmanned aerial vehicle images, obtains remote sensing images with appropriate resolution and coverage, and performs large-area image acquisition. Through the feature information of the image, the lodging identification analysis is performed. Field detection is mainly applied to small-area, detailed and timely scenes. At present, field detection technology includes traditional manual patrol, traditional image recognition scheme and wheat lodging automatic identification technology based on computer vision or deep learning.

[0005] Remote sensing technology has many advantages in identifying wheat lodging, but also has some disadvantages, such as remote sensing depends on the quality and resolution of remote sensing images, and the morphology and characteristics of lodging wheat in different regions have environmental differences, and remote sensing technology cannot obtain more detailed information features, and cannot meet the real-time information requirements in the unmanned driving process. Artificial patrol identification of the field lodging area has the problems of wide identification information, and cannot be effectively converted into geographic information data for use by the unmanned driving system. The above scheme only identifies the lodging area in advance, and records the action plan in advance to realize lodging harvesting during unmanned driving, and cannot realize real-time and on-site identification functions. The traditional visual image recognition lodging area method is based on artificial experience and rule setting, and when processing different lodging wheat morphology and wheat with different appearance characteristics due to different locations and environments, it is difficult to extract uniform features for image processing due to large interference. SUMMARY

[0006] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a wheat lodging identification method, device and storage medium for unmanned harvester.

[0007] In the first aspect, the present application provides a wheat lodging identification method for unmanned harvester, which comprises the following steps:

[0008] obtaining the original image of the scene in front of the unmanned harvester;

[0009] preprocessing the original image of the scene;

[0010] detecting the wheat lodging of the preprocessed image;

[0011] judging the wheat lodging area according to the detection result;

[0012] generating an information code according to the judgment result;

[0013] sending the information code.

[0014] Preferably, the step of obtaining the original image of the scene in front of the unmanned harvester comprises the following steps:

[0015] installing a camera on the unmanned harvester;

[0016] obtaining the original image of the scene in front of the unmanned harvester through the camera.

[0017] Preferably, the preprocessing of the original image of the scene comprises the following steps:

[0018] brightness adjustment processing of the original image of the scene;

[0019] size adjustment processing of the original image of the scene.

[0020] Preferably, the step of detecting wheat lodging in the pre-processed image comprises the steps of:

[0021] obtaining a pre-trained neural network model;

[0022] inputting the pre-processed image into the neural network model;

[0023] obtaining a lodging area mask output by the neural network model;

[0024] predicting a lodging area position and a lodging area size according to the lodging area mask.

[0025] Preferably, the step of obtaining a pre-trained neural network model comprises the steps of:

[0026] collecting wheat lodging data and constructing a data set;

[0027] performing data pre-processing on the data set;

[0028] training a YOLOv5 model using pre-processed data;

[0029] obtaining a trained neural network model.

[0030] Preferably, the step of performing data pre-processing on the data set comprises the steps of:

[0031] performing color space conversion on the data set;

[0032] performing image enhancement on the data set.

[0033] Preferably, the step of generating an information code according to the judgment result comprises the steps of:

[0034] obtaining a lodging area matrix;

[0035] obtaining a judgment area matrix;

[0036] calculating a product sum of the lodging area matrix and the judgment area matrix;

[0037] judging whether the product sum is greater than 0;

[0038] if yes, generating a down pressure unmanned harvester header and deceleration instruction;

[0039] if no, generating a recovery unmanned harvester header height instruction.

[0040] In a second aspect, the present application provides an unmanned harvester wheat lodging identification device, comprising:

[0041] an image acquisition module, configured to acquire a scene original image in front of the unmanned harvester;

[0042] An image processing module is configured to pre-process the original image of the scene.

[0043] A lodging detection module is configured to detect lodging of the wheat in the pre-processed image.

[0044] A region judging module is configured to judge a lodging region of the wheat according to the detection result.

[0045] An information code generating module is configured to generate an information code according to the judging result.

[0046] An information code sending module is configured to send the information code.

[0047] In a third aspect, an air conditioner control device is provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus.

[0048] The memory is configured to store a computer program.

[0049] The processor is configured to execute the program stored in the memory, so as to realize the steps of the unmanned harvester wheat lodging identification method according to any one of the embodiments of the first aspect.

[0050] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to realize the steps of the unmanned harvester wheat lodging identification method according to any one of the embodiments of the first aspect.

[0051] Compared with the prior art, the above technical solution provided by the embodiments of the present application has the following advantages:

[0052] The unmanned harvester wheat lodging identification method, device and storage medium provided by the present application can solve the harvesting scheme of the lodging wheat in the unmanned driving process, can solve the problem that the lodging wheat form and features are difficult to extract, and can solve the problem that other regions of the picture are too dark due to the direct sunlight on the camera. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present application and, together with the specification, serve to explain the principles of the application.

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained based on these drawings without creative labor.

[0055] Figure 1is a flowchart of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0056] Figure 2 is a structural diagram of a wheat lodging recognition device of an unmanned harvester provided by an embodiment of the present application.

[0057] Figure 3 is a structural diagram of an electronic device provided by the present application.

[0058] Figure 4 is a structural diagram of a non-transitory computer-readable storage medium provided by the present application.

[0059] Figure 5 is an image preprocessing process diagram of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0060] Figure 6 is an image sky brightness information calculation process diagram of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0061] Figure 7 is an image brightness adjustment process diagram of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0062] Figure 8 is an information decision process diagram of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0063] Figure 9 is an information sending process diagram of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application. DETAILED DESCRIPTION

[0064] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0065] Figure 1 is a flowchart of a wheat lodging recognition method of an unmanned harvester provided by an embodiment of the present application.

[0066] The present application provides a wheat lodging recognition method of an unmanned harvester, which comprises the following steps:

[0067] S1: acquiring a scene original image in front of the unmanned harvester;

[0068] In the embodiment of the present application, the step of acquiring the original image of the scene in front of the unmanned harvester comprises:

[0069] A camera is installed on the unmanned harvester.

[0070] The original image of the scene in front of the unmanned harvester is acquired by the camera.

[0071] Specifically, a camera assembly is installed on the unmanned harvester, and the original image of the scene in front of the current unmanned harvester is acquired by the camera assembly.

[0072] S2: pre-processing the original image of the scene;

[0073] In the embodiment of the present application, the step of pre-processing the original image of the scene comprises:

[0074] The original image of the scene is subjected to brightness adjustment processing.

[0075] The original image of the scene is subjected to size adjustment processing.

[0076] Specifically, the brightness and size of the original image are automatically pre-processed according to pre-set parameters, the image size is uniformly cropped or scaled to 640*640 by a dimension reduction method, and brightness adjustment helps to solve the problem of inability to identify wheat lodging caused by information loss in low-light areas under backlight view angle.

[0077] The image pre-processing process is as shown in Figure 5 First, the sky brightness information of the image is acquired under non-backlight conditions (normal illumination). Since the camera installation position is fixed, the calculated non-backlight sky brightness information can be used as the sky brightness reference a of the camera. Subsequently, only the sky brightness information b of the real-time picture needs to be calculated to complete the brightness adjustment. The sky part is reduced in brightness and the non-sky part is increased in brightness according to a and b, and then the image size is adjusted to 640*640.

[0078] The image sky brightness information calculation process is as shown in Figure 6 The image is composed of three channels of RGB, so the matrix of the three channels of RGB of the intercepted sky part is added according to the element position, and then the average value of the three channels is calculated as the sky brightness information. When the camera installation position is fixed, the sky part is basically fixed.

[0079] The image brightness adjustment process is as shown in Figure 7 The current camera real-time picture is calculated according to the process of Figure 4 The sky brightness information b is calculated, and the sky brightness reference a is known. Then, the formula y = ∑|b-a| / (w×h sky) Calculate the image brightness adjustment coefficient γ, where β is the sky brightness information of the current image, α is the sky brightness reference of the current camera, w is the image width or the width of the sky in the image, h sky is the height of the sky in the image,

[0080] Figure 7 In The obtained adjustment coefficient γ is used for brightness reduction processing on the sky part of the image, that is, each pixel is reduced by γ, and brightness gain processing is performed on the non-sky (wheat field) part of the image, that is, each pixel is added by γ. After the processing is completed, the problem of reduced brightness of the wheat field caused by backlight can be improved.

[0081] S3: Perform wheat lodging detection on the preprocessed image;

[0082] In the embodiments of the present application, the wheat lodging detection on the preprocessed image comprises the steps of:

[0083] Obtaining a pre-trained neural network model;

[0084] Inputting the preprocessed image into the neural network model;

[0085] Obtaining a lodging area mask output by the neural network model;

[0086] According to the lodging area mask, the position and size of the lodging area are predicted.

[0087] Specifically, the trained neural network model is applied in the program, given an input image x_input, and a segmentation mask m_predicted of the predicted lodging area is obtained through an inference function f_inference of the model, and the process is

[0088] m_predicted=f_inference(x_input, θ_optimal).

[0089] In order to map the identified lodging area mask m_predicted back to the original size image, coordinate change operation is performed through a scaling factor S and a translation vector T of pixel coordinates, and the formula is:

[0090] (x, y, w, h)_original=(x, y, w, h)_predicted / S+T,

[0091] Where (x, y, w, h)_original represents the position and size of the lodging area mapped back to the original size image, and (x, y, w, h)_predicted represents the predicted position and size of the lodging area. The lodging area mask is mapped back to the size of the original image, which is convenient for subsequent information decision.

[0092] In the embodiments of the present application, the step of obtaining the pre-trained neural network model comprises:

[0093] Collecting wheat lodging data and constructing a data set;

[0094] Data preprocessing is performed on the data set;

[0095] The preprocessed data is used to train the YOLOv5 model;

[0096] Obtaining the trained neural network model.

[0097] Specifically, the wheat lodging data is collected in advance, and the data set contains wheat lodging images and labeled lodging area positions and sizes, and the training set image D={x i , y i}, wherein x i represents the i-th image, and y i represents the lodging labeled area of the i-th image. Data preprocessing includes color space conversion and image enhancement steps to improve image quality and contrast and highlight lodging area features. The input image is preprocessed by a data preprocessing function F p (x i ), and the preprocessed image is x′ i =F p (x i ). The YOLOv5 model is used for training, and the model parameters are optimized to accurately predict the position and size of the lodging area. The training process can be represented as a minimization problem of the optimization objective, that is, solving the loss function L minimization of model parameters θ_optimal, represented as θ_optimal=argminL(θ), L(θ) represents the loss function between the model prediction result and the labeled result.

[0098] In the embodiments of the present application, the step of data preprocessing on the data set comprises:

[0099] Color space conversion is performed on the data set;

[0100] Image enhancement is performed on the data set.

[0101] Specifically, data preprocessing includes color space conversion and image enhancement steps to improve image quality and contrast and highlight lodging area features.

[0102] S4: judging the wheat lodging area according to the detection result;

[0103] S5: generating an information code according to the judgment result;

[0104] In the embodiments of the present application, the generating the information code according to the judgment result comprises the steps of:

[0105] obtaining a lodging area matrix;

[0106] obtaining a judgment area matrix;

[0107] calculating a product sum of the lodging area matrix and the judgment area matrix;

[0108] judging whether the product sum is greater than 0;

[0109] if yes, generating a down-press unmanned harvester header and deceleration instruction;

[0110] if no, generating a recovery unmanned harvester header height instruction.

[0111] Specifically, the generation of the instruction is realized by detecting the distance between the recognized lodging and the unmanned harvester and whether the distance is within the judgment area. For example, the judgment area is within 5 meters in front of the harvester. A single-channel zero matrix with the same size as the original image is established, and a 5-meter-wide and car-body-length car front area is drawn as the judgment area, and the element values in the area are replaced with 1. The information decision process is as shown in Figure 6 First, the lodging wheat area recognized by the visual recognition subsystem is extracted without changing the size and position of the lodging area, and then the lodging area and the judgment area are multiplied and summed, and the formula is th = ∑(f mat *are mat ),

[0112] where f mat is the lodging area matrix, and are mat is the judgment area matrix. When th > 0, it indicates that there is a lodging area in the judgment area, and therefore the generated instruction is to down-press the harvester header and decelerate. When th < 0, it indicates that there is no lodging area in the judgment area, and therefore the generated instruction is to recover the header height.

[0113] S6: sending the information code.

[0114] Specifically, when the information decision subsystem generates the control instruction, the signal sending subsystem quickly sends the control instruction to the unmanned vehicle control end in the form of a CAN signal, and the information sending process is as shown in Figure 7As shown, since there is no cutting table in the picture taken by the camera, it cannot be considered that the cutting table has been completely harvested when the fallen area disappears in the picture, and the time difference from the disappearance of the fallen area to the complete harvesting needs to be determined as the basis for the delay lifting of the cutting table according to the current speed and the length of the cutting table. Therefore, the delay calculation formula is delay = l / v, where l is the length of the cutting table, and v is the current speed of the vehicle. When the instruction is to press the cutting table, in order to avoid missing the fallen wheat, the instruction is not delayed. At the same time, it is ensured that the instruction will not be lost, and the signal sending subsystem periodically sends the current obtained instruction.

[0115] As Figure 2 The application provides a wheat lodging recognition device of unmanned harvester, which comprises:

[0116] An image acquisition module 10 is configured to acquire a scene original image in front of the unmanned harvester.

[0117] An image processing module 20 is configured to pre-process the scene original image.

[0118] A lodging detection module 30 is configured to detect wheat lodging in the pre-processed image.

[0119] A region judgment module 40 is configured to judge a wheat lodging region according to the detection result.

[0120] An information code generation module 50 is configured to generate an information code according to the judgment result.

[0121] An information code sending module 60 is configured to send the information code.

[0122] The application provides a wheat lodging recognition device of unmanned harvester, which can execute the above-mentioned steps to provide a wheat lodging recognition method of unmanned harvester.

[0123] It should be understood that the above specific embodiments of the application are only used for illustrative or explanatory purposes of the principles of the application, and do not constitute a limitation of the application. Therefore, any modification, equivalent replacement, improvement, etc. made without departing from the spirit and scope of the application shall be included in the protection scope of the application. In addition, the appended claims of the application are intended to cover all changes and modifications falling within the scope and boundary of the appended claims, or the equivalent forms of such scope and boundary.

[0124] Reference will now be made to Figure 3FIG. 1 shows a structural diagram of an electronic device 100 suitable for implementing embodiments of the present disclosure. The electronic device in embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (Personal Digital Assistant), a PAD (Tablet PC), a PMP (Portable Multimedia Player), a car terminal (e.g., a car navigation terminal), and the like, as well as a stationary terminal such as a digital TV, a desktop computer, and the like. Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functions and use range of embodiments of the present disclosure.

[0125] As shown in FIG. 1, the electronic device 100 can include a processing device (e.g., a central processor, a graphic processor, etc.) 101 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 102 or loaded into a random access memory (RAM) 103 from a storage device 108. In the RAM 103, various programs and data required for the operation of the electronic device 100 are also stored. The processing device 101, the ROM 102, and the RAM 103 are connected to each other through a bus 104. An input / output (I / O) interface 105 is also connected to the bus 104. Figure 3

[0126] Generally, the following devices can be connected to the I / O interface 105: input devices 106 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; output devices 107 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; storage devices 108 including, for example, a magnetic tape, a hard disk, and the like; and communication devices 109. The communication devices 109 can allow the electronic device 100 to communicate wirelessly or wiredly with other devices to exchange data. Although the electronic device 100 having various devices is shown in the drawing, it should be understood that all of the devices shown are not required to be implemented or possessed. More or less devices can be alternatively implemented or possessed.

[0127] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present disclosure include a computer program product including a computer program carried on a computer readable medium, the computer program containing program codes for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 109, or installed from the storage devices 108, or installed from the ROM 102. When the computer program is executed by the processing device 101, the above-described functions defined in the methods of the present disclosure are performed.

[0128] Reference will now be made to Figure 4 ​It shows a structural diagram of a computer readable storage medium suitable for being used to implement the embodiments of the present disclosure, the computer readable storage medium stores a computer program, and the computer program can implement the unmanned harvester wheat lodging identification method as described in any of the above when executed by a processor.

[0129] The unmanned harvester wheat lodging identification method, device and storage medium provided by the present application can solve the harvesting scheme of the lodged wheat in the unmanned driving process, can solve the problem that the form and features of the lodged wheat are difficult to extract, and can solve the problem that the other areas of the picture are too dark due to the direct sunlight of the camera.

[0130] It should be noted that, in this document, relational terms such as "first" and "second", and the like can be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... " does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0131] The above description is merely one specific implementation of the present application, which enables those skilled in the art to understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to these embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying lodging of wheat using an unmanned harvester, characterized in that, The method includes the following steps: The camera installed on the unmanned harvester captures the original image of the scene in front of the harvester; The original image of the scene is preprocessed; Wheat lodging detection is performed on the preprocessed images; Determine the areas of wheat lodging based on the test results; Generate an information code based on the judgment result; Send the aforementioned information code; The preprocessing of the original scene image includes: obtaining the sky brightness reference α of the camera under non-backlight conditions; calculating the sky brightness information β of the original scene image; calculating the image brightness adjustment coefficient γ based on the sky brightness reference α and the sky brightness information β, and performing brightness reduction processing on the sky portion of the original scene image and brightness gain processing on the non-sky portion; wherein, the image brightness adjustment coefficient γ is calculated using the following formula. Calculate, where, w It refers to the image width or the width of the sky in the image. It is the height of the sky in the image; The step of generating the information code based on the judgment result includes the following steps: Obtain the matrix of collapsed regions; Obtain the determination region matrix; Calculate the sum of the products of the collapsed region matrix and the determination region matrix; Determine whether the sum of the products is greater than 0; If so, generate a command to press down the unmanned harvester's header and decelerate; If not, generate a command to restore the header height of the unmanned harvester.

2. The method for identifying lodging of wheat using an unmanned harvester according to claim 1, characterized in that, The wheat lodging detection of the preprocessed image includes the following steps: Obtain a pre-trained neural network model; The preprocessed image is input into the neural network model; Obtain the mask of the collapsed area output by the neural network model; The location and size of the collapsed area are predicted based on the collapsed area mask.

3. The method for identifying lodging of wheat using an unmanned harvester according to claim 2, characterized in that, The steps of obtaining the pre-trained neural network model include: Collect wheat lodging data and construct a dataset; Perform data preprocessing on the dataset; The YOLOv5 model was trained using preprocessed data; Obtain the trained neural network model.

4. The method for identifying lodging of wheat using an unmanned harvester according to claim 3, characterized in that, The data preprocessing of the dataset includes the following steps: Perform color space conversion on the dataset; Image enhancement is performed on the dataset.

5. An unmanned harvester wheat lodging identification device for implementing the unmanned harvester wheat lodging identification method according to any one of claims 1-4, characterized in that, include: The image acquisition module is used to acquire original images of the scene in front of the unmanned harvester; The image processing module is used to preprocess the original image of the scene; The lodging detection module is used to detect wheat lodging in preprocessed images; The area determination module is used to determine the lodging area of ​​wheat based on the detection results; The information code generation module is used to generate an information code based on the judgment result; An information code sending module is used to send the information code.

6. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; When a processor executes a program stored in memory, it implements the steps of the unmanned harvester wheat lodging identification method according to any one of claims 1-4.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the unmanned harvester wheat lodging identification method as described in any one of claims 1-4.

Images