IMAGE ANALYSIS METHOD AND SELF-PROPELLED FIELD CHOPPER
Patent Information
- Application Number
- AT2024168633T
- Authority / Receiving Office
- AT · AT
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2023-06-22
- Filing Date
- 2024-04-05
- Publication Date
- 2026-06-15
- Estimated Expiration
- 2044-04-05
Abstract
Description
[0001] The present invention relates to an image analysis method according to the preamble of claim 1. Furthermore, the present invention relates to a self-propelled forage harvester according to the preamble of claim 12.
[0002] A method for the computer-implemented determination of the degree of grain disruption of grains within a crop stream processed by the working units of a forage harvester, which comprises whole grains and crushed grains as grain components, as well as non-grain components, is known from DE 10 2014 011 308 A1. Images of the crop stream are cyclically captured by a camera system and transmitted to an image analysis device for evaluation. Using the method, images of grain-like particles are identified in the images and subsequently sorted into two size fractions. The degree of grain disruption is determined by determining the thickness of the two size fractions. At least one working unit is controlled depending on the determined degree of grain disruption.
[0003] DE 10 2020 122 202 A1 discloses an image recognition algorithm based on machine learning. The forage harvester described in DE 10 2020 122 202 A1 has a camera system for recording image data of the crop contained in the crop stream. This data is analyzed using the image recognition algorithm to determine non-grain components from the geometric properties according to a predetermined calculation rule in order to determine the structural content of the non-grain components in the crop stream.
[0004] Based on the above-mentioned prior art, the invention is based on the object of developing an image analysis method of the type mentioned at the outset, which improves the prediction of the determination of whole grains and crushed grains in the crop stream for the computer-implemented determination of the degree of grain disruption.
[0005] This object is achieved according to the invention by an image analysis method having the features of claim 1. Advantageous further developments are the subject of the dependent claims.
[0006] According to claim 1, an image analysis method for the computer-implemented determination of the degree of grain disruption of grains within a crop stream processed by working units of a forage harvester, which comprises whole grains and crushed grains as grain components as well as non-grain components, is proposed, wherein images of the crop stream are cyclically recorded by means of a camera system and transmitted to an image analysis device for evaluation, wherein at least one working unit is controlled depending on the determined degree of grain disruption.
[0007] The image analysis method according to the invention is characterized in that, in order to determine the degree of grain disruption, image pixels contained in the images are classified into grain components and non-grain components by the image analysis device, within the image pixels of a recorded image classified as grain components, a classification of whole grains and crushed grains is carried out by means of a segmentation model, and that a loss function used by the segmentation model is weighted with an adjustable weighting factor.
[0008] The image analysis method according to the invention is based on the consideration that, when creating training data obtained from image data through a manual annotation process, non-grain components such as corn cob or corn stalk fragments or the like are identified and classified as grains and crushed grains, which is referred to as "false positives" (FP), and that grains and crushed grains are not identified and classified as such, which is referred to as "false negatives" (FN). It was found that the proportion of grains classified as false positives (FP) or crushed grains generally predominates, which ultimately reduces the accuracy in determining the degree of grain disruption.
[0009] This influence on the accuracy of the proposed determination of the degree of grain disruption is compensated by weighting a loss function used by the segmentation model with an adjustable weighting factor.
[0010] The weighting factor takes into account the ratio of recognition rate and recognition accuracy.
[0011] For this purpose, the weighting factor can be set to a value greater than 0 and less than 1, in particular to a value between 0.2 and 0.5. Setting the weighting factor in the above-mentioned range results in the sensitivity, i.e. the probability with which a positive object is correctly classified as positive, being weighted more highly than the positive prediction value, i.e. the proportion of objects correctly classified as positive. With a weighting factor greater than 0 and less than 1, the sensitivity is weighted more heavily, so that grain components, whole grains and crushed grains, are more likely to be predicted if there is a high probability that these are actually grain components that are used to determine the degree of grain disruption. If the probability is lower, the particles are more likely not to be predicted.On the other hand, a value for the weighting factor greater than 1 leads to an increase in the probability that an object is classified as positive, whereby these are "True Positive" (TP), i.e. grain components correctly classified as grains or crushed grains, as well as "False Positive" (FP) objects, i.e. non-grain components are incorrectly classified as grains or crushed grains.
[0012] In particular, a weighting factor from a previously performed harvesting process on the field to be processed can be set as the initial weighting factor. This allows access to a weighting factor that has already been optimized in the past and can serve as the basis for the image analysis method. Alternatively, an initial weighting factor can be set within the value range of claim 2.
[0013] Furthermore, the initially set weighting factor can be adjusted iteratively. This can preferably be done as part of a regeneration of the segmentation model. In particular, the regeneration of the segmentation model can be performed during the year or in the following year.
[0014] In particular, images with a resolution in the range between 128 x 128 pixels and 1024 x 1024 pixels can be fed to the image analysis device for evaluation as input data from the camera system. Images with a resolution in the range between 256 x 256 pixels and 512 x 512 pixels are particularly preferably fed as input data from the camera system, since an image size in this range has sufficient resolution to enable even small grain components to be displayed with sufficient clarity in order to learn their characteristics during training and to predict them with a high degree of determination during subsequent image analysis. A higher resolution when acquiring the images, however, does not lead to a significant improvement in prediction accuracy. The increase in resolution, which is associated with an increased amount of data, leads to an increase in the inference time, which impacts the evaluation time.
[0015] Preferably, the weighting factor can be set as a function of limit values for inference time and achievable coefficient of determination of the segmentation model, wherein the limit value for the inference time is less than 30 ms and the limit value for the coefficient of determination is greater than 70%.
[0016] In particular, the image analysis method can be carried out by at least one neural network that uses a U-Net architecture as a segmentation model. The extraction of characteristic image features of the image using the neural network can take place in a common feature sub-module. A feature sub-module is also referred to as an encoder and, specifically in this case, as a feature extractor. MobileNet is preferably used here as the feature extractor. In particular, a U-Net architecture, whose input and output sizes are each selected to be 256 x 256 pixels, is characterized by an inference time of less than 30 ms, whereby the required coefficient of determination of greater than 70% is achieved.This makes it possible to carry out the image analysis process on the mobile hardware of a forage harvester, which is characterized by lower computing power than stationary hardware, such as might be found on a farm, for example.
[0017] Further preferably, the classification data determined by means of the segmentation model and the training data used for whole grains and crushed grains can be fed to the loss function, from which a loss value is determined which is used in an optimization step to adapt the weighting factor.
[0018] In particular, to classify whole grains and crushed grains, the length of a long major axis and a short major axis of each classified grain component can be determined using a length-width comparison. To calculate the degree of grain disruption, the quotient can be formed from the sum of the area of classified grain components that fall below an adaptive threshold for the length of the short major axes and the sum of the area of all classified grain components. The use of an adaptive threshold, particularly a dynamic one, to determine the degree of grain disruption, taking into account crop properties, takes into account the external influences that affect the actual grain size during plant growth.
[0019] For this purpose, the adaptive limit value can be adapted automatically and / or manually.
[0020] With automatic adjustment of the threshold, it is conceivable that a stored or retrievable preset initial threshold value is used at the beginning of the harvesting process, which is then adapted as the process progresses. For example, historical data containing information on previously cultivated crops and previous harvesting processes can be used as part of the documentation for a field.
[0021] Manual adaptation of the limit value can be performed by selecting from a predefined or predefinable value range for values of a minimum grain size and a maximum grain size and / or by entering at least one value of an average grain size that is valid for the harvesting process. Manual adaptation of the limit value can be supported by the image analysis method in such a way that, in the event of a significant deviation of the calculated mean value representing the average grain size from the manually specified value, a notification is generated for the operator. This notification can preferably contain a suggestion for manual adjustment of the limit value.
[0022] In particular, the adaptive limit can be adapted cyclically at intervals. Cyclical adaptation refers to the repeated adjustment of the adaptive limit within a definable period of time and / or depending on a definable crop throughput or a definable distance traveled in a field during the harvesting process.
[0023] From the sum of the areas of whole grains determined within the interval, a mean value for the visible area representing the mean grain size can be calculated. From this mean value, the limit value to be adapted is dynamically derived as a fraction of the long major axis and / or the short major axis. Thus, half of the short and / or long major axis can be used as the fraction, so that the grain is considered quartered. Alternatively, other fractions, such as thirds or fifths of the short and / or long major axis, are also conceivable. The adapted limit value updated in this way forms the basis for the image analysis method for determining the degree of grain disruption.
[0024] Furthermore, the object stated at the outset is achieved by a field chopper having the features of claim 12.
[0025] According to claim 12, a self-propelled forage harvester is proposed, comprising an attachment as a working unit for picking up crop material, working units for processing a crop flow generated from the picked up crop material, a driver assistance system which is designed and configured to control the working units, a camera system which cyclically records images of the crop flow and transmits them to an image analysis device which is designed and configured for image analysis using the image analysis method according to one of claims 1 to 11, in order to determine a degree of grain disruption of grains in the crop flow, wherein the driver assistance system is designed and configured to control a working unit designed as a post-shredding device depending on the determined degree of grain disruption.
[0026] Reference may be made to all statements relating to the image analysis method according to the invention.
[0027] In particular, the image analysis device can be designed with a machine learning algorithm which is implemented as a neural network in the form of a U-Net architecture of a convolutional neural network or as a recurrent neural network.
[0028] Preferably, the forage harvester can have a camera system which is designed and configured to record and evaluate the crop flow processed by the working units, wherein the camera system comprises an RGB camera which is configured to record the crop flow flowing through a discharge spout of the forage harvester and is arranged in a housing arranged on the discharge spout, wherein a translucent viewing window is arranged in the discharge spout, past which the crop flow to be detected flows, as well as at least one light source arranged opposite the viewing window, the light beams of which are directed onto the crop flow, at least one mirror which deflects light reflected from the crop flow into a lens arranged on the RGB camera, wherein the RGB camera transmits images recorded from the crop flow to the image analysis device for evaluation.
[0029] Particularly preferably, the RGB camera can capture images at a frame rate in the range of 20 frames / second to 40 frames / second, the exposure time is between 5 microseconds and 25 microseconds, and the lens of the RGB camera has a focal length between 7 mm and 10 mm.
[0030] Particularly preferably, the RGB camera can capture images of the crop flow, in particular at a frame rate in the range of 25 frames / second to 35 frames / second. Further preferably, the exposure time can be in particular between 9 microseconds and 21 microseconds.
[0031] It is essential to consider optimally adapting the design of the RGB camera of the camera system, as well as the parameters essential for capturing images with the RGB camera, to the conditions prevailing in the discharge spout, in particular the flow velocity of the crop stream after exiting a secondary shredding device, which lies in the range of 15 m / s to 20 m / s. The images with the preferred resolution in the range between 256 x 256 pixels and 512 x 512 pixels are fed from the camera system as input data to the image analysis device, which is then processed with the machine learning algorithm, which is implemented as a neural network in the form of a U-Net architecture of a convolutional neural network or as a recurrent neural network.
[0032] With the inventive design and proposed parameterization of the RGB camera, an image analysis method for the computer-implemented determination of the degree of grain disintegration within the crop stream processed by the forage harvester's working units can be carried out using the image analysis device of the camera system. This method enables the differentiation of grain components and non-grain components with the required accuracy and, based on this, the differentiation between whole grains and crushed grains by optical sieving. The required accuracy for the differentiation of grain components and non-grain components, as well as the differentiation between whole grains and crushed grains, using the image analysis method is based on a predefined coefficient of determination.
[0033] The present invention is explained in more detail below with reference to an embodiment shown in the drawings.
[0034] They show: Fig. 1 schematically and exemplarily shows a forage harvester; Fig. 2 schematically and exemplarily shows a simplified representation of a camera system; Fig. 3 schematically and exemplarily shows an image of chopped crop taken by the camera system; Fig. 4 schematically and exemplarily shows a binary image of the image analyzed by an image analysis method according to Fig. 3 ; Fig. 5 schematically and exemplarily an enlarged section of the binary image according to Fig. 4 ; Fig. 6 schematically and exemplarily an evaluation of the binary image according to Fig. 5; Fig. 7 schematically and exemplarily a visualization of whole grains and crushed grains in an image of the chopped crop taken by the camera system; Fig. 8 schematically and exemplarily a pixel mask of the image analyzed by the image analysis method according to Fig. 7 ; Fig. 9 shows an exemplary diagram in which a curve for a coefficient of determination and a curve for a detected number of grain components are shown over a weighting factor; and Fig. 10 shows a simplified flow diagram of the image analysis method according to the invention.
[0035] Fig. 1shows schematically and by way of example a forage harvester 1 according to the invention harvesting a crop of plants, in particular maize plants 2, in a field. A receiving device 3 of the forage harvester 1 comprises, in a conventional manner, an interchangeable attachment 4 adapted to the plant material to be harvested, and an intake device 5 with several pairs of rollers 6, 7, which receives the crop from the attachment 4 to feed it to a chopping device 8.
[0036] The chopping device 8 comprises a rotationally driven chopping drum 9 and a counter-blade 10, over which the corn plants 2 are pushed by the adjacent pair of rollers 7 of the intake device 5 to be shredded by the interaction of the counter-blade 10 with the chopping drum 9. Downstream of the chopping device 8 is a secondary shredding device 13, also referred to as a corn cracker, with a pair of conditioning or cracker rollers 11 that define a gap 12 of adjustable width, hereinafter also referred to as the cracker gap, and rotate at different speeds to shred corn kernels contained in the material flow passing through the gap 12.A post-accelerator 14 imparts the shredded crop, here the corn plants 2, conditioned in the post-shredding device 13, the necessary speed to pass through a discharge spout 15 and be transferred into an accompanying vehicle (not shown). The discharge spout 15 has a substantially rectangular cross-section along its longitudinal extent. The discharge spout 15 has a continuously closed upper side 35 and a partially open lower side. Side walls are arranged orthogonally to the upper side 35 of the discharge spout 15, which laterally delimit and guide a crop flow 21 (illustrated by arrows) conveyed through the discharge spout 15.
[0037] At least one camera system 16 is arranged on the discharge spout 15 to generate images 44 of the crop flow 21 conveyed through the discharge spout 15. Furthermore, an NIR sensor 22 can be arranged on the discharge spout 15. Crop properties can be determined using the NIR sensor 22. Here, the NIR sensor 22 is preferably positioned upstream of the camera system 16 on the top side of the discharge spout 16.
[0038] The front attachment 4, the intake device 5, the chopping device 8, the secondary shredding device 13 as well as the secondary accelerator 14 and their respective components are working units 20 of the forage harvester 1, which serve to harvest the maize plants 2 of a field and / or to process the maize plants 2 of the field as part of the harvesting process.
[0039] Within the crop stream 21 processed by the working units 20 of the forage harvester 1 there are whole grains 23 and crushed grains 24 as grain components 25 as well as non-grain components 26, such as stems, leaves and the like.
[0040] The camera system 16 has an RGB camera 32 for recording image data of the crop contained in the crop stream 21. The RGB camera 32 records spatially resolved image data. The term "spatially resolved" here means that it is possible to distinguish details of the crop in the image data. The RGB camera 32 therefore has at least enough pixels to enable the proposed image analysis, which will be explained later. In a measurement routine, the camera system 16 uses the RGB camera 32 to capture image data of the crop in the crop stream 21, here the chopped corn plants 2. This measurement routine is carried out accordingly during operation of the forage harvester 1.
[0041] The images generated by the camera system 16 are transmitted to an image analysis device 27 and evaluated by it.
[0042] The image analysis device 27 is connected to a driver assistance system 17 or can be implemented as a component of the driver assistance system 17. The driver assistance system 17 can be connected to an input / output unit 18 in a driver's cab 19 of the forage harvester 1 in order to output evaluation results thereto. The driver assistance system 17 controls at least one actuator for adjusting the gap width of the cracker gap 12 and / or the differential speed and / or the speed levels of the rollers 11 of the secondary shredding device 13.
[0043] During operation, the rollers 11 of the secondary comminution device 13 each rotate at a speed that can be set as a parameter, with the gap 12 remaining between the rollers with a gap width that can be set as a parameter. Furthermore, the rollers 11 have a speed difference that can be set as a parameter, by which the speeds of the rollers 11 differ. The driver assistance system 17 controls at least one of the parameters depending on a grain disruption degree CSPS opt to be determined.
[0044] The reason for controlling the post-shredding device 13 depending on the degree of grain disruption is that, particularly when the harvested material is used as animal feed and in biogas plants, it is important that the grain components 25 of the harvested material are disrupted, i.e., disrupted. Disruption of the grain components 25 is important so that the starch contained therein becomes accessible and is not protected by the shell of the grain component 25. Disruption of the grain components 25 occurs, on the one hand, by chopping the harvested material and, on the other hand, essentially by the post-shredding device 13. The post-shredding device 13 can be adjusted to ensure that all grain components 25 contained in the harvested material stream 21 are disrupted, but this entails increased energy and fuel consumption.For example, to achieve maximum comminution and thus a high processing quality of the grain components, the gap width could be set to a minimum. This unnecessarily high energy consumption cannot be converted into an increase in travel speed, resulting in a correspondingly reduced area performance inherent in the system.
[0045] The proposed method for computer-implemented determination of the degree of grain disintegration (CSPS opt) of the grains 23 is explained below. For this purpose, cyclically recorded images 28 of the crop stream 21 are transmitted by the optical detection device 16 to the image analysis device 27 for evaluation using an image analysis method.
[0046] The schematically illustrated camera system 16 has an optical system in addition to the RGB camera 32. The optical system comprises a mirror 30, a lens 31 arranged on the RGB camera 32, and at least one light source 33. The RGB camera 32 has a field of view 34 in which it can detect light reflected from the crop stream 21. The RGB camera 32 and the optical system are arranged in a housing 28 of the camera system 16, which is attached to the top of the discharge spout 15. A translucent viewing window 29 is arranged on the side of the housing 28 facing the discharge spout 15. The viewing window 29 is preferably made of sapphire glass. The viewing window 29 can be round or polygonal.
[0047] The housing 28 of the camera system 16, located on the top side of the discharge spout 15, is arranged in the second half of the discharge spout 15, relative to its longitudinal extent. The housing 28 is releasably attached to the top side 35 of the discharge spout 15 by means of two mounting devices 36.
[0048] Fig. 2shows a schematic and exemplary simplified representation of the camera system 16. In the upper side 35 of the discharge spout 15, an opening is provided, into which the viewing window 29 is recessed, flush with the surface of the upper side 35 of the discharge spout 15 facing the crop stream 21. Here and preferably, the viewing window 29 and the opening are essentially circular. Alternatively, the viewing window 29 and the opening can be polygonal. The viewing window 29 can be glued into an essentially annular holder 38. The holder 38 is fastened in the housing 28. The holder 38 can be detachably fastened to the housing 28.
[0049] The viewing window 29 has a visible diameter D 29 that can be detected by the lens 31 and is greater than 7 cm and less than 13 cm. Particularly preferably, the viewing window 29 can have a detectable visible diameter D 29 that is greater than or equal to 9 cm and less than or equal to 12 cm. D denotes an overall diameter of the viewing window 29, which is round in the illustrated embodiment and includes an edge region between 2 mm and 4 mm, which serves to support the holder 38.
[0050] In the case of a polygonal design of the viewing window 29, an at least square design, the visible diameter D 29 detectable by the lens 31 is taken into account by the respective edge length.
[0051] When the viewing window 29 is glued into the holder 38, the necessary edge-side support surface is taken into account in the visible diameter D 29 of the viewing window 29, which can be detected by the lens 31 and is effective when recording the crop flow. The visible diameter D 29 of the viewing window 29, which can be detected by the lens 31, limits the field of view 34.
[0052] Preferably, the thickness of the viewing window 29 can be in the range between 2 mm and 4 mm. The thickness of the viewing window 29 depends essentially on the overall diameter D or, in the case of a polygonal design, the edge lengths of the viewing window 29.
[0053] The RGB camera 32 records images of the crop stream 21 at a frame rate in the range of 20 frames / second to 40 frames / second. Particularly preferably, the frame rate is in the range of 25 frames / second to 35 frames / second. The exposure time here is preferably between 5 microseconds and 25 microseconds. Particularly preferably, the exposure time is in the range between 9 microseconds and 20 microseconds. The lens 31 of the RGB camera 32 has a focal length between 7 mm and 10 mm. Furthermore, the lens 31 here and preferably has an angle of view in the range of 32° to 37°. Particularly preferably, the lens 31 can have an angle of view in the range of 34° to 35°.
[0054] The at least one mirror 30 in the housing 28 results in an object distance in the range of 175 mm to 195 mm. In particular, the dimensions of the mirror 30 are selected such that the angle of view of the lens 31 is taken into account and the entire viewing window of the viewing pane 29, which corresponds to the light-transmitting diameter D 29, can be visualized. The object distance in the range of 175 mm to 195 mm is realized by means of the at least one mirror 30, so that exceeding the maximum permissible height due to the height of the housing on the discharge spout 15 in road traffic is avoided. By using a focal length between 7 mm and 10 mm for an object distance in the range of 175 mm to 195 mm, undesirable artifacts such as image curvatures can be avoided, which are to be expected with shorter focal lengths and simultaneously shorter object distances.Particularly preferably, the object width can be in the range of 180 mm to 190 mm.
[0055] The camera system may include a control unit 37 for controlling the at least one light source 33. The control unit 37 for controlling the at least one light source 33 may preferably be arranged in the housing 28. At least one matrix LED spotlight is preferably used as the light source 33.
[0056] The position of the at least one light source 33 in the housing 28 can be adjusted vertically and / or horizontally relative to the viewing window 29 and the mirror 30. For this purpose, the light source 33 can be arranged in the housing 28 by means of a holding device 39, which has components that are movable relative to the housing 28. This enables calibration and fine adjustment.
[0057] In the illustrated embodiment, the holding device 39 comprises essentially L-shaped metal sheets 40, 41 as relatively movable components, which are arranged in pairs. One pair of the essentially L-shaped holding elements 40, 41 is arranged on each side of the light source 33. The holding elements 40, which extend in sections in the longitudinal direction of the housing 28, have horizontally extending, parallel elongated holes 42, within which the holding elements 40 can be displaced relative to the housing 28. The holding elements 41 have vertically extending, parallel elongated holes 43, within which the holding elements 41 can be displaced relative to the holding elements 40 and the housing 28, respectively.
[0058] The distance of the at least one light source 33 from the center of the viewing window 29 is preferably between 120 mm and 130 mm. The at least one light source 33 is preferably arranged at an angle of between 31° and 34° to the surface of the viewing window 29.
[0059] In addition, the RGB camera with the lens 31 arranged thereon can also be adjustable in the vertical direction and / or in the horizontal direction and / or its inclination.
[0060] In Fig. 3An image 44 of chopped crop material, here the corn plant 2, taken by the camera system 16 is shown schematically and by way of example. The grain components 25 and non-grain components 26 of the chopped corn plant 2 are visible in the image 44 taken by the camera system 16. The detection of grain components 25 and the differentiation of whole grains 23 from crushed grains 24 is only possible to a very limited extent using such an image 44.
[0061] In Fig. 4 is schematically and exemplarily a binary image 45 of the image 44 processed by an image analysis method according to Fig. 3The binary image 45 generated by semantic segmentation shows only grain components 25, while all non-grain components 26 visible in image 44 are not displayed. For this purpose, the detected grain components 25 can be displayed, for example, in white, while the detected non-grain components 26 are displayed uniformly, for example, in black, so that the detected non-grain components 26 form a uniform background for the visualization in the binary image 45.
[0062] In a first stage of the image analysis process, image pixels contained in the images 44 captured by the camera system 16 are classified into grain components 25 and non-grain components 26. This can be performed here, preferably using semantic image segmentation. Other computer-implemented computer vision methods that can be used for image analysis include, for example, object recognition or instance segmentation.
[0063] In a second stage of the image analysis process, a length determination of a long major axis 46 and a short major axis 47 of each classified grain component 25 is carried out by means of a length-width comparison, as in Fig. 5 shown as an example.
[0064] Fig. 5 shows schematically and exemplarily an enlarged section 45a of the binary image 45 according to Fig. 4. The largest value r min determined for the length of the short major axis 47 of each grain component 25 is used as the basis for the further image analysis procedure.
[0065] The first and second stages of the image analysis method are carried out by at least one neural network. The at least one neural network is a component of the image analysis device 27. Here, the at least one neural network preferably uses a U-Net architecture as a segmentation model. The extraction of characteristic image features of the image using the at least one neural network can take place in a common feature sub-module. A feature sub-module is also referred to as an encoder and, specifically in this case, as a feature extractor. Here, the MobileNet is preferably used as the feature extractor.
[0066] The determination of the long major axis 46 and the short major axis 47 of each classified grain constituent 25 for area determination is carried out cyclically at time-spaced intervals.
[0067] In Fig. 6 is schematically and exemplarily an extended evaluation of the binary image 45 according to Fig. 4 By multi-class classification using semantic image segmentation, an image 45b can be generated in which a distinction is made between the whole grains 23 as one class of the multi-class classification and the crushed grains 24 as another class of the multi-class classification.
[0068] According to the method of the invention, the multi-class classification can be used to determine an average grain size. This is important given that maize plants 2 from different regions and / or harvest years have different average grain sizes, which influences the determination of the degree of grain disruption.
[0069] The image analysis process begins with the capture of image 44 by camera system 16, which is transmitted to image analysis device 27. Subsequently, image 44 is classified using semantic segmentation to generate binary image 45. Based on the image pixels contained in the respective image 44, the grain components 25 and non-grain components 26 are classified. The binary image 45 thus generated, corresponding to image 44, contains only information about the grain components 25.
[0070] The length of the long major axis 46 and the short major axis 47 of each classified grain component 25 is then determined using a length-width comparison. Binarization allows the visible surface of the respective grain component 25 to be determined from the thus determined grain components 25 based on the pure number of pixels, and the length of the long major axis 46 and the short major axis 47 to be determined. The calculation of the grain fragmentation degree CSPS opt based on the image analysis is generally performed according to the following formula [1]. CSPS opt = ∑ A KB r 47 < r min A KBG
[0071] In this formula, CSPS opt denotes the degree of grain disruption determined by optical sieving, r min a limit value for the maximum length of the short major axis 47 of the detected grain components 25, A KB the visible area of grain components 25 whose length of the short major axis 47 is less than the limit value r min, and A KBG the visible area of all detected grain components 25. The limit value r min corresponds to a sieve opening width of an optical sieve. If grain components 25 pass through this optical sieve opening width, i.e. if a maximum length r 47 of the short major axis 47 of a grain component 25 falls below the limit value r min, the detected grain component 25 corresponds to a crushed grain 24. The limit value r min can correspond to the limit value of 4.75 mm used in laboratory tests according to the state of the art.
[0072] In order to be able to react to fluctuations in the actual grain size during the harvesting process, the grain disruption degree CSPS opt is calculated according to the following formula [2]. CSPS opt = ∑ A KB r 47 < r adapt A KBG
[0073] Here, r adapt denotes an adaptive limit value for the maximum length r 47 of the short main axis 47, exceeding which classifies the respective grain component 25 as a whole grain 23 and falling below which classifies the respective grain component 25 as a crushed grain 24.
[0074] The adaptive limit value r adapt is not kept constant during a harvesting process, but is cyclically adjusted in order to be able to react to fluctuations in the actual grain size.
[0075] The use of the, in particular dynamically adapted, limit value r adapt to determine the adaptive grain disruption rate CSPS opt taking into account changing crop properties takes into account the external influences that affect the actual grain size during plant growth. Thus, the grain disruption rate CSPS opt can be dynamically adapted to the actual harvesting conditions, which has a beneficial effect on the control of working units 20, in particular the post-shredding device 13, of the forage harvester 1.
[0076] Preferably, the adaptive limit value r adapt can be adapted cyclically at intervals. Cyclical adaptation refers to the repeated adaptation of the limit value r adapt within a definable period of time and / or depending on a definable crop throughput or a definable distance traveled on a field during the harvesting process. For this purpose, the multi-class classification is carried out as described above with reference to Fig. 6 was described. Using the multi-class classification, a further distinction is made between the whole grains 23 and the crushed grains 24 as grain components 25.
[0077] The length determination of the long major axis 46 and the short major axis 47 of each classified whole grain 23 as a grain component 25 is subsequently evaluated. A mean value A mKB representing the mean area of the whole grains 23 is calculated from the sum of the areas A KB of the whole grains 23 determined within the specific time interval. This mean value A mKB is used as a criterion for the mean grain size. The mean grain size can be determined from the polygons of the grain components 25 classified as "whole grain 23" using the determined short major axis 47 and long major axis 46 of the whole grain 23.An inertia factor, the time-spaced intervals, can be implemented in the process flow so that the mean grain size determined by length-width comparison is only updated within a selected time window, based on all whole grains 23 detected in the interval. A mKB = ∑ A KB B
[0078] The mean value A mKB calculated from the mean area of the whole grains 23 represents the actual grain size of the processed harvested crop. The adaptive limit value r adapt is then dynamically derived from the mean value A mKB as a fractional value B of the long main axis 46 and / or the short main axis 47 (see formula [4]). r adapt = A mKB B
[0079] For this purpose, the calculated adaptive limit value r adapt is formed as the quotient of the mean value A mKB and the fractional value B. Half of the short major axis 47 and / or the long major axis 46 can be used as the fractional value, so that the processed grain component 25 is considered quartered. Alternatively, other fractional values B, such as thirds or fifths of the short major axis 47 and / or the long major axis 46, are also conceivable.
[0080] The threshold value r adapt used in a previous interval is then compared with the adaptive threshold value r adapt calculated in the previous step. If a deviation is found, the previously calculated threshold value r adapt is used when performing the calculation. The adaptive threshold value r adapt can be adapted automatically and / or manually.
[0081] The secondary comminution device 13 and the driver assistance system 17 form an automated process unit. The automated process unit is designed and configured to optimize the parameters for controlling the secondary comminution device 13 depending on the determined grain separation degree CSPS opt and to specify the optimized parameters of the secondary comminution device 13.
[0082] For this purpose, the determined value for the grain breakdown degree CSPS opt is transmitted to the driver assistance system 17. The driver assistance system 17 uses the value for the grain breakdown degree CSPS opt determined according to the method to control at least the post-shredding device 13 depending on this value. The process unit formed by the post-shredding device 13 and the driver assistance system 17 is designed and configured to optimize at least one of the parameters of the post-shredding device 13 depending on the determined grain breakdown degree CSPS opt and to specify it to the post-shredding device 13. This allows the efficiency and quality of the chopping process to be improved.
[0083] The images 44 with a resolution in the range between 128 x 128 pixels and 512 x 512 pixels are fed to the image analysis device 27 as input data from the camera system 16 for continuous evaluation by the image analysis method.
[0084] The representation in Fig. 7 shows a schematic and exemplary visualization of whole grains 23 and chopped grains 24 in an image 48 of the chopped crop taken by the camera system 16. The polygons detected or annotated as chopped grains 24 have a length of their short main axis 47 that is less than the limit value r min . The polygons detected or annotated as whole grains 23 have a length of their short main axis 47 that exceeds the limit value r min .
[0085] The representation in Fig. 8 shows schematically and exemplarily a pixel image 49 of the image 48 analyzed by the image analysis method according to Fig. 7. In it, the whole grains 23, the crushed grains 24 and the non-grain components 26 are analogous to the binary image 45 according to Fig. 3 shown.
[0086] In contrast to the binary image 45, the pixel image 49 uses at least one neural network to illustrate the polygons calculated here, preferably using the U-Net architecture as a segmentation model. For this purpose, the values of a confusion matrix, which compares the frequency of the classification results and the test results, are determined in the pixel image 49. On the one hand, the frequency of classification of non-grain components 26, such as corn cob or corn stalk fragments or the like, as grains 23 and crushed grains 24, which is referred to as "false positive" (False Positive FP), and the frequency of classification of grains 23 and crushed grains 24 that are not identified and classified as such, which is referred to as "false negative" (False Negative FN), are determined and recorded in the confusion matrix.Secondly, the frequency of classification of grain components 25 correctly classified as grains 23 or crushed grains 24, which is a "True Positive" (TP) classification, as well as the frequency of correct classification of non-grain components 26, which is referred to as "True Negative" (TN), are determined and also recorded in the confusion matrix. In pixel image 49, the non-grain components 26 classified as "True Negative" (TN) are displayed in their entirety against a solid color background, analogous to images 45 and 45b.
[0087] The image analysis method according to the invention is based on the consideration that, when creating training data obtained from the image data or images 48 through a manual annotation process, non-grain components 26, such as corn cob or corn stalk fragments or the like, are identified and classified as grains 23 and crushed grains 24, and that grains 23 and crushed grains 24 are not identified and classified as such. In this case, it was found that the proportion of false positive (FP) classified grains 23 or crushed grains 24 generally predominates, which ultimately reduces the accuracy in determining the degree of grain disruption CSPS opt.
[0088] This influence on the accuracy of the proposed determination of the grain breakdown degree CSPS opt is compensated by weighting a loss function used by the segmentation model with an adjustable weighting factor β.
[0089] For this purpose, the weighting factor β can be set to a value greater than 0 and less than 1, in particular to a value between 0.2 and 0.5. Setting the weighting factor β in the above-mentioned range results in the sensitivity, i.e. the probability with which a positive object is correctly classified as positive, being weighted more highly than the positive prediction value, i.e. the proportion of objects correctly classified as positive. With a weighting factor greater than 0 and less than 1, the sensitivity is weighted more heavily, so that grain components 25, whole grains 23 and crushed grains 24 are more likely to be predicted, where there is a high probability that these are actually grain components 25 on which the determination of the grain breakdown degree CSPS opt is based. If there is a lower probability, grain components 25 are more likely not to be predicted.In contrast, a value for the weighting factor β greater than 1 leads to an increase in the probability that an object is classified as positive, whereby these are "True Positive" (TP) objects, ie grain components correctly classified as grains 23 or crushed grains 24, as well as "False Positive" (FP) objects, ie non-grain components 26 are incorrectly classified as grains 23 or crushed grains 34.
[0090] Fig. 9 shows an example of a diagram in which a curve 50 for a coefficient of determination R 2< as well as a curve 51 for a detected number n KB of grain components 25, which are determined on the basis of the training data provided by the at least one neural network of the image analysis device 27. As can be seen from the diagram according to Fig. 9As can be seen, the coefficient of determination R 2< resulting from the U-Net architecture as a segmentation model is highest for the weighting factor β, which is set to a value between 0.2 and 0.5.
[0091] The representation in Fig. 10shows a simplified flowchart of the image analysis method according to the invention. In method step 60, the images 44 generated by the camera system 16 are fed as input data to the image analysis device 27. In step 61, an initial weighting factor β i is fed to a first processing stage 62 of the U-Net architecture. The result of the first processing stage 62 is fed to at least one subsequent processing stage 63, which generates prediction data in the form of the binary image 45b in method step 64. In parallel, in method step 65, image data from an analyzed and annotated test data set is fed, which is processed in the subsequent method step 66 using a loss function, taking into account the initial weighting factor β i preset in method step 61, in order to determine a loss score therefrom in method step 67.In the subsequent method step 68, the weighting factor β is determined on the basis of the loss factor determined in the preceding method step 67, which is fed to the first processing stage 62 for further image analysis in method step 69.
[0092] The loss function used in step 66 can be a dice-loss function. Good results can be achieved using the dice-loss function, which ensures that the results of the rare classes are taken into account when the classes are unevenly distributed.
[0093] The initially set weighting factor β i is iteratively adjusted during the harvesting process to be performed by forage harvester 1. The weighting factor β of a previously performed harvesting process on the field to be processed can be set as the initial weighting factor β i. This allows access to a previously optimized weighting factor β that can be used as the basis for the image analysis process. Alternatively, an initial weighting factor β i can be set automatically or manually within the value range specified above. List of reference symbols
[0094] 1 forage harvester 34 Field of view 2 corn plant 35 Top of 15 3 Recording device 36 Mounting device 4 Attachment 37 Control unit 5 Feeding device 38 bracket 6 pair of rollers 39 Holding device 7 pair of rollers 40 Holding element 8 Chopping device 41 Holding element 9 chopping drum 42 slot 10 Counter blade 43 slot 11 Conditioning or cracker roller 44 Picture 12 gap 45 Binary image 13 Post-shredding device 45a Excerpt from 45 14 Post-accelerator 45b Binary image 15 discharge spout 46 Long main axis 16 Recording device 47 Short main axis 17 Driver assistance system 48 Picture 18 Input-output unit 49 Pixel image 19 Driver's cab 50 Course for coefficient of determination 20 Working unit 51 History for number of grain components 21 Crop flow 22 NIR sensor 60 Process step 23 Whole grains 61 Process step 24 Crushed grains 62 Processing level 25 Grain component 63 Processing level 26 Non-grain component 64 Process step 27 Image analysis device 65 Process step 28 Housing 66 Process step 29 Viewing window 67 Process step 30 Mirror 68 Process step 31 lens 69 Process step 32 RGB camera β Weighting factor 33 light source β i Initial weighting factor D Total diameter of 29 CSPS opt Grain disintegration degree D 29 diameter n KB Number of grain components TP Really positive R 2< Coefficient of determination TN Really negative r min limit FP False positives r adapt Adaptive limit FN False Negative
Claims
1. Image analysis method for computer-implemented determination of the degree of grain disruption (CSPS opt ) of grains (23) within a crop flow (21) processed by working units (20) of a forage harvester (1), which comprises whole grains (23) and comminuted grains (24) as grain components (25) as well as non-grain components (26), wherein by means of a camera system (16) images (44) of the crop flow (21) are cyclically recorded and transmitted to an image analysis device (27) for evaluation, wherein at least one working unit (20) depending on the determined grain disruption degree (CSPS opt ) is controlled, characterized in that to determine the degree of grain disruption (CSPS opt) by the image analysis device (27) - image pixels contained in the images (44) are classified into grain components (25) and non-grain components (26), - within the image pixels of a recorded image (44) classified as grain components (25), a classification of whole grains (23) and crushed grains (24) is carried out by means of a segmentation model, and - that a loss function used by the segmentation model is weighted with an adjustable weighting factor (β).
2. Method according to claim 1, characterized in that the weighting factor (β) is set to a value greater than 0 and less than 1, in particular to a value between 0.2 and 0.
5.
3. Method according to claim 1 or 2, characterized in that as initial weighting factor (β i ) a weighting factor (β) of a previously carried out harvesting process is set on the field to be worked.
4. Method according to claim 3, characterized in that the initially set weighting factor (β) is adjusted iteratively.
5. Method according to one of claims 1 to 4, characterized in that the images (44) with a resolution in the range between 128 x 128 pixels and 512 x 512 pixels are fed as input data from the camera system (16) to the image analysis device (27) for evaluation.
6. Method according to one of claims 1 to 5, characterized in that the weighting factor (β) is set depending on limit values for inference time and coefficient of determination of the segmentation model used, whereby the limit value for the inference time is less than 30 ms and the limit value for the coefficient of determination is greater than 70%.
7. Method according to one of the preceding claims, characterized in that the image analysis process is carried out by at least one neural network which uses a U-Net architecture as a segmentation model.
8. Method according to one of the preceding claims, characterized in that the loss function is fed with the classification data determined by means of the segmentation model and the corresponding training data of whole grains (23) and crushed grains (24), from which a loss value is determined which is used in an optimization step to adapt the weighting factor (β).
9. Method according to one of claims 1 to 8, characterized in that for the classification of whole grains (23) and crushed grains (24), a length determination of a long main axis (46) and a short main axis (47) of each classified grain component (25) is carried out by means of a length-width comparison, and that for the calculation of the grain breakdown ratio (CSPS opt ) is the quotient of the sum of the area of classified grain components (25) that meet an adaptive threshold (r adapt) for the length of the short main axes (47), and the sum of the area of all classified grain components (25).
10. Method according to claim 9, characterized in that the adaptive limit (r adapt ) is adapted automatically and / or manually.
11. Method according to claim 9 or 10, characterized in that the adaptive limit (r adapt ) is adapted cyclically at intervals.
12. Self-propelled forage harvester (1), comprising an attachment (4) as a working unit (20) for picking up harvested material, working units (20) for processing a crop flow (21) generated from the picked up harvested material, a driver assistance system (17) which is designed and configured to control the working units (20), a camera system (16) which cyclically records images (44) of the crop flow (21) and transmits them to an image analysis device (27) which is designed and configured for image analysis using the image analysis method according to one of claims 1 to 11, in order to determine a grain breakdown degree (CSPS opt ) of grains (23) in the crop stream (21), characterized in that the driver assistance system (17) for controlling a working unit (20) designed as a secondary crushing device (13) depending on the determined degree of grain disruption (CSPS opt ) is executed and set up.
13. Field chopper (1) according to claim 12, characterized in that the image analysis device (27) is designed with an algorithm for machine learning, which is designed as a neural network in the form of a U-Net architecture of a convolutional neural network or as a recurrent neural network.
14. Field chopper (1) according to claim 12 or 13, characterized in thatthe forage harvester (1) has a camera system (16) which is designed and configured to capture and evaluate the crop stream (21) processed by the working units (20), wherein the camera system (16) comprises an RGB camera (32) which is configured to capture the crop stream (21) flowing through a discharge chute (15) of the forage harvester (1) and is arranged in a housing (28) arranged on the discharge chute (15), wherein a translucent viewing window (29) is arranged in the discharge chute (15), past which the crop stream (21) to be detected flows, as well as at least one light source (33) arranged opposite the viewing window (29), the light beams of which are directed onto the crop stream (21), at least one mirror (30) which projects light reflected from the crop stream (21) into a lens (31) arranged on the RGB camera (32) redirects,wherein the RGB camera (32) transmits images (44) taken from the crop stream (21) to the image analysis device (27) for evaluation.
15. Field chopper according to claim 14, characterized in that the RGB camera (28) records the images (44) at a frame rate in the range of 20 frames / second to 40 frames / second, that the exposure time is between 5 microseconds and 258 microseconds and that the lens (31) of the RGB camera (32) has a focal length between 7 mm and 10 mm.