Method and system for predicting fruit yield for target site in greenhouse

Abstract

A method for predicting a fruit yield for a target site in a greenhouse, according to the present invention, may comprise: obtaining a plurality of images reflecting fruit growing at a target site over a predetermined period of time; obtaining fruit size classification values and fruit maturity classification values for the fruit depicted in the plurality of images; obtaining count values of fruit belonging to each of M*N fruit state classes on the basis of the fruit size classification values and the fruit maturity classification values for the fruit to generate first site state information for the target site; applying the first site state information to a pre-established fruit state transition model to generate first site predicted state information; and obtaining fruit yield prediction information for the target site on the basis of the first site predicted state information.

Description

Method and system for predicting fruit yield for a target site in a greenhouse

[0001] The present invention relates to a fruit yield prediction system, and more particularly, to a fruit yield prediction method and system for a target site in a greenhouse.

[0002]

[0003] Agriculture is an industrial sector that provides the fundamental energy for people to live, and has been evaluated as a large-scale industrial sector since ancient times.

[0004] However, the rate at which advanced technologies are applied appears to be low compared to other industries in comparison to its scale.

[0005] However, recently, there has been a movement to apply cutting-edge technologies in agricultural fields, which can be considered as a foreign field of cutting-edge technology, such as smart farms and smart greenhouses, and related technologies are being researched.

[0006] However, only technologies that can acquire greenhouse-related data are currently being researched.

[0007] Therefore, there is a need to develop a method and system for generating information that can be significantly helpful in greenhouse operation by utilizing data acquired from the greenhouse.

[0008]

[0009] One object of the present invention is to provide a method for predicting fruit yield for a target site in a greenhouse based on an image of the crop.

[0010] The problems to be solved by the present invention are not limited to the problems described above, and problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention pertains from this specification and the attached drawings.

[0011]

[0012] According to one embodiment of the present invention, there is provided a method for predicting a fruit yield for a target site of a greenhouse, comprising: acquiring a plurality of images reflecting fruits growing at the target site during a predetermined period of time, wherein the plurality of images are acquired through at least one image acquisition device mounted on a monitoring robot moving within the greenhouse; acquiring a fruit size classification value and a fruit maturity classification value for a plurality of fruits reflected in the plurality of images, wherein the fruit size classification value is acquired as a value corresponding to any one of M fruit size classes, and the fruit maturity classification value is acquired as a value corresponding to any one of N fruit maturity classes; acquiring a count value of fruits belonging to each of M*N fruit status classes based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits, thereby generating first site status information for the target site; wherein the M*N fruit status classes are classified by the M fruit size classes and the N fruit maturity classes. A method for predicting fruit yield may be provided, comprising: applying the first site state information to a pre-prepared fruit state transition model to generate first site predicted state information, wherein the first site predicted state information includes a predicted count value for each of M*N fruit state classes; and obtaining fruit yield predicted information for the target site based on the first site predicted state information.

[0013] The solutions to the problems of the present invention are not limited to the solutions described above, and solutions that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from this specification and the attached drawings.

[0014]

[0015] According to one embodiment of the present invention, a method for predicting fruit yield for a target site in a greenhouse based on an image of a crop may be provided.

[0016] The effects of the present invention are not limited to the effects described above, and effects not mentioned can be clearly understood by a person skilled in the art to which the present invention pertains from this specification and the attached drawings.

[0017]

[0018] FIG. 1 is a diagram for explaining a yield prediction system according to one embodiment.

[0019] FIG. 2 is a drawing for explaining a fruit detection module according to one embodiment.

[0020] FIG. 3 is a drawing for explaining a fruit detection module according to one embodiment.

[0021] FIG. 4 is a drawing for explaining a fruit size judgment module according to one embodiment.

[0022] FIG. 5 is a diagram for explaining a fruit maturity judgment module according to one embodiment.

[0023] FIG. 6 is a drawing for exemplarily explaining information about fruits according to one embodiment.

[0024] FIG. 7 is a diagram illustrating a site status information generation module according to one embodiment.

[0025] FIG. 8 is a diagram for explaining a yield prediction module according to one embodiment.

[0026] FIG. 9 is a diagram for explaining site status information obtained from past work according to one embodiment.

[0027] FIG. 10 is a diagram for explaining a fruit state transition probability matrix according to one embodiment.

[0028] FIG. 11 is a diagram for explaining a fruit state transition probability matrix according to one embodiment.

[0029] FIG. 12 is a diagram for explaining the creation of a fruit state transition model based on matrix operations according to one embodiment.

[0030] FIG. 13 is a diagram for explaining the creation of a fruit state transition model based on machine learning according to one embodiment.

[0031] FIG. 14 is a diagram illustrating selection of site state information obtained from past operations to obtain a fault state transition model for generating next parking site prediction state information according to one embodiment.

[0032] FIG. 15 is a diagram illustrating a selection of site state information obtained from past work to obtain a fault state transition model for generating specific parking site prediction state information according to one embodiment.

[0033] FIG. 16 is a diagram for explaining a method for predicting fruit yield for a target site according to one embodiment.

[0034] FIG. 17 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0035] FIG. 18 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0036] FIG. 19 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0037] FIG. 20 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0038]

[0039] Since the embodiments described in this specification are intended to clearly explain the idea of ​​the present invention to a person having ordinary skill in the art to which the present invention pertains, the present invention is not limited to the embodiments described in this specification, and the scope of the present invention should be interpreted to include modified or altered examples that do not depart from the idea of ​​the present invention.

[0040] The terms used in this specification have been selected from widely used terms, taking into account the functions of the present invention. However, these terms may vary depending on the intentions of those skilled in the art, precedents, or the emergence of new technologies. However, if a specific term is defined and used with an arbitrary meaning, the meaning of that term will be described separately. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the term and the overall content of this specification, rather than simply the name of the term.

[0041] The drawings attached to this specification are intended to facilitate explanation of the present invention, and the shapes depicted in the drawings may be exaggerated as necessary to help understand the present invention, and therefore the present invention is not limited by the drawings.

[0042] When an element or layer described herein is referred to as being “on” or “on” another element or layer, it may include not only directly on top of the other element or layer, but also cases where there are other layers or other components interposed therebetween.

[0043] Throughout this specification, identical reference numbers may, in principle, represent identical components.

[0044] The numbers (e.g., first, second, etc.) used in the description of this specification can be understood as identification symbols to distinguish one component from another.

[0045] The suffixes "module" and "part" used for components in the description of this specification are used or used interchangeably depending on the ease of writing the specification, and may not have distinct meanings or roles in themselves.

[0046] In this specification, if it is determined that a detailed description of the composition or function of a known document related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted as necessary.

[0047] According to one embodiment of the present invention, there is provided a method for predicting a fruit yield for a target site of a greenhouse, comprising: acquiring a plurality of images reflecting fruits growing at the target site during a predetermined period of time, wherein the plurality of images are acquired through at least one image acquisition device mounted on a monitoring robot moving within the greenhouse; acquiring a fruit size classification value and a fruit maturity classification value for a plurality of fruits reflected in the plurality of images, wherein the fruit size classification value is acquired as a value corresponding to any one of M fruit size classes, and the fruit maturity classification value is acquired as a value corresponding to any one of N fruit maturity classes; acquiring a count value of fruits belonging to each of M*N fruit status classes based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits, thereby generating first site status information for the target site; wherein the M*N fruit status classes are classified by the M fruit size classes and the N fruit maturity classes. A method for predicting fruit yield may be provided, comprising: applying the first site state information to a pre-prepared fruit state transition model to generate first site predicted state information, wherein the first site predicted state information includes a predicted count value for each of M*N fruit state classes; and obtaining fruit yield predicted information for the target site based on the first site predicted state information.

[0048] Here, the count value of the above fruits may be the number value of fruits belonging to each of the M*N fruit status classes.

[0049] Here, the above M and the above N may be identical to each other.

[0050] Here, the M fruit size classes may include a first fruit size class, a second fruit size class, and a third fruit size class, wherein the second fruit size class may be a class corresponding to a fruit larger than a fruit corresponding to the first fruit size class, and the third fruit size class may be a class corresponding to a fruit larger than a fruit corresponding to the second fruit size class, and the N fruit maturity classes may include a first fruit maturity class, a second fruit maturity class, and a third fruit maturity class, wherein the second fruit maturity class may be a class corresponding to a fruit larger than a fruit corresponding to the first fruit maturity class, and the third fruit maturity class may be a class corresponding to a fruit larger than a fruit corresponding to the second fruit maturity class.

[0051] Here, the M*N fruit condition classes include first to ninth fruit condition classes, wherein the first fruit condition class is a fruit condition class specified by the first fruit size class and the first fruit maturity class, the second fruit condition class is a fruit condition class specified by the second fruit size class and the first fruit maturity class, the third fruit condition class is a fruit condition class specified by the third fruit size class and the first fruit maturity class, the fourth fruit condition class is a fruit condition class specified by the first fruit size class and the second fruit maturity class, the fifth fruit condition class is a fruit condition class specified by the second fruit size class and the second fruit maturity class, the sixth fruit condition class is a fruit condition class specified by the third fruit size class and the second fruit maturity class, and the seventh fruit condition The class may be a fruit state class specified by a first fruit size class and a third fruit maturity class, the eighth fruit state class may be a fruit state class specified by a second fruit size class and a third fruit maturity class, and the ninth fruit state class may be a fruit state class specified by a third fruit size class and a third fruit maturity class.

[0052] Here, the pre-prepared fault state transition model includes a fault state transition relationship for the first to ninth fault state classes, and according to the fault state transition relationship, the degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site status information belong to the second fault state class of the first site prediction status information is greater than the degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site status information belong to the fourth fault state class of the first site prediction status information, and the degree to which the fault state transitions so that the fruits belonging to the second fault state class of the first site status information belong to the first fault state class of the first site prediction status information is less than the degree to which the fault state transitions so that the fruits belonging to the second fault state class of the first site status information belong to the fifth fault state class of the second site prediction status information. there is.

[0053] Here, the pre-prepared fault state transition model includes a fault state transition relationship for the first to ninth fault state classes, and according to the fault state transition relationship, when the count value for the other fault state classes except the count value for the first fault state class of the first site state information is corrected to 0 and the corrected first site state information is applied to the fault transition model, the count value for the second fault state class of the first site prediction state information may be greater than the count value for the fourth fault state class.

[0054] Here, the pre-prepared fault state transition model includes a fault state transition relationship for the M*N fault state classes, and the fault state transition relationship may include a fault state transition probability matrix.

[0055] Here, the fault state transition probability matrix may include probability values ​​that the faults belonging to each of the M*N fault state classes included in the first site state information will transition to each of the M*N fault state classes included in the first site prediction state information.

[0056] Here, the number of probability values ​​included in the above-mentioned fruit state transition probability matrix can be (M*N)^2.

[0057] Here, the M fruit size classes include a first fruit size class, a second fruit size class, and a third fruit size class, the second fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the first fruit size class, the third fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the second fruit size class, the N fruit maturity classes include a first fruit maturity class, a second fruit maturity class, and a third fruit maturity class, the second fruit maturity class is a class corresponding to a fruit larger than a fruit corresponding to the first fruit maturity class, the third fruit maturity class is a class corresponding to a fruit larger than a fruit corresponding to the second fruit maturity class, and the M*N fruit state classes include first to ninth Including a fruit state class, a first fruit state class is a fruit state class specified by a first fruit size class and a first fruit maturity class, a second fruit state class is a fruit state class specified by a second fruit size class and a first fruit maturity class, a third fruit state class is a fruit state class specified by a third fruit size class and a first fruit maturity class, a fourth fruit state class is a fruit state class specified by a first fruit size class and a second fruit maturity class, a fifth fruit state class is a fruit state class specified by a second fruit size class and a second fruit maturity class, a sixth fruit state class is a fruit state class specified by a third fruit size class and a second fruit maturity class, and a seventh fruit state class is a fruit state class specified by a first fruit size class and a third fruit maturity class. The 8th fruit condition class is a fruit condition class specified by the 2nd fruit size class and the 3rd fruit maturity class.If the 9th fruit state class is a fruit state class specified by the 3rd fruit size class and the 3rd fruit maturity class, the fruit state transition probability matrix is ​​designed so that the probability that fruits belonging to the 2nd fruit state class, the 5th fruit state class, and the 8th fruit state class, whose fruit size class is the 2nd fruit size class, can transition to the 1st fruit state class, the 4th fruit state class, and the 7th fruit state class, whose fruit size class is the 1st fruit size class, is 0, and the probability that fruits belonging to the 4th fruit state class, the 5th fruit state class, and the 6th fruit state class, whose fruit maturity class is the 2nd fruit maturity class, can transition to the 1st fruit state class, the 2nd fruit state class, and the 3rd fruit state class, whose fruit maturity class is the 1st fruit maturity class, is 0. can be designed to be 0.

[0058] Here, the probability that the fruit belonging to the fourth fault state class can transition to the fifth, sixth, eighth, and ninth fault state classes is designed to be 0, and the probability that the fruit belonging to the fifth fault state class can transition to the fourth, sixth, seventh, and ninth fault state classes is designed to be 0.

[0059] Here, the plurality of images may be images captured by at least one image acquisition device mounted on the monitoring robot while the monitoring robot moves along a rail positioned in the greenhouse within the predetermined period of time.

[0060] Here, the difference between the time at which the first image among the plurality of images is acquired and the time at which the last image among the plurality of images is acquired may be shorter than the predetermined period.

[0061] Here, the plurality of images may not include images expressing the same crop row for different days.

[0062] Here, the fruit size classification value is determined based on the length of the predetermined axis of the target fruit area or the width of the target fruit area, and the fruit maturity classification value can be determined based on the pixel values ​​of pixels included in the target fruit area.

[0063] Here, the pre-prepared fault state transition model includes a plurality of fault state transition models, and applying the first site state information to the pre-prepared fault state transition model to generate the first site predicted state information may include: obtaining information about the current time; selecting a fault state transition model based on the information about the current time; and applying the first site state information to the selected fault state transition model to generate the first site predicted state information.

[0064] Here, generating first site predicted state information by applying the first site state information to a pre-prepared fault state transition model may include: obtaining information about a specific point in time; determining the number of times the fault state transition model is applied based on the information about the specific point in time; and generating first site predicted state information by applying the fault state transition model to the first site state information the number of times it is applied determined.

[0065] Here, obtaining fruit yield prediction information for the target site based on the first site prediction status information may include: obtaining information on fruit condition classes that are targets of harvesting; specifying prediction count values ​​belonging to the fruit condition classes that are targets of harvesting among prediction count values ​​included in the first site prediction status information; and obtaining fruit yield prediction information based on the specified prediction count values.

[0066] Here, in order to obtain fruit yield prediction information based on specific predicted count values, pre-stored reference weight value information for each fruit condition class or each fruit size class can be used.

[0067] According to another embodiment of the present invention, a method for predicting a yield in a greenhouse comprises: acquiring images for a first period through at least one image acquisition device mounted on a monitoring robot, wherein the images represent fruits set at various points in time before the first period; acquiring fruit property information about fruits in the greenhouse based on the acquired image data, wherein the fruit property information includes first property information related to fruit growth in which a degree of change in an early stage of a fruit development period is greater than a degree of change in a later stage, and second property information related to fruit ripening in which a degree of change in a later stage of the fruit development period is greater than a degree of change in an early stage; A method for predicting yield of a greenhouse may be provided, including: generating site state information for a first period of the greenhouse by reflecting fruit attribute information for fruits in the greenhouse, wherein the site state information includes a distribution of fruit state classes of fruits, and the fruit states are specified according to the first attribute information and the second attribute information; generating site predicted state information for a second period by applying the site state information for the first period to a fruit state transition model, wherein the site predicted state information includes fruit state classes corresponding to the site state information; and generating predicted yield information for the second period by using the site predicted state information for the second period.

[0068]

[0069] I. [Problems of prior art and contents disclosed in this application to solve them]

[0070] Predicting yields in a greenhouse where crops are grown is directly related to predicting greenhouse sales, and farming strategies can be established based on the predicted yields, so predicting yields in a greenhouse where crops are grown is important.

[0071] Recently, there has been an active trend toward smart farms that acquire sensing data on various environmental indicators of greenhouses (e.g., temperature, CO2 concentration, etc.), use this data to derive greenhouse-related information, and control the greenhouse.

[0072] In this smart farm field, technologies have been developed to predict greenhouse yields by considering various sensing data obtained from various sensors placed in the greenhouse.

[0073] However, it may be difficult to view the various environmental indicators of the greenhouse as information about the crops located in the greenhouse.

[0074] Therefore, in order to predict greenhouse yields more accurately, technologies need to be developed to predict greenhouse yields based on images of crops.

[0075] At this time, the image in which at least one crop is expressed can be understood as containing information about the surface shape and color of the crop located in the greenhouse.

[0076] However, even if an image of at least one crop is considered, it is difficult to predict the future state of the crop using only specific state information at a specific point in time. Therefore, in order to predict the yield by considering an image of at least one crop, it is necessary to consider changes between states at multiple points in time.

[0077] However, in order to track changes between the states of crops at various points in time, it is necessary to identify the same fruit at a previous and subsequent point in time, but in reality, it is difficult to track changes in the state of the same fruit at a previous and subsequent point in time in a greenhouse.

[0078] This may be because, in the case of greenhouses where crops are actually grown, monitoring robots are used to move around the greenhouse and acquire images of the crops to obtain images of at least one crop.

[0079] To be more specific, if the monitoring robot acquires an image of the first crop at the first point in time on the first day and the monitoring robot acquires an image of the first crop at the second point in time on the second day, even in the case of images of the same first crop, the locations of fruits located on the first crop may differ greatly between the first point in time and the second point in time, so it can be said that it is practically impossible to identify the same fruits at the first point in time and the second point in time and track the change in status.

[0080] Accordingly, according to the contents disclosed in the present application, a method for predicting yield based on images of crops can be provided by collectively indexing the 'individual state of fruits' and based on this, so as to solve the difficulties of the prior art.

[0081]

[0082] II. [Design of a method for predicting yield by grouping the 'individual condition of fruits' and using this]

[0083] 1. [Design of a site to collectively index the 'individual status of fruits']

[0084] According to the contents disclosed by the present application, a collective index for the 'individual state of fruits' can be obtained for each designated site.

[0085] For example, for a first site where the first to tenth crops are located, a first group index, which is a group index of the 'individual state of fruits' located in the first to tenth crops, can be obtained, and for a second site where the eleventh to twentieth crops are located, a second group index, which is a group index of the 'individual state of fruits' located in the eleventh to twentieth crops, can be obtained.

[0086] Additionally, according to the contents disclosed by the present application, group indices acquired at different points in time for each site can be used to track changes in the 'individual status of fruits' grouped for that site.

[0087] For example, for a first site where first to tenth crops are located, a first group index, which is a group index of the 'individual state of fruits' located in the first to tenth crops at a first point in time, and a second group index, which is a group index of the 'individual state of fruits' located in the first to tenth crops at a second point in time after the first point in time, can be used to track changes in the 'individual state of fruits' group-indexed for the first site.

[0088] At this time, the term "site" may mean a part of an area within a greenhouse where fruits that are the subject of group indexing are located, and may mean a part of an area within a greenhouse where crops that are the subject of group indexing are placed.

[0089] Additionally, the site described above may be the entire greenhouse, a section of the greenhouse, a collection of specific rows of crops in the greenhouse, a specific row in the greenhouse, or a portion of a specific row in the greenhouse.

[0090] At this time, the size and number of the above-described sites can be designed based on the speed of the monitoring robot, the battery of the monitoring robot, the number of monitoring robots, the size of the greenhouse, the number of crop rows located in the greenhouse, etc.

[0091] The design of the site as described above may be a design to solve conventional technical problems as a result of reflecting the locational characteristics of crops and fruits located in the crops in the greenhouse.

[0092] That is, although the locations of fruits in a single crop may change significantly over time, the locations of each crop in the greenhouse can be controlled to remain within a designated location range, so the design of the site for obtaining the aforementioned site-specific group index can be a design that can track changes in the 'individual status of fruits' located in crops that can be tracked as the same crop, rather than tracking fruits that cannot be tracked individually.

[0093]

[0094] 2. [Design of parameters to collectively index the 'individual status of fruits']

[0095] According to the contents disclosed by the present application, parameters for collectively indexing the 'individual state of fruits' can be designed as attributes related to the size of the fruits and attributes related to the maturity of the fruits.

[0096] At this time, the attribute related to the size of the fruit may mean attribute information related to the size of the fruit obtained using information measured from at least one sensor.

[0097] For example, the attribute related to the size of the fruit may mean attribute information related to the size of the fruit judged based on an image in which at least one crop is expressed.

[0098] For a more specific example, the attribute related to the size of the fruit may mean information determined based on the size of the pixel area when a pixel area corresponding to one fruit is specified within an image in which at least one crop is expressed, and the size of the pixel area may be a value calculated as a length in one axis direction of the pixel area, an area of ​​the pixel area, etc., and a method for obtaining the attribute related to the size of the fruit will be described in more detail below.

[0099] Additionally, at this time, the attribute related to the maturity of the fruit may mean attribute information related to the maturity of the fruit obtained using information measured from at least one sensor.

[0100] For example, the attribute related to the maturity of the fruit may mean attribute information related to the maturity of the fruit judged based on an image in which at least one crop is expressed.

[0101] For a more specific example, the attribute related to the maturity of the fruit may mean information determined based on pixel values ​​(e.g., color values, R, G, B values, etc.) of pixels corresponding to one fruit within an image in which at least one crop is expressed, and a method for obtaining the attribute related to the maturity of the fruit will be described in more detail below.

[0102] In addition, according to the contents disclosed by the present application, designing the parameters for collectively indexing the 'individual state of fruits' as attributes related to the size of the fruits and attributes related to the maturity of the fruits enables tracking the change in the collectively indexed 'individual state of fruits' more accurately than using other attributes of the fruits as parameters or using only one of the attributes related to the size of the fruits or the attributes related to the maturity of the fruits as parameters.

[0103] This may be because typical fruits develop through the fruit set stage, fruit growth stage, and fruit ripening stage during the fruit development period, which is the period from the time the fruit is formed until the fruit becomes a ripe fruit.

[0104] To explain more specifically, typical fruits go through a fruit growth stage after the fruit setting stage, and a fruit ripening stage after the fruit growth stage. During the fruit growth stage, the fruit enlarges through cell division and cell enlargement of the fruit, which can be observed as a phenomenon in which the size of the fruit changes on the surface, and during the fruit ripening stage, the various components that make up the fruit change so that it has a taste and appearance unique to the variety, which can be observed as a phenomenon in which the color of the fruit changes on the surface.

[0105] That is, rather than using other attributes of the fruit as parameters for group indexing the 'individual state of the fruits', using attributes related to the size of the fruit and attributes related to the maturity of the fruit as parameters for group indexing the 'individual state of the fruits' can be seen as more directly measuring and utilizing the superficial changes of the fruits during the fruit development period, making it possible to track the changes in the 'individual state of the fruits' group indexed more accurately.

[0106] In addition, if only the attributes related to the size of the fruit are used as parameters to collectively index the 'individual state of the fruits', it may be possible to track changes in the fruits going through the fruit growth stage, but it is almost impossible to track changes in the fruits going through the fruit ripening stage. In addition, if only the attributes related to the maturity of the fruit are used as parameters to collectively index the 'individual state of the fruits', it may be possible to track changes in the fruits going through the fruit ripening stage, but it is almost impossible to track changes in the fruits going through the fruit growth stage.

[0107] Therefore, in many cases, fruits located at a specific point in time in a greenhouse are set at different times rather than at the same time, and in such a situation, designing the parameters for group indexing the 'individual state of fruits' as attributes related to fruit size and attributes related to fruit maturity enables tracking the changes in the group-indexed 'individual state of fruits' more accurately than using only one of the attributes related to fruit size or fruit maturity as a parameter.

[0108]

[0109] 3. [Design of site status information to collectively index the 'individual status of fruits']

[0110] As described above, information that collectively indexes the 'individual status of fruits' by attributes related to the size of the fruits and attributes related to the maturity of the fruits for each site can be referred to as site status information.

[0111] Below, the site status information disclosed in the present application for solving the problems of the prior art will be described.

[0112] According to the contents disclosed by the present application, the site status information for collectively indexing the 'individual status of fruits' can be designed as a set of count values ​​of fruits belonging to each of a plurality of classes distinguished according to attributes related to the size of the fruits and attributes related to the maturity of the fruits.

[0113] At this time, the count value of the fruits may mean a value related to the distribution of fruits, such as the number of fruits belonging to each of multiple classes and the ratio of fruits.

[0114] That is, according to the contents disclosed by the present application, the site status information may mean a set of the number of fruits belonging to each of a plurality of fruit status classes specified by a class of attributes related to the size of the fruit and a class of attributes related to the maturity of the fruit.

[0115] For example, the site status information may be a set of the number of fruits belonging to a first fruit condition class specified by a first fruit size class and a first fruit maturity class, the number of fruits belonging to a second fruit condition class specified by a first fruit size class and a second fruit maturity class, the number of fruits belonging to a third fruit condition class specified by a second fruit size class and a first fruit maturity class, and the number of fruits belonging to a fourth fruit condition class specified by a second fruit size class and a second fruit maturity class.

[0116] In addition, according to the contents disclosed by the present application, designing the site status information as a set of count values ​​of fruits belonging to each of a plurality of classes distinguished according to attributes related to the size of the fruit and attributes related to the maturity of the fruit enables tracking changes in the 'individual status of fruits' grouped as group indicators more accurately than designing the site status information as a one-dimensional group indicator such as the average value of the attribute related to the size of the fruit and the average value of the attribute related to the maturity of the fruit.

[0117] To be more specific, designing the site status information as a set of count values ​​of fruits belonging to each of a plurality of classes distinguished by attributes related to the size of the fruit and attributes related to the maturity of the fruit enables statistical tracking of status changes of each of a plurality of classes corresponding to a plurality of fruit states specified by attributes related to the size of the fruit and attributes related to the maturity of the fruit, and thus enables tracking of changes in the 'individual status of fruits' grouped as group indices more accurately than designing them as one-dimensional group indices, such as the average value of attributes related to the size of the fruit and the average value of attributes related to the maturity of the fruit.

[0118]

[0119] 4. [Design of a method for estimating site prediction status information using site status information]

[0120] In order to predict yield in a greenhouse, information on the condition of the fruits to be predicted at the time when the yield is to be predicted may be required, and for this purpose, site condition information designed as described above may be used.

[0121] Below, we will describe the design of a method for estimating site prediction status information using site status information.

[0122] According to the contents disclosed by the present application, site state information obtained from past operations can be used to estimate site prediction state information using site state information, and more specifically, a state transition model obtained based on site state information obtained from past operations can be used.

[0123] In addition, according to the contents disclosed by the present application, the state transition model may be a state transition function (e.g., a state transition matrix) acquired based on site state information acquired in past operations, an artificial intelligence model learned based on site state information acquired in past operations, etc., and various acquisition methods of the state transition model will be described in more detail below.

[0124] Additionally, according to the contents disclosed by the present application, site prediction status information can be obtained using site status information and a status transition model.

[0125] For example, by applying site state information acquired for a time interval corresponding to the present to a previously stored state transition model, site predicted state information for a time interval corresponding to the future can be acquired.

[0126] At this time, the site predicted status information may be information expressed in the same data format as the site status information described above, and may mean site status information predicted for a time period corresponding to the future.

[0127] That is, the site prediction status information may mean a set of predicted count values ​​of fruits belonging to each of a plurality of classes (as described above, the plurality of classes are distinguished based on attributes related to the size of the fruit and attributes related to the maturity of the fruit) for a specific time interval in the future.

[0128]

[0129] 5. [Design of a greenhouse yield prediction method using site prediction status information]

[0130] Greenhouse yields can vary depending on which fruits are harvested (the harvest threshold) and how much fruit exceeds the harvest threshold, and are typically expressed in terms of the weight of the harvested fruit.

[0131] Therefore, in order to predict the yield of a greenhouse using the site prediction status information, it may be necessary to set a harvest standard and estimate the fruit weight, and a specific description of a method for predicting the yield of a greenhouse using the site prediction status information will be provided below.

[0132]

[0133] III. [Yield Prediction System]

[0134] 1. [Yield Prediction System]

[0135] FIG. 1 is a diagram for explaining a yield prediction system according to one embodiment.

[0136] Referring to FIG. 1, a yield prediction system (1000) according to one embodiment may include a monitoring robot (1050) and a server (1060).

[0137] At this time, the yield prediction system (1000) according to one embodiment may be a system for predicting the yield of a greenhouse based on data on at least one crop located in the greenhouse (1010).

[0138] For example, a yield prediction system (1000) according to one embodiment may be a system for predicting the yield of a greenhouse based on an image representing at least one crop located in the greenhouse (1010).

[0139] A greenhouse (1010) according to one embodiment may mean a specific space in which a plurality of crops are located and a plurality of crops are grown, and a crop row in which a plurality of crops are arranged may be located in the greenhouse (1010).

[0140] For example, the greenhouse (1010) may have a first crop row (1020) and a second crop row (1030) in which a plurality of crops are arranged.

[0141] Additionally, at least one crop row positioned in a greenhouse (1010) according to one embodiment may be configured to include at least one sub-crop row.

[0142] For example, a first crop row (1020) located in the greenhouse (1010) may be configured to include a first sub-crop row (1021) and a second sub-crop row (1022), and a second crop row (1030) may be configured to include a third sub-crop row (1031) and a fourth sub-crop row (1032).

[0143] At this time, the above-described sub-crop row may mean a physically separated crop row as shown in Fig. 1, but is not limited thereto, and may mean different parts of a crop growing in different directions in one crop row.

[0144] For example, the first sub-crop row (1021) may refer to a row of crops specified by stems growing to the right among crops positioned in the first crop row (1020), and the second sub-crop row (1022) may refer to a row of crops specified by stems growing to the left among crops positioned in the first crop row (1020).

[0145] In addition, the concept of the sub-crop row described above is only described for convenience of explanation, and may be expressed by replacing it with various terms to express the right, left, etc. of a single crop row.

[0146] In one embodiment, the greenhouse (1010) may be positioned with at least one rail to guide the movement path of the monitoring robot (1050).

[0147] For example, a first rail (1040) may be positioned in the greenhouse (1010) to guide the movement path of the monitoring robot (1050).

[0148] At this time, at least one rail positioned in the greenhouse (1010) may be positioned between the plurality of crop rows so that the monitoring robot (1050) can move between the plurality of crop rows positioned in the greenhouse (1010).

[0149] For example, the first rail (1040) positioned in the greenhouse (1010) may be positioned between the first crop row (1020) and the second crop row (1030).

[0150] Additionally, for example, the first rail (1040) positioned in the greenhouse (1010) may be positioned between the first sub-crop row (1021) included in the first crop row (1020) and the third sub-crop row (1031) included in the second crop row (1030).

[0151] Of course, at least one rail positioned in the greenhouse (1010) may be placed in various areas, such as an area for entering between multiple crop rows in addition to between multiple crop rows.

[0152] A monitoring robot (1050) according to one embodiment may be a robot for obtaining monitoring data on crops located in the greenhouse (1010).

[0153] For example, a monitoring robot (1050) according to one embodiment may be a robot equipped with at least one image acquisition device and configured to move around the greenhouse (1010) to acquire at least one image depicting at least one crop.

[0154] At this time, the monitoring robot (1050) may include at least one communication unit.

[0155] For example, the monitoring robot (1050) may include at least one communication unit capable of wireless communication such as WiFi, Bluetooth, Beacon, Near Field Communication (NFC), ZigBee, WiGig, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), Ultra Wideband (UWB), and WiHD, but is not limited thereto.

[0156] Additionally, at this time, the monitoring robot (1050) may include at least one input unit.

[0157] For example, the monitoring robot (1050) may include at least one input unit such as a keyboard, a mouse, a key pad, a dome switch, a touch pad (static / capacitive), a button, etc., but is not limited thereto.

[0158] Additionally, at this time, the monitoring robot (1050) may include at least one positioning unit.

[0159] For example, the monitoring robot (1050) may include at least one positioning unit among various positioning units, such as a positioning unit through satellite communication such as a Global Positioning System (GPS) or a Global Navigation Satellite System (GNSS), a positioning unit through short-range communication such as Bluetooth, a Beacon, NFC (Near Field Communication), ZigBee, WiGig, RFID (Radio Frequency Identification), infrared Data Association (IrDA), and UWB (Ultra Wideband).

[0160] Additionally, the monitoring robot (1050) according to one embodiment can be moved on at least one rail located in the greenhouse (1010).

[0161] For example, the monitoring robot (1050) may be moved on the first rail (1040) located in the greenhouse (1010), but is not limited thereto.

[0162] Additionally, the monitoring robot (1050) according to one embodiment may include at least one sensor for obtaining monitoring data.

[0163] For example, the monitoring robot (1050) may include, but is not limited to, a camera device for obtaining an image of at least one crop located in the greenhouse (1010).

[0164] Additionally, for example, the monitoring robot (1050) may include, but is not limited to, a depth camera device for obtaining a depth image of at least one crop located in the greenhouse (1010).

[0165] At this time, the depth image can be expressed as a distance image, etc., and the depth value included in the depth image data can be expressed as a distance value, etc.

[0166] In addition, the monitoring robot (1050) according to one embodiment may be moved by human manipulation, but is not limited thereto, and may move automatically by sensing the surroundings.

[0167] Additionally, a monitoring robot (1050) according to one embodiment can move on at least one rail located in the greenhouse (1010) and obtain monitoring data on crops.

[0168] For example, the monitoring robot (1050) may move in a first direction on a first rail (1040) located in the greenhouse (1010) and obtain a plurality of images for the first crop row (1020), and may move in a second direction on a first rail (1040) located in the greenhouse (1010) and obtain a plurality of images for the second crop row (1030), but is not limited thereto.

[0169] In addition, for example, the monitoring robot (1050) may be moved in a first direction on a first rail (1040) located in the greenhouse (1010) and may acquire a plurality of images for a first sub-crop row (1021) included in the first crop row (1020), and may be moved in a second direction on a first rail (1040) located in the greenhouse (1010) and may acquire a plurality of images for a third sub-crop row (1031) included in the second crop row (1030), but is not limited thereto.

[0170] Additionally, the monitoring robot (1050) according to one embodiment can transmit the acquired monitoring data to the server (1060).

[0171] For example, the monitoring robot (1050) may move in a first direction on a first rail (1040) located in the greenhouse (1010) and transmit a plurality of depth images of the first crop row (1020) obtained through the communication unit to the server (1060), but is not limited thereto.

[0172] In addition, for example, the monitoring robot (1050) may move in the second direction on the first rail (1040) located in the greenhouse (1010) and transmit a plurality of depth images of the second crop row (1030) obtained through the communication unit to the server (1060), but is not limited thereto.

[0173] Additionally, the monitoring robot (1050) according to one embodiment can transmit at least one piece of identification information through a communication unit.

[0174] For example, the monitoring robot (1050) may transmit the ID information of the monitoring robot (1050) to the server (1060) through the communication unit, but is not limited thereto.

[0175] Additionally, the monitoring robot (1050) according to one embodiment can transmit at least one positioning information through a communication unit.

[0176] For example, the monitoring robot (1050) may transmit positioning information obtained through the positioning unit to the server (1060) through the communication unit, but is not limited thereto.

[0177] Additionally, the monitoring robot (1050) according to one embodiment can transmit at least one piece of information at various times through the communication unit.

[0178] For example, the monitoring robot (1050) may transmit at least one piece of information to the server in real time through the communication unit, but is not limited thereto.

[0179] Additionally, for example, the monitoring robot (1050) may transmit at least one piece of information to the server at regular intervals, but is not limited thereto.

[0180] Additionally, for example, the monitoring robot (1050) may transmit at least one piece of information to the server at a time when a specific trigger is acquired, but is not limited thereto.

[0181] A server (1060) according to one embodiment may include a fruit detection module (1061), a fruit size determination module (1062), a fruit maturity determination module (1063), a site status information generation module (1064), and a yield prediction module (1065).

[0182] At this time, the fruit detection module (1061), fruit size judgment module (1062), fruit maturity judgment module (1063), site status information generation module (1064), and yield prediction module (1065) may be performed in at least one processor, and are only specific for convenience of explanation, and each module does not mean only a physically distinct unit, but may also mean a functional unit that performs the functions described below.

[0183] A fault detection module (1061) according to one embodiment may be a module for detecting a fault expressed in monitoring data obtained from the monitoring robot (1050).

[0184] For example, the above-described fruit detection module (1061) may be a module for detecting areas corresponding to fruits expressed in a plurality of images acquired from the monitoring robot (1050) and extracting areas corresponding to fruits, but is not limited thereto.

[0185] The error detection module (1061) according to one embodiment will be described in more detail below using other drawings.

[0186] In addition, the fruit size judgment module (1062) according to one embodiment may be a module for judging and outputting the size of fruits located in the greenhouse (1010) based on monitoring data obtained from the monitoring robot (1050).

[0187] For example, the fruit size judgment module (1062) may be a module for judging and outputting the size of each fruit expressed in a plurality of images acquired from the monitoring robot (1050), and may be a module for judging and outputting the size of each fruit detected by the fruit detection module (1061) among the fruits expressed in the plurality of images, but is not limited thereto.

[0188] The fruit size judgment module (1062) according to one embodiment will be described in more detail below using other drawings.

[0189] In addition, the fruit maturity judgment module (1063) according to one embodiment may be a module for judging and outputting the maturity of fruits located in the greenhouse (1010) based on monitoring data obtained from the monitoring robot (1050).

[0190] For example, the fruit maturity judgment module (1063) may be a module for judging and outputting the maturity of each fruit expressed in a plurality of images obtained from the monitoring robot (1050), and may be a module for judging and outputting the maturity of each fruit detected by the fruit detection module (1061) among the fruits expressed in the plurality of images, but is not limited thereto.

[0191] The fruit maturity judgment module (1063) according to one embodiment will be described in more detail below using other drawings.

[0192] Additionally, the site status information generation module (1064) according to one embodiment may be a module for generating site status information for a specific site of the greenhouse (1010) based on monitoring data obtained from the monitoring robot (1050).

[0193] For example, the site status information generation module (1064) may be a module for generating site status information for a specific site based on the size and maturity of each fruit expressed in a plurality of images determined by the fruit size determination module (1062) and the fruit maturity determination module (1063), but is not limited thereto.

[0194] At this time, the above-described contents may be applied to the site and site status information, so redundant descriptions will be omitted.

[0195] In addition, the specific site may mean the entire greenhouse (1010), may mean a part of the greenhouse (1010), or may mean a part of the greenhouse (1010) where crops expressed in a plurality of images acquired from the monitoring robot (1050) are located, but is not limited thereto.

[0196] The site status information generation module (1064) according to one embodiment will be described in more detail below using other drawings.

[0197] Additionally, the yield prediction module (1065) according to one embodiment may be a module for generating yield prediction information for a specific site of the greenhouse (1010) based on monitoring data obtained from the monitoring robot (1050).

[0198] For example, the yield prediction module (1065) may be a module for generating predicted status information for a specific site based on the site status information generated by the site status information generation module (1064), and generating yield prediction information for a specific site based on the generated predicted status information, but is not limited thereto.

[0199] The yield prediction module (1065) according to one embodiment will be described in more detail below using other drawings.

[0200]

[0201] 2. [Fruit Detection Module]

[0202] FIG. 2 is a drawing for explaining a fruit detection module according to one embodiment.

[0203] Referring to FIG. 2, a fruit detection module (1100) according to one embodiment may be a module for outputting at least one of a pixel coordinate value (1130) for at least one fruit area (1120) included in a crop, a depth value (1140) for the fruit area (1120), and a cropped image (1150) for the fruit area (1120) based on an image (1110) in which at least one crop is expressed.

[0204]

[0205] An image (1110) depicting at least one crop according to one embodiment may be an image obtained from a monitoring robot included in a yield prediction system.

[0206] For example, the first image included in the image (1110) in which at least one crop is expressed may be an image acquired from a monitoring robot at a first point in time, but is not limited thereto.

[0207] Additionally, the image (1110) in which at least one crop is expressed according to one embodiment may be image data including at least a portion of at least one crop.

[0208] For example, the image (1110) in which at least one crop is expressed may be image data including at least a portion of a plurality of crops, but is not limited thereto.

[0209] Additionally, the image (1110) in which at least one crop is expressed according to one embodiment may be image data including pixel coordinates, pixel values ​​for color, and pixel values ​​for distance.

[0210] For example, the image (1110) in which at least one crop is expressed may be depth image data including pixel coordinates, R, G, B values ​​corresponding to the pixel coordinates, and depth values ​​corresponding to the pixel coordinates, but is not limited thereto.

[0211] Additionally, an image (1110) representing at least one crop according to one embodiment may include an area corresponding to at least one fruit area included in the crop.

[0212] For example, the image (1110) in which at least one crop is represented may include, but is not limited to, an area for a first fruit included in a first crop.

[0213]

[0214] Additionally, the fruit area (1120) according to one embodiment may mean an area determined to correspond to a fruit within an image (1110) in which at least one crop is expressed.

[0215] In addition, the fruit area (1120) according to one embodiment may be specified as a bounding box shape as illustrated in FIG. 2, but is not limited thereto, and may be specified as various shapes, such as a shape in which the area corresponding to the fruit is segmented.

[0216]

[0217] Additionally, a pixel coordinate value (1130) corresponding to a fruit area (1120) according to one embodiment may be selected as one pixel coordinate among a plurality of pixel coordinates included in the fruit area (1120).

[0218] For example, the first pixel coordinate value corresponding to the first fruit area may be selected as the second pixel coordinate, which is one of a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0219] Additionally, a pixel coordinate value (1130) corresponding to a fruit area (1120) according to one embodiment can be calculated based on a plurality of pixel coordinates included in the fruit area (1120).

[0220] For example, the first pixel coordinate value corresponding to the first fruit area may be calculated as the center pixel coordinate of a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0221]

[0222] Additionally, a depth value (1140) corresponding to a fruit area (1120) according to one embodiment may be selected as one of a plurality of depth values ​​corresponding to a plurality of pixel coordinates included in the fruit area (1120).

[0223] For example, the first depth value corresponding to the first fruit area may be selected as the first depth value corresponding to the first pixel coordinate located at the center among a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0224] Additionally, a depth value (1140) corresponding to a fruit area (1120) according to one embodiment can be calculated based on a plurality of depth values ​​corresponding to a plurality of pixel coordinates included in the fruit area (1120).

[0225] For example, a first depth value corresponding to a first fruit area may be calculated by weighting a plurality of depth values ​​corresponding to a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0226] For a more specific example, the first depth value corresponding to the first fruit area may be calculated by applying a Gaussian-shaped weight to a plurality of depth values ​​corresponding to a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0227] For a more specific example, the first depth value corresponding to the first fruit area may be calculated by applying a weight of a Laplacian shape to a plurality of depth values ​​corresponding to a plurality of pixel coordinates included in the first fruit area, but is not limited thereto.

[0228]

[0229] Additionally, the fruit detection module (1100) according to one embodiment may be implemented using a machine learning method. For example, the fruit detection module (1100) according to one embodiment may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0230] In addition, the fruit detection module (1100) according to one embodiment may be implemented with at least one artificial neural network (ANN). For example, the fruit detection module (1100) according to one embodiment may include, but is not limited to, at least one artificial neural network layer among various artificial neural network layers such as a feedforward neural network, a radial basis function network, a Cohen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or gated recurrent units (GRUs).

[0231] Additionally, at least one artificial neural network layer included in the fruit detection module (1100) according to one embodiment may use the same or different activation functions.

[0232] At this time, the activation function may include, but is not limited to, a sigmoid function, a hyperbolic tangent function, a Relu function (rectified linear unit function), a leaky Relu function, an ELU function (exponential linear unit function), a softmax function, etc., and may include various activation functions (including custom activation functions) for outputting a result value or transmitting it to another artificial neural network layer.

[0233] Additionally, the fruit detection module (1100) according to one embodiment can be trained using a learning data set including at least one image data for a crop.

[0234] For example, the fruit detection module (1100) according to one embodiment may be trained using a training data set including at least one image data of a crop and an annotation corresponding thereto, but is not limited thereto.

[0235] Additionally, the error detection module (1100) according to one embodiment can be trained using at least one loss function.

[0236] At this time, the at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0237] Additionally, at least one optimizer may be used for learning of the error detection module (1100) according to one embodiment.

[0238] At this time, the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0239]

[0240] FIG. 3 is a drawing for explaining a fruit detection module according to one embodiment.

[0241] Referring to FIG. 3, a fruit detection module (1200) according to one embodiment may be a module for outputting at least one representative value (1220) for a plurality of fruit areas included in a plurality of crops based on a plurality of images (1210) in which at least one crop is expressed.

[0242] At this time, since the above-described contents can be applied to the above-described fault detection module (1200), redundant descriptions will be omitted.

[0243] A plurality of images (1210) depicting at least one crop according to one embodiment may be images obtained from a monitoring robot included in a yield prediction system.

[0244] For example, the first image included in the plurality of images (1210) in which at least one crop is expressed may be an image obtained from a monitoring robot at a first time point, the second image may be an image obtained from the monitoring robot at a second time point, and the third image may be an image obtained from the monitoring robot at a third time point, but is not limited thereto.

[0245] Additionally, the plurality of images (1210) representing at least one crop according to one embodiment may be images obtained by a monitoring robot included in the yield prediction system moving on at least one rail located in a greenhouse.

[0246] For example, the first image included in the plurality of images (1210) in which at least one crop is expressed may be an image acquired at a first point in time when the monitoring robot is moved on a first rail located in the greenhouse and is located at a first location, the second image may be an image acquired at a second point in time when the monitoring robot is moved on the first rail and is located at a second location, and the third image may be an image acquired at a third point in time when the monitoring robot is moved on the first rail and is located at a third location, but is not limited thereto.

[0247] Additionally, a plurality of images (1210) depicting at least one crop according to one embodiment may be image frames continuously acquired while a monitoring robot included in the yield prediction system moves on at least one rail located in a greenhouse.

[0248] Additionally, the plurality of images (1210) representing at least one crop according to one embodiment may include multiple images representing the same crop.

[0249] For example, the plurality of images (1210) expressing at least one crop may include, but are not limited to, a first image expressing at least a portion of a first crop, a second image expressing at least another portion of the first crop, and a third image expressing at least another portion of the first crop.

[0250]

[0251] Additionally, since the plurality of images (1210) representing at least one crop according to one embodiment include multiple image data for one crop, multiple detections for one fruit can be performed.

[0252] For example, detection of a first fruit may be performed in a first image included in a plurality of images (1210) in which at least one crop is represented according to one embodiment, detection of the first fruit may be performed in a second image, and detection of the first fruit may be performed in a third image.

[0253] Accordingly, the fault detection module (1200) according to one embodiment may include various algorithms to determine the same fault as one fault.

[0254] For example, a fruit detection module (1200) according to one embodiment may include an algorithm for determining identity based on the positional relationship between each of the fruit areas in order to determine that a first fruit area, which is a fruit area for a first fruit included in a first image, a second fruit area, which is a fruit area for the first fruit included in a second image, and a third fruit area, which is a fruit area for the first fruit included in a third image, are the same fruit area, but is not limited thereto, and may include various algorithms for determining that the same object included in consecutive image frames is the same object.

[0255]

[0256] Additionally, at least one representative value (1220) for a plurality of fruit areas according to one embodiment may include a depth value for each of the plurality of fruit areas.

[0257] For example, at least one representative value (1220) for a plurality of fruit areas according to one embodiment may include, but is not limited to, a first depth value for a first fruit area and a second depth value for a second fruit area.

[0258] Additionally, at least one representative value (1220) for a plurality of fruit regions according to one embodiment can be obtained based on a depth value for the fruit region obtained from each of the plurality of image data.

[0259] For example, the first depth value for the first fruit area may be obtained based on, but is not limited to, the second depth value for the second fruit area for the first fruit included in the first image data, and the third depth value for the third fruit area for the first fruit included in the second image data.

[0260] Additionally, at least one representative value (1220) for a plurality of fruit areas according to one embodiment may include an index value for the plurality of fruit areas.

[0261] At this time, the index value may correspond to the number of detected errors.

[0262]

[0263] 3. [Fruit Size Judgment Module]

[0264] FIG. 4 is a drawing for explaining a fruit size judgment module according to one embodiment.

[0265] Referring to FIG. 4, a fruit size judgment module (1300) according to one embodiment may be a module for outputting a fruit size classification value (1320) for a fruit area based on at least one image (1310) of the fruit.

[0266] At least one image (1310) of a fruit according to one embodiment may be an image obtained from a monitoring robot included in a yield prediction system.

[0267] For example, the first image included in at least one image (1310) of the above-described fruit may be, but is not limited to, at least a portion of an image acquired from a monitoring robot at a first point in time.

[0268] Additionally, at least one image (1310) of a fruit according to one embodiment may be an image cropped from an image in which at least one crop is represented.

[0269] For example, the first image included in at least one image (1310) of the above-described fruit may be an image obtained from a monitoring robot at a first point in time and generated by cropping an area corresponding to the fruit by the above-described fruit detection module, but is not limited thereto.

[0270] In addition, the fruit size judgment module (1300) according to one embodiment may be a module for outputting a fruit size classification value (1320) for a fruit area based on at least one representative value for the fruit area generated by the fruit detection module described above.

[0271] For example, the above-described fruit size judgment module (1300) may be a module for outputting a fruit size classification value (1320) based on a depth value and crop image for a fruit area generated by the above-described fruit detection module, but is not limited thereto.

[0272] Additionally, according to the fruit size judgment module (1300) according to one embodiment, the fruit size classification value (1320) for the fruit area can be judged based on the size of at least one image (1310) for the fruit.

[0273] For example, the fruit size classification value (1320) for the fruit area may be determined as a class corresponding to 'small' if the predetermined axial length of at least one image (1310) for the fruit is within a first length range, determined as a class corresponding to 'medium' if the predetermined axial length is within a second length range, and determined as a class corresponding to 'large' if the predetermined axial length is within a third length range, but is not limited thereto.

[0274] In addition, for example, the fruit size classification value (1320) for the fruit area may be determined as a class corresponding to 'small' if the area of ​​at least one image (1310) for the fruit is within a first area range, determined as a class corresponding to 'medium' if the area is within a second area range, and determined as a class corresponding to 'large' if the area is within a third area range, but is not limited thereto.

[0275] At this time, the predetermined axial length and width of at least one image (1310) of the fruit can be determined based on the number of pixels included in the at least one image (1310) of the fruit, coordinate values ​​of the pixels, etc.

[0276] Additionally, at this time, the predetermined axial length and width of at least one image (1310) of the fruit can be determined by further considering the depth value corresponding to at least one image (1310) of the fruit.

[0277] Additionally, various algorithms for determining the size of an area corresponding to a fruit can be applied to the fruit size determination module (1300) according to one embodiment.

[0278] Additionally, the fruit size determination module (1300) according to one embodiment may be implemented using a machine learning method. For example, the fruit size determination module (1300) according to one embodiment may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0279] In addition, the fruit size judgment module (1300) according to one embodiment may be implemented with at least one artificial neural network (ANN). For example, the fruit size judgment module (1300) according to one embodiment may include at least one artificial neural network layer among various artificial neural network layers such as a feedforward neural network, a radial basis function network, a Cohen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or gated recurrent units (GRUs), but is not limited thereto.

[0280] Additionally, at least one artificial neural network layer included in the fruit size judgment module (1300) according to one embodiment may use the same or different activation functions.

[0281] At this time, the activation function may include, but is not limited to, a sigmoid function, a hyperbolic tangent function, a Relu function (rectified linear unit function), a leaky Relu function, an ELU function (exponential linear unit function), a softmax function, etc., and may include various activation functions (including custom activation functions) for outputting a result value or transmitting it to another artificial neural network layer.

[0282] Additionally, the fruit size judgment module (1300) according to one embodiment can be trained using a learning data set including at least one image data for the fruit.

[0283] For example, the fruit size judgment module (1300) according to one embodiment may be trained using a learning data set including at least one image data of a fruit and an annotation corresponding thereto, but is not limited thereto.

[0284] Additionally, the fruit size judgment module (1300) according to one embodiment can be trained using at least one loss function.

[0285] At this time, the at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0286] Additionally, at least one optimizer may be used for learning of the fruit size judgment module (1300) according to one embodiment.

[0287] At this time, the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0288] Additionally, the fruit size classification value (1320) according to one embodiment may be provided as a probability value.

[0289] For example, the above fruit size classification value (1320) may be provided as a probability value that the class corresponds to 'small', a probability value that the class corresponds to 'medium', and a probability value that the class corresponds to 'large', but is not limited thereto.

[0290] Additionally, a fruit size classification value (1320) according to one embodiment may be provided as a classification result.

[0291] For example, the above fruit size classification value (1320) may be provided as a classification result regarding which class of the image of the input fruit corresponds to among ‘large’, ‘medium’, and ‘small’, but is not limited thereto.

[0292] In addition, the classes for 'large', 'medium', and 'small' described above are merely class names for convenience of explanation, and class distinction is not limited to naming.

[0293] In addition, in FIG. 4, the fruit size classification value (1320) is described as three classes for convenience of explanation, but the fruit size classification value (1320) may be determined as two classes, or as four, five, six, seven, eight, nine, or ten classes, and the number of classes of the fruit size classification value (1320) is not limited to the above-described content.

[0294] In addition, the fruit size judgment module (1300) described in FIG. 4 can be performed for each of the fruit areas detected by the above-described fruit detection module, and accordingly, a fruit size classification value (1320) for each of the fruit areas detected by the above-described fruit detection module can be obtained.

[0295]

[0296] 4. [Fruit Maturity Judgment Module]

[0297] FIG. 5 is a diagram for explaining a fruit maturity judgment module according to one embodiment.

[0298] Referring to FIG. 5, a fruit maturity judgment module (1400) according to one embodiment may be a module for outputting a fruit maturity classification value (1420) for a fruit area based on at least one image (1410) of the fruit.

[0299] At least one image (1410) of a fruit according to one embodiment may be an image obtained from a monitoring robot included in a yield prediction system.

[0300] For example, the first image included in at least one image (1410) of the above-described fruit may be, but is not limited to, at least a portion of an image acquired from a monitoring robot at a first point in time.

[0301] Additionally, at least one image (1410) of a fruit according to one embodiment may be an image cropped from an image in which at least one crop is represented.

[0302] For example, the first image included in at least one image (1410) of the fruit may be an image obtained from a monitoring robot at a first point in time and generated by cropping an area corresponding to the fruit by the fruit detection module described above, but is not limited thereto.

[0303] Additionally, according to the fruit maturity judgment module (1400) according to one embodiment, the fruit maturity classification value (1420) for the fruit area can be judged based on the pixel value (color value) of at least one image (1410) for the fruit.

[0304] For example, the fruit maturity classification value (1420) for the fruit area may be determined as a class corresponding to 'unripe' if the average of the pixel values ​​(e.g., R, G, B values) of at least one image (1310) for the fruit is within a first color range, may be determined as a class corresponding to 'ripening' if the average of the pixel values ​​is within a second color range, and may be determined as a class corresponding to 'ripe' if the average of the pixel values ​​is within a third color range, but is not limited thereto.

[0305] Additionally, various algorithms can be applied to the fruit maturity judgment module (1400) according to one embodiment to determine the maturity classification value of the area corresponding to the fruit.

[0306] Additionally, the fruit maturity assessment module (1400) according to one embodiment may be implemented using a machine learning method. For example, the fruit maturity assessment module (1400) according to one embodiment may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, etc.

[0307] In addition, the fruit maturity judgment module (1400) according to one embodiment may be implemented with at least one artificial neural network (ANN). For example, the fruit maturity judgment module (1400) according to one embodiment may include, but is not limited to, at least one artificial neural network layer among various artificial neural network layers such as a feedforward neural network, a radial basis function network, a Cohen self-organizing network, a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory network (LSTM), or gated recurrent units (GRUs).

[0308] Additionally, at least one artificial neural network layer included in the fruit maturity judgment module (1400) according to one embodiment may use the same or different activation functions.

[0309] At this time, the activation function may include, but is not limited to, a sigmoid function, a hyperbolic tangent function, a Relu function (rectified linear unit function), a leaky Relu function, an ELU function (exponential linear unit function), a softmax function, etc., and may include various activation functions (including custom activation functions) for outputting a result value or transmitting it to another artificial neural network layer.

[0310] Additionally, the fruit maturity judgment module (1400) according to one embodiment can be trained using a learning data set including at least one image data for the fruit.

[0311] For example, the fruit maturity judgment module (1400) according to one embodiment may be trained using a learning data set including at least one image data of a fruit and an annotation corresponding thereto, but is not limited thereto.

[0312] Additionally, the fruit maturity judgment module (1400) according to one embodiment can be trained using at least one loss function.

[0313] At this time, the at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0314] Additionally, at least one optimizer may be used for learning of the fruit maturity judgment module (1400) according to one embodiment.

[0315] At this time, the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0316] Additionally, the fruit maturity classification value (1420) according to one embodiment may be provided as a probability value.

[0317] For example, the above fruit maturity classification value (1420) may be provided as a probability value corresponding to the class 'unripe', a probability value corresponding to the class 'ripening', and a probability value corresponding to the class 'ripe', but is not limited thereto.

[0318] Additionally, a fruit maturity classification value (1420) according to one embodiment may be provided as a classification result.

[0319] For example, the above fruit maturity classification value (1420) may be provided as a classification result regarding which class of the image of the input fruit corresponds to among ‘ripe’, ‘ripening’, and ‘unripe’, but is not limited thereto.

[0320] In addition, the classes for 'ripe', 'ripening' and 'not ripe' described above are merely class names for convenience of explanation, and class distinction is not limited to naming.

[0321] In addition, in FIG. 5, the fruit maturity classification value (1420) is described as three classes for convenience of explanation, but the fruit maturity classification value (1420) may be determined as two classes, or as four, five, six, seven, eight, nine, or ten classes, and the number of classes of the fruit maturity classification value (1420) is not limited to the above-described contents.

[0322] In addition, the fruit maturity judgment module (1400) described in FIG. 5 can be performed for each of the fruit areas detected by the fruit detection module described above, and accordingly, a fruit size classification value (1420) for each of the fruit areas detected by the fruit detection module described above can be obtained.

[0323]

[0324] 5. [Information about the fruits]

[0325] According to the above-described fruit detection module, fruit size judgment module, and fruit maturity judgment module, the monitoring robot can move to a specific site and information on fruits detected within the specific site can be generated using the acquired multiple images.

[0326] FIG. 6 is a drawing for exemplarily explaining information about fruits according to one embodiment.

[0327] Referring to FIG. 6, information (1500) on fruits according to one embodiment may include a depth value (1510), a fruit size classification value (1520), and a fruit maturity classification value (1530) for each of the detected fruits.

[0328] At this time, when multiple fruit regions are detected within different images for each of the detected fruits, various algorithms can be applied to determine the same fruit as a single fruit, as described in FIG. 3. Therefore, redundant descriptions will be omitted.

[0329] Referring again to FIG. 6, the depth value (1510) for each of the detected fruits included in the information on the fruits (1500) may be a representative depth value obtained based on the depth values ​​of multiple fruit areas for the same fruit.

[0330] For example, the first depth value (D1), which is a depth value for the first fruit (1540), may be a depth value obtained as an average value of a depth value for a first fruit area for the first fruit expressed in a first image, a depth value for a second fruit area for the first fruit expressed in a second image, and a depth value for a third fruit area for the first fruit expressed in a third image, but is not limited thereto, and the first depth value (D1), which is a depth value for the first fruit (1540), may be obtained by various methods for obtaining a representative depth value based on a plurality of depth values.

[0331] Additionally, the fruit size classification value (1520) for each of the detected fruits included in the information on the fruits (1500) may be a representative fruit size classification value obtained based on the fruit size classification values ​​of multiple fruit areas for the same fruit.

[0332] For example, the first fruit size classification value (S1), which is a fruit size classification value for the first fruit (1540), may be a fruit size classification value obtained as an average value of a fruit size classification value for a first fruit area for the first fruit expressed in a first image, a fruit size classification value for a second fruit area for the first fruit expressed in a second image, and a fruit size classification value for a third fruit area for the first fruit expressed in a third image, but is not limited thereto, and the first fruit size classification value (S1), which is a fruit size classification value for the first fruit (1540), may be obtained by various methods for obtaining a representative fruit size classification value based on a plurality of fruit size classification values.

[0333] In addition, the fruit maturity classification value (1530) for each of the detected fruits included in the information on the fruits (1500) may be a representative fruit maturity classification value obtained based on the fruit maturity classification values ​​of multiple fruit areas for the same fruit.

[0334] For example, the first fruit maturity classification value (R1), which is a fruit maturity classification value for the first fruit (1540), may be a fruit maturity classification value obtained by an average value of a fruit maturity classification value for a first fruit area for the first fruit expressed in a first image, a fruit maturity classification value for a second fruit area for the first fruit expressed in a second image, and a fruit maturity classification value for a third fruit area for the first fruit expressed in a third image, but is not limited thereto, and the first fruit maturity classification value (R1), which is a fruit maturity classification value for the first fruit (1540), may be obtained by various methods for obtaining a representative fruit maturity classification value based on a plurality of fruit maturity values.

[0335] Below, the site status information generation module that generates site status information based on information about errors detected within a specific site and the site status information will be described in more detail.

[0336]

[0337] 6. [Site Status Information Generation Module]

[0338] FIG. 7 is a diagram illustrating a site status information generation module according to one embodiment.

[0339] Referring to FIG. 7, a site status information generation module (1600) according to one embodiment may be a module for generating site status information (1620) based on information (1610) on errors obtained according to the above-described methods.

[0340] Since the above-described contents can be applied to information on fruits according to one embodiment (1610), redundant descriptions will be omitted.

[0341] Additionally, site status information (1620) according to one embodiment may include M*N fruit status classes distinguished according to M fruit size classes and N fruit maturity classes.

[0342] For example, site status information (1620) according to one embodiment may include nine fruit status classes distinguished according to three fruit size classes and three fruit maturity classes.

[0343] For a more specific example, site status information (1620) according to one embodiment may include nine fruit status classes distinguished according to three fruit size classes corresponding to 'large', 'medium', and 'small' respectively, and three fruit maturity classes corresponding to 'ripe', 'ripening', and 'unripe' respectively.

[0344] At this time, the above M and the above N may be identical to each other as described above, but are not limited thereto and may be different from each other.

[0345] Site status information (1620) according to one embodiment may include count values ​​of fruits belonging to each of a plurality of fruit status classes distinguished according to fruit size class and fruit maturity class.

[0346] For example, the site status information (1620) may include a count value (X_11) of fruits belonging to a first fruit status class specified according to a first fruit size class and a first fruit maturity class, a count value (X_12) of fruits belonging to a second fruit status class specified according to a second fruit size class and a first fruit maturity class, a count value (X_21) of fruits belonging to a third fruit status class specified according to a first fruit size class and a second fruit maturity class, and a count value (X_22) of fruits belonging to a fourth fruit status class specified according to a second fruit size class and a second fruit maturity class.

[0347] At this time, the count value of the above fruits can be provided as the number value of fruits belonging to each of the plurality of fruit status classes.

[0348] For example, the count value (X_11) of fruits belonging to the first fruit status class specified according to the first fruit size class and the first fruit maturity class included in the site status information (1620) can be obtained as the number value of fruits having a fruit size classification value corresponding to the first fruit size class and a fruit maturity classification value corresponding to the first fruit maturity class in the information about the fruits (1610).

[0349] Of course, for the convenience of explanation, the above-described example was explained based on the count value (X_11) of fruits belonging to the first fault status class, but the contents explained for the count value (X_11) of fruits belonging to the first fault status class can also be applied to each of the M*N fault status classes.

[0350] Additionally, at this time, the count value of the above fruits may be provided as a ratio value of fruits belonging to each of the plurality of fruit status classes.

[0351] For example, the count value (X_11) of fruits belonging to the first fruit status class specified according to the first fruit size class and the first fruit maturity class included in the site status information (1620) can be obtained as a ratio value obtained by dividing the number value of fruits having a fruit size classification value corresponding to the first fruit size class and a fruit maturity classification value corresponding to the first fruit maturity class by the number value of all fruits in the information about the fruits (1610).

[0352] Of course, for the convenience of explanation, the above-described example was explained based on the count value (X_11) of fruits belonging to the first fault status class, but the contents explained for the count value (X_11) of fruits belonging to the first fault status class can also be applied to each of the M*N fault status classes.

[0353] As described above, according to the site status information generation module (1600) according to one embodiment, the site status information (1620), which is a set of count values ​​of fruits belonging to each of M*N fruit status classes distinguished according to M fruit size classes and N fruit maturity classes, can be obtained based on information about fruits (1610), and below, a yield prediction module that predicts yield based on the obtained site status information (1620) will be described in more detail.

[0354] However, for the convenience of explanation, the data format of the above site status information (1620) is described and explained in matrix format, but it is obvious that it can be obtained in various data formats to express a set of count values ​​of fruits belonging to each of M*N fruit status classes distinguished according to M fruit size classes and N fruit maturity classes.

[0355] In addition, for convenience of explanation, the site status information (1620) will be described below as a set of count values ​​of fruits belonging to each of nine fruit status classes distinguished according to three fruit size classes corresponding to 'large', 'medium', and 'small' respectively, and three fruit maturity classes corresponding to 'ripe', 'ripening', and 'unripe' respectively.

[0356]

[0357] 7. [Yield Prediction Module]

[0358] FIG. 8 is a diagram for explaining a yield prediction module according to one embodiment.

[0359] Referring to FIG. 8, a yield prediction module (1700) according to one embodiment may be a module for outputting a yield prediction value (1730) based on generated site status information (1710).

[0360] At this time, since the above-described contents can be applied to the site status information (1710), redundant descriptions will be omitted.

[0361] Referring again to FIG. 8, a yield prediction module (1700) according to one embodiment may include a fruit state transition model (1701) for receiving the site state information (1710) and generating site prediction state information (1720), and a yield calculation model (1702) for calculating the yield prediction value (1730) based on the generated site prediction state information (1720).

[0362] At this time, the above-mentioned fruit state transition model (1701) will be described in more detail using other drawings below.

[0363] Additionally, the site prediction status information (1720) according to one embodiment may include information corresponding to the site status information (1710), and may be information expressed through the same data format as the site status information (1710).

[0364] For example, if the site status information (1710) includes M*N fruit status classes distinguished according to M fruit size classes and N fruit maturity classes, the site prediction status information (1720) may include M*N fruit status classes distinguished according to M fruit size classes and N fruit maturity classes.

[0365] In addition, for example, if the site status information (1710) includes count values ​​of fruits belonging to each of a plurality of fruit status classes distinguished according to a fruit size class and a fruit maturity class, the site prediction status information (1720) may include count values ​​of fruits belonging to each of a plurality of fruit status classes distinguished according to a fruit size class and a fruit maturity class.

[0366] At this time, the count value of the fruits included in the site prediction status information (1720) may be a value corresponding to the number of fruits predicted to belong to each of the plurality of fruit status classes in the time interval corresponding to the future, and this may be provided as a non-integer value according to the fruit status transition model (1701).

[0367] Additionally, site prediction status information (1720) according to one embodiment may mean information on the predicted status of a site for a future time period that is the target of yield prediction.

[0368] For example, site prediction status information (1720) according to one embodiment may mean information about the predicted status of the site for the next week of the week in which the site status information (1710) was obtained.

[0369] Additionally, for example, site prediction status information (1720) according to one embodiment may mean information about the predicted status of the site for a week eight weeks after the week in which the site status information (1710) was obtained.

[0370] At this time, various fault state transition models (1701) can be applied in various ways to generate site prediction state information (1720) for a specific future time interval, and this will be explained in more detail below using other drawings.

[0371] In addition, a yield calculation model (1702) according to one embodiment can calculate a fruit count value belonging to the fruit condition classes to be harvested for a future time interval to be the target of yield prediction using information on the fruit condition classes to be harvested and the site prediction condition information (1720), and can calculate a yield prediction value (1730) using reference weight values ​​for the fruit condition classes.

[0372] At this time, information on the fruit condition class to be harvested can be stored in advance according to the user's input.

[0373] For example, if the first to third fruit size classes each correspond to 'small', 'medium', and 'large', and the first to third fruit maturity classes each correspond to 'unripe', 'ripening', and 'ripe', and if input is obtained from the user that fruits whose fruit size classification value belongs to 'medium' and 'large' and whose fruit maturity classification value belongs to 'ripe' are to be harvested, the eighth fruit condition class specified by the second fruit size class and the third fruit maturity class and the ninth fruit condition class specified by the third fruit size class and the third fruit maturity class can be stored as information about the fruit condition classes to be harvested.

[0374] Accordingly, according to the yield calculation model (1702) according to one embodiment, the count values ​​(Y_32) of fruits belonging to the eighth fruit condition class and the count values ​​(Y_33) of fruits belonging to the ninth fruit condition class can be specified as fruit count values ​​belonging to fruit condition classes to be harvested.

[0375] Accordingly, according to a yield calculation model (1702) according to one embodiment, the yield prediction value (1730) can be calculated by adding a value calculated by multiplying the count value (Y_32) of fruits belonging to the eighth fruit condition class by the reference weight value for the eighth fruit condition class and a value calculated by multiplying the count value (Y_33) of fruits belonging to the ninth fruit condition class by the reference weight value for the ninth fruit condition class.

[0376] At this time, the reference weight values ​​for the above fruit condition classes may be stored as reference weight values ​​for the fruit size classes, and may also be stored as reference weight values ​​for each of the fruit condition classes.

[0377] In addition, for the convenience of explanation, the above-described examples have been described assuming a situation in which the 8th fruit condition class and the 9th fruit condition class are specified. However, information on the fruit condition class to be harvested may differ depending on the user's input, and it can be easily applied by modifying it accordingly, so redundant descriptions will be omitted.

[0378]

[0379] Referring again to FIG. 8, since the fruit state transition model (1701) according to one embodiment is a model for receiving the site state information (1710) and generating site prediction state information (1720), in order to more accurately calculate the yield prediction value (1730), the fruit state transition model (1701) needs to be designed to more accurately generate site prediction state information (1720).

[0380] Therefore, below, we will describe in more detail various methods for generating a fruit state transition model using site state information obtained from past works.

[0381]

[0382] IV. [Fruit State Transition Model]

[0383] 1. [Site status information obtained from past work]

[0384] In greenhouses equipped with yield prediction systems or monitoring robots, site status information can be generated based on monitoring data obtained from the monitoring robot during the crop growing season.

[0385] At this time, the above site status information can be generated daily or on a weekly basis.

[0386] Below, for convenience of explanation, the explanation will be based on site status information obtained from past works generated on a weekly basis.

[0387] However, this is only a description of an example suitable for a greenhouse that requires yield prediction on a weekly basis, and it is self-evident that the content described below can also be applied to site status information generated on a daily basis or site status information generated on a monthly basis.

[0388]

[0389] FIG. 9 is a diagram for explaining site status information obtained from past work according to one embodiment.

[0390] Referring to FIG. 9, according to one embodiment, site status information (1800) obtained from past work may include first status information (1810) which is site status information for the first week, second status information (1820) which is site status information for the second week, third status information (1830) which is site status information for the third week, and Nth status information (1840) which is site status information for the Nth week.

[0391] At this time, since the contents described in relation to the site status information can be applied to each status information, redundant descriptions will be omitted.

[0392] Referring again to FIG. 9, according to one embodiment, according to the site state information (1800) acquired in the past operation, information on count values ​​of fruits belonging to each of a plurality of fruit state classes distinguished according to the fruit size class and fruit maturity class corresponding to each parking lot in the past operation can be acquired, and accordingly, when the site state information (1800) acquired in the past operation is appropriately combined and selected, a fruit state transition model can be acquired.

[0393]

[0394] Below, we will describe in more detail the design of the fruit state transition model and the acquisition of various types of fruit state transition models through a combination of site state information (1800) obtained from past works.

[0395]

[0396] 2. [Fault State Transition Model Using Fault State Transition Probability Matrix]

[0397] (1) [Design of the probability matrix of the state transition]

[0398] FIG. 10 is a diagram for explaining a fruit state transition probability matrix according to one embodiment.

[0399] Referring to FIG. 10, exemplary site state information (1910) and a fault state transition probability matrix (1920) are illustrated to illustrate a fault state transition probability matrix according to one embodiment.

[0400] At this time, referring to the site status information (1910), the first fruit status class is a fruit status class specified by the first fruit size class and the first fruit maturity class, the second fruit status class is a fruit status class specified by the second fruit size class and the first fruit maturity class, the third fruit status class is a fruit status class specified by the third fruit size class and the first fruit maturity class, the fourth fruit status class is a fruit status class specified by the first fruit size class and the second fruit maturity class, the fifth fruit status class is a fruit status class specified by the second fruit size class and the second fruit maturity class, the sixth fruit status class is a fruit status class specified by the third fruit size class and the second fruit maturity class, and the seventh fruit status class is a fruit status class specified by the first fruit size class and The third fruit maturity class may be a fruit condition class specified by the third fruit maturity class, the eighth fruit condition class may be a fruit condition class specified by the second fruit size class and the third fruit maturity class, and the ninth fruit condition class may be a fruit condition class specified by the third fruit size class and the third fruit maturity class.

[0401] Additionally, at this time, the above-mentioned fault state transition probability matrix (1920) may be a matrix of the probability that fruits belonging to each fault state class will transition to belong to each fault state class.

[0402] For example, referring to the above-described fault state transition probability matrix (1920), P00 may mean a probability that the fault state of fruits belonging to the first fault state class will be maintained so that they belong to the first fault state class, and P01 may mean a probability that the fault state of fruits belonging to the first fault state class will transition so that they belong to the second fault state class.

[0403] That is, when each probability of the above-mentioned fault state transition probability matrix (1920) is expressed as PMN, PMN may mean the probability that the fault state will transition so that the fruits belonging to the M-th fault state class belong to the N-th fault state class.

[0404] Therefore, when each probability value of the above-described fault state transition probability matrix (1920) is obtained using the site state information obtained from the above-described past work, a fault state transition model based on the fault state transition probability matrix (1920) can be created.

[0405] At this time, in the case of the obtained fruit state transition probability matrix (1920), the sum of P00 to P08 can be 1, the sum of P10 to P18 can also be 1, and so on, the sum of PM0 to PM8 can be 1.

[0406] This may mean that fruits belonging to one fault state class are changed to belong to the first to ninth fault state classes.

[0407] In addition, the fruit state transition probability matrix (1920) described above can be designed by considering the growth direction of fruits going through the fruit growth stage before obtaining each probability value using the site state information obtained from the past work described above.

[0408] Below, we will describe a fruit state transition probability matrix designed by considering the growth direction of fruits passing through the fruit growth section.

[0409]

[0410] FIG. 11 is a diagram for explaining a fruit state transition probability matrix according to one embodiment.

[0411] Referring to FIG. 11, exemplary site state information (2010) and a fault state transition probability matrix (2020) are illustrated to illustrate a fault state transition probability matrix according to one embodiment.

[0412] At this time, referring to the site status information (2010), the first fruit status class is a fruit status class specified by the first fruit size class and the first fruit maturity class, the second fruit status class is a fruit status class specified by the second fruit size class and the first fruit maturity class, the third fruit status class is a fruit status class specified by the third fruit size class and the first fruit maturity class, the fourth fruit status class is a fruit status class specified by the first fruit size class and the second fruit maturity class, the fifth fruit status class is a fruit status class specified by the second fruit size class and the second fruit maturity class, the sixth fruit status class is a fruit status class specified by the third fruit size class and the second fruit maturity class, and the seventh fruit status class is a fruit status class specified by the first fruit size class and The third fruit maturity class may be a fruit condition class specified by the third fruit maturity class, the eighth fruit condition class may be a fruit condition class specified by the second fruit size class and the third fruit maturity class, and the ninth fruit condition class may be a fruit condition class specified by the third fruit size class and the third fruit maturity class.

[0413] In addition, at this time, the first fruit size class may be a fruit size class corresponding to a fruit size of 'small', the second fruit size class may be a fruit size class corresponding to a fruit size of 'medium', the third fruit size class may be a fruit size class corresponding to a fruit size of 'large', the first fruit maturity class may be a fruit maturity class corresponding to 'unripe', the second fruit maturity class may be a fruit maturity class corresponding to 'ripening', and the third fruit maturity class may be a fruit maturity class corresponding to 'ripe'.

[0414] In addition, at this time, since the above-described contents can be applied to the above-described fruit state transition probability matrix (2020), redundant descriptions will be omitted.

[0415] In addition, during the fruit growth stages that normal fruits go through, the fruit becomes larger through cell division and cell enlargement, so the extent to which the size of the fruit changes in the direction of increasing size may be greater than the extent to which it changes in the direction of decreasing size.

[0416] That is, normal fruits only increase in size during the fruit growth stage and do not decrease in size.

[0417] Additionally, as the various components that make up the fruit change during the fruit ripening stages that regular fruits go through, so that it acquires the flavor and appearance characteristic of the variety, the degree to which the fruit changes toward ripening may be greater than the degree to which it changes toward unripening.

[0418] That is, in the case of normal fruits, the maturity of the fruit changes only toward ripeness and not toward unripeness during the fruit ripening stage.

[0419] Therefore, the fruit state transition probability matrix (2020) considering the change in fruit in the fruit growth section described above can be designed so that at least some of the transition probabilities have low values.

[0420] For example, the above-described fruit state transition probability matrix (2020) can be designed so that the probability of a fruit state transition such that fruits belonging to the second fruit state class, the fifth fruit state class, and the eighth fruit state class, whose fruit size class is the second fruit size class, can belong to the first fruit state class, the fourth fruit state class, and the seventh fruit state class, whose fruit size class is the first fruit size class, is 0.

[0421] Additionally, for example, the above-described fault state transition probability matrix (2020) can be designed such that the probability that the fault state can transition to belong to the 1st, 2nd, 4th, 5th, 7th and 8th fault state classes, where the fruit size class is the 1st or 2nd fault size class, for the 3rd fault state class, the 6th fault state class and the 9th fault state class, is 0.

[0422] In addition, for example, the fruit state transition probability matrix (2020) may be designed such that the probability of a fruit state transition such that fruits belonging to the fourth fruit state class, the fifth fruit state class, and the sixth fruit state class, where the fruit maturity class is the second fruit maturity class, can belong to the first fruit state class, the second fruit state class, and the third fruit state class, where the fruit maturity class is the first fruit maturity class, is 0.

[0423] Additionally, for example, the fruit state transition probability matrix (2020) may be designed such that the probability of a fruit state transition such that fruits belonging to the 7th fruit state class, the 8th fruit state class, and the 9th fruit state class, which are the 3rd fruit maturity class, can belong to the 1st, 2nd, 3rd, 4th, 5th, and 6th fruit state classes, which are the 1st or 2nd fruit maturity classes, is 0.

[0424] In addition, for example, although not shown in FIG. 11, since normal fruits can go through a fruit growth stage and then a fruit maturity stage during the fruit growth period, the fruit state transition probability matrix (2020) can be designed so that the probability that fruits belonging to the fourth to ninth fruit state classes, in which the fruit maturity class is the second or third fruit maturity class, can transition to a fruit state class in which the fruit size class is changed is 0.

[0425] For a more specific example, the above-mentioned fault state transition probability matrix (2020) can be designed so that the probability that the fault state can transition so that the fruits belonging to the fourth fault state class belong to the fifth, sixth, eighth, and ninth fault state classes is 0.

[0426] Additionally, for a more specific example, the fault state transition probability matrix (2020) may be designed such that the probability that the fault state can transition so that the fruits belonging to the fifth fault state class belong to the fourth, sixth, seventh, and ninth fault state classes is 0.

[0427]

[0428] Below, a method for generating a fault state transition model based on the fault state transition probability matrix described through FIGS. 10 and 11 and the site state information obtained from past work described through FIG. 9 will be described in more detail.

[0429]

[0430] (2) [Fruit state transition model based on matrix operations]

[0431] FIG. 12 is a diagram for explaining the creation of a fruit state transition model based on matrix operations according to one embodiment.

[0432] Referring to FIG. 12, in order to generate a fault state transition model according to one embodiment, a fault state transition probability matrix (2110), at least one input site state information (2120), and at least one output site state information (2130) may be included.

[0433] At this time, since the above-described contents can be applied to the above-described fruit state transition probability matrix (2110), redundant descriptions will be omitted.

[0434] Additionally, at this time, the at least one input site state information (2120) and the at least one output site state information (2130) may mean a pair of site state information selected to obtain the fault state transition probabilities included in the fault state transition probability matrix (2110).

[0435] For example, the at least one input site status information (2120) and the at least one output site status information (2130) may mean a pair of first parking site status information and second parking site status information, wherein the first parking site status information may be included in the at least one input site status information (2120), and the second parking site status information may be included in the at least one output site status information (2130).

[0436] Additionally, at this time, the at least one input site state information (2120) and the at least one output site state information (2130) may include a plurality of site state information pairs selected to obtain the fault state transition probabilities included in the fault state transition probability matrix (2110).

[0437] For example, the at least one input site status information (2120) and the at least one output site status information (2130) may include a first pair in which the first parking lot site status information is at least one input site status information (2120) and the second parking lot site status information is at least one output site status information (2130), and a second pair in which the second parking lot site status information is at least one input site status information (2120) and the third parking lot site status information is at least one output site status information (2130).

[0438] Additionally, the at least one input site state information (2120) and the at least one output site state information (2130) can be arranged in an appropriate form to obtain the fault state transition probabilities included in the fault state transition probability matrix (2110).

[0439] For example, as illustrated in FIG. 12, the at least one input site status information (2120) and the at least one output site status information (2130) may be arranged in nine rows per one column with the fruit count values ​​belonging to the first to ninth fruit status classes described above.

[0440] Referring again to FIG. 12, the fault state transition probabilities included in the fault state transition probability matrix (2110) can be calculated based on matrix operations.

[0441] For example, the count value (y11) of the first fault status class included in at least one output site status information (2130) may be P00*x11+P10*x12+P20*x13+P30*x21 + P40*x22+P50*x23+P60*x31+P70*x32+P80*x33.

[0442] In this way, for each of the count values ​​(y11, y12, y13, y21, y22, y23, y31, y32, y33) of the first to ninth fault state classes, a relationship between at least one input site state information (2120) and the fault state transition probability matrix (2110) can be derived by matrix operation.

[0443] At this time, the number of fault state transition probabilities included in the fault state transition probability matrix (2110) is 81, and 9 relationships can be derived for one site state information pair. Therefore, when at least 9 site state information pairs are used, information on the fault state transition probabilities included in the fault state transition probability matrix (2110) can be obtained.

[0444] However, for the convenience of explanation, this is explained using a fruit state transition probability matrix designed based on site state information having three fruit size classes and three fruit maturity classes each. If there are N fruit size classes and M fruit maturity classes, the number of fruit state classes included in the site state information can be N*M. In this case, the fruit state transition probability matrix can include (N*M)^2 fruit state transition probabilities, and N*M relationships can be derived for one site state information pair. Therefore, when at least N*M site state information pairs are used, information on the fruit state transition probabilities included in the fruit state transition probability matrix can be obtained.

[0445] Accordingly, as described above, values ​​for each fault state transition probability included in the fault state transition probability matrix (2110) can be obtained based on matrix operations, and a fault state transition model can be created based on the obtained fault state transition probabilities.

[0446] At this time, the acquired state transition probabilities can be expressed as state transition relationships.

[0447] However, the above-mentioned fault state transition model can be obtained using various site state information pairs depending on the purpose, etc., and this will be explained in more detail below using other drawings.

[0448]

[0449] (3) [Machine Learning-Based Fruit State Transition Model]

[0450] FIG. 13 is a diagram for explaining the creation of a fruit state transition model based on machine learning according to one embodiment.

[0451] Referring to FIG. 13, according to one embodiment, a hypothesis (H(X)) can be used to generate a fruit state transition model based on machine learning.

[0452] At this time, the above hypothesis (H(X)) may be a function designed to find a weight (w) and a bias (B), and in order to generate the above-described fault state transition model, the weight (w) may be designed as the above-described fault state transition probability matrix.

[0453] In addition, at this time, a learning data set can be used to derive the weight (w) and bias (B) of the above hypothesis (H(X)), and the above-described site information pair can be used as the learning data set.

[0454] In addition, at this time, learning can be performed using the above hypothesis (H(X)) and the above learning data set, and a learned fruit state transition model can be generated as a result of the learning.

[0455] Additionally, at this time, the above-described learning can be learned so that the cost or loss is minimized, and in this case, at least one loss function can be used.

[0456] In addition, at this time, the at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0457] In addition, at this time, the cost or loss can be obtained by applying the value obtained when input site status information is input into the hypothesis (H(X)) and the result value (Y) to at least one loss function.

[0458] Additionally, at this time, at least one optimizer may be used, and the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0459] In addition, at this time, at least one input site status information and one output site status information can be used to perform the above-described learning, and the above-described contents can be applied to this, so redundant descriptions will be omitted.

[0460] In addition, in Fig. 13, an exemplary hypothesis (H(X)) is used for convenience of explanation, but various machine learning techniques can be used.

[0461] Therefore, as described above, the fault state transition model can be created based on machine learning in which the fault state transition probability matrix and the hypothesis formula designed based on the same are input.

[0462] At this time, the fault state transition model generated based on the machine learning may store a fault state transition relationship, and the fault state transition relationship may mean a weight (w) and bias (B) acquired through learning, but is not limited thereto.

[0463]

[0464] It is described that a fruit state transition model can be created through matrix operations or machine learning through Figures 12 and 13.

[0465] Below, we will explain various fault state transition models according to the selection of site status information pairs.

[0466]

[0467] (4) [Fruit state transition model for generating next parking site prediction status information]

[0468] FIG. 14 is a diagram illustrating selection of site state information obtained from past operations to obtain a fault state transition model for generating next parking site prediction state information according to one embodiment.

[0469] In FIG. 14, the first parking site status information (2210), the second parking site status information (2220), the third parking site status information (2230), the N-1 parking site status information (2240), and the N-th parking site status information (2250) are illustrated. The first to N-th parking site status information (2250) may be site status information obtained from past work, and the above-described contents may be applied to this, so that redundant descriptions will be omitted.

[0470] In addition, in FIG. 14, for convenience of explanation, the first fruit size class is indicated as S1, the second fruit size class is indicated as S2, the third fruit size class is indicated as S3, the first fruit maturity class is indicated as R1, the second fruit maturity class is indicated as R2, and the third fruit maturity class is indicated as R3.

[0471] Again, referring to FIG. 14, in order to obtain a fault state transition model for generating the next parking site prediction state information according to one embodiment, a plurality of site state information pairs may be selected based on site state information obtained from past operations.

[0472] For example, a first pair (2260) that uses the first parking lot site status information (2210) as input site status information and the second parking lot site status information (2220) as output site status information, a second pair (2270) that uses the second parking lot site status information (2220) as input site status information and the third parking lot site status information (2230) as output site status information, and a third pair (2280) that uses the N-1 parking lot site status information (2240) as input site status information and the N-th parking lot site status information (2250) as output site status information can be selected.

[0473] In addition, in order to obtain a fault state transition model for generating next parking site prediction state information according to one embodiment, when a plurality of site state information pairs are selected based on site state information acquired in past operations, the fault state transition model obtained using the selected plurality of site state information pairs may reflect fault state transition relationships that allow next parking site prediction state information to be generated when site state information at the current point in time is input.

[0474] That is, the fault state transition model obtained by the multiple site state information pairs described through FIG. 14 may be a suitable model for obtaining site prediction state information of the next consecutive week, and this may be described as a 'general-next week fault state transition model' through this specification.

[0475] However, since the growth trend or mutation trend of fruits grown in a greenhouse may change depending on the surrounding environment, a separate model of the fruit state transition to the next week can be obtained 'by week', 'by quarter', or 'by season', and this will be described in more detail below.

[0476]

[0477] 1) [Parking-by-Parking - A state transition model for generating predicted status information for the next parking site]

[0478] If site status information has been acquired over several years and through several operations, it can be used to obtain a model of the next parking lot negligence status transition for each parking lot.

[0479] For example, for each of the first to Nth years, site status information for the first week and site status information for the second week can be obtained, and in this case, N pairs of site status information can be selected in which the site status information for the first week is input site status information and the site status information for the second week is output site status information, and a first week fault state transition model can be created using the selected N pairs of site status information.

[0480] In addition, in this case, since one year consists of 52 weeks, when the method according to the example described above is applied to the site status information of weeks 1 to 51, a fault state transition model of week 1 to a fault state transition model of week 51 can be generated.

[0481] Accordingly, according to the above, fault state transition models for generating next parking site prediction status information for each parking lot can be created, which can be described as a 'parking lot-to-next parking lot fault state transition model' through this specification.

[0482]

[0483] 2) [Quarterly - Fruit state transition model for generating next week site prediction status information]

[0484] A quarter can refer to a year divided into four periods, with the first quarter being January to March, the second quarter being April to June, the third quarter being July to September, and the fourth quarter being October to December. Therefore, since the growth or mutation trends of fruits may differ by quarter, creating a quarterly fruit status transition model can be an example of generating more accurate site status prediction information.

[0485] If site status information has been acquired over several years and several periods, it can be used to obtain a model of the next parking lot fault status transition by quarter.

[0486] For example, for each of the first quarters of Years 1 to N, site status information for Weeks 1 to 12 can be obtained, and in this case, a first quarter fruit status transition model can be generated using the site status information pairs selected according to the method described through FIG. 14.

[0487] In addition, in this case, since one year consists of four quarters, if the method according to the example described above is applied to the site status information obtained in each of the first to fourth quarters, a first quarter fault state transition model to a fourth quarter fault state transition model can be generated.

[0488] Accordingly, according to the above, fault state transition models for generating quarterly next-week parking site prediction status information can be created, which can be described herein as 'quarterly-next-week fault state transition models'.

[0489]

[0490] 3) [Seasonal - Fruit state transition model for generating next parking site prediction status information]

[0491] At this time, the season can mean a unit promised to explain the environmental changes that occur according to the latitude and longitude on the Earth where each country is located, and can include spring, summer, fall, and winter based on Korea, and generally, March, April, and May can be divided into spring, June, July, and August into summer, September, October, and November into fall, and December, January, and February into winter.

[0492] However, the definitions of seasons are not limited to the above-described definitions, and definitions of seasons that are distinguished according to the social conventions of the country to which the examples described in this application are applied may be applied.

[0493] That is, since the growth trends or mutation trends of fruits may differ by season, creating a seasonal fruit condition transition model can be an example of creating more accurate site prediction condition information.

[0494] If site status information has been acquired over several years and several periods, it can be used to obtain a model of the next parking lot fruit condition transition by season.

[0495] For example, for each of the first seasons of the first to Nth years, site status information for the first to 12th weeks can be obtained, and in this case, a first season fruit status transition model can be created using the site status information pairs selected according to the method described through FIG. 14.

[0496] In addition, in this case, since one year in Korea consists of four seasons, if the method according to the example described above is applied to the site status information obtained in each of the first to fourth seasons, a first season fruit status transition model to a fourth season fruit status transition model can be generated.

[0497] Accordingly, according to the above, fault state transition models can be created to generate seasonal next parking site prediction information, which can be described herein as 'quarterly-next week fault state transition models'.

[0498]

[0499] 4) [Progress-by-progress - Fruit state transition model for generating predicted status information for the next parking lot site]

[0500] Since crops located in a greenhouse may show different growth trends or mutation trends of fruits over time after the start of the growing season, creating a fruit condition transition model according to the progress of the growing season can be an example of creating more accurate site prediction condition information.

[0501] If site status information has been acquired over several years of operation, it can be used to obtain a model of the next parking lot fault status transition according to the operation progress.

[0502] For example, site status information of the first and second weeks of each of the first to Nth operations can be acquired, and in this case, N pairs of site status information can be selected in which the first week site status information is input site status information and the second week site status information is output site status information, and a first week fault status transition model of the operation can be created using the selected N pairs of site status information.

[0503] At this time, if one crop for a fruit called A is composed of 16 weeks, and the method according to the example described above is applied to the site status information of the first to the fifteenth weeks, a fruit status transition model of the first week of the crop to a fruit status transition model of the 15th week of the crop can be generated.

[0504] Accordingly, according to the above, fault state transition models for generating next parking lot site prediction status information by progress of the work can be created, which can be described as 'Factory progress - next parking lot fault state transition model' through this specification.

[0505]

[0506] (5) [Fruit state transition model for generating predicted state information for a specific parking site]

[0507] FIG. 15 is a diagram illustrating a selection of site state information obtained from past work to obtain a fault state transition model for generating specific parking site prediction state information according to one embodiment.

[0508] In Fig. 15, the A parking site status information (2310), the N parking site status information (2320), the A+1 parking site status information (2330), the N+1 parking site status information (2340), the A+M parking site status information (2350), and the N+M parking site status information (2360) are illustrated. This may be site status information acquired in a past work, and the above-described contents may be applied to this, so redundant descriptions will be omitted.

[0509] In addition, in FIG. 15, for convenience of explanation, the first fruit size class is indicated as S1, the second fruit size class is indicated as S2, the third fruit size class is indicated as S3, the first fruit maturity class is indicated as R1, the second fruit maturity class is indicated as R2, and the third fruit maturity class is indicated as R3.

[0510] Again, referring to FIG. 15, in order to obtain a fault state transition model for generating specific parking site prediction state information according to one embodiment, a plurality of site state information pairs may be selected based on site state information obtained from past operations.

[0511] For example, a first pair (2370) that uses the A-th parking site status information (2310) as input site status information and the N-th parking site status information (2320) as output site status information, a second pair (2380) that uses the A+1-th parking site status information (2330) as input site status information and the N+1-th parking site status information (2340) as output site status information, and a third pair (2390) that uses the A+M-th parking site status information (2350) as input site status information and the N+M-th parking site status information (2360) as output site status information can be selected.

[0512] At this time, the difference between the Nth parking lot and the A parking lot may correspond to the specific parking lot.

[0513] For example, if the above-mentioned fault state transition model is a model for generating site prediction state information 9 weeks later, the Nth week may mean the week 9 weeks later from the Ath week.

[0514] In this case, the site status information of the 1st and 10th weeks, the 2nd and 11th weeks, and the 3rd and 12th weeks can be selected in pairs.

[0515] In addition, in order to obtain a fault state transition model for generating a specific parking site prediction state information according to one embodiment, when a plurality of site state information pairs are selected based on site state information obtained from past operations, a fault state transition model obtained using the selected plurality of site state information pairs may reflect fault state transition relationships that allow site prediction state information after a specific parking lot to be generated when site state information at the current point in time is input.

[0516] That is, the fault state transition model obtained by the multiple site state information pairs described through FIG. 15 may be a suitable model for obtaining site prediction state information after a specific consecutive parking, and this may be described as a 'general-specific parking fault state transition model' through this specification.

[0517] In addition, since the contents of selecting site status information for generating the above-described 'by parking week', 'by quarter', 'by season' or 'by crop progress' fault status transition model can be applied to the fault status transition model for generating the above-described specific parking site prediction status information, redundant descriptions will be omitted.

[0518]

[0519] 3. [Fruit State Transition Model Using AI]

[0520] The contents described through FIGS. 10 to 15 were about a method for designing a fault state transition probability matrix and creating a fault state transition model using the designed fault state transition probability matrix.

[0521] However, a deep learning model can be used to create a fruit state transition model based on site state information obtained from past work, and this will be described in more detail below.

[0522] A fruit state transition model according to one embodiment may be implemented with at least one artificial neural network for learning sequential data. For example, the fruit state transition model according to one embodiment may include, but is not limited to, at least one artificial neural network layer among various artificial neural network layers such as a recurrent neural network (RNN), a long short-term memory network (LSTM), or gated recurrent units (GRUs).

[0523] Additionally, at least one artificial neural network layer included in the fault state transition model according to one embodiment may use the same or different activation functions.

[0524] At this time, the activation function may include, but is not limited to, a sigmoid function, a hyperbolic tangent function, a Relu function (rectified linear unit function), a leaky Relu function, an ELU function (exponential linear unit function), a softmax function, etc., and may include various activation functions (including custom activation functions) for outputting a result value or transmitting it to another artificial neural network layer.

[0525] Additionally, the fault state transition model according to one embodiment can be learned using at least one loss function.

[0526] At this time, the at least one loss function may include, but is not limited to, MSE (Mean Squared Error), RMSE (Root Mean Squared Error), Binary Crossentropy, Categorical Crossentropy, Sparse Categorical Crossentropy, etc., and may include various functions (including custom loss functions) for calculating the difference between the predicted result value and the actual result value.

[0527] Additionally, at least one optimizer may be used to learn a state transition model according to one embodiment.

[0528] At this time, the at least one optimizer may include, but is not limited to, Gradient descent, Batch Gradient Descent, Stochastic Gradient Descent, Mini-batch Gradient Descent, Momentum, AdaGrad, RMSProp, AdaDelta, Adam, NAG, NAdam, RAdam, AdamW, etc.

[0529] Additionally, a learning data set obtained based on site state information obtained in the above-described past work can be used to learn a fruit state transition model according to one embodiment.

[0530] At this time, the learning data set may be composed of learning input data and learning output data, and the learning input data and the learning output data may be composed of at least one of the site status information obtained from the above-described past work.

[0531] For example, in order to learn the above-described fault state transition model according to one embodiment, when the site state information of the 'i' week is used as learning input data, the site state information of the 'i+1' week can be used as learning output data, and in this case, the above-described fault state transition model can be learned as a model corresponding to the above-described 'general-next week fault state transition model'.

[0532] In addition, for example, in order to learn the above-described fault state transition model according to one embodiment, when the site state information of the 'i' week from 'iM' is used as learning input data, the site state information of the 'i+1' week can be used as learning output data, and in this case, when the site state information of the current point in time and the site state information of the past M weeks are input, a fault state transition model that generates the site state information of the next consecutive week can be created.

[0533] In addition, for example, in order to learn the fault state transition model according to one embodiment, when the site state information of the 'i' parking lot is used as learning input data, the site state information of the 'i+N' parking lots can be used as learning output data, and in this case, the fault state transition model can be learned as a model corresponding to the 'general-specific parking fault state transition model' described above, and in particular, can be learned as a model corresponding to a fault state transition model that generates site prediction state information after the Nth parking lot.

[0534] In addition, for example, in order to learn the above-described fault state transition model according to one embodiment, when the site state information from 'iM' to 'i' weeks is used as learning input data, the site state information of 'i+N' weeks can be used as learning output data, and in this case, when the site state information of the current point in time and the site state information of the past M weeks are input, a fault state transition model that generates predicted site state information after N weeks can be created.

[0535]

[0536] Below, the yield prediction method using the fruit state transition model described above will be explained in more detail.

[0537]

[0538] V. [Yield Prediction Method]

[0539] 1. [Yield Prediction Method]

[0540] FIG. 16 is a diagram for explaining a method for predicting fruit yield for a target site according to one embodiment.

[0541] Referring to FIG. 16, a method (3000) for predicting a fruit yield for a target site according to an embodiment may include obtaining a plurality of images reflecting fruits growing in a target site for a predetermined period of time (S3010), obtaining a fruit size classification value and a fruit maturity classification value for a plurality of fruits reflected in the plurality of images (S3020), obtaining a count value of fruits belonging to each of M*N fruit status classes based on the fruit size classification value and the fruit maturity classification value for the plurality of fruits, thereby generating first site status information for the target site (S3030), applying the first site status information to a fruit status transition model prepared in advance to generate first site prediction status information (S3040), and obtaining fruit yield prediction information for the target site based on the first site prediction status information (S3050).

[0542] In obtaining a plurality of images reflecting fruits growing at a target site for a predetermined period of time according to one embodiment (S3010), the predetermined period of time may mean a unit period for monitoring the status of the fruits.

[0543] For example, if the condition of fruits located in a greenhouse is monitored on a weekly basis, the predetermined period may mean one week.

[0544] In addition, in acquiring a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), acquiring a plurality of images during the predetermined period of time may mean not only acquiring a plurality of images continuously during the predetermined period of time, but also acquiring images at least once within the predetermined period of time.

[0545] For example, if the predetermined period is one week, acquiring multiple images during the predetermined period may mean acquiring multiple images continuously for one week, but is not limited thereto, and may also be interpreted to include acquiring multiple images for 8 hours on Monday, 8 hours on Wednesday, and 8 hours on Friday within one week.

[0546] In addition, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the predetermined period of time may be defined as a period of time during which the monitoring robot moved.

[0547] For example, if the monitoring robot was moved inside the greenhouse from 9:00 AM to 5:00 PM on November 2, the predetermined period may be defined as the period from 9:00 AM to 5:00 PM on November 2.

[0548] In addition, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the predetermined period of time may be defined as a period of time including a period during which the monitoring robot moved.

[0549] For example, if the monitoring robot was moved inside the greenhouse from 9:00 AM to 5:00 PM on November 2, the preset period may be defined as November 2.

[0550] In addition, in acquiring a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the predetermined period of time may be defined as a unit period representing images acquired through at least one image acquisition device mounted on the monitoring robot.

[0551] For example, if a monitoring robot moves inside a greenhouse and acquires images from 9:00 AM to 5:00 PM on November 2, and if a monitoring robot moves inside a greenhouse and acquires images from 9:00 AM to 5:00 PM on November 3, the first period can be defined as a week from October 30 to November 5.

[0552] In addition, in obtaining a plurality of images reflecting fruits growing in a target site during a predetermined period of time according to one embodiment (S3010), the target site may be the entire greenhouse, a portion of the greenhouse, a set of specific crop rows in the greenhouse, a specific row in the greenhouse, or a portion of a specific row in the greenhouse.

[0553] In addition, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the plurality of images reflecting fruits growing at the target site may mean a plurality of images in which at least some fruits growing at the target site are at least partially expressed.

[0554] For example, if the target site means the entire greenhouse, the plurality of images reflecting fruits growing in the target site may mean a plurality of images representing at least some of the fruits growing in a set of specific crop rows in the greenhouse.

[0555] In addition, in acquiring a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the difference between the time at which a first image among the plurality of images is acquired and the time at which a last image among the plurality of images is acquired may be shorter than the predetermined period of time.

[0556] For example, the predetermined period may be one week, the first image acquired among the plurality of images may be acquired at 8:00 AM on Monday, and the last image acquired among the plurality of images may be acquired at 4:59 PM on Friday.

[0557] In addition, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the plurality of images can be obtained through at least one image acquisition device mounted on the monitoring robot.

[0558] For example, the plurality of images may be acquired by at least one image acquisition device mounted on the monitoring robot while the monitoring robot moves between specific rows of crops located in the greenhouse.

[0559] Additionally, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), the plurality of images may not include images expressing the same crop rows for different days.

[0560] For example, images representing at least a portion of a first crop row located in the greenhouse may be acquired on a first day, images representing at least a portion of a second crop row located in the greenhouse may be acquired on a second day, and so on. The plurality of images may not include images representing the same crop row for different days.

[0561] In addition, in obtaining a plurality of images reflecting fruits growing at a target site during a predetermined period of time according to one embodiment (S3010), images expressing at least a portion of the same crop row among the plurality of images can be obtained on the same day.

[0562] For example, images representing at least a portion of a first crop row located in a greenhouse may be acquired on a first day, images representing at least a portion of a second crop row located in the greenhouse may be acquired on a second day, and so on, images representing at least a portion of the same crop row among the plurality of images may be acquired on the same day.

[0563] In obtaining fruit size classification values ​​and fruit maturity classification values ​​for a plurality of fruits reflected in the plurality of images according to one embodiment (S3020), the contents of the fruit size judgment module described above can be applied to obtaining fruit size classification values ​​for a plurality of fruits reflected in the plurality of images, so redundant descriptions will be omitted.

[0564] For example, the fruit size classification values ​​for the plurality of fruits reflected in the plurality of images can be determined based on the size of the target fruit area, and the size of the fruit area can be determined based on a predetermined axial length or an area of ​​the fruit area, and in this case, the depth value for the fruit area can be considered, and the contents of the fruit size determination module described above can be applied to obtaining the fruit size classification values ​​for the plurality of fruits reflected in the plurality of images.

[0565] In obtaining fruit size classification values ​​and fruit maturity classification values ​​for a plurality of fruits reflected in the plurality of images according to one embodiment (S3020), the fruit size classification values ​​can be obtained as values ​​corresponding to any one of the M fruit size classes.

[0566] For example, the above fruit size classification value can be obtained as a value corresponding to any one of three fruit size classes corresponding to 'small', 'medium', and 'large'.

[0567] In addition, in obtaining fruit size classification values ​​and fruit maturity classification values ​​for a plurality of fruits reflected in the plurality of images according to one embodiment (S3020), the contents of the fruit maturity judgment module described above can be applied to obtaining fruit maturity classification values ​​for a plurality of fruits reflected in the plurality of images, so redundant descriptions will be omitted.

[0568] For example, the fruit maturity classification values ​​for the multiple fruits reflected in the multiple images can be determined based on the pixel values ​​(color values) of the target fruit area, and the contents of the fruit maturity judgment module described above can be applied to obtaining the fruit maturity classification values ​​for the multiple fruits reflected in the multiple images.

[0569] In obtaining fruit size classification values ​​and fruit maturity classification values ​​for a plurality of fruits reflected in the plurality of images according to one embodiment (S3020), the fruit maturity classification values ​​can be obtained as values ​​corresponding to any one of N fruit maturity classes.

[0570] For example, the above fruit maturity classification value can be obtained as a value corresponding to any one of three fruit maturity classes corresponding to ‘unripe’, ‘ripening’, and ‘ripe’.

[0571] In one embodiment, based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits, count values ​​of fruits belonging to each of M*N fruit condition classes are obtained to generate first site condition information for the target site (S3030), wherein the M*N fruit condition classes can be distinguished by M fruit size classes and N fruit maturity classes.

[0572] For example, the above M*N fruit condition classes may mean nine fruit condition classes distinguished according to three fruit size classes corresponding to 'large', 'medium', and 'small' respectively, and three fruit maturity classes corresponding to 'ripe', 'ripening', and 'unripe' respectively.

[0573] At this time, the fruit size class may be a class of fruit attribute information related to fruit enlargement, and the fruit maturity class may be a class of fruit attribute information related to fruit maturity.

[0574] In this case, the M*N fruit condition classes may include first to ninth fruit condition classes, the first fruit condition class is a fruit condition class specified by the first fruit size class and the first fruit maturity class, the second fruit condition class is a fruit condition class specified by the second fruit size class and the first fruit maturity class, the third fruit condition class is a fruit condition class specified by the third fruit size class and the first fruit maturity class, the fourth fruit condition class is a fruit condition class specified by the first fruit size class and the second fruit maturity class, the fifth fruit condition class is a fruit condition class specified by the second fruit size class and the second fruit maturity class, the sixth fruit condition class is a fruit condition class specified by the third fruit size class and the second fruit maturity class, and the seventh The fruit condition class may be a fruit condition class specified by the first fruit size class and the third fruit maturity class, the eighth fruit condition class may be a fruit condition class specified by the second fruit size class and the third fruit maturity class, and the ninth fruit condition class may be a fruit condition class specified by the third fruit size class and the third fruit maturity class.

[0575] At this time, the first fruit size class may be a fruit size class corresponding to a fruit size of 'small', the second fruit size class may be a fruit size class corresponding to a fruit size of 'medium', the third fruit size class may be a fruit size class corresponding to a fruit size of 'large', the first fruit maturity class may be a fruit maturity class corresponding to 'unripe', the second fruit maturity class may be a fruit maturity class corresponding to 'ripening', and the third fruit maturity class may be a fruit maturity class corresponding to 'ripe'.

[0576] In addition, in generating first site status information for the target site (S3030), the count values ​​of fruits belonging to each of M*N fruit status classes are obtained based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits according to one embodiment, wherein the M and N may be the same, but are not limited thereto, and may be different from each other.

[0577] In addition, in generating first site status information for the target site (S3030), the count value of fruits belonging to each of M*N fruit status classes is obtained based on the fruit size classification value and the fruit maturity classification value for the plurality of fruits according to one embodiment, and the count value of fruits may be a number value of fruits belonging to each of the M*N fruit status classes, but is not limited thereto, and may be provided as a ratio value of fruits belonging to each of the M*N fruit status classes.

[0578] In addition, in generating first site status information for the target site (S3030), the count values ​​of fruits belonging to each of M*N fruit status classes are obtained based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits according to one embodiment, and therefore, the contents of the above-described site status information can be applied to the first site status information, and therefore, redundant descriptions will be omitted.

[0579] In generating first site prediction state information by applying the first site state information according to one embodiment to a pre-prepared fault state transition model (S3040), since the above-described contents can be applied to the fault state transition model, redundant descriptions will be omitted.

[0580] In addition, in generating first site prediction state information by applying the first site state information to a pre-prepared fault state transition model according to one embodiment (S3040), applying the first site state information to the pre-prepared fault state transition model may include inputting the first site state information into the pre-prepared fault state transition model.

[0581] In addition, in generating first site prediction state information by applying the first site state information to a pre-prepared fault state transition model according to one embodiment (S3040), applying the first site state information to the pre-prepared fault state transition model may include applying the pre-prepared fault state transition model to the first site state information multiple times.

[0582] In addition, in generating first site prediction state information by applying the first site state information according to one embodiment to a pre-prepared fault state transition model (S3040), the pre-prepared fault state transition model may include at least one fault state transition model.

[0583] For example, the above-mentioned pre-prepared fault state transition model may include at least one or more fault state transition models from a 'general-next week fault state transition model', a 'weekly-next week fault state transition model', a 'quarterly-next week fault state transition model', a 'seasonal-next week fault state transition model', a 'cropping progress-next week fault state transition model', and a 'general-specific parking fault state transition model'.

[0584] In addition, in generating first site prediction state information by applying the first site state information according to one embodiment to a pre-prepared fault state transition model (S3040), the pre-prepared fault state transition model may include a fault state transition relationship.

[0585] At this time, assuming that the M*N fault state classes include the first to ninth fault state classes, according to the fault state transition relationship, the degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site state information belong to the second fault state class of the first site prediction state information may be greater than the degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site state information belong to the fourth fault state class of the first site prediction state information.

[0586] In addition, at this time, when assuming that the M*N fault state classes include the first to ninth fault state classes, according to the fault state transition relationship, the degree to which the fault states of the fruits belonging to the second fault state class of the first site state information transition to the first fault state class of the first site prediction state information may be smaller than the degree to which the fault states of the fruits belonging to the second fault state class of the first site state information transition to the fifth fault state class of the second site prediction state information.

[0587] In addition, at this time, according to the above-mentioned fault state transition relationship, when the count values ​​for other fault state classes except the count value for the first fault state class of the first site state information are corrected to 0 and the corrected first site state information is applied to the fault transition model, the count value for the second fault state class of the first site prediction state information may be greater than the count value for the fourth fault state class.

[0588] In addition, in generating first site prediction state information by applying the first site state information to a pre-prepared fault state transition model according to one embodiment (S3040), the form and method of applying the first site state information to the pre-prepared fault state transition model may vary depending on the current point in time represented by the first site state information or the specific point in time in the future represented by the first site prediction state information, and this will be described in more detail below using FIGS. 17 to 20.

[0589] In addition, in generating first site predicted state information by applying the first site state information according to one embodiment to a pre-prepared fruit state transition model (S3040), the first site predicted state information may be information expressed through the same data format as the site state information, may mean predicted site state information representing a specific point in time in the future, and may mean a set of predicted count values ​​of fruits belonging to each of a plurality of fruit state classes (as described above, the plurality of fruit state classes are distinguished according to a fruit size class and a fruit maturity class) for a specific point in time in the future, and since the contents of the above-described site predicted state information may be applied to the first site predicted state information, redundant descriptions will be omitted.

[0590] In obtaining fruit yield prediction information for the target site based on the first site prediction status information according to one embodiment (S3050), the fruit yield prediction information may be fruit yield prediction information for a specific point in the future represented by the first site prediction status information.

[0591] For example, if the first site prediction status information represents the next week, the fruit yield prediction information may be fruit yield prediction information for the next week.

[0592] Additionally, for example, if the first site prediction status information represents a week 8 weeks later, the fruit yield prediction information may be fruit yield prediction information for a week 8 weeks later.

[0593] Additionally, for example, if the first site prediction status information represents a specific future week, the fruit yield prediction information may be fruit yield prediction information for the specific future week.

[0594] In obtaining fruit yield prediction information for the target site based on the first site prediction status information according to one embodiment (S3050), the fruit yield prediction information may correspond to the above-described yield prediction value, and in obtaining fruit yield prediction information for the target site based on the first site prediction status information according to one embodiment (S3050), the contents of the above-described yield calculation model may be applied, so redundant descriptions will be omitted.

[0595]

[0596] Hereinafter, various embodiments of generating first site predicted state information by applying the first site state information according to one embodiment to a pre-prepared fault state transition model (S3040) will be described in more detail.

[0597] This is to specifically explain this because the application of the first site status information may vary depending on whether the first site prediction status information is prediction status information representing a specific point in the future, and the application of the first site status information may vary depending on the current point in time at which the first site status information is acquired.

[0598] Accordingly, the various embodiments described below may be applied to generating first site prediction state information by applying the first site state information described through FIG. 16 to a pre-prepared fault state transition model (S3040), or may be applied by replacing generating first site prediction state information by applying the first site state information to a pre-prepared fault state transition model (S3040).

[0599]

[0600] 2. [Various embodiments for generating first site prediction status information]

[0601] (1) [Method for generating the first site prediction status information representing the next week]

[0602] In the case where the above-described pre-prepared fruit state transition model according to one embodiment is provided to include the above-described 'weekly-next week fruit state transition model', 'quarterly-next week fruit state transition model', 'seasonal-next week fruit state transition model' or 'cropping progress-next week fruit state transition model', it is necessary that the fruit state transition model be selected according to the current point in time represented by the above-described first site state information.

[0603] Accordingly, below, a method for generating first site prediction information by applying first site status information to a pre-prepared fault state transition model, taking into account the current point in time represented by the first site status information, will be described in more detail.

[0604] FIG. 17 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0605] Referring to FIG. 17, a method (3100) for generating first site prediction information according to one embodiment may include obtaining information about a current point in time (S3110), selecting a fault state transition model based on the information about the current point in time (S3120), and generating first site prediction state information by applying the first site state information to the selected fault state transition model (S3130).

[0606] At this time, the method (3100) for generating first site prediction information according to one embodiment can be understood as more specifically describing generating first site prediction status information by applying the first site status information described through FIG. 16 to a pre-prepared fault status transition model (S3040).

[0607] In obtaining information about the current point in time according to one embodiment (S3110), the current point in time is described as a point in time for convenience of explanation, but may be understood as a current period, a representative point in time representing the current period, a representative period, a representative day, or a representative week.

[0608] In addition, in obtaining information about the current point in time according to one embodiment (S3110), the current point in time can be obtained as a representative value for the period in which the above-described plurality of images were obtained.

[0609] For example, if the above-described multiple images were acquired between January 2, 2023 and January 8, 2023, the present point in time can be acquired as any one day between January 2, 2023 and January 8, 2023, and can be acquired as the first week of 2023.

[0610] Additionally, in obtaining information about the current point in time according to one embodiment (S3110), the current point in time may mean a point in time represented by the first site status information.

[0611] For example, if a plurality of images for obtaining the first site status information are obtained between January 2, 2023 and January 8, 2023, the fruits growing at the target site reflected in the plurality of images are reflected in the status between January 2, 2023 and January 8, 2023, and therefore, the first site status information can be understood as a group indicator of the status of the fruits growing at the target site between January 2, 2023 and January 8, 2023, and therefore, if the current point in time is obtained by specifying any one day from January 2, 2023 to January 8, 2023 or is obtained in the first week of 2023, the current point in time can mean a time represented by the first site status information.

[0612] Selecting a fault state transition model based on information about the current point in time according to one embodiment (S3120) can be understood as selecting a fault state transition model corresponding to the current point in time among the fault state transition models prepared in advance based on information about the current point in time, and this can operate differently depending on the fault state transition model prepared in advance.

[0613] For example, if the above-mentioned pre-prepared fault state transition model is configured as a 'week-to-next-week fault state transition model' and includes a fault state transition model for the 1st to 52nd week, selecting the fault state transition model based on information about the current time point according to one embodiment (S3120) may include determining a parking corresponding to the current time point based on information about the current time point, and selecting a fault state transition model corresponding to the determined parking lot.

[0614] That is, if the information about the current point in time is acquired as the first week of 2023, according to selecting a fault state transition model based on the information about the current point in time according to one embodiment (S3120), the fault state transition model may be selected as the first week fault state transition model.

[0615] In addition, for example, if the pre-prepared fault state transition model is configured as a 'quarterly-next week fault state transition model' and includes the first to fourth quarter fault state transition models, selecting the fault state transition model based on information about the current point in time according to one embodiment (S3120) may include determining a branch corresponding to the current point in time based on information about the current point in time, and selecting a fault state transition model corresponding to the determined branch.

[0616] That is, if the information about the current point in time is acquired as the first week of 2023, according to selecting a fault state transition model based on the information about the current point in time according to one embodiment (S3120), the branch corresponding to the current point in time can be determined as the first quarter, and the fault state transition model can be selected as the first quarter fault state transition model.

[0617] In addition, for example, if the pre-prepared fruit state transition model is configured as a 'seasonal-next week fruit state transition model' and includes spring, summer, fall, and winter fruit state transition models, selecting the fruit state transition model based on information about the current point in time according to one embodiment (S3120) may include determining a season corresponding to the current point in time based on information about the current point in time, and selecting a fruit state transition model corresponding to the determined season.

[0618] That is, if the information about the current point in time is acquired in the first week of 2023, according to selecting a fruit state transition model based on the information about the current point in time according to one embodiment (S3120), the season corresponding to the current point in time can be determined to be winter, and the fruit state transition model can be selected as the winter fruit state transition model.

[0619] In addition, for example, if the pre-prepared fault state transition model is configured as a 'fine state transition model by crop progress - next week' and includes a fault state transition model of the first to Nth week of cropping, selecting the fault state transition model based on information about the current point in time according to one embodiment (S3120) may include determining the crop progress week corresponding to the current point in time by further considering information about the current point in time and information about the crop start time, and selecting the fault state transition model corresponding to the determined crop progress week.

[0620] That is, if the information about the current point in time is acquired as the 1st week of 2023, and the information about the start of the crop is acquired as the 50th week of 2022, then according to selecting the fault state transition model based on the information about the current point in time according to one embodiment (S3120), the week of the crop progress corresponding to the current point in time can be determined to be the 4th week, and the fault state transition model can be selected as the 4th week fault state transition model of the crop.

[0621] Regarding generating first site predicted state information by applying first site state information to a selected fault state transition model according to one embodiment (S3130), the contents of generating first site predicted state information by applying the first site state information to a pre-prepared fault state transition model (S3040) described above through FIG. 16 can be applied, so redundant descriptions will be omitted.

[0622]

[0623] (2) [Method for generating first site prediction status information representing a specific parking lot]

[0624] In order to obtain fruit yield prediction information for a specific point in time that is in the future, rather than the week immediately following the current time represented by the first site status information, according to one embodiment, it may be necessary to generate first site prediction status information representing a specific week corresponding to a specific point in the future.

[0625] At this time, if the pre-prepared fault state transition model is a model for generating site prediction state information after a specific parking lot, such as a 'general-specific parking fault state transition model', the first site prediction state information representing a specific parking lot corresponding to a specific point in the future can be generated according to the above-described method.

[0626] However, if the pre-prepared fault state transition model is a model for generating next parking site prediction information, it may not be easy to generate first site prediction state information representing a specific parking lot corresponding to a specific point in the future rather than the parking lot immediately following the current point in time.

[0627] Therefore, in the following, when a pre-prepared fault state transition model is a model for generating next parking site prediction information, a specific method for generating first site prediction state information representing a specific parking lot corresponding to a specific point in the future rather than the parking lot immediately following the current point in time will be described.

[0628]

[0629] FIG. 18 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0630] Referring to FIG. 18, a method (3200) for generating first site prediction information according to one embodiment may include obtaining information about a specific point in time (S3210), determining the number of times a fault state transition model is applied based on the information about the specific point in time (S3220), and generating first site prediction state information by applying the fault state transition model to the first site state information the number of times it is applied determined (S3230).

[0631] In obtaining information about a specific point in time according to one embodiment (S3210), the information about the specific point in time can be obtained through a user's input.

[0632] In addition, in obtaining information about a specific point in time according to an embodiment (S3210), the specific point in time is described as a point in time for convenience of explanation, but may be understood as a specific period, a representative point in time representing a specific period, a representative period, a representative day, or a representative week.

[0633] For example, information about the specific point in time above can be obtained as the 52nd week of 2023, November 29, 2023, the 3rd week of the current period, the week 9 weeks from the current time, the day 100 days from the current time, etc.

[0634] In determining the number of applications of the fault state transition model based on information about a specific point in time according to an embodiment (S3220), the number of applications can be determined based on the type of the fault state transition model and the period from the current point in time to the specific point in time.

[0635] For example, if the above-mentioned fault state transition model is a fault state transition model for generating next parking site prediction status information, and the specific point in time is a parking lot 9 weeks from the current point in time, the number of applications may be determined to be 9 times.

[0636] Additionally, for example, if the above-mentioned fault state transition model is a fault state transition model for generating site prediction state information for the next day, and the specific point in time is 100 days from the current point in time, the number of applications may be determined to be 100.

[0637] In addition, for example, if the above-mentioned fault state transition model is a fault state transition model for generating next parking site prediction state information, and the specific point in time is the 52nd week of 2023 and the current point in time is the 49th week of 2023, the number of applications may be determined to be 3 times.

[0638] Generating first site predicted state information by applying the fault state transition model to the first site state information a determined number of times according to one embodiment (S3230) may include generating first site predicted state information by multiplying the fault state transition model to the first site state information a determined number of times.

[0639] For example, if the fault state transition model includes a fault state transition probability matrix, generating first site predicted state information by applying the fault state transition model to the first site state information a determined number of times according to one embodiment (S3230) may include generating first site predicted state information by multiplying the fault state transition probability matrix by the number of times applied to the first site state information.

[0640] In addition, generating first site predicted state information by applying the fault state transition model to the first site state information a determined number of applications according to one embodiment (S3230) may include obtaining first preliminary state information by applying the first site state information to the fault state transition model, and obtaining second preliminary state information by applying the obtained first preliminary state information to the fault state transition model, so that the number of times the first site state information and the preliminary state information are applied to the fault state transition model becomes the determined number of applications.

[0641] For example, if the determined number of applications is 3, generating the first site predicted state information by applying the fault state transition model to the first site state information the determined number of applications (S3230) according to one embodiment may include obtaining first preliminary state information by applying the first site state information to the fault state transition model, obtaining second preliminary state information by applying the obtained first preliminary state information to the fault state transition model, and generating the first site predicted state information by applying the obtained second preliminary state information to the fault state transition model.

[0642] At this time, the contents of the site prediction status information described above can be applied to the preliminary status information, and the preliminary status information is similar to the site prediction status information in that it is information obtained by applying the site status information to the fault state transition model, but the term preliminary status information is used only to distinguish it from the first site prediction status information that is finally obtained, and therefore, a detailed description of overlapping contents to which the contents of the site prediction status information described above can be applied will be omitted.

[0643]

[0644] The methods described above can be easily applied to the 'general-next week negligence state transition model'.

[0645] However, in the case of 'Weekly-Next Week Fault Status Transition Model', 'Quarterly-Next Week Fault Status Transition Model', 'Seasonally-Next Week Fault Status Transition Model', or 'Crop Progress-Next Week Fault Status Transition Model', the model to be applied may differ depending on the current point in time or a specific point in time, and the order of application of the models or the number of times the models are applied need to be determined accordingly.

[0646] Below, a specific method for generating first site predicted state information representing a specific parking lot corresponding to a specific point in the future rather than the current point in time, which can be easily applied to a 'week-by-week-next-week fault state transition model' or a 'crop progress-by-next-week fault state transition model', will be described.

[0647]

[0648] FIG. 19 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0649] Referring to FIG. 19, a method (3300) for generating first site prediction information according to one embodiment may include obtaining information about a current point in time and information about a specific point in time (S3310), selecting a plurality of fault state transition models based on the information about the current point in time and the information about the specific point in time, determining an application order of the selected plurality of fault state transition models (S3320), and applying the plurality of selected fault state transition models to the first site state information according to the determined application order to generate first site prediction state information (S3330).

[0650] In obtaining information about the current point in time and information about a specific point in time according to one embodiment (S3310), the above-described contents may be applied to the information about the current point in time and the information about the specific point in time, so redundant descriptions will be omitted.

[0651] In one embodiment, a plurality of fault state transition models are selected based on information about a current point in time and information about a specific point in time, and an application order of the selected plurality of fault state transition models is determined (S3320), wherein the plurality of fault state transition models can be selected based on the type of the fault state transition model and the period from the current point in time to the specific point in time.

[0652] For example, if the above-mentioned fault state transition model is composed of a 'weekly-next week fault state transition model' and includes a 1st to 52nd week fault state transition model, and the specific point in time is a week 9 weeks after the current point in time and the current point in time is the 1st week of 2023, the plurality of fault state transition models can be selected as the 1st to 9th week fault state transition models.

[0653] In addition, for example, if the above-mentioned fault state transition model is configured as a 'daily-next-day fault state transition model' and includes the first to 365th day fault state transition models, and the specific point in time is 100 days after the current point in time and the current point in time is January 2, 2023, the plurality of fault state transition models may be selected as the second to 101st day fault state transition models.

[0654] In addition, for example, if the above-mentioned fault state transition model is configured as a 'week-to-next-week fault state transition model' and includes a fault state transition model for the 1st to 52nd weeks, and the specific point in time is the 52nd week of 2023 and the current point in time is the 49th week of 2023, the plurality of fault state transition models may be selected as fault state transition models for the 49th, 50th, and 51st weeks.

[0655] In addition, in selecting a plurality of fault state transition models based on information about a current point in time and information about a specific point in time according to one embodiment, and determining an application order of the selected plurality of fault state transition models (S3320), the application order of the selected plurality of fault state transition models may be determined according to a time-series order of target periods that are targets of each of the selected plurality of fault state transition models.

[0656] For example, if the plurality of selected fault state transition models are first to ninth week fault state transition models, the order of application of the plurality of selected fault state transition models can be determined so that the first to ninth week fault state transition models are applied sequentially.

[0657] Generating first site prediction status information by applying a plurality of selected fault state transition models to the first site status information according to an embodiment of the present invention (S3330) may include generating first site prediction status information by multiplying the plurality of selected fault state transition models to the first site status information according to the determined application order.

[0658] For example, if the above-described fault state transition model includes a fault state transition probability matrix, and the selected plurality of fault state transition models are first to third week fault state transition models, and the first to third week fault state transition models are determined to be sequentially applied, generating first site predicted state information by applying the selected plurality of fault state transition models to the first site state information according to the determined application order (S3330) may include multiplying the first site state information by the fault state transition probability matrix of the first week fault state transition model, then multiplying the first site state information by the fault state transition probability matrix of the second week fault state transition model, and then multiplying the first site state information by the fault state transition probability matrix of the third week fault state transition model to generate the first site predicted state information.

[0659] In addition, the generating of the first site predicted state information by applying the plurality of selected fault state transition models to the first site state information according to one embodiment in a determined application order (S3330) may include applying the first site state information to the fault state transition model determined to be applied first to obtain first preliminary state information, and applying the obtained first preliminary state information to the fault state transition model determined to be applied second to obtain second preliminary state information, so that the first site state information and the preliminary state information are sequentially applied to the plurality of selected fault state transition models in a determined application order.

[0660] For example, if the selected plurality of fault state transition models are first to third week fault state transition models, and the first to third week fault state transition models are determined to be sequentially applied, generating first site predicted state information by applying the selected plurality of fault state transition models to the first site state information in the determined application order (S3330) according to one embodiment may include: obtaining first preliminary state information by applying the first site state information to the first week fault state transition model, obtaining second preliminary state information by applying the obtained first preliminary state information to the second week fault state transition model, and generating the first site predicted state information by applying the obtained second preliminary state information to the third week fault state transition model.

[0661]

[0662] Below, a specific method for generating first site predicted state information representing a specific week corresponding to a specific point in the future rather than the week immediately following the current point in time, which can be easily applied to a 'quarterly-next week fault state transition model' or a 'seasonal-next week fault state transition model', will be described.

[0663]

[0664] FIG. 20 is a diagram for explaining a method for generating first site prediction information according to one embodiment.

[0665] Referring to FIG. 20, a method (3400) for generating first site prediction information according to one embodiment may include obtaining information about a current point in time and information about a specific point in time (S3410), selecting a plurality of fault state transition models based on the information about the current point in time and the information about the specific point in time, determining the number of applications and the application order of each of the selected plurality of fault state transition models (S3420), and generating first site prediction state information by applying the plurality of selected fault state transition models to the first site state information a determined number of applications according to the determined application order (S3430).

[0666] Since the above-described contents can be applied to obtaining information about the current point in time and information about a specific point in time according to one embodiment (S3410), redundant descriptions will be omitted.

[0667] In one embodiment, a plurality of fault state transition models are selected based on information about a current point in time and information about a specific point in time, and the number of times and the order of application of each of the selected plurality of fault state transition models are determined (S3420), wherein the plurality of fault state transition models can be selected based on the type of the fault state transition model and the period from the current point in time to the specific point in time.

[0668] For example, if the above-mentioned fault state transition model is configured as a 'quarterly-next week fault state transition model' and includes a first quarter to fourth quarter fault state transition model, and the specific point in time is a week nine weeks after the current point in time and the current point in time is the eighth week of 2023, the plurality of fault state transition models can be selected as the first quarter and second quarter fault state transition models.

[0669] In addition, for example, if the above-mentioned fruit state transition model is configured as a 'seasonal-next week fruit state transition model' and includes the first to fourth seasonal fruit state transition models, and the specific point in time is the week 9 weeks after the current point in time and the current point in time is the fifth week of 2023, the plurality of fruit state transition models may be selected as the fourth season fruit state transition model corresponding to winter and the first season fruit state transition model corresponding to spring.

[0670] In addition, in selecting a plurality of fault state transition models based on information about the current point in time and information about a specific point in time according to one embodiment, and determining the number of applications and the application order of each of the selected plurality of fault state transition models (S3420), the number of applications of each of the plurality of fault state transition models can be selected based on the type of the fault state transition model and the period from the current point in time to the specific point in time.

[0671] For example, if the above-mentioned fault state transition model is configured as a 'quarterly-next week fault state transition model' and includes a first quarter to fourth quarter fault state transition model, and the specific point in time is a week nine weeks after the current point in time and the current point in time is the eighth week of 2023, the number of applications of the first quarter fault state transition model may be determined to be five times, and the number of applications of the second quarter fault state transition model may be determined to be four times.

[0672] In addition, for example, if the above-mentioned fruit state transition model is configured as a 'seasonal-next week fruit state transition model' and includes the first to fourth seasonal fruit state transition models, and the specific point in time is the week 9 weeks after the current point in time and the current point in time is the fifth week of 2023, the number of applications of the fourth seasonal fruit state transition model corresponding to winter may be determined to be 4 times, and the number of applications of the first seasonal fruit state transition model corresponding to spring may be determined to be 5 times.

[0673] In addition, in selecting a plurality of fault state transition models based on information about a current point in time and information about a specific point in time according to one embodiment, and determining the number of times and the order of application of each of the plurality of selected fault state transition models (S3420), the order of application of the plurality of selected fault state transition models may be determined according to the time-series order of the target period that is the target of each of the plurality of selected fault state transition models.

[0674] For example, if the plurality of selected fault state transition models are first-quarter and second-quarter fault state transition models, the order of application of the plurality of selected fault state transition models can be determined so that the first-quarter fault state transition model and the second-quarter fault state transition model are applied sequentially.

[0675] In addition, for example, when the plurality of selected fruit state transition models are a fourth season fruit state transition model and a first season fruit state transition model, the order of application of the plurality of selected fruit state transition models can be determined so that the fourth season fruit state transition model and the first season fruit state transition model are applied sequentially.

[0676] Generating first site prediction status information by applying a plurality of selected fault state transition models to the first site status information according to an embodiment of the present invention a number of times determined according to a determined application order (S3430) may include generating first site prediction information by multiplying the plurality of selected fault state transition models to the first site status information a number of times determined according to a determined application order.

[0677] For example, if the above-described fault state transition model includes a fault state transition probability matrix, and the selected plurality of fault state transition models are first and second branch fault state transition models, and the first and second branch fault state transition models are determined to be sequentially applied, and the number of times the first branch fault state transition model is applied is determined to be five times, and the number of times the second branch fault state transition model is applied is determined to be four times, generating first site predicted state information by applying the selected plurality of fault state transition models to the first site state information in the determined application order a determined number of times (S3430) according to one embodiment may include multiplying the first site state information by the fault state transition probability matrix of the first branch fault state transition model five times, and then multiplying the first site state information by the fault state transition probability matrix of the second branch fault state transition model four times to generate the first site predicted state information.

[0678] In addition, generating first site prediction status information by applying a plurality of selected fault state transition models to the first site status information according to one embodiment a determined number of applications according to a determined application order (S3430) may include generating first site prediction status information by applying the first site status information to the fault state transition model determined to be applied first a determined number of applications, and then applying the first site status information to the fault state transition model determined to be applied second a determined number of applications.

[0679] For example, if the selected plurality of fault state transition models are first and second branch fault state transition models, and the first and second branch fault state transition models are determined to be applied sequentially, and the number of times the first branch fault state transition model is applied is determined to be two times, and the number of times the second branch fault state transition model is applied is determined to be one time, generating first site predicted state information by applying the selected plurality of fault state transition models to the first site state information a number of times determined according to the determined application order (S3430) may include applying the first site state information to the first branch fault state transition model to obtain first preliminary state information, applying the obtained first preliminary state information to the first branch fault state transition model to obtain second preliminary state information, and applying the obtained second preliminary state information to the second branch fault state transition model to generate the first site predicted state information.

[0680] In the above, various examples of applying a pre-prepared fault state transition model according to the type of pre-prepared fault state transition model through FIGS. 17 to 20, the current point in time represented by the first site state information, or the specific point in the future represented by the first site predicted state information are described.

[0681] However, this is only to specify and disclose the types of specific models for convenience of explanation, and the methods described in each may be used with types of fault state transition models that are not described.

[0682] In addition, according to the contents described through FIGS. 1 to 20 of the present specification, a method for predicting yield based on group indexing of 'individual states of fruits' can be provided so as to solve the difficulties of the prior art while considering more 'direct information' about crops.

[0683]

[0684] The method according to the embodiment may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the medium may be those specially designed and configured for the embodiment or may be those known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and hardware devices specially configured to store and execute program commands, such as ROMs, RAMs, and flash memories. Examples of the program commands include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, etc. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

[0685] Although the embodiments described above have been described by way of limited examples and drawings, those skilled in the art will appreciate that various modifications and variations can be made based on the above teachings. For example, appropriate results can still be achieved even if the described techniques are performed in a different order than described, and / or components of the described systems, structures, devices, circuits, etc. are combined or combined in a different manner than described, or are replaced or substituted with other components or equivalents.

[0686] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims described below.

[0687]

[0688] As described above, the best mode for carrying out the invention has been described in detail.

Claims

1. A method for predicting fruit yield for a target site in a greenhouse, Acquiring a plurality of images reflecting fruits growing at a target site over a predetermined period of time, wherein the plurality of images are acquired by at least one image acquisition device mounted on a monitoring robot moving within the greenhouse; Obtaining fruit size classification values ​​and fruit maturity classification values ​​for multiple fruits reflected in the multiple images, wherein the fruit size classification value is obtained as a value corresponding to any one of M fruit size classes, and the fruit maturity classification value is obtained as a value corresponding to any one of N fruit maturity classes; Generating first site status information for the target site by obtaining count values ​​of fruits belonging to each of M*N fruit status classes based on the fruit size classification values ​​and fruit maturity classification values ​​for the plurality of fruits, wherein the M*N fruit status classes are distinguished by the M fruit size classes and the N fruit maturity classes; Applying the first site state information to a pre-prepared fault state transition model to generate the first site predicted state information, wherein the first site predicted state information includes a predicted count value for each of M*N fault state classes; and Obtaining fruit yield prediction information for the target site based on the first site prediction status information; A method for predicting fruit yield.

2. In paragraph 1, The count value of the above fruits is the number of fruits belonging to each of the M*N fruit status classes. A method for predicting fruit yield.

3. In paragraph 1, The above M and N are identical to each other. A method for predicting fruit yield.

4. In paragraph 3, The above M fruit size classes include a first fruit size class, a second fruit size class, and a third fruit size class, wherein the second fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the first fruit size class, and the third fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the second fruit size class. The above N fruit maturity classes include a first fruit maturity class, a second fruit maturity class, and a third fruit maturity class, wherein the second fruit maturity class is a class corresponding to a fruit having a greater degree of maturity than a fruit corresponding to the first fruit maturity class, and the third fruit maturity class is a class corresponding to a fruit having a greater degree of maturity than a fruit corresponding to the second fruit maturity class. A method for predicting fruit yield.

5. In paragraph 4, The above M*N fault condition classes include the first to ninth fault condition classes, The first fruit condition class is a fruit condition class specified by the first fruit size class and the first fruit maturity class, the second fruit condition class is a fruit condition class specified by the second fruit size class and the first fruit maturity class, the third fruit condition class is a fruit condition class specified by the third fruit size class and the first fruit maturity class, the fourth fruit condition class is a fruit condition class specified by the first fruit size class and the second fruit maturity class, the fifth fruit condition class is a fruit condition class specified by the second fruit size class and the second fruit maturity class, the sixth fruit condition class is a fruit condition class specified by the third fruit size class and the second fruit maturity class, and the seventh fruit condition class is a fruit condition specified by the first fruit size class and the third fruit maturity class. The 8th fruit condition class is a fruit condition class specified by the 2nd fruit size class and the 3rd fruit maturity class, and the 9th fruit condition class is a fruit condition class specified by the 3rd fruit size class and the 3rd fruit maturity class. A method for predicting fruit yield.

6. In paragraph 5, The above pre-prepared fruit state transition model is Contains the fault state transition relationships for the first to ninth fault state classes above, According to the above-mentioned fruit state transition relationship, The degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site status information belong to the second fault state class of the first site prediction status information is greater than the degree to which the fault state transitions so that the fruits belonging to the first fault state class of the first site status information belong to the fourth fault state class of the first site prediction status information. The degree to which the fault state transitions so that the fruits belonging to the second fault state class of the first site status information belong to the first fault state class of the first site prediction status information is smaller than the degree to which the fault state transitions so that the fruits belonging to the second fault state class of the first site status information belong to the fifth fault state class of the second site prediction status information. A method for predicting fruit yield.

7. In paragraph 5, The above pre-prepared fruit state transition model is Contains the fault state transition relationships for the first to ninth fault state classes above, According to the above-mentioned fruit state transition relationship, When the count value for the other fault status classes of the first site status information is corrected to 0, except for the count value for the first fault status class, and the corrected first site status information is applied to the fault transition model, the count value for the second fault status class of the first site prediction status information is greater than the count value for the fourth fault status class. A method for predicting fruit yield.

8. In paragraph 1, The above pre-prepared fruit state transition model is Contains the fruit state transition relationship for the above M*N fruit state classes, The above fault state transition relationship includes a fault state transition probability matrix. A method for predicting fruit yield.

9. In paragraph 8, The above-mentioned fruit state transition probability matrix is The probability values ​​that the fruits belonging to each of the M*N fruit condition classes included in the first site condition information will transition to each of the M*N fruit condition classes included in the first site prediction condition information A method for predicting fruit yield.

10. In paragraph 9, The number of probability values ​​included in the above-mentioned fruit state transition probability matrix is ​​(M*N)^2 individuals. A method for predicting fruit yield.

11. In paragraph 8, The above M fruit size classes include a first fruit size class, a second fruit size class, and a third fruit size class, wherein the second fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the first fruit size class, and the third fruit size class is a class corresponding to a fruit larger than a fruit corresponding to the second fruit size class. The above N fruit maturity classes include a first fruit maturity class, a second fruit maturity class, and a third fruit maturity class, wherein the second fruit maturity class is a class corresponding to a fruit that is more mature than a fruit corresponding to the first fruit maturity class, and the third fruit maturity class is a class corresponding to a fruit that is more mature than a fruit corresponding to the second fruit maturity class. The above M*N fault condition classes include the first to ninth fault condition classes, The first fruit condition class is a fruit condition class specified by the first fruit size class and the first fruit maturity class, the second fruit condition class is a fruit condition class specified by the second fruit size class and the first fruit maturity class, the third fruit condition class is a fruit condition class specified by the third fruit size class and the first fruit maturity class, the fourth fruit condition class is a fruit condition class specified by the first fruit size class and the second fruit maturity class, the fifth fruit condition class is a fruit condition class specified by the second fruit size class and the second fruit maturity class, the sixth fruit condition class is a fruit condition class specified by the third fruit size class and the second fruit maturity class, and the seventh fruit condition class is a fruit condition specified by the first fruit size class and the third fruit maturity class. If the 8th fruit condition class is a fruit condition class specified by the 2nd fruit size class and the 3rd fruit maturity class, and the 9th fruit condition class is a fruit condition class specified by the 3rd fruit size class and the 3rd fruit maturity class, The above-mentioned fruit state transition probability matrix is The probability that the fruit state can transition to the second fruit state class, the fifth fruit state class, and the eighth fruit state class, where the fruit size class is the second fruit size class, is designed to be 0, so that the fruit state can transition to the first fruit state class, the fourth fruit state class, and the seventh fruit state class, where the fruit size class is the first fruit size class. The probability that the fruit state can transition from the fourth fruit state class, the fifth fruit state class and the sixth fruit state class, whose fruit maturity class is the second fruit maturity class, to the first fruit state class, the second fruit state class and the third fruit state class, whose fruit maturity class is the first fruit maturity class, is 0. A method for predicting fruit yield.

12. In paragraph 11, The probability that the fruit belonging to the above 4th fault state class can transition to the 5th, 6th, 8th, and 9th fault state classes is designed to be 0. The probability that the fault state can be transitioned so that the fruits belonging to the 5th fault state class belong to the 4th, 6th, 7th and 9th fault state classes is designed to be 0. A method for predicting fruit yield.

13. In paragraph 1, The above plurality of images are images captured by at least one image acquisition device mounted on the monitoring robot while the monitoring robot moves along a rail positioned in the greenhouse within the above predetermined period of time. A method for predicting fruit yield.

14. In paragraph 1, The difference between the time at which the first image among the plurality of images was acquired and the time at which the last image among the plurality of images was acquired is shorter than the predetermined period. A method for predicting fruit yield.

15. In paragraph 1, The above multiple images do not contain images showing the same crop row on different days. A method for predicting fruit yield.

16. In paragraph 1, The above fruit size classification value is determined based on the length of the predetermined axis of the target fruit area or the width of the target fruit area. The above fruit maturity classification value is determined based on the pixel values ​​of pixels included in the target fruit area. A method for predicting fruit yield.

17. In paragraph 1, The above-mentioned pre-prepared fault state transition model includes multiple fault state transition models, The first site state information is applied to a pre-prepared fault state transition model to generate the first site predicted state information. Obtain information about the current point in time; Select a fault state transition model based on information about the current point in time; and generating the first site predicted state information by applying the first site state information to the selected fault state transition model; A method for predicting fruit yield.

18. In paragraph 1, The first site state information is applied to a pre-prepared fault state transition model to generate the first site predicted state information. Obtain information about a specific point in time; Determine the number of applications of the fault state transition model based on information about a specific point in time; and generating first site predicted state information by applying the fault state transition model to the first site state information a determined number of times; A method for predicting fruit yield.

19. In paragraph 1, Obtaining fruit yield prediction information for the target site based on the first site prediction status information Obtain information about the fruit condition classes that are the target of harvest; Among the predicted count values ​​included in the first site predicted status information, the predicted count values ​​belonging to the fruit status classes to be harvested are specified; Obtaining fruit yield prediction information based on specific predicted count values; including; A method for predicting fruit yield.

20. In paragraph 19, In order to obtain fruit yield prediction information based on specific predicted count values, pre-stored reference weight value information for each fruit condition class or fruit size class is used. A method for predicting fruit yield.

Images