Crop harvesting support system, crop harvesting support method, and crop harvesting support program

Abstract

We provide a crop harvesting support system that enables efficient harvesting by identifying the optimal harvesting time for crops. [Solution] The system comprises a harvesting robot, a management server configured to communicate with the robot, a reference information collection means for collecting reference information for estimating the harvest time for crops, a learning model storage unit that stores a learning model that has been machine-learned to determine the correlation between input data including the reference information for estimating the harvest time and output data including the harvest time of crops, and a time synchronization means for synchronizing the harvesting robot, the management server, and the reference information collection means at predetermined intervals. The system inputs the reference information for estimating the harvest time into the learning model to estimate the harvest time of crops, and sends a harvest command to the harvesting robot for crops whose estimated harvest time has arrived.

Description

Technical Field

[0001] The present invention relates to a crop harvesting support system, a crop harvesting support method, and a crop harvesting support program for supporting the harvesting of crops such as tomatoes.

Background Art

[0002] Recently, due to the aging of agricultural workers and the departure of young people from agriculture, the shortage of labor during the peak season has become particularly serious (labor shortage). In addition, in harvesting by hand, there are variations in determining the maturity, and it may be difficult to manage the quality (difficulty in quality control of harvesting). Furthermore, the harvesting work done by hand takes time, and in particular, in large-scale farms, improving the work efficiency becomes an issue (demand for work efficiency improvement). Moreover, harvesting by hand is costly in terms of labor costs, and in particular, the cost tends to increase as the labor shortage progresses (demand for cost management).

[0003] Therefore, conventionally, as a technology for supporting the shortage of farmers' labor and improving profitability, a self-driving robot that utilizes the light spectrum measurement technology and artificial intelligence (AI) to distinguish and pick red tomatoes has been developed (see Non-Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, even with the techniques described above, it is not possible to predict the optimal harvesting time or manage data on harvested tomatoes (harvesting time, location, ripeness, etc.). This invention has been made in view of the above circumstances, and its main objective is to provide a crop harvesting support system, a crop harvesting support method, and a crop harvesting support program that enable efficient harvesting by determining the optimal harvesting timing for crops. [Means for solving the problem]

[0006] To achieve the above objectives, the crop harvesting support system according to the present invention is A harvesting robot that harvests crops in a farm, A management server configured to communicate with the harvesting robot, A means for collecting reference information to collect reference information for estimating the harvest time for each of the aforementioned crops, A learning model storage unit stores a learning model that has been trained to determine the correlation between input data including the harvest time estimation criteria information and output data including the harvest time of the crops. The harvesting robot, the management server, and the reference information collection means are provided with time synchronization means for synchronizing the time of the harvesting robot, the management server, and the reference information collection means at predetermined intervals. The aforementioned management server A harvest time estimation means that inputs estimation input data, including the harvest time estimation reference information collected by the reference information collection means, into the learning model to estimate the harvest time of the crop, A harvest command means transmits a harvest command to the harvest robot for the crops whose harvest time has arrived as estimated by the harvest time estimation means, It is characterized by having [this feature].

[0007] Therefore, the management server uses the harvest time estimation means to input estimation input data, including harvest time estimation reference information collected by the reference information collection means, into a learning model to estimate the harvest time of crops, and the harvest command means transmits a harvest command to the harvesting robot for crops whose estimated harvest time has arrived.

[0008] In this system, the harvesting robot, management server, and information gathering means are synchronized at predetermined intervals by a time synchronization means, allowing for the collection of accurate information in real time. Based on this information, harvesting operations are carried out, enabling appropriate and efficient harvesting of crops.

[0009] Here, the harvest timing can be determined using information obtained from captured images, such as color, shape, and size. For example, in the case of tomatoes, the surface color changes from green to yellow and finally to red, so the maturity of the crop can be determined by capturing the change in color. Also, the shape of crops tends to change during the maturation process; for example, mature tomatoes have a more uniform shape than average. Therefore, it is possible to represent the maturity using an index that evaluates shape (shape index). Furthermore, the size of a crop is determined to be suitable for harvest when it reaches a certain size. Thus, it is possible to determine the harvest timing based on size.

[0010] Furthermore, meteorological or soil information may be added as information for determining the harvest time. Meteorological information may include temperature, humidity, sunshine hours, and precipitation. Soil information may include soil moisture and soil acidity.

[0011] Furthermore, when controlling harvesting robots accurately in real time, especially when controlling multiple harvesting robots, it is necessary to synchronize the time of each harvesting robot. Therefore, the following method can be used. In other words, a plurality of first devices are installed at intervals in the farm and collect location information in the farm, The harvesting robot is equipped with a second device, A control reference information calculation means calculates the propagation time of the information or signal between each of the multiple first devices and the second device, and the time difference between the first device and the second device, based on the bidirectional transmission and reception times of the information or signal between each of the multiple first devices and the second device. Distance calculation means for calculating the distance between each of the plurality of first devices and the second device based on the propagation time, A position identification means that identifies the position of the second device based on the distance between each of the first and second devices calculated by the distance calculation means, and the position information of each of the first devices. Based on the aforementioned time difference, the device synchronization means synchronizes the time of the second device and the first device, It has, By synchronizing the time of the multiple first devices with the management server at predetermined intervals using the time synchronization means, it is preferable to synchronize the time of the second devices using the device synchronization means.

[0012] Here, the first device from which location information in the farm is collected includes not only cases where the location of the first device within the farm has been collected in advance, but also cases where it has been collected retrospectively by some means. Furthermore, the location information collected by the first device may be stored in a readable format in its own memory, or it may be compiled into a database and stored in another storage device.

[0013] For the location information of the first device within the farm, two-dimensional location information is sufficient on flat ground, but if the terrain is uneven or the farm is on a slope, it is preferable to use three-dimensional location information. Furthermore, the installation method of the first device within the farm is not particularly limited; it may be attached to farm structures (such as the inside of a greenhouse or the surface of a light fixture), or it may be embedded in a structure. It may also be fixed to a dedicated pole.

[0014] The second device installed on the harvesting robot includes not only cases where the second device is directly installed on the harvesting robot (attached to the second device by appropriate means such as adhesive or brackets), but also cases where it is housed in a storage section provided on the harvesting robot. Furthermore, it is desirable to intentionally vary the height at which the first device is installed. By managing the height of the first device, it becomes possible to more accurately determine the three-dimensional position of the second device.

Advantages of the Invention

[0015] As described above, according to the crop harvesting support system, crop harvesting support method, and crop harvesting support program according to the present invention, after synchronizing the harvesting robot, the management server, and the reference information collection means in time at a predetermined cycle, the input data for estimation including the harvesting time estimation reference information collected by the reference information collection means is input into the learning model to estimate the harvesting time of the crops. Since a harvesting command is sent to the harvesting robot for the crops for which the estimated harvesting time has arrived, it is possible to efficiently harvest the crops at an optimal harvesting timing.

Brief Description of the Drawings

[0016] [Figure 1] It is a diagram showing an example of a farm where the crop harvesting support system according to the present invention is applied, specifically an example of cultivating tomatoes inside a greenhouse. [Figure 2] It is a diagram showing a configuration example of the crop harvesting support system according to the present invention. [Figure 3] It is a schematic configuration diagram showing a harvesting time estimation system. [Figure 4] It is a flowchart showing an example of a schematic control process in the management server. [Figure 5] It is a schematic configuration diagram showing a robot position measurement system. (a) is a plan view showing an installation example of a first device and a second device in a farm (cultivation greenhouse), and (b) is a block diagram of (a). [Figure 6] It is a block diagram showing a configuration example of the first device. [Figure 7] It is a block diagram showing a configuration example of the second device. [Figure 8] It is a block diagram showing the configuration of the management server. [Figure 9] It is a flowchart showing a distance calculation process. [Figure 10] It is a flowchart showing a position identification process. [Figure 11]This flowchart shows an example of the control operation of a harvesting robot used in the crop harvesting support system according to the present invention. [Modes for carrying out the invention]

[0017] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0018] Embodiments of the present invention will be described below with reference to the drawings. Figures 1 and 2 show a schematic configuration of the crop harvesting support system S1 according to the present invention. This crop harvesting support system S1 is used in farms that cultivate crops (for example, tomatoes), such as when cultivating tomatoes in greenhouse H shown in Figure 1. As shown in Figure 2, it consists of a harvesting robot 10, a management server 20 configured to communicate with the harvesting robot 10, a reference information collection unit 30 consisting of various sensors (sensors that collect weather data and soil data, etc.) and imaging devices (cameras installed on the harvesting robot 10 to acquire image information of crops, cameras mounted on the drone 40, cameras fixed in predetermined positions, etc.) for collecting reference information for estimating the harvest time for individual crops, and a time synchronization server 50 for managing time synchronization.

[0019] As shown in Figure 3, the management server 20 comprises a reference information collection unit 30, a crop image database 3, an image analysis device 4 (image analysis device 4a that analyzes images taken by camera 2a and image analysis device 4b that analyzes images taken by camera 2b) within the reference information collection unit 30 (camera and other shooting devices 2a that capture images used in the learning phase and shooting device 2b that captures images used in the estimation phase), a machine learning device 5, a harvest time estimation device 6, and a display device 7, thereby constructing the harvest time estimation system S2.

[0020] The imaging device 2 utilizes cameras mounted on the harvesting robot 10, cameras pre-installed near the harvested crops, and cameras mounted on the drone 40. It consists of high-resolution cameras capable of capturing specific wavelengths (for example, near-infrared cameras) and is used to photograph the appearance (color, shape, size) of crops during their growth process. The imaging device 2a is used to take these photographs to record the growth of the crops and to collect images that will be used to form input data for creating a learning model. The imaging device 2b is used to photograph the crops to be evaluated (hereinafter referred to as "evaluation crops"). In other words, it is used to collect images that will be used to form estimation input data that will be input into the trained learning model 8. Here, the imaging device 2a used in the learning phase and the imaging device 2b used in the estimation phase may be the same or different.

[0021] The crop image database 3 stores image data captured by the camera 2a, linked to the location and date / time of the crop. The image data stored in this crop image database 3 is used as needed during the learning phase to form a learning model. In the learning phase, the image data captured by the camera 2a is temporarily stored in the crop image database 3 for record-keeping purposes, but it may also be sent directly to the image analysis device 4a. In the estimation phase, the images captured by the camera 2b are sent directly to the image analysis device 4b, but they may also be temporarily stored in an image database (not shown) for pre-processing, such as removing unnecessary parts of the image.

[0022] Image analysis device 4 is used to analyze images of crops captured by the camera 2 and obtain harvest time estimation criteria information (color, shape, size, etc.) for estimating the harvest time of the crops. Image analysis device 4a, used in the learning phase, is used to output the harvest time estimation criteria information obtained here to the machine learning device 5. Image analysis device 4b, used in the estimation phase, is used to output the harvest time estimation criteria information obtained here to the harvest time estimation device 6.

[0023] The machine learning device 5 operates as the main component in the learning phase, generating a learning model used to estimate the harvest time of crops using machine learning, based on information obtained by analyzing images of crops in the growing area (color, shape, size, etc.), as well as weather and soil data that are incorporated as needed. Therefore, the information obtained by analyzing these images (color, shape, size, etc.), weather data, and soil data constitute the harvest time estimation reference information that serves as the basis for estimating the harvest time.

[0024] The harvest timing estimation device 6 operates as the main component of the estimation phase. Using a pre-trained model 8 generated by the machine learning device 5, it estimates the harvest timing of the crop based on information obtained by analyzing images of the crop, as well as weather data and soil data. The display unit 7 can be replaced by the display screen of various terminals and displays information such as the harvest time estimated by the harvest time estimation device 6.

[0025] The machine learning device 5 consists of a general-purpose or dedicated computer. This computer may be a stationary computer or a portable computer, and may be a client computer, a server computer, or a cloud computer.

[0026] The training data used to form the learning model consists of, as input data, crop image analysis information (color, shape, size) obtained by analyzing images of crops obtained by the imaging device 2, and, if necessary, weather data and soil data. The output data consists of the actual harvest time (or an assessment of maturity by an expert). Furthermore, the data analyzed using image analysis information may be used as the primary source of information, while meteorological and soil data may be considered as supplementary factors influencing the maturation rate. In addition, during extreme weather events, the contribution of meteorological data may be increased.

[0027] Here, the color of crops can be represented using RGB values ​​or values ​​converted to a specific color space (e.g., HSV). For example, if the crop is a tomato, its ripeness changes from green to yellow and finally to red, so by tracking this color change, it is possible to estimate its ripeness.

[0028] Furthermore, the shape of crops can also change during the maturation process. For example, if the crop is a tomato, a fully mature tomato will have a more uniform shape than an earlier tomato. By tracking this change in shape, it becomes possible to estimate the degree of maturity.

[0029] Furthermore, crop size also changes as they mature, and they become ready for harvest when they reach a certain size. For example, with tomatoes, by tracking changes in size information measured from images such as diameter and area, and determining when they are ready for harvest when they reach a certain size, it becomes possible to estimate maturity by tracking the shape of the crop.

[0030] In addition, meteorological data such as temperature (daily average temperature, maximum temperature, minimum temperature, etc.), humidity (daily average humidity, humidity at a specific time, etc.), sunshine duration (total sunshine duration per day), precipitation (daily precipitation, cumulative precipitation at a specific time, etc.), and CO2 concentration are used. Weather stations measuring temperature, humidity, sunshine hours, and precipitation are installed at high elevations near the farm. CO2 sensors, which measure CO2 concentration, are installed inside and outside the greenhouses and are also used to evaluate the photosynthetic efficiency of crops and optimize ventilation. These weather data are related to crop growth and are collected daily or weekly during the crop (tomato) growing season, and used as input data for a learning model.

[0031] Soil data used includes soil moisture content and soil acidity. Soil moisture is measured by a soil moisture sensor, which is installed on the ground inside the greenhouse. When the soil maintains an appropriate level of moisture, plant roots can efficiently absorb water, allowing physiological processes such as photosynthesis and transpiration to function normally and promoting plant growth. Furthermore, soil moisture dissolves nutrients and supplies them to plant roots, so appropriate moisture facilitates nutrient movement and enables efficient absorption. This, in turn, affects the growth rate of crops, and therefore, the harvest time.

[0032] Furthermore, soil acidity (pH) is a crucial factor that directly affects crop growth, nutrient absorption, and microbial activity. Different crops have different optimal pH ranges. Soil acidity is measured using soil sensors, which are installed on the ground inside greenhouses. Tomatoes, for example, are known to grow best in neutral to slightly acidic soil with a pH of 6.0 to 7.0. Thus, soil acidity influences the growth rate of crops, and therefore, their harvest time.

[0033] (Machine learning methods: The process of building a learning model) The learning model that estimates the harvest time using the harvest time determination information (color, shape, size, etc.) described above is formed through the following process. [Data collection] Images of crops (tomatoes) are taken periodically during their growth process, and data on color, shape, and size is extracted for each crop. [Data labeling] A label (output data) is placed between the extracted data and the actual harvest time (or the expert's assessment of maturity). [Feature selection and preprocessing] Select the features that contribute most to the prediction, and then normalize and transform the data. [Model training] The machine learning model is trained using the selected features and labels. Various algorithms are tested at this stage, including multilayer perception, random forests, and support vector machines. [Evaluation and adjustment of the learning model] The performance of the trained model is evaluated using the validation dataset, and parameters are adjusted as needed.

[0034] Furthermore, the development of a learning model that uses weather information to predict harvest times is carried out through the following process. [Data collection and preprocessing] Collect historical weather data and corresponding tomato maturity data (harvest date, maturity assessment, etc.). Clean and normalize the data as needed. [Feature Selection] Identify the weather factors that have the greatest impact on tomato growth and maturation, and select them as input features for the model. [Model training] A machine learning model is trained using selected weather data and corresponding maturity data. Algorithms such as regression analysis, random forests, and neural networks are used at this stage. [Evaluation and adjustment of the learning model] The performance of the trained model is evaluated using the validation dataset, and parameters are adjusted as needed.

[0035] Furthermore, the development of a learning model that uses soil information to predict harvest time is carried out through the following process. Data acquisition and preprocessing: Collect historical soil data and corresponding tomato maturity data (harvest date, maturity assessment, etc.). Clean and normalize the data as needed. [Feature Selection] Identify the soil factors that have the greatest impact on tomato growth and maturation, and select them as input features for the model. [Model training] A machine learning model is trained using selected soil data and corresponding maturity data. Algorithms such as regression analysis, random forests, and neural networks are used at this stage. [Model evaluation] Evaluate the model's performance using a validation dataset and adjust parameters as needed.

[0036] Therefore, by using any of the learning models described above, or a combination of them as appropriate, it becomes possible to predict the optimal harvest time for crops. In the above, the harvest time determination information is calculated separately from the color, shape, and size identified from images of the crop being evaluated, the harvest time calculated from weather information, and the harvest time calculated from soil information. However, it is also possible to form a learning model that estimates the harvest time by combining all of the above harvest time determination information into input information, and then estimate the optimal harvest time based on that learning model.

[0037] Using the above-described learning model, the management server 20 performs harvest control as shown in Figure 4, for example. First, the management server 20 receives estimation input data, including harvest time estimation reference information, which is periodically collected by the reference information collection unit 30 (step S101). Then, it inputs the received harvest time estimation reference information as estimation input data into the learning model 8 of the harvest time estimation device 6 to estimate the harvest time of the evaluated crop (step S102). Subsequently, for the evaluated crop whose harvest time has been estimated by the harvest time estimation device, the management server 20 sends a harvest command to the harvest robot 10 along with the location information of the evaluated crop (step S103), causing the harvest robot 10 to perform the harvesting operation.

[0038] Incidentally, precise location information of the crops to be harvested is necessary to identify the crops to be harvested, to identify the targets for data collection during their growth process, and to enable efficient target identification by harvesting robots. Furthermore, precise location information of the harvesting robots is also necessary for efficient crop harvesting and to avoid collisions between harvesting robots.

[0039] Therefore, in the case of agricultural products, the precise location information of the products can be identified by attaching RFID tags (attaching RFID tags is useful not only for determining the precise location at harvest time, but also for tracking agricultural products after harvest). In addition, location information can also be determined by image recognition technology.

[0040] Alternatively, the harvesting robot's position may be determined by mounting a GPS module on the robot and receiving GPS signals. However, there are limitations to the positioning accuracy of GPS, and it becomes difficult to confirm the position of a robot harvesting crops grown in a greenhouse, during bad weather, or when the farm is located in a mountainous area or other place with radio wave barriers. Therefore, in order to determine the precise location of the harvesting robot without using GPS, a method is effective in which the first device is placed at intervals in appropriate locations on the farm, the second device is placed on the harvesting robot 10, and the location of the harvesting robot 10 is determined by calculating the distance between each of the multiple first devices and the second device.

[0041] Figure 5 shows an overview of the robot positioning system S3 for determining the position of harvesting robots. This robot positioning system S2 comprises a first device 11 installed at intervals in the farm (inside the greenhouse), a second device 12 installed on the harvesting robots 10 located in the farm (inside the greenhouse), and the management server 20.

[0042] The first device 11 can be installed at any suitable mounting location within the farm (inside the greenhouse) that is at a height suitable for transmitting and receiving radio waves (e.g., on the upper side or ceiling of the cultivation greenhouse, or on an object installed inside the greenhouse (e.g., a light fixture or a dedicated mounting pole)). The first device 11 may be fixed to the surface of the mounting location or object by appropriate means such as screws, adhesive, or brackets, or it may be installed by embedding it in the mounting location or object. Furthermore, each first device 11 has its own location information measured and identified.

[0043] The location information of the first device 11 may be determined in advance and stored in a readable format inside the first device, or it may be determined retrospectively by some means after the system has been started. Alternatively, the location information of the first device 11 may be compiled into a database and stored in the storage unit (storage unit 33 described later) of the management server 20. Here, it is preferable to use three-dimensional location information, which may be expressed by latitude, longitude, and ellipsoidal height in the WGS84 coordinate system, or by a unique three-dimensional coordinate system set for each area indoors.

[0044] The first device 11 and the second device 12 can communicate directly with each other. Furthermore, the first device 11 can connect to the management server 20 via the communication network 14, and the second device 12 can also connect to the management server 20 via the communication network 14.

[0045] Each of the first device 11 and second device 12 has an internal clock, which can be synchronized to a reference time by a method described later. By synchronizing them, it is possible to obtain accurate positional information of multiple second devices at the same time.

[0046] Furthermore, the first device 11 can also function as the first device 11 for multiple second devices 12, and if there are multiple second devices 12, each of these multiple second devices 12 may be configured to function as the first device for multiple other second devices. In other words, if the precise location of a second device can be determined, the distance between that second device and other second devices can be calculated and used to determine the location of the other second devices. In this embodiment, we will describe a case where only the first device is used to locate the second device.

[0047] (Regarding the first device) As shown in Figure 6, the first device 11 comprises a control unit 101, an RF chip 102, and an oscillator 103, each connected by a bus. It also includes a RAM 104 and a storage unit 105, each connected to the control unit 101 by a bus.

[0048] The control unit 101 consists of a CPU and ROM, and executes programs stored in ROM to control the first device 11. The RF chip 102 is equipped with at least a clock 106, but may also be equipped with a phase detector. The RF chip 102 also has the function of processing the transmission and reception of wireless signals, and the data received by the RF chip 102 is subject to calculation processing by the control unit 101. The RAM 104 is the work area of ​​the control unit 101, and the storage unit 105 is a storage area for saving programs, data, etc.

[0049] The oscillator 103 oscillates at a predetermined frequency and outputs a signal to provide the operating timing for each part of the device. A crystal oscillator or an atomic oscillator can be used as the oscillator 103. The clock 106 keeps time using the output signal of the oscillator 103 as the source oscillation and outputs the time. The time kept by the clock 106 is controlled by the control unit 101 to be transmitted to the second device 12 via the RF chip 102. If a phase detector is also provided, it detects the phase of the carrier wave that constitutes the information received from the second device 12, and also detects the phase of the signal transmitted by the oscillator 103 of the first device 11.

[0050] The RF chip 102 is capable of sending and receiving data with other computer devices. Data received by the RF chip 102 is stored in the RAM 104 or storage unit 105 and is subject to calculation processing by the control unit 101. When the 3D position information of the first device 11 is received via the RF chip 102, it is stored in the RAM 104 or storage unit 105 and controlled by the control unit 101 to be transmitted to the second device 12 via the RF chip 102.

[0051] In this indoor positioning system S2, the installation location of the first device 11 was described as being inside a cultivation greenhouse. However, in greenhouses with good radio wave penetration, such as vinyl greenhouses, the device may be installed in a location that has a clear line of sight from the greenhouse, such as outside the cultivation greenhouse, to objects (utility poles, streetlights, transmission towers, etc.) or buildings.

[0052] If the farm is on flat land, the first device 11 only needs to identify the two-dimensional position information of the harvesting robot 10. However, if the farm is located on a slope or uneven terrain, it is preferable to identify the three-dimensional position information of the harvesting robot 10.

[0053] Furthermore, it is desirable that the first device 11 be installed comprehensively in and around the greenhouse, and that it be placed appropriately in locations where GPS signals can be easily received in order to cover the entire area inside the greenhouse.

[0054] Furthermore, the location information of the installation site of the first device 11 may be stored in its own storage unit 105 in association with identification information that can identify the first device 11, or it may be stored in the storage unit 303 of the management server 20, or it may be made available via the communication network 14 from other management servers that manage location information.

[0055] (Regarding the second device) Next, the second device 12 will be described. This second device 12 is installed on all harvesting robots 10 located within the farm, and may be integrated with the harvesting robot 10, fixed to the surface of the harvesting robot by appropriate means, or embedded in the robot. The second device 12 may be installed in a manner that it is directly attached to the robot, or it may be attached to an accessory that moves with the robot and is always accompanied by it.

[0056] As shown in Figure 7, the second device 12 comprises a control unit 201, an RF chip 202, and an oscillator 203, each connected by a bus. It also includes a RAM 204 and a storage unit 205, each connected to the control unit 201 by a bus.

[0057] The RF chip 202 includes at least a clock 26, but may also include a phase detector if necessary.

[0058] The control unit 201 is configured with a CPU and ROM, executes programs stored in the storage unit 205, and controls the second device 12. The RAM 204 is the work area of ​​the control unit 201, and the storage unit 205 is a storage area for saving programs and data. The control unit 201 performs calculation processing based on programs and data read from the RAM 204 and the storage unit 205, as well as data input from an input unit (not shown).

[0059] The RF chip 202 is capable of sending and receiving data with other computer devices. The data received by the RF chip 202 is loaded into the RAM 24 and subjected to calculation processing by the control unit 21.

[0060] The oscillator 203 oscillates at a predetermined frequency and outputs a signal to provide timing for the operation of each part of the device. A crystal oscillator or an atomic oscillator can be used as the oscillator 203. The clock 206 keeps time using the output signal of the oscillator 203 as the source oscillation and outputs the time. The time kept by the clock is controlled by the control unit 201 to be transmitted to the first device 11 via the RF chip 202. If a phase detector is also present, it detects the phase of the carrier wave that constitutes the information received from the first device 11, and also detects the phase of the signal oscillated by the oscillator 203 of the second device 12.

[0061] (Regarding the management server) Next, the management server 20 of the present invention will be described. The management server 20 can acquire location information from the second device 12.

[0062] The acquired location information is stored in the management server 20 as location information for the harvesting robot 10 (second device 12). The location information for the harvesting robot 10 (second device 12) is transmitted from the second device 12 to the management server 20, for example, by associating identification information that can identify the second device 12 with the time when the location information was identified. The management server 20 may also enable communication between the first device 11 and the second device 12 via a smart meter installed in the farm's management facilities, etc.

[0063] Figure 8 is a block diagram showing the configuration of a management server 20 according to an embodiment of the present invention. The management server 20 comprises at least a control unit 301, a RAM 302, a storage unit 303, and a communication interface 304, each connected by an internal bus. It also includes a database 35 for storing information received from the first device 11 and the second device 12. The location information of the first device 11 may also be stored in this database 35 after being compiled into a database.

[0064] The control unit 301 consists of a CPU, ROM, etc., and executes programs stored in the storage unit 303 to control the management server 20. The control unit 301 also has an internal timer for timing. The RAM 302 is the work area of ​​the control unit 301. The storage unit 303 is a storage area for saving programs and data. The control unit 301 reads programs and data from the storage unit 303 and RAM 302, and, based on information received from the first device 11 or the second device 12, executes various control processes in the control unit according to the program.

[0065] (Distance calculation process) Using the above configuration, the process for calculating the distance between the first device 11 and the second device 12 will now be described.

[0066] This distance calculation process calculates the distance between each of the first devices 11 and the second device 12 based on the propagation time Tp of the information or signal between each of the first devices 11 and the second device 12, provided that the first devices 11 and the second device 12 are within a distance range from which they can mutually send and receive information or signals.

[0067] The distance calculation process is performed at predetermined time intervals (for example, every minute) or whenever predetermined conditions are met, and the processes in steps S1 to S16 shown in Figure 9 are carried out. For convenience, here we will explain the case of calculating the distance between one first device 11 and one second device 12.

[0068] First, information or a signal is transmitted from the first device 11 to the second device 12 (step S1). The information or signal transmitted from the first device 11 to the second device 12 is not particularly limited.

[0069] In the first device 11, the time (T11) when information or a signal was transmitted in step S1 is recorded (step S2), and this recorded time is stored in the memory or storage unit 105 within the control unit 101 (step S3).

[0070] Subsequently, the second device 12 receives the information or signal from the first device 11 (step S4). The second device 12 records the time (T21) when the information or signal was received in step S4 (step S5). The recorded time (including the measured phase if one is measured) is then stored in the memory or storage unit 205 within the control unit 201 (step S6).

[0071] Next, the second device 12 transmits information or a signal to the first device 11 (step S7). The information or signal transmitted from the second device 12 to the first device 11 is not particularly limited. The second device 12 records the time (T22) when the information or signal was transmitted in step S7 (step S8). Then, the recorded time is stored in the memory or storage unit 205 of the control unit 201 (step S9).

[0072] The first device 11 receives the information or signal transmitted in step S7 (step S10). The first device 11 records the time (T12) at which it received the information or signal in step S10 (step S11). The recorded time (including the measured phase if the phase is measured) is then stored in the memory or storage unit 105 within the control unit 101 (step S12).

[0073] Subsequently, the first device 11 transmits to the second device 12 via its RF chip 102 the information stored in step S3 regarding the time (T11) when the signal was transmitted in step S1, and the information stored in step S12 regarding the time (T12) when the signal was received in step S10 (step S13). At this time, the position information of the first device 11 is also transmitted to the second device 12.

[0074] Then, the second device 12 receives information regarding the time (T11) when the first device 11 transmitted information or a signal in step S1, and information regarding the time (T12) when the first device 11 received information or a signal in step S10 (step S14).

[0075] Next, the second device 12 calculates the distance between the first device 11 and the second device 12 (step S15). This distance is calculated in the following manner.

[0076] Information regarding the time (T11) of the first device's clock is transmitted to the second device 12 via radio waves. The difference between this time and the time (T21) of the second device 12's clock when it receives this information is recorded as ΔTa on the second device side. That is, if we define the time of the first device's clock when it transmits information or a signal from the first device 11 to the second device 12 as T11, and the time of the second device 12's clock when it receives the information or signal transmitted from the first device 11 and sets time as T21, and the difference between them as ΔTa, then this ΔTa (the difference in transmission and reception times when information or a signal is transmitted from the first device 11 to the second device 12) is the difference between the time of the first device 11's clock and the second device 12's clock (time difference: T20-T10) plus the propagation time (propagation delay) Tp, resulting in the relationship shown in Equation 1. This time difference (T20-T10) would be zero if the clocks of the first device 11 and the second device 12 were synchronized, but here we assume that a time difference (T20-T10) exists (they are not synchronized). [Formula 1] ΔTa=T21-T11=(T20-T10)+Tp

[0077] To determine this propagation time Tp, the second device 12 also sends information about the time of its clock (T22) to the first device 11, and the difference between this time and the time of the first device 11's clock (T12) when the first device 11 receives it is recorded as ΔTb on the first device side. That is, if we define the time of the second device 12's clock when it transmits information or a signal from the second device 12 to the first device 11 as T22, and the time of the first device 11's clock when it receives the information or signal transmitted from the second device 12 and sets time as T12, and the difference between them is ΔTb, then this ΔTb (the difference in transmission and reception times when information or a signal is transmitted from the second device 12 to the first device 11) is the difference between the time of the first device 11's clock and the second device 12's clock (time difference: T10-T20) plus the propagation time (propagation delay) Tp, resulting in the relationship shown in Equation 2. Here too, the time difference (T10-T20) would be zero if the clocks of the first device 11 and the second device 12 were synchronized, but here we assume that a time difference (T10-T20) exists (they are not synchronized). [Formula 2] ΔTb=T12−T22=(T10−T20)+Tp

[0078] The time differences between the two clocks, (T20-T10) and (T10-T20), are added when transmitting from the first device to the second device, and the same amount of time difference is subtracted when transmitting from the second device to the first device. Therefore, to find the propagation time Tp, we add equations 1 and 2, which cancels out the terms for the time differences (T20-T10) and (T10-T20), resulting in the relationship in equation 3. [Formula 3] Tp=(ΔTa+ΔTb) / 2 =((T21-T11)+(T12-T22)) / 2

[0079] Therefore, the propagation time Tp can be calculated based only on the time read by the clock of the first device 11 and the time read by the clock of the second device 12.

[0080] Incidentally, the time difference (T10-T20) between the clock of the first device 11 and the clock of the second device 12 is given by the relationship in Equation 4, obtained by [Equation 1] - [Equation 2]. [Formula 4] (T10-T20)=(ΔTa−ΔTb) / 2

[0081] Subsequently, the distance between the first device 11 and the second device 12 is calculated by multiplying the propagation time calculated using Equation 3 by the propagation speed of the information or signal (for example, high speed) (step S15).

[0082] Then, the distance between the first device 11 and the second device 12 calculated in step S15 is stored in the memory or storage unit 205 of the control unit 201 (step S16). By executing step S16, the distance calculation process is completed.

[0083] Therefore, since equation (3) for calculating the propagation time Tp does not include a term for the time difference (time difference: T20-T10) between the clocks of the first device 11 and the second device 12, it is possible to calculate the propagation time for information or signals to propagate between the first device 11 and the second device 12, regardless of whether there is a time difference between the clocks of the first device 11 and the second device 12 (independent of the time difference (time difference: T10-T20) between the clocks of the first device 11 and the second device 12).

[0084] [Location identification process] Next, we will explain the process of determining the position of the harvesting robot 10 on which the second device 12 is installed. This position determination process determines the position of the second device 12 based on the distances between each of the multiple first devices 11 and the second device 12, which were calculated in the distance calculation process. Since the second device 12 is installed on the harvesting robot 10, this process can be said to determine the position of the harvesting robot 10.

[0085] It is desirable that this position determination process be performed immediately after the distance calculation process is completed. Furthermore, in order to determine the position of the second device 12, it is assumed that the distance calculation device has calculated the distance to each of the multiple first devices 11 for each of the second devices 12.

[0086] In other words, when obtaining three-dimensional positional information of the harvesting robot 10 (to obtain x, y, and z coordinates), the position of the second device 12 can be determined by a well-known multi-point surveying calculation method based on the distance between one second device 12 and at least four first devices 11, and the positional information of each of the four first devices 11 used to calculate this distance. Therefore, since this system can determine the three-dimensional position of the second device 12 if four or more data points representing the distance between the first device 11 and the second device 12 are available, it is advisable to appropriately distribute the first devices so that even if the second device 12 moves, the second device 12 can send and receive information or signals with at least four first devices 11. In particular, in locations where positional accuracy is required, it is necessary to pre-adjust the number and three-dimensional position of the first device 11 to achieve the required accuracy.

[0087] Figure 10 shows a flowchart of the location identification process according to an embodiment of the present invention. This location identification process can be performed on the first device 11, the second device 12, or the management server 20. When the location identification process is performed on the first device 11 or the management server 20, the distance between each of the multiple first devices 11 and the second device 12, as well as the location information of the first device 11, can be associated with the identification information of the second device 12, transmitted to the first device 11 or the management server 20, and used.

[0088] First, the position determination process requires that distance information for at least four different first devices 11 and second devices 12 be obtained at the same time or close together. Here, "close together" means that the time at which the distances of the four first devices 11 and second devices 12 used to determine the position of the second device 12 are calculated is within a range that does not hinder the capture of the movement of the second device. If the distances are not calculated at the same time or close together (for example, if the time at which the propagation time of information or signals between each of the multiple first devices 11 and the second device 12 is measured is the same time or close together), it becomes difficult to accurately determine the position of the second device 12 (harvesting robot 10) assuming that it is moving.

[0089] Therefore, first, it is determined whether four or more data points of the distance between the first device 11 and the second device 12 have been acquired within a predetermined time range (step S21).

[0090] If four or more distance data points between the first device 11 and the second device 12 are not acquired within a predetermined time range, accurate three-dimensional positional information cannot be obtained even using this positioning method. Therefore, the system waits until four or more distance data points are obtained within a predetermined time range. In contrast, if four or more distance data points between the first device 11 and the second device 12 can be acquired within a predetermined time range, three-dimensional position information can be obtained with high accuracy using this position determination method that utilizes wireless bidirectional time comparison. Then, the current position of the second device 12 is determined using the multi-point surveying calculation method described above (step S22), and display processing is performed such as displaying the current position of the second device 12 (harvesting robot 10) on a display screen (not shown) of the management server 20 (step 23). At the same time, it is preferable to store the position information of the second device 12 along with the time it was calculated in the storage unit 303 of the management server 20 for use in subsequent processing.

[0091] Therefore, if there are four or more first devices 11 that can transmit and receive information from the second device 12 installed on the harvesting robot 10 within a predetermined time range, the three-dimensional position of the second device 12 is determined by a position determination process based on the distance between each first device 11 and the second device 12 calculated by the distance calculation process, and the position information of each first device 11 used in this distance calculation. As a result, the harvesting robot 10 on which the second device 12 is installed moves, and the four first devices 11 from which distance calculation is possible are switched sequentially, making it possible to continuously capture the position of the second device 12. Thus, if there are four or more first devices 11 capable of calculating distance, it becomes possible to determine the three-dimensional position of the second device 12. By adjusting the mounting locations and heights of the first devices 11 and scattering them appropriately, it becomes possible to capture the position of the displaced second device 12 (harvesting robot 10) in real time.

[0092] In the above description, we have explained the case in which the second device 12 is installed on the harvesting robot 10 to determine its position. However, the second device 12 may also be installed on the photography drone 40, and the drone's position may be captured in real time using a similar method, thereby controlling the drone's position.

[0093] (Time synchronization) However, in the above configuration, if the collection time of maturity data obtained from each harvested crop, the data collection time of each sensor, and the time of the internal clock of the harvesting robot 10 (or the time of the internal clocks of the first device 11 or the second device 12) are not synchronized with the time of the management server 20, real-time synchronized data collection and accurate control of the harvesting robot will not be possible.

[0094] Therefore, it is necessary to accurately synchronize the timestamps of the data and to share the location information of multiple harvesting robots 10 and drones 40 in real time so that they can be positioned appropriately. In response to these requirements, this system is equipped with a means for synchronizing the time of all devices (various sensors, harvesting robots, etc.). This time synchronization is performed at predetermined intervals, and can be based on either Coordinated Universal Time (UTC) or Japan Standard Time (JST).

[0095] If each device is equipped with a GPS module, it could be synchronized with the atomic clock on a GPS satellite. Furthermore, if various sensors and harvesting robots are connected to the internet, time information can be obtained using NTP (Network Time Protocol) to synchronize the time. Alternatively, as shown in Figure 2, a dedicated time synchronization server 50 may be installed within the farmland, and all devices may obtain time information from this time synchronization server 50 to perform time synchronization.

[0096] In particular, in mountainous areas or farmland near tall buildings, it can be difficult to receive GPS satellite signals, raising concerns about the inconvenience of time synchronization. For mobile devices such as harvesting robots and drones equipped with cameras to photograph crops, being able to determine their location and synchronize their time regardless of location is crucial for collecting crop data and controlling harvesting operations.

[0097] Therefore, by calculating the time difference between the first device 11 and the second device 12 based on the bidirectional transmission and reception times of information or signals between each of the aforementioned multiple first devices 11 and the second device 12 (see Equation 4), and then synchronizing the second device 12 with the first device 11 based on that time difference, it becomes possible to accurately synchronize the time of the harvesting robot 10 and the drone 40 even if they are not connected to GPS or the internet.

[0098] As a result, by synchronizing the time of all devices, including various sensor devices, the management server 20, the harvesting robot 10, and the drone 40, the following problems can be avoided. [Inconsistency between data collection and analysis] If tomato growth data collected from sensors (temperature, humidity, soil conditions, etc.) is not precisely time-synchronized, accurate analysis of growth conditions becomes difficult. This can lead to missing important agricultural insights, such as growth rate and stress responses under specific environmental conditions. When using data collected from multiple sensors simultaneously for time-series analysis, a one-second time difference can cause data inconsistencies. This can distort the results of the data analysis and lead to erroneous decision-making. If work logs transmitted from harvesting robots and other agricultural machinery are not properly synchronized with the time, it becomes difficult to analyze work efficiency and optimize planning. Furthermore, inaccurate work records make it impossible to identify areas for improvement. [Errors in time-series analysis of sensor data] When tracking tomato growth, comparing environmental data collected at different points in time (e.g., sunlight hours, temperature) helps identify optimal conditions for each growth stage. Poor time synchronization can lead to inaccurate correlations between these data points, increasing the risk of developing flawed growth management strategies. [Failure of coordinated operation between robots] When multiple harvesting robots work in the same field, precise time synchronization is essential to avoid overlapping tasks and efficiently distribute tasks. If the timing is off, the risk of overlapping tasks and collisions between robots increases, reducing the efficiency and safety of the harvesting process. When multiple robots are operated simultaneously, even a one-second difference can cause a collision. For example, if robot A enters robot B's work area one second earlier than scheduled, robot B may still be cleaning that area. This can lead to an unexpected collision, potentially damaging the robots or crops. [Issues related to traceability and quality control] To ensure the traceability of harvested tomatoes, it is necessary to accurately record which robot harvested what, when, and where. Inaccurate time synchronization makes it difficult to trace the cause of quality or safety issues. From the perspective of quality control and traceability of harvested crops, even a one-second discrepancy in harvest records can make it difficult to identify the cause of problems when they occur. [Hindering the optimization of automated systems] In automating harvesting operations, analysis based on historical data is essential to improve work efficiency. Insufficient time synchronization reduces data reliability, making it difficult to optimize operations and plan for the future. When automated systems perform time-critical tasks (such as completing harvesting before sunset), even a one-second delay can accumulate and potentially delay the entire work schedule.

[0099] As described above, insufficient and rigorous time synchronization can lead to numerous problems in managing tomato growth and efficiently operating harvesting robots. These problems can result in decreased productivity, economic losses, and crop quality issues, making regular time synchronization essential. This precise time synchronization is crucial for understanding the growth status of crops and predicting the optimal harvest time by synchronizing data collected from various sensors (temperature, humidity, soil conditions, etc.), and is also necessary for the harvesting robot 10 to perform its scheduled tasks at the appropriate time. Furthermore, time synchronization is crucial for managing data on harvested tomatoes (harvesting time, location, ripeness, etc.). Information on which tomatoes were harvested by the harvesting robot, when, and where is necessary for quality control, traceability, and inventory management. Thus, time synchronization plays a crucial role in maintaining data integrity and improving the overall efficiency and accuracy of the system, particularly in predicting optimal harvest timing and managing harvested tomato data.

[0100] (Examples of using this system) It is advisable to use the above indoor positioning system to control the harvesting robot as shown in Figure 11. First, the time of each device (management server 20, reference information collection unit 30, harvesting robots 10 (first device 11, second device 12)) is synchronized (step S31). Then, the current position of each harvesting robot 10 is confirmed (step S32), and the status of each harvesting robot (whether it is moving, harvesting, or waiting) is confirmed (step S33). If it is determined that a harvesting robot is moving, the distance from other harvesting robots is confirmed, as well as the distance to the destination point (harvesting area or waiting area) (step S34). This distance to the destination point may be calculated by referring to the farm information database 9 (see Figures 2 and 5), which stores information such as a map of the farm and the locations where agricultural products are being grown, to determine the shortest possible route that avoids collisions with other harvesting robots. Finally, after reaching the target point, the harvesting operation begins (step S35).

[0101] Furthermore, if it is determined that the harvesting robot is in the process of harvesting, the system checks whether there are any crops that can be harvested in the harvesting area, i.e., crops that have reached the harvesting time estimated by the harvesting time estimation device 6 (step S36). If the harvesting of all crops that can be harvested in the harvesting area is completed, the system switches to a standby state (step S37). After that, it checks whether harvesting has been completed in all harvesting areas (step S38). If harvesting has been completed in all harvesting areas, the harvesting operation is terminated, and if there are other harvesting areas that require harvesting, the process from step S31 onwards is repeated. The waiting harvesting robots await instructions from the management server 20 regarding the next harvesting area (step S39), and if instructed, they move to that harvesting area (step S40).

[0102] In this type of control, if the time synchronization of each harvesting robot is not maintained, although the position of the harvesting robot itself can be accurately determined using the method described above, the position of the harvesting robot at a certain time recognized by the management server 20 will differ from the actual position of the harvesting robot 10. Therefore, if the harvesting robots are controlled while there is a time difference between the management server 20 and the harvesting robot 10, the distance between the robots cannot be accurately managed, and there is a risk of collision. However, since each device (management server, various sensors, and harvesting robots) is sequentially synchronized in step S01, the management server 20 can accurately control each of the two harvesting robots.

[0103] Therefore, with the above system, even when it is difficult to perform location confirmation and time synchronization using GPS, it is possible to collect accurate location information of crops on a farm, collect accurate location information of harvesting robots in real time, and synchronize the time of all devices, thereby enabling efficient management of crop growth and efficient operation of harvesting robots. [Explanation of Symbols]

[0104] 10 Harvesting Robots 20 Management Server 30. Standards Information Collection Department 50 Time Synchronization Server 11 1st device 12 Second device S1 Indoor Positioning System S2 Robot Positioning System

Claims

1. A harvesting robot that harvests crops in a farm, A management server configured to communicate with the harvesting robot, A means for collecting reference information to collect reference information for estimating the harvest time for each of the aforementioned crops, A learning model storage unit stores a learning model that has been trained to determine the correlation between input data including the harvest time estimation criteria information and output data including the harvest time of the crops. The harvesting robot, the management server, and the reference information collection means are provided with time synchronization means for synchronizing the time of the harvesting robot, the management server, and the reference information collection means at predetermined intervals. The aforementioned management server A harvest time estimation means that inputs estimation input data, including the harvest time estimation reference information collected by the reference information collection means, into the learning model to estimate the harvest time of the crop, A harvest command means transmits a harvest command to the harvest robot for the crops whose harvest time has arrived as estimated by the harvest time estimation means, A crop harvesting support system characterized by having the following features.

2. The crop harvesting support system according to claim 1, characterized in that the harvest time estimation criteria information is the color, shape, and size obtained from the photographed image of the crop.

3. The crop harvesting support system according to claim 2, further characterized in that at least one of weather information or soil information at the location of the farm is added as the harvest time estimation criterion information.

4. Multiple first devices are installed at intervals in the aforementioned farm and collect location information in the aforementioned farm, The harvesting robot is equipped with a second device, A control reference information calculation means calculates the propagation time of the information or signal between each of the multiple first devices and the second device, and the time difference between the first device and the second device, based on the bidirectional transmission and reception times of the information or signal between each of the multiple first devices and the second device. Distance calculation means for calculating the distance between each of the plurality of first devices and the second device based on the propagation time, A position determination means that determines the position of the second device based on the distance between each of the first and second devices calculated by the distance calculation means, and the position information of each of the first devices, Based on the aforementioned time difference, the device synchronization means synchronizes the time of the second device and the first device, It has, The time synchronization means synchronizes the multiple first devices with the management server at predetermined intervals, and the device synchronization means also synchronizes the second device's time. A crop harvesting support system according to any one of features 1 to 3.

5. A harvesting robot that harvests crops in a farm, A management server configured to communicate with the harvesting robot, A means for collecting reference information to collect reference information for estimating the harvest time for each of the aforementioned crops, A learning model storage unit stores a learning model that has been trained to determine the correlation between input data including the harvest time estimation criteria information and output data including the harvest time of the crops. A time synchronization means for synchronizing the harvesting robot, the management server, and the reference information collection means at a predetermined time interval, A crop harvesting support method used in a crop harvesting support system equipped with, A harvest time estimation step involves inputting estimation input data, including the harvest time estimation reference information collected by the reference information collection means, into the learning model of the management server to estimate the harvest time of the crop, A harvest command step in which a harvest command is transmitted to the harvest robot for the crop whose harvest time has arrived as estimated in the harvest time estimation step, A method for supporting crop harvesting, characterized by having the following features.

6. The aforementioned crop harvesting support system is Multiple first devices are installed at intervals in the aforementioned farm and collect location information in the aforementioned farm, The harvesting robot further comprises a second device installed on the harvesting robot, A control reference information calculation step that calculates the propagation time of the information or signal between each of the multiple first devices and the second device, and the time difference between the first device and the second device, based on the bidirectional transmission and reception times of the information or signal between each of the multiple first devices and the second device. A distance calculation step in which the distance between each of the plurality of first devices and the second device is calculated based on the propagation time, A position determination step in which the position of the second device is determined based on the distance between each of the first and second devices calculated in the distance calculation step, and the position information of each of the first devices, The system includes a device synchronization step that synchronizes the time of the second device and the first device based on the aforementioned time difference, The time synchronization means synchronizes the multiple first devices with the management server at predetermined intervals, and the device synchronization step also synchronizes the second device's time. The method for supporting crop harvesting according to feature 5.

7. A crop harvesting support program for causing a computer to perform each step of the crop harvesting support method according to claim 5 or claim 6.

Images