Farm information management method, farm information management system, and program

Abstract

To efficiently provide video footage related to work performed in the fields. [Solution] The farming information management method according to the embodiment includes inputting input information indicating the characteristics of the target work performed by a work device in the field into a learning model that has been trained using video information indicating other work performed at a different time or in the same field as the target work, thereby generating a generated video showing the state of the target work, and outputting output information indicating the generated generated video. The farming information management method may include inputting operational information indicating at least one of the position and operating state of the work device performing the target work as input information into the learning model.

Description

Technical Field

[0001] The present invention relates to a farming information management method, a farming information management system, and a program.

Background Art

[0002] In recent years, as the informatization of agriculture progresses, technologies for managing images (for example, still images or moving images) of operations performed in fields have been developed for use in farming activities.

[0003] For example, Patent Document 1 discloses a technique for providing information indicating the growth status of crops and images of the crops to potential purchasers based on images of crops cultivated in a plant and a growth status model. The system of Patent Document 1 generates a three-dimensional composite image including the appearance of the crops based on the image information of the imaged crops and the imaging position.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique of Patent Document 1, in order to provide images of target crops in a timely manner, it is necessary to image the images of all possible target crops. Today, images of operations performed in fields and the like may be used for various purposes such as work management, crop sales promotion, branding or recruitment using image videos of operations in fields. When providing videos using the technique of Patent Document 1, it is necessary to image and store videos of all agricultural operations or crops. In such a system, the costs required for management such as image imaging, storage, and extraction of necessary images may increase. Thus, in the conventional technique, there is a problem of low efficiency in providing videos of operations performed in fields.

[0006] In light of the above circumstances, one of the purposes of this disclosure is to efficiently provide video footage related to work performed in the field. Other purposes can be understood from the following description and the description of the embodiments. [Means for solving the problem]

[0007] The means for solving the problem are described below using the numbers and symbols used in the embodiments for carrying out the invention. These numbers and symbols are added in parentheses for reference to show an example of the correspondence between the claims and the embodiments for carrying out the invention. Therefore, the claims should not be interpreted restrictively because of the parenthetical statements.

[0008] The farming information management method according to the embodiment includes inputting input information indicating the characteristics of the target work performed by the work device (30) in the field (F) into a learning model (AI_1~AI_2) that has been learned using video information indicating other work performed at a different time or in a different field than the target work, thereby generating a generated video showing the situation during the target work, and outputting output information indicating the generated generated video.

[0009] Another embodiment of the farming information management method includes generating video information showing multiple images of the status of a target operation performed by a work device (30) in a field (F), receiving user input indicating the characteristics of the video desired by the user, inputting the input information indicating the characteristics of the video indicated by the received user input into a learning model (AI_3) to extract an extracted video from among the multiple images shown in the video information that includes an image corresponding to the characteristics desired by the user, and outputting output information indicating the extracted video.

[0010] The farming information management system (1) according to this embodiment includes a video generation unit (130) that generates generated video showing the process of target work by inputting input information indicating the characteristics of target work performed by a work device (30) in a field (F) into a learning model (AI_1~AI_2) that has been learned using video information showing other work performed at a different time or in a different field than the target work, and an information output unit (140) that outputs output information indicating the generated video generated by the video generation unit (130).

[0011] The programs (P1 to P3) according to the embodiment cause the computers (14, 24, 34) to input input information indicating the characteristics of the target work performed by the work device (30) in the field (F) into a learning model (AI_1 to AI_2) that has been learned using video information showing videos of other work performed at a different time or in a different field than the target work, thereby generating generated video showing the situation during the target work, and outputting output information showing the generated generated video. [Effects of the Invention]

[0012] According to the above embodiment, it is possible to efficiently provide video footage related to work performed in the field. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the configuration of the farming information management system according to the first embodiment. [Figure 2] This is a block diagram showing the configuration of the farming information management device according to the first embodiment. [Figure 3] This is a block diagram showing the configuration of a terminal device according to the first embodiment. [Figure 4] This is a block diagram showing the configuration of the work apparatus according to the first embodiment. [Figure 5] This is a block diagram showing the functional configuration of the farming information management system according to the first embodiment. [Figure 6] This figure shows an example of device information according to the first embodiment. [Figure 7] This figure shows an example of work information according to the first embodiment. [Figure 8] It is a flowchart showing the processes executed by the farming information management system according to the first embodiment. [Figure 9A] It is a flowchart showing the processes executed by the farming information management system according to the first embodiment. [Figure 9B] It is a flowchart showing the processes executed by the farming information management system according to the first embodiment. [Figure 10] It is a block diagram showing the functional configuration of the farming information management system according to the second embodiment. [Figure 11A] It is a flowchart showing the processes executed by the farming information management system according to the second embodiment. [Figure 11B] It is a flowchart showing the processes executed by the farming information management system according to the second embodiment.

Modes for Carrying Out the Invention

[0014] (Embodiment 1) Hereinafter, the farming information management system according to the embodiment will be described with reference to the drawings. As shown in FIG. 1, the farming information management system 1 includes a farming information management device 10, one or more terminal devices 20, and one or more working devices 30 that perform work in the field F. The farming information management device 10, one or more terminal devices 20, and the working device 30 can communicate with each other via a network NT. The network NT is, for example, the Internet. Also, the working device 30 can receive a positioning signal transmitted by the positioning satellite GP.

[0015] The working device 30 may be, for example, a vehicle incorporating an agricultural machine capable of working in the field F, such as a combine or a harvester. Alternatively, the working device 30 may be a tractor that pulls various working machines. Note that the working device 30 may be an agricultural drone that performs work in the field F. Each of the one or more working devices 30 may perform agricultural work with a predetermined working width in the direction perpendicular to the traveling direction in one or more fields F when the power source (for example, an engine or a motor) is in an operating state.

[0016] As will be described later, the working device 30 continuously measures its own position based on the positioning signals of the positioning satellites GP. Hereinafter, the position of the working device 30 measured by itself may be referred to as the positioning position, the time when the positioning position is measured may be referred to as the positioning time, and the information indicating the positioning position may be referred to as the positioning information. Further, the working device 30 may use sensors (for example, the measuring device 39 in FIG. 4) to measure the operating state (for example, speed, engine speed, or on / off state of the clutch, etc.) of the working device 30 during work and generate state information indicating the measurement result. Note that information including at least one of the positioning information and the state information may be referred to as the operating information. Further, the working device 30 may use an imaging device (for example, the imaging device 33 in FIG. 4) to capture an image of the work in progress. In the present embodiment, still images or moving images are collectively referred to as images. Further, the moving image may include sound.

[0017] The farming information management system 1 in FIG. 1 may synthesize an image of the work desired by the user using a learning model (for example, the learning models AI_1, AI_2, etc. in FIG. 5) learned using the images of the work in progress captured in past work. At this time, the farming information management system 1 in FIG. 1 generates a composite image corresponding to the actual state of the target work based on the operating information during the execution of the target work that is the subject of the synthesis process or the work information indicating the content of the work (for example, climate, type of work, type of crop), and displays it on the screen S of the terminal device 20. Therefore, the farming information management system 1 can provide the image desired by the user without capturing images of all the work and saving the image information indicating the captured images.

[0018] The configuration of the farming information management system 1 will be described. The farming information management device 10 included in the farming information management system 1 includes an input / output device 12, an arithmetic device 14, a communication device 16, and a storage device 18, as shown in FIG. 2. The farming information management device 10 is, for example, a computer having a server function. Note that the functions of the farming information management device 10 may be provided in the cloud via the network NT.

[0019] The input / output device 12 receives information for the arithmetic unit 14 to perform processing. The input / output device 12 also outputs the results of the processing performed by the arithmetic unit 14. The input / output device 12 includes various input and output devices, such as a keyboard, mouse, microphone, display, speaker, and touch panel.

[0020] The communication device 16 is connected to the network NT in a communicative manner and communicates with external devices (e.g., terminal device 20, work device 30, or external server) of the farming information management device 10 via the network NT. The communication device 16 transfers information acquired from the external device to the arithmetic unit 14. It also transfers information generated by the arithmetic unit 14 to the external device. The communication device 16 includes various interface devices with data communication functions, such as a NIC (Network Interface Card) and a USB (Universal Serial Bus).

[0021] The storage device 18 stores a program P1 and the like, which includes various instructions for the farm information management device 10 of this embodiment to perform the processing described later. The storage device 18 is used as a non-transitory tangible storage medium for storing this data and instructions. The program P1 may be provided as a computer program product recorded on a computer-readable storage medium M1. The storage medium M1 may be a portable physical medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or USB (Universal Serial Bus) memory. Alternatively, the storage medium M1 may be a storage device of an external server that stores the program P1. In this case, the program P1 may be provided as a computer program product that can be downloaded from the server.

[0022] The arithmetic unit 14 reads a program P1 containing instructions and data for executing at least a part of the processing described later from the storage device 18 and executes it. The arithmetic unit 14 includes, for example, a central processing unit (CPU).

[0023] As shown in Figure 3, the terminal device 20 included in the farming information management system 1 comprises an input / output device 22, an arithmetic unit 24, a communication device 26, and a storage device 28. The terminal device 20 is, for example, a mobile device such as a tablet or a smartphone. However, the terminal device 20 may also be a stationary personal computer or a notebook computer.

[0024] The input / output device 22 receives information for the arithmetic unit 24 to perform processing. The input / output device 22 also outputs the results of the processing performed by the arithmetic unit 24. The input / output device 22 includes various input and output devices. Furthermore, the input / output device 22 includes a touch panel or display that functions as a screen S displaying the trajectory of the work device 30. The input / output device 22 may also include a speaker for outputting sound. In cases where the terminal device 20 is a personal computer, the input / output device 22 may include a keyboard, mouse, microphone, etc.

[0025] The communication device 26 is connected to the network NT in a communicative manner and communicates with an external device (for example, the farming information management device 10) of the terminal device 20 via the network NT. The communication device 26 transfers information acquired from the external device to the computing device 24. It also transfers information generated by the computing device 24 to the external device. The communication device 26 includes various interface devices with communication functions, such as transceivers used in wireless communication such as wireless LAN (Local Area Network) and cellular networks.

[0026] The storage device 28 stores a program P2 and the like, which includes various instructions for the farm information management system 1 of this embodiment to perform the processing described later. The storage device 28 is used as a non-temporary storage medium for storing this data and instructions. The program P2 may be provided as a computer program product recorded on a computer-readable storage medium M2. The storage medium M2 may be a portable physical medium such as a CD, DVD, or USB memory. Alternatively, the storage medium M2 may be a storage device of an external server that stores the program P2. In this case, the program P2 may be provided as a computer program product that can be downloaded from the server.

[0027] The arithmetic unit 24 reads a program P2 containing instructions and data for executing at least a part of the processing described later from the storage device 28 and executes it. For example, the arithmetic unit 24 includes a central processing unit (CPU).

[0028] As shown in Figure 4, the work apparatus 30 includes an input / output device 32, an imaging device 33, a calculation device 34, a communication device 36, a storage device 38, and a measuring device 39.

[0029] The input / output device 32 receives information for the arithmetic unit 34 to perform processing. The input / output device 32 also outputs the results of the processing performed by the arithmetic unit 34. The input / output device 32 may also include various input and output devices such as speakers, touch panels, keyboards, mice, microphones, and displays.

[0030] The imaging device 33 comprises one or more camera units, including a lens and an image sensor, which are installed on the work device 30. Based on control signals supplied from the arithmetic unit 34 or the input / output device 32, the imaging device 33 captures images of the work of the work device 30, such as the ground in field F, the crops planted in field F, or a part of the work device 30 working in field F. The imaging device 33 may also include a microphone to detect sounds in the vicinity of the work device 30.

[0031] The communication device 36 is connected to the network NT in a communicative manner and communicates with external devices (e.g., the farming information management device 10) of the work device 30 via the network NT. The communication device 36 transfers information acquired from the farming information management device 10 to the computing device 34. It also transfers information generated by the computing device 34 to the farming information management device 10. The communication device 36 includes various interface devices with wireless communication capabilities, such as transceivers for cellular networks and wireless LANs.

[0032] The storage device 38 stores a program P3 containing various instructions for the farm information management system 1 of this embodiment to perform the processing described later. The storage device 38 is used as a non-temporary storage medium for storing this data and instructions. The program P3 may be provided as a computer program product recorded on a computer-readable storage medium M3. The storage medium M3 may be a portable physical medium such as a CD, DVD, or USB memory. Alternatively, the storage medium M3 may be a storage device of an external server that stores the program P3. In this case, the program P3 may be provided as a computer program product that can be downloaded from the server.

[0033] The measuring device 39 continuously measures the operating status of the work device 30 and generates operating information indicating that operating status. The measuring device 39 may also measure the current position and time of the work device 30. For example, the measuring device 39 is equipped with a GNSS (Global Navigation Satellite System) receiver and uses this receiver to receive positioning signals from positioning satellites (GP) to measure the position and time of the work device 30. Alternatively, the measuring device 39 may measure the position of the work device 30 by self-position estimation using a quantum compass.

[0034] Furthermore, the operating status measured by the measuring device 39 may include, for example, the rotational speed of the power source of the work device 30 and the wheel speed of the work device 30. In this case, the measuring device 39 includes a rotational speed sensor and a wheel speed sensor. The measuring device 39 may also include a sensor device that measures the operating status of the agricultural machinery, including the work device 30. This sensor device may measure, for example, whether the agricultural machinery is in operation or not, or, if it is in operation, physical quantities related to the work (e.g., the amount of pesticide sprayed per unit area, the angle of the arm, the depth of tilling, etc.).

[0035] The arithmetic unit 34 reads a program P3 containing instructions for executing at least a part of the processing described later from the storage device 38 and executes it. For example, the arithmetic unit 34 includes a central processing unit (CPU). The arithmetic unit 34 may also be, for example, an ECU (Electronic Control Unit) incorporated into the work device 30 and controlling various parts of the work device 30.

[0036] Next, the functions of the farm information management system 1 will be explained with reference to Figure 5. The work device 30 realizes the functions of the sampling unit 310 and output unit 320 in Figure 5 when the calculation unit 34 in Figure 4 executes program P3.

[0037] As described later, the sampling unit 310 of the work device 30 continuously measures the position and time of the work device 30 using the measuring device 39 at predetermined intervals (for example, every second) while the power source of the work device 30 is running. The sampling unit 310 also continuously measures the operating state of the work device 30 using the measuring device 39 at predetermined intervals (for example, every minute) while the power source of the work device 30 is running. The sampling unit 310 generates operating information indicating the measured position and time (i.e., the positioning position and positioning time of the work device 30) and the operating state.

[0038] Furthermore, while the power source of the work device 30 is running, the sampling unit 310 uses the imaging device 33 to generate video (still images or video) showing the work scene of the work device 30. For example, the sampling unit 310 captures the work scene at a predetermined period (e.g., every second) and generates still images based on the capture results. Alternatively, the sampling unit 310 captures video at a predetermined frame rate (e.g., 30fps). As an example, the sampling unit 320 captures frame images at a period corresponding to the frame rate and generates a video including those frame images. The sampling unit 310 may also measure sound using a microphone and generate a video including that sound.

[0039] The output unit 320 of the work device 30 transmits the operation information and video information generated by the sampling unit 310 to the farming information management device 10 using the communication device 36. The operation information output by the output unit 320 may include information that identifies the work device 30 (for example, an identifier).

[0040] The farm information management device 10 of the farm information management system 1 realizes the functions of the information acquisition unit 110, selection unit 120, image generation unit 130, information output unit 140, learning unit 150, and information storage unit 160 shown in Figure 5, when the calculation unit 14 in Figure 2 executes program P1.

[0041] The information storage unit 160 stores information necessary for the farming information management device 10 to perform processing. For example, before starting the processing described later, the information storage unit 160 pre-stores device information D1, work information D2, and one or more learning models (e.g., learning model AI_1, learning model AI_2, ...). Note that if one or more learning model AIs are not distinguished from each other, they may simply be referred to as learning model AI. The information storage unit 160 may also store setting information such as thresholds used in the processing described later.

[0042] Device information D1 may store, for example as shown in Figure 6, information identifying each work device 30, information indicating the group to which each work device 30 belongs, and information indicating the learning model corresponding to each work device 30, in association with each other. In the example in Figure 6, device information D1 stores a "device ID" indicating the identifier of the work device 30, a "group" indicating the group to which the work device 30 belongs, and a "learning model" indicating the learning model AI corresponding to the work device 30. For example, each group indicated by device information D1 may correspond to a user (individual or organization) that owns one or more work devices 30. Alternatively, each group may correspond to users who have mutually authorized each other to use the images captured by their own work devices 30 as learning information for the learning model AI. In this embodiment, device information D1 has different learning model AIs registered as corresponding models for each of the multiple groups.

[0043] Work information D2 stores information indicating the content of work performed or planned to be performed by one or more work devices 30 that are processed by the farming information management system 1, as shown in Figure 7, for example. In the example in Figure 7, work information D2 stores information that identifies the work ("work ID"), information that identifies the work device 30 that performs the work ("device ID"), information that indicates the period of the work ("work date"), information that indicates the field F where the work is performed ("field"), information that indicates the crop ("crop"), information that indicates the type of work ("work type"), and information that indicates the weather at the time of the work ("weather").

[0044] In Figure 7, "Work Date" indicates that the work will be performed within a specified time on that date (for example, from 8:00 to 17:00). "Field" stores information indicating the geographical extent of field F where the work will be performed (for example, if field F is a polygon, information indicating the latitude and longitude of each vertex). "Crop" indicates the type of crop cultivated in field F where the work will be performed. "Work Type" indicates which of several types of work can be performed in field F the work in question is performed. "Weather" stores information indicating which of several types of weather conditions, such as sunny, cloudy, and rainy, was present in field F at the time of the work. If the weather changes during the work, "Weather" may include information indicating multiple types of weather and the duration of that weather. For future work, "Weather" may store information indicating NULL, or information indicating the weather for the work period estimated by the weather forecast. Note that work information D2 may be information entered by the user using the input / output device 12. Furthermore, the information acquisition unit 110 may acquire some of the work information D2 (for example, "weather" information) from an external server (for example, a server that provides weather information).

[0045] The information storage unit 160 in Figure 5 stores one or more learning model AIs (e.g., learning model AI_1, learning model AI_2), each of which is a model realized using, for example, a neural network or deep learning. In this embodiment, the learning model AI is a deep generative model that has been trained to output video corresponding to instruction information (e.g., a prompt) that indicates the content of a task. Each of the one or more learning model AIs may have been trained by the learning unit 150 using at least partially different training data, as will be described later.

[0046] The information acquisition unit 110 acquires information necessary for the processing described later from the information storage unit 160 or an external device (for example, the terminal device 20, the work device 30, or an external server). The information acquisition unit 110 may provide the acquired information to the selection unit 120, the image generation unit 130, the information output unit 140, the learning unit 150, and the information storage unit 160.

[0047] As will be described later, the selection unit 120 selects a learning model AI from among the one or more learning model AIs stored in the information storage unit 160 that corresponds to the work device 30 that performed the task to be processed.

[0048] As will be described later, the video generation unit 130 generates a video showing the process of the target work using the learning model AI selected by the selection unit 120.

[0049] As will be described later, the information output unit 140 outputs output information indicating the generated video generated by the video generation unit 130.

[0050] As will be described later, the learning unit 150 trains a learning model AI using training data.

[0051] The terminal device 20 realizes the functions of the display unit 210 and the reception output unit 220 shown in Figure 5 when the arithmetic unit 24 shown in Figure 3 executes the program P2.

[0052] As will be described later, the display unit 210 displays video footage showing the work being performed by the work device 30 in a manner recognizable to the user, based on the output information output by the farming information management device 10.

[0053] The reception output unit 220 uses the input / output device 22 to receive user input indicating the content desired by the user, such as the type of work shown in the generated video. The reception output unit 220 also outputs input information indicating the received user input to the farming information management device 10.

[0054] Next, we will explain the process of training the learning model AI that the farm information management system 1 executes. When the power source of the work device 30 is started to perform work, the farm information management system 1 starts the process shown in Figure 8.

[0055] In the process shown in Figure 8, first, in step S1002, the sampling unit 310 of the work device 30 measures the operating status of the work device 30. For example, the sampling unit 310 measures the current position and time of the work device 30 using the measuring device 39. The sampling unit 310 also measures the current operating status of the work device 30 using the measuring device 39. As an example, the sampling unit 310 measures the wheel speed of the work device 30 and whether the agricultural machinery (e.g., a combine harvester or pesticide sprayer) of the work device 30 is in operation or not. If the agricultural machinery is in operation, the sampling unit 310 also measures the amount of work done by the agricultural machinery (e.g., the amount of pesticide sprayed or the rotation speed of the threshing drum of a harvester). Note that if the current time differs from the sampling time determined according to the sampling period of the measuring device 39, the sampling unit 310 may skip step S1002.

[0056] Next, in step S1004, the sampling unit 310 captures images showing the work in progress. For example, the sampling unit 310 uses the imaging device 33 to capture images of one or more of the following: the field F during work, the crops being cultivated in field F, a part of the work equipment 30 during work, and the operator of the work equipment 30. The sampling unit 310 may skip step S1004 if the current time differs from the imaging time determined according to the imaging cycle of the imaging device 33. The sampling unit 310 captures still images showing the scenery being captured at a predetermined imaging cycle (e.g., 1 second). Alternatively, the sampling unit 310 may use the imaging device 33 to capture video frame images showing the scenery of the work equipment 30 during work at a predetermined imaging cycle (e.g., 1 / 30 second). The sampling unit 310 may also record audio at the time of imaging using the microphone of the imaging device 33.

[0057] Next, in step S1006, the sampling unit 310 determines whether the currently running operation has finished. For example, the sampling unit 310 determines that the operation has not finished when the power source of the work device 30 is still running (step S1006; NO). In this case, the process returns to step S1002, and the processes of steps S1002 to S1006 are executed at the next sampling time or imaging time. On the other hand, the sampling unit 310 may determine that the operation has finished when the power source transitions from the running state to the non-running state (step S1006; YES). In this case, step S1008 is executed next. If the sampling period in step S1002 and the imaging period in step S1004 are different, the sampling unit 310 may repeat the loop of steps S1002 to S1006 with the shorter of the two periods.

[0058] Next, in step S1008, the output unit 320 outputs operation information and video information corresponding to the current operation. For example, the output unit 320 uses the communication device 36 to output operation information indicating the operating status of the work device 30 measured in step S1002 at each sampling time, and video information including images (for example, multiple still images or video frames) captured in step S1004 at each imaging time, to the terminal device 20. The operation information output by the output unit 320 may include an identifier for the work device 30. The video information may include information indicating the imaging time of each image (for example, multiple still images or video frames) included in the video information.

[0059] Next, in step S1010, the information acquisition unit 110 of the farming information management device 10 acquires the operation information and video information output in step S1008.

[0060] Next, in step S1012, the information acquisition unit 110 acquires device information D1 and work information D2 from the information storage unit 160. For example, based on the operation information, the information acquisition unit 110 extracts information (also called content information) indicating the content of the work performed this time from the work information D2. For example, the information acquisition unit 110 extracts from the work information D2 a row in which the positioning time indicated by the operation information is included in the work period indicated by "date", and the positioning location of the work device 30 included in the operation information is included in the geographical range indicated by "field", as content information indicating the content of the work performed this time. If there is no information regarding the current work in the work information D2, the information acquisition unit 110 may use the input / output device 32 to receive user input indicating content information for the current work and generate content information based on said user input.

[0061] Next, in step S1014, the selection unit 120 selects a learning model AI corresponding to the work device 30 from among the multiple learning model AIs stored in the information storage unit 160. For example, the selection unit 120 determines which of the multiple work devices 30 indicated by the device information D1 matches the work device 30 indicated by the operation information acquired in step S1010. Then, the selection unit 120 selects the learning model AI indicated in the "Learning Model" of the row corresponding to the determined work device 30 in the device information D1 as the learning model AI corresponding to the work device 30.

[0062] Next, in step S1016, the learning unit 150 learns a learning model AI. For example, the learning unit 150 trains the learning model AI selected in step S1014 using training data. The training data includes, for example, video information acquired in step S1012. Alternatively, the learning unit 150 may train the learning model AI selected in step S1014 using information obtained by adding operational information acquired in step S1012 to the video information as training data. In this case, the target learning model AI may learn that when work is performed with the positioning location changes indicated by the operational information, an image like that shown by the video information can be obtained. Alternatively, the target learning model AI may learn that when work is performed with the operational state indicated by the state information included in the operational information, an image like that shown by the video information can be obtained.

[0063] Furthermore, when the learning unit 150 trains the learning model AI selected in step S1014, it may use video information with content information of the work indicated by the work information D2 acquired in step S1012 added to the operational information, in addition to or instead of operational information. In this case, the target learning model AI may learn that when the work indicated by the content information is performed, the video shown in the video information will be obtained. Note that the operational information of the work device 30 during operation and the content information indicating the content of the work may be referred to as manner information indicating the manner of the work.

[0064] When step S1016 is completed, the process shown in Figure 8 is finished. Through the process shown in Figure 8, the farming information management system 1 can learn a corresponding learning model AI using video information captured of the work performed by a work device 30 owned by the same user or by users who have mutually granted permission to use the video. In addition, in the process shown in Figure 8, the learning unit 150 of the farming information management system 1 may learn the learning model AI using video and behavioral information captured during a different period or during other work performed in a different field F than the work targeted in the processes shown in Figures 9A and 9B.

[0065] The farm information management system 1 may train its learning model AI by performing the process shown in Figure 8 for each of the multiple tasks performed on multiple fields F.

[0066] After the learning model AI has been trained in advance by the process shown in Figure 8, the farming information management system 1 starts the processes shown in Figures 9A and 9B when the power source of the work device 30 is activated in order to perform the target work.

[0067] In the process shown in Figure 9A, first, in step S2002, the sampling unit 310 of the work device 30 measures the operating state of the work device 30, similar to step S1002 in Figure 8.

[0068] Next, in step S2004 in Figure 9A, the sampling unit 310 determines whether the work is finished or not, similar to step S1006 in Figure 8. If it determines that the work is not finished (step S2004; NO), the processes from steps S2002 to S2004 are repeated until the work is finished. On the other hand, if it determines that the work is finished (step S2004; YES), then step S2006 is executed.

[0069] Next, in step S2006 of Figure 9A, the output unit 320 outputs operational information. For example, the output unit 320 uses the communication device 36 to output operational information to the terminal device 20, indicating the operational status of the work device 30 measured in step S2002 at each sampling time. The output unit 320 may also output the operational information with an identifier indicating the work device 30 that performed the target work.

[0070] Next, in step S2008, the information acquisition unit 110 of the farming information management device 10 acquires the operational information output in step S2006.

[0071] Next, in step S2010, the reception output unit 220 of the terminal device 20 receives user input indicating the characteristics of the target task that is the subject of the process for generating the generated video. For example, the reception output unit 220 displays information on the screen S of the input / output device 22 of the terminal device 20 prompting the user to input a prompt to generate the generated video for the learning model AI (for example, a string of characters such as "Please enter the characteristics of the task for which you want to generate a video"). Then, when the user inputs a prompt using the input / output device 220, the reception output unit 220 receives the prompt as user input indicating the content of the generated video.

[0072] The prompt received in step S2010 may contain information indicating the characteristics of the work for which the user wishes to generate a composite video. For example, this prompt may include at least some of the operational information obtained in step S2008 and a specification to generate a video of the work performed using that operational information. As an example, this prompt may be a string of characters such as, "Generate a video of the work. The operational status of the work equipment during the work is as indicated in the operational information (file name). However, this is limited to work performed between 12:30 and 12:35."

[0073] Next, in step S2012, the reception output unit 220 outputs instruction information. For example, the reception output unit 220 uses the communication device 26 to output instruction information to the farming information management device 10, indicating the content of the prompt received in step S2010. The instruction information may include information indicating the characteristics of the target work for which the generated video is to be produced, as indicated by the user operation received in step S2010.

[0074] Next, in step S2014, the information acquisition unit 110 of the farming information management device 10 acquires the instruction information output in step S2012.

[0075] Next, in step S2016 of Figure 9B, the selection unit 120 selects a learning model AI corresponding to the work device 30 that performed the target operation. For example, similar to step S1014 of Figure 8, the selection unit 120 may select a corresponding learning model AI from among the one or more learning model AIs stored in the information storage unit 160 based on the device information D1 and the identifier of the work device 30 indicated by the operation information acquired in step S2008 of Figure 9A.

[0076] Next, in step S2018 of Figure 9B, the video generation unit 130 generates a generated video using the learned model AI. For example, the video generation unit 130 generates input information that indicates the characteristics of the target work desired by the user, as indicated by the instruction information acquired in step S2012 of Figure 9A. Then, the video generation unit 130 inputs this input information to the learned model AI selected in step S2016 of Figure 9B to generate a generated video that reproduces the appearance of the target work. Note that if content information of the target work is stored in the work information D2, the video generation unit 130 may add this content information to the input information.

[0077] Next, in step S2020, the information output unit 140 outputs output information. For example, the information output unit 140 uses the communication device 16 to output output information indicating the generated video generated in step S2018 to the terminal device 20.

[0078] Next, in step S2022, the display unit 210 of the terminal device 20 displays the generated video. For example, based on the output information output in step S2020, the display unit 210 displays the generated video generated in step S2018 on the screen S of the input / output device 22. When step S2022 is completed, the processes shown in Figures 9A and 9B are finished.

[0079] As described above, the farming information management system 1 of this embodiment can generate generated video showing the state of a target operation using a learned model AI, based on input information indicating the characteristics of the target operation for which the video is generated. Therefore, the farming information management system 1 can generate a variety of videos related to operations without the need to capture video for all operations and manage the information of the captured videos. Furthermore, it is possible to select and use a learned model AI that has been trained using only images captured by the imaging device 33 of the operation devices 30 belonging to the same group. For this reason, the farming information management system 1 can provide generated video generated using a learned model AI trained using licensed videos.

[0080] (Second Embodiment) Next, the farming information management system 2 according to the second embodiment will be described. The farming information management system 2 differs from the farming information management system 1 according to the first embodiment in that it extracts images corresponding to the features indicated by the input information from the images captured during the target work using a learning model. Note that the farming information management system 2 of this embodiment may be omitted from the explanation if it is common with the farming information management system 1 of the first embodiment.

[0081] As shown in Figure 10, the farm information management system 2 includes a farm information management device 10a, a terminal device 20, and a work device 30. The farm information management device 10a of the farm information management system 2 has the same configuration as the farm information management device 10 shown in Figure 2. Furthermore, the terminal device 20 and the work device 30 of the farm information management system 2 have the same configuration as the terminal device 20 in Figure 3 and the work device 30 in Figure 4.

[0082] Functionally, the work device 30 of the farm information management system 2 has a sampling unit 310 and an output unit 320, similar to the work device 30 in Figure 5. The functions of the sampling unit 310 and the output unit 320 are the same as the functional units of the same name in the work device 30 of Embodiment 1.

[0083] Functionally, the farming information management device 10a comprises an information acquisition unit 110, an extraction unit 132, an information output unit 140, and an information storage unit 160, as shown in Figure 10.

[0084] The information storage unit 160 stores information necessary for the farming information management device 10 to perform processing. For example, the information storage unit 160 pre-stores the learning model AI_3 before starting the processing described later.

[0085] The learning model AI_3 stored in the information storage unit 160 is a model implemented using, for example, technologies such as neural networks and deep learning. In this embodiment, the learning model AI_3 is a learning model that has been trained to extract video corresponding to the instruction information when instruction information (e.g., a prompt) indicating a desired feature is input from the video being worked on. The learning model AI_3 may be a model trained using any known method. As an example, the learning model AI_3 may be a large-scale language model trained using video information and text information that can be collected on the internet.

[0086] The information acquisition unit 110 acquires information necessary for the processing described later from the information storage unit 160 or an external device (for example, the terminal device 20, the work device 30, or an external server). The information acquisition unit 110 may provide the acquired information to the extraction unit 132, the information output unit 140, and the information storage unit 160.

[0087] As will be described later, the extraction unit 132 uses the learning model AI_3 to extract images (also called extracted images) that correspond to the features desired by the user from the images captured by the work device 30 during the target work.

[0088] The information output unit 140 outputs an output image showing the extracted video extracted by the extraction unit 132 to the terminal device 20, for example, as will be described later.

[0089] The terminal device 20 of the farming information management system 2 has, functionally, the same configuration as the terminal device 20 in Figure 5, a display unit 210 and a reception output unit 220. The functions of the display unit 210 and the reception output unit 220 are the same as those of the correspondingly named functions of the terminal device 20 in Embodiment 1.

[0090] Next, we will explain the process of extracting video using the learning model AI_3, which is executed by the farm information management system 2. When the power source of the work device 30 is started to perform the target work, the farm information management system 2 starts the process shown in Figures 11A and 11B.

[0091] In the process shown in Figure 11A, first, in step S3002, the sampling unit 310 of the work device 30 captures an image showing the work being performed by the work device 30, similar to step S1004 in Figure 8.

[0092] Next, in step S3004 in Figure 11A, the sampling unit 310 determines whether the work is finished or not, similar to step S1006 in Figure 8. If the sampling unit 310 determines that the work is not finished (step S3004; NO), the processes from steps S3002 to S3004 are repeated until the work is finished. On the other hand, if the sampling unit 310 determines that the work is finished (step S3004; YES), then step S3006 is executed.

[0093] Next, in step S3006 of Figure 11A, the output unit 320 outputs video information. For example, the output unit 320 uses the communication device 36 to output video information indicating the video captured in step S3002 at each imaging time to the farming information management device 10a.

[0094] Next, in step S3008, the information acquisition unit 110 of the farming information management device 10a acquires the video information output in step S3006.

[0095] Next, in step S3010, the reception output unit 220 of the terminal device 20 receives user input indicating the characteristics of the video to be extracted. For example, the reception output unit 220 displays information on the screen S of the input / output device 22 of the terminal device 20 prompting the user to enter a prompt to allow the learning model AI_3 to extract the video (for example, a string such as "Please enter the characteristics of the video you want to extract"). When the user enters a prompt using the input / output device 22, the reception output unit 220 accepts the prompt as user input indicating the content of the video to be extracted.

[0096] For example, a prompt received by the reception output unit 220 may include information specifying whether the image to be extracted is a still image or a video, how many still images to extract, or how long the video should be if it is a video. Furthermore, a prompt received by the reception output unit 220 may include information indicating the specific characteristics of the image to be extracted. For instance, this prompt might instruct the system to extract images in which the worker performing the target task is smiling. Such a prompt might include a string like, "Extract five videos in which the worker is smiling. Each video should be approximately 5 seconds long." Alternatively, a prompt received by the reception output unit 220 might instruct the system to extract images in which the crops being cultivated in the target task are prominently featured (for example, a string like, "Extract three photos in which tomatoes are prominently featured. The photos should show tomatoes that are as red and free of blemishes as possible, and each photo should be taken at least one minute apart").

[0097] Next, in step S3012, the reception output unit 220 outputs instruction information in the same manner as in step S2012 in Figure 9A.

[0098] Next, in step S3014, the information acquisition unit 110 of the farming information management device 10a acquires the instruction information output in step S3012.

[0099] Next, in step S3016 of Figure 11B, the extraction unit 132 extracts images using the learning model AI_3. For example, the extraction unit 132 generates input information indicating the video features desired by the user, as shown by the instruction information acquired in step S3014 of Figure 11A. Then, the extraction unit 132 inputs the video information acquired in step S3008 and the input information to the learning model AI_3 to extract images corresponding to the features desired by the user.

[0100] Next, in step S3018 of Figure 11B, the information output unit 140 outputs output information. For example, the information output unit 140 uses the communication device 16 to output output information indicating the extracted video extracted in step S3016 to the terminal device 20.

[0101] Next, in step S3020, the display unit 210 of the terminal device 20 displays the extracted video. For example, based on the output information output in step S3018, the display unit 210 displays the extracted video extracted in step S3016 on the screen S of the input / output device 22. When step S3020 is completed, the processes shown in Figures 11A and 11B are finished.

[0102] As described above, the farming information management system 2 of this embodiment can extract images with features desired by the user from among the images captured during the target work. Therefore, the farming information management system 2 can efficiently provide the necessary images without requiring the user to review all of them.

[0103] (modified version) The configuration described in the embodiment is just one example, and the configuration can be changed as long as it does not impair the functionality.

[0104] For example, one or more of the functional units of the farming information management device 10 or the farming information management device 10a (for example, the information acquisition unit 110, the selection unit 120, the image generation unit 130, the extraction unit 132, the information output unit 140, the learning unit 150, and the information storage unit 160) may be implemented in a distributed manner by two or more computers. Alternatively, for example, one or more of the functions of the farming information management device 10 or the farming information management device 10a may be provided by the terminal device 20.

[0105] Furthermore, if the operational information and video information generated by the work device 30 can be obtained from an external device (for example, an external server device that collects information about the work device 30), the information acquisition unit 110 may acquire the operational information from this external device instead of the work device 30. In this case, the farming information management system 1 or the farming information management system 2 does not need to include the work device 30.

[0106] Furthermore, the farming information management device 10 or farming information management device 10a may include a functional unit corresponding to the display unit 210 and the receiving output unit 220 of the terminal device 20. In this case, the functional unit corresponding to the display unit 210 may display a screen showing the generated image or extracted image on the screen of the display device of the input / output device 12 of the farming information management device 10 in step S1012 of Figure 8. In this case, the farming information management system 1 or farming information management system 2 does not need to include the terminal device 20.

[0107] Furthermore, the farming information management system 1 may perform processes different from those described above. For example, the farming information management system 1 may omit some of the processes described in Embodiments 1 and 2. As an example, if there are no issues with the rights to the video to be learned, step S2016 in Figure 9B may be omitted. In this case, the information storage unit 160 may store only one learning model AI.

[0108] Furthermore, if the generated video produced in step S2018 of Figure 9B includes the appearance of a worker, the video generation unit 130 may set the appearance of the worker to be that of a specific person model. In this case, the learning unit 150 may train the learning model AI using learning data that includes videos of the appearances of multiple person models and the names of those person models. In addition, the reception output unit 220 may receive a prompt in step S2010 of Figure 9A to specify the video information of a person to be used for the person model as user input. In this case, the instruction information output in step S2012 includes the video information of the person itself, or information indicating the location of the video information. Also, in step S2018, the video generation unit 130 may input input information including the video information of the person to the learning model AI.

[0109] Furthermore, the farming information management system of the present invention may also include a system that has both the functions of Embodiment 1 and Embodiment 2. For example, the farming information management system 1 of Embodiment 1 may have an extraction unit 132 that extracts images having features specified by the user from the images generated by the image generation unit 130. In this case, the farming information management system 1 may execute the processes from steps S3010 in Figure 11A to steps S3020 in Figure 11B after step S2018 in Figure 9B.

[0110] Furthermore, the farm information management device 10a of the farm information management system 2 of Embodiment 2 may have a learning unit 150 that learns a learning model AI_3 using learning data that includes characteristic images of farm work and text data indicating the characteristics of said farm work. In this case, the information storage unit 160 of the farm information management device 10a may store multiple learning model AIs according to the group to which the work device 30 that captured the learning images belongs. The farm information management device 10a may also have a selection unit 120 that, during learning, selects a learning model AI to be learned according to the group to which the work device 30 that captured the images belongs. In addition, the selection unit 120 of the farm information management system 2 of this modified example may perform the same processing as in step S2016 of Figure 9B before step S3016 of Figure 11B, and select a learning model AI corresponding to the work device 30 that performed the target work as the learning model AI to be used in the extraction work in step S3016.

[0111] Furthermore, the generated video of the target work performed in field F, which is generated by the farming information management system 1, may not be limited to work already performed in field F, but may also be a video showing work that may be performed in the future, or even a hypothetical work that is not planned. In this case, the processing in steps S2002 to S2008 in Figure 9A may be omitted. In this modified example, in step S2010, the reception output unit 220 may receive user input indicating the characteristics of the target work. In this case, in step S2010, the reception output unit 220 may receive a prompt that includes content indicating one or more of the following as the nature of the target work: the weather during the execution of the work, the type of target work, the type of crop cultivated in field F where the target work should be performed, and the geographical range of field F. As an example, this prompt may include a string such as, "Please create a video of wheat harvesting work, assuming it was performed in field F on September 12th. Assume the weather was sunny." Furthermore, the input output unit 220 may also accept a prompt that indicates a modification to some of the information indicated by the work information D2 regarding the actual work performed, such as a string of characters like, "Assuming the weather was sunny, generate a video of the wheat harvest work that took place on September 12th."

[0112] The embodiments and modifications described above are merely examples, and the configurations described in each embodiment may be arbitrarily changed and / or combined as long as they do not impair the function. Furthermore, some of the functions described in the embodiments may be omitted if the necessary functions can be achieved.

[0113] (Note) The farming information management method, farming information management system, and program described in each embodiment can be described as follows.

[0114] The farming information management method relating to the first aspect is: By inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in a different field than the target operation, a generated video showing the operation in progress is produced. Outputting output information showing the generated video, Includes.

[0115] The farming information management method relating to the second aspect is the farming information management method relating to the first aspect, The generation of the aforementioned video includes inputting operational information, which indicates at least one of the position and operating state of the work device performing the aforementioned work, into the learning model as input information.

[0116] The farming information management method relating to the third aspect is the farming information management method relating to the second aspect, The method further includes measuring at least one of the position and operating status of the work device that performs the aforementioned work, and generating the aforementioned operating information.

[0117] The farming information management method relating to the fourth aspect is the farming information management method relating to the second aspect, The system measures at least one of the position and operating status of the work device that performs the other tasks, and generates operating information for the work device that performs the other tasks. The process involves capturing video information of the work of the work apparatus performing the other work, and generating video information showing the video of the other work. The learning model is further trained using learning data that includes the video information of the other work and the operational information of the other work.

[0118] The farming information management method relating to the fifth aspect is a farming information management method relating to any of the first to fourth aspects, The input information includes one or more of the following: the weather during the execution of the target operation, the type of the target operation, and the type of crop cultivated in the field where the target operation was performed.

[0119] The farming information management method relating to the sixth aspect is a farming information management method relating to the fifth aspect, The learning model is further trained using learning data that includes the video information of the other work and characteristic information indicating one or more of the following: the weather during the execution of the target work, the type of the target work, and the type of crop cultivated in the field where the target work was performed.

[0120] The farming information management method relating to the seventh aspect is a farming information management method relating to any of the first to sixth aspects, The method further includes training the learning model using, as training data, video information of work performed by work devices belonging to the same group as the work device that performed the target work, from among the video information of work performed by multiple work devices.

[0121] The farming information management method relating to the eighth aspect is a farming information management method relating to the seventh aspect, The further includes selecting, from among multiple learning models, a learning model that has been trained using video information and behavioral information of work performed by a work device belonging to the same group as the work device that performed the target work, as the learning model to be used to generate the video of the work.

[0122] The farming information management method relating to the ninth aspect is the farming information management method relating to the eighth aspect, The aforementioned plurality of learning models include two or more learning models that have been trained using video information and behavioral information of work performed by the work devices belonging to different groups.

[0123] The farming information management method relating to the 10th aspect is a farming information management method relating to any of the 1st to 9th aspects, Generating the generated video includes inputting input information into the learning model that specifies setting the appearance of a person that may be included in the generated video to the appearance of a specific person model.

[0124] The farming information management method relating to the 11th aspect is: To generate video information showing multiple images of the target operation performed by the work equipment in the field, The system accepts user input indicating the desired characteristics of the video, By inputting input information indicating the characteristics of the video shown by the received user input into the learning model, the system extracts extracted video from among multiple images shown by the video information that includes images corresponding to the characteristics desired by the user. Outputting output information that shows the extracted video, Includes.

[0125] The farming information management method relating to the 12th aspect is a farming information management method relating to the 11th aspect, The extracted video includes the appearance of the worker performing the target task, with a smile on their face.

[0126] The farming information management system relating to the 13th aspect is: A video generation unit generates generated video showing the process of the target operation by inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in the same field as the target operation. An information output unit that outputs output information indicating the generated video generated by the video generation unit, Includes.

[0127] The farming information management system according to the 14th aspect is a farming information management system according to the 13th aspect, The system further includes a display unit that displays the generated video based on the output information output by the information output unit.

[0128] The program relating to the aspect of the first fourteenth is: On the computer, By inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in a different field than the target operation, a generated video showing the operation in progress is produced. Outputting the generated video, Make it run. [Explanation of Symbols]

[0129] 1, 2... Farm Management Information System 10, 10a... Farming information management device 12… Input / Output Devices 14...Arithmetic device 16…Communication equipment 18...Storage device 110…Information acquisition department 120...Selection section 130...Video generation unit 132...Extraction part 140... Information output unit 150…Learning Department 160...Information storage unit 20…Terminal device 22… Input / Output Devices 24...Arithmetic device 26…Communication equipment 28…Storage device 210...Reception output section 220…Display section 30…Working equipment 32… Input / Output Devices 33…Imaging device 34...Arithmetic device 36…Communication equipment 38…Storage device 39... Measuring device 310...Sampling section 320...Output section F...field A1~A4…area NT... Network M1, M2, M3…Storage medium P1, P2, P3... Program D1…Device information D2...Work Information AI_1~AI_3...Learning Models GP... Positioning satellite S...Screen

Claims

1. By inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in a different field than the target operation, a generated video showing the operation in progress is produced. Outputting output information showing the generated video, including, Methods for managing farming information.

2. The generation of the aforementioned video includes inputting operational information indicating at least one of the position and operating state of the work device performing the aforementioned work as input information to the learning model. The farming information management method according to claim 1.

3. The process further includes measuring at least one of the position and operating status of the work device that performs the aforementioned target work, and generating the aforementioned operating information. The farming information management method according to claim 2.

4. The system measures at least one of the location and operating status of a work device that performs tasks other than the aforementioned target task, and generates operating information for the work device that performs the other tasks. The process involves capturing video information of the work of the work apparatus performing the other work, and generating video information showing the state of the other work. The learning model is further trained using learning data that includes the video information of the other work and the operational information of the other work. The farming information management method according to claim 2.

5. The input information includes one or more of the following: the weather during the execution of the target operation, the type of the target operation, and the type of crop being cultivated in the field where the target operation was performed. The farming information management method according to claim 1.

6. The learning model is further trained using learning data that includes the video information of the other work, and characteristic information indicating one or more of the following: the weather during the execution of the target work, the type of the target work, and the type of crop cultivated in the field where the target work was performed. The farming information management method according to claim 5.

7. The process further includes training the learning model using, as training data, video information of work performed by work devices belonging to the same group as the work device that performed the target work, from among the video information of work performed by multiple work devices. The farming information management method according to claim 1.

8. This further includes selecting, from among multiple learning models, a learning model trained using video information and behavioral information of work performed by a work device belonging to the same group as the work device that performed the target work, as the learning model to be used to generate the video of the work. The farming information management method according to claim 7.

9. The plurality of learning models include two or more learning models that have been trained using video information and behavioral information of work performed by the work devices belonging to different groups. The farming information management method according to claim 8.

10. The process of generating the generated video includes inputting input information into the learning model that specifies that the appearance of a person that may be included in the generated video should be set to the appearance of a specific person model. The farming information management method according to claim 1.

11. To generate video information showing multiple images of the target operation performed by the work equipment in the field, The system accepts user input indicating the desired characteristics of the video, By inputting input information indicating the characteristics of the video shown by the received user input into the learning model, the system extracts extracted video from among multiple images shown by the video information that includes images corresponding to the characteristics desired by the user. Outputting output information that shows the extracted video, including, Methods for managing farming information.

12. The extracted video includes the appearance of the worker smiling during the aforementioned task. The farming information management method according to claim 11.

13. A video generation unit generates generated video showing the process of the target operation by inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in the same field as the target operation. An information output unit that outputs output information indicating the generated video generated by the video generation unit, including, Agricultural information management system.

14. The system further includes a display unit that displays the generated video based on the output information output by the information output unit, The farming information management system according to claim 13.

15. On the computer, By inputting input information indicating the characteristics of the target operation performed by the work device in the field into a learning model that has been trained using video information showing other operations performed at a different time or in a different field than the target operation, a generated video showing the operation in progress is produced. Outputting the generated video, A program to execute.

Images