Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program that can identify the variety of a plant even when it does not contain fruits or leaves. [Solution] The information processing device of this embodiment includes an acquisition unit that acquires a plant image including at least a part of the organs of the plant, and an identification unit that identifies the plant variety included in the plant image by inputting the information including the plant image acquired by the acquisition unit into a trained model that takes information including the plant image as input and outputs the plant variety included in the plant image.

Description

Technical Field

[0004] , , , ,

[0006] , , , , , , , , , , , ,

[0005] , , , , , , ,

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] Conventionally, as a method for determining infringement of plant breeder's rights, there are a first method of comparing characteristics on a variety registration book through a cultivation test, a second method of using a presumption rule for comparison with a characteristic table, and a third method of using DNA variety identification technology.

[0003] Also, a technology for recognizing an object depicted in an image captured by a camera using deep learning is known (see, for example, Patent Document 1, Non-Patent Documents 1 and 2). Patent Document 1, Non-Patent Documents 1 and 2 disclose a technology for recognizing a plant and specifying its variety by inputting an image of a plant fruit or leaf into a deep learning model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005] [[ID=�5]]

Non-Patent Document 1

Non-Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, the first method described above requires actually cultivating plants and comparing their characteristics, which can make it time-consuming to determine whether plant breeder's rights have been infringed. The second method can be difficult for non-expert users to use to determine infringement. The third method can also be time-consuming to determine infringement. Furthermore, when plants are distributed without fruits or leaves, it is impossible to photograph images including these, making it impossible to use the technologies described in Patent Document 1, Non-Patent Documents 1 and 2. For example, this problem becomes particularly pronounced in situations such as detecting the export of registered varieties abroad or the unauthorized propagation of registered varieties, where plants are often distributed without fruits or leaves.

[0007] One aspect of the present invention has been made in consideration of these circumstances, and aims to provide an information processing device, an information processing method, and a program that can identify the variety of a plant even when it does not contain fruits or leaves. [Means for solving the problem]

[0008] An information processing device according to a first aspect of the present invention is an information processing device comprising: an acquisition unit that acquires a plant image including at least a part of the organs of a plant; and an identification unit that identifies the plant variety included in the plant image by inputting the information including the plant image acquired by the acquisition unit into a trained model that takes information including the plant image as input and outputs the plant variety included in the plant image.

[0009] A second aspect of the present invention is an information processing device in which the plant image further includes at least one of the dormant branches and green branches of the plant.

[0010] A third aspect of the present invention is an information processing device in which the plant image is an image that includes one or more nodes of the plant.

[0011] A fourth aspect of the present invention is an information processing device which further includes a first determination unit that refers to registered breeder's right information, which is a breeder's right established for a variety, and determines whether the variety identified by the identification unit is the registered breeder's right.

[0012] A fifth aspect of the present invention is an information processing device in which the breeder's rights information is information registered in association with the registered variety and the variety registration information relating to the breeder's rights covering the registered variety, and the device further includes an information output unit that, when the variety identified by the identification unit is the registered variety, refers to the breeder's rights information and outputs the variety registration information relating to the plant included in the plant image.

[0013] A sixth aspect of the present invention is an information processing device in which the acquisition unit acquires auxiliary information which is information relating to the plant image, and a second determination unit that determines whether or not there is an infringement of breeder's rights relating to the variety identified by the identification unit, based on the auxiliary information acquired by the acquisition unit and the variety registration information output by the information output unit.

[0014] A seventh aspect of the present invention is an information processing device which further comprises a learning unit that generates a trained model based on information relating the plant image and the variety, and the identification unit identifies the variety included in the plant image by inputting the plant image acquired by the acquisition unit into the trained model learned by the learning unit.

[0015] An information processing device according to an eighth aspect of the present invention comprises: an acquisition unit that acquires a plant image including at least a part of the organs of a plant; a learning unit that takes information including the plant image as input and generates one of a plurality of trained models that output the plant species included in the plant image, according to the type of plant organ included in the plant image; and an identification unit that identifies the plant species included in the plant image by inputting the plant image acquired by the acquisition unit into the trained model generated by the learning unit.

[0016] The information processing method according to the ninth aspect of the present invention is an information processing method in which a computer acquires a plant image including at least a part of a plant organ, inputs information including the plant image, and inputs the information including the acquired plant image to a learned model that outputs the variety of the plant included in the plant image, thereby specifying the variety included in the plant image.

[0017] The information processing method according to the tenth aspect of the present invention is an information processing method in which a computer generates a learned model that takes, as input, information including a plant image that is an image including at least one of a dormant branch and a green branch of a plant and / or an image including one or more nodes of the plant, and that outputs the variety of the plant, and inputs, to the generated learned model, information including the plant image including at least a part of a plant organ, thereby specifying the variety included in the plant image.

[0018] The program according to the eleventh aspect of the present invention is a program that causes a computer to acquire a plant image including at least a part of a plant organ, input information including the plant image, and input the information including the acquired plant image to a learned model that outputs the variety of the plant included in the plant image, thereby causing the variety included in the plant image to be specified.

Advantages of the Invention

[0019] According to the present invention, even a plant that does not include fruits or leaves can be identified as to its variety.

Brief Description of the Drawings

[0020] [Figure 1] It is a configuration diagram of an information processing system 1 to which the information processing apparatus of the embodiment is applied. [Figure 2] It is a diagram showing an example of the functional configuration of the terminal device 100. [Figure 3]It is a diagram showing an example of the functional configuration of the information processing apparatus 200. [Figure 4] It is a diagram showing an example of the content of the breeder's right information 254. [Figure 5] It is a diagram for explaining a method of generating the learned model 252 learned by the learning unit 242. [Figure 6] It is a diagram showing an example of a plant image input when generating the learned model 252. [Figure 7] It is an example of an image of an ear of grain including nodes. [Figure 8] It is a diagram showing the number of data used when generating the learned model 252. [Figure 9] It is a diagram showing the output result when test data is input to the learned model 252. [Figure 10] It is a diagram showing an example of the image IM10 output to the display unit 132. [Figure 11] It is a diagram showing an example of the image IM20 provided by the information providing unit 246 of the information processing apparatus 200. [Figure 12] It is a flowchart showing an example of the processing executed by the information processing apparatus 200 in the embodiment. [Figure 13] It is a diagram showing an example of the functional configuration of the terminal device 100A.

Embodiments for Carrying Out the Invention

[0021] Hereinafter, embodiments of an information processing apparatus, an information processing method, and a program of the present invention will be described with reference to the drawings. In the following, an example in which the information processing apparatus is applied to an information processing system that acquires information including a plant image and specifies the variety of a plant or determines the presence or absence of infringement of the breeder's right using the information of the specified variety will be described.

[0022] [Overall Configuration] Figure 1 is a diagram showing the configuration of an information processing system 1 to which the information processing device of the embodiment is applied. The information processing system 1 includes, for example, one or more terminal devices 100-1 to 100-N and an information processing device 200. Since the same configuration can be applied to the terminal devices 100-1 to 100-N, they will be collectively referred to as "terminal device 100" below. The terminal device 100 and the information processing device 200 are connected in a way that allows communication, for example, via a network NW. The network NW includes, for example, a Wi-Fi network, a cellular network, the Internet, a WAN (Wide Area Network), a LAN (Local Area Network), provider equipment, a wireless base station, etc. The network NW may be configured by combining multiple networks.

[0023] The terminal device 100 is, for example, a smartphone, tablet, or camera. The terminal device 100 comprises at least a camera that captures images of the surroundings and a communication unit that transmits the captured images to the information processing device 200 via a network. The number and type of terminal devices 100 are not limited. The terminal device 100 may be mounted on a flying object such as a drone or a small unmanned aerial vehicle (UAV) equipped with a camera and a communication unit. The terminal device 100 captures images of plants. The plant images are, for example, images that include at least some of the organs of a plant. Plant organs include, for example, vegetative reproductive organs such as dormant branches (scions), green branches (scions), tubers, and rhizomes, as well as mature leaves, fruits, berries, seeds, tree shape, petals, and corollas. Vegetative reproductive organs are not limited to those mentioned above; any plant organ that performs vegetative reproduction, which is one of the modes of asexual reproduction, is acceptable. Furthermore, plant organs may be limited to vegetative reproductive organs. The images captured by the terminal device 100 may be taken by a user of the terminal device 100, or they may be taken while the device is installed in a fixed position.

[0024] The information processing device 200 may be, for example, a general-purpose PC (Personal Computer) or a server device. Alternatively, the information processing device 200 may be a cloud computing system implemented by a server device or storage device. Furthermore, the information processing device 200 may be a communication terminal (terminal device) such as a smartphone or tablet. The information processing device 200 acquires image information, location information, etc., from the terminal device 100 via a network NW, manages the acquired images, identifies the plant varieties contained in the images, and determines whether or not there is an infringement of plant breeder's rights based on the identification results. The information processing device 200 also manages the identified varieties and the infringement determination results.

[0025] The functional configurations of the terminal device 100 and the information processing device 200 will be described in detail below. [Terminal device] Figure 2 shows an example of the functional configuration of the terminal device 100. The configuration shown in Figure 2 is an example of the configuration when the terminal device 100 is a smartphone or a tablet device. The terminal device 100 includes, for example, a communication unit 110, an imaging unit 120, an output unit 130, an information acquisition unit 140, a control unit 150, an application execution unit 160, and a storage unit 170. Some or all of the imaging unit 120, the information acquisition unit 140, the control unit 150, and the application execution unit 160 are realized by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Furthermore, some or all of these components may be implemented by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and SOC (System On Chip), or by the collaboration of software and hardware. The program may be stored in advance in a storage device such as an HDD or flash memory (a storage device equipped with a non-transient storage medium), or it may be stored in a removable storage medium such as a DVD or CD-ROM (a non-transient storage medium) and installed in the storage device when the storage medium is inserted into a drive device or a card slot of terminal device 100.

[0026] The storage unit 170 may be implemented using the various storage devices described above, or an SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 170 stores, for example, an information processing application 172, a program, and various other information. The information processing application 172 is an application program that works in conjunction with the information processing device 200, or the terminal device 100 alone, to identify a plant variety from a plant image, etc., and to determine whether or not there is an infringement of breeder's rights using the information of the identified variety. The information processing application 172 may be configured as separate applications for the process of identifying a plant variety from a plant image, etc., and the process of determining whether or not there is an infringement of breeder's rights using the information of the identified variety.

[0027] The communication unit 110 communicates with the information processing device 200 and other external devices, for example, via a network NW. The communication unit 110 may also communicate with other terminal devices 100 via a short-range communication network or a network NW.

[0028] The imaging unit 120 is a digital camera that utilizes a solid-state imaging element such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor). The imaging unit 120 may also be a stereo camera. The imaging unit 120 captures a plant image that includes at least a part of the plant's organs. The captured image may be a visible image, a multispectral image, a hyperspectral image, or other digital image.

[0029] The imaging unit 120 may be fixedly installed or held by the user. The imaging unit 120 may also take pictures periodically and repeatedly or take pictures based on user operation. The captured images may be transmitted to the information processing device 200 via the communication unit 110 each time they are taken, or they may be temporarily stored in the storage unit 170 and transmitted to the information processing device 200 at a predetermined timing. The imaging unit 120 may be configured in any way that enables it to capture plant images including at least a part of a plant organ.

[0030] The output unit 130 includes, for example, a display unit 132. The output unit 130 outputs predetermined information to the display unit 132. The display unit 132 may be, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. Alternatively, the display unit 132 may be an interface for connecting an image display device to the terminal device 100. In this case, the display unit 132 generates a video signal for displaying image data and outputs the video signal to the image display device connected to it. The display unit 132 may also be configured as a touch panel integrated with the information acquisition unit 140.

[0031] The output unit 130 outputs an image corresponding to the information provided by the information processing device 200. The output unit 130 also outputs, for example, an image captured by the imaging unit 120, information to be input by the user, and information entered by the user.

[0032] The information acquisition unit 140 acquires various types of information input via input devices such as keyboards, pointing devices (mouse, tablet, etc.), buttons, and touch panels. For example, the information acquisition unit 140 acquires auxiliary information, which is information about plant images input by a user operating the input device. The auxiliary information includes, for example, at least one of the following: identification information (name, etc.) of a person or organization intending to cultivate (propagate, etc.), sell, import, or export the plants included in the plant image; identification information (user ID) of the user who took the plant image; location information of the shooting location; and date information regarding the shooting date and time. This auxiliary information can be any information necessary to acquire information regarding plant breeder's rights. The auxiliary information may be associated with the plant image taken by the shooting unit 120 and transmitted to the information processing device 200 via the communication unit 110. The information acquisition unit 140 may also acquire information input via the input device, such as whether or not to allow the first determination unit 243 or the second determination unit 245, described later, to perform a determination. Furthermore, the information acquisition unit 140 may acquire the contents of the variety registration information input by the input device.

[0033] Furthermore, the information acquisition unit 140 may acquire the above information using a microphone and a speech recognition device, etc. In this case, the information acquisition unit 140 acquires an acoustic signal generated by the user's speech, recognizes speech from the acquired signal, and acquires string information of the recognition result. Speech recognition processing may be performed by the control unit 150. In addition, the information acquisition unit 140 may acquire plant images taken by another terminal device 100 via the communication unit 110.

[0034] The control unit 150 controls the overall functions of the terminal device 100. For example, the control unit 150 controls communication by the communication unit 110, controls output by the output unit 130, and controls the execution of the information processing application 172 by the application execution unit 160.

[0035] The application execution unit 160 is realized by executing an information processing application 172 stored in the storage unit 170. The information processing application 172 is downloaded from an external device via a network NW, installed on the terminal device 100, and stored in the storage unit 170. For example, the information processing application 172 transmits plant images taken by the shooting unit 120 and information acquired by the information acquisition unit 140 (e.g., auxiliary information, plant images) to the information processing device 200 to perform processes such as identifying the plant variety and determining whether or not there is an infringement of breeder's rights using the identified variety information, and obtains the execution result. When the information processing application 172 transmits information to the information processing device 200, it may also transmit information such as identification information to identify the terminal device 100 and identification information to identify the user. The information processing application 172 also receives information transmitted from the information processing device 200 and outputs the received information to the output unit 130 or stores it in the storage unit 170.

[0036] Furthermore, if the terminal device 100 is a camera device, it may not have to have some of the configurations of the output unit 130 among the functions described above, and applications other than the information processing application 172 may be installed.

[0037] [Information Processing Device] Figure 3 shows an example of the functional configuration of the information processing device 200. The information processing device 200 includes, for example, a communication unit 210, an acquisition unit 220, an output unit 230, a processing unit 240, and a storage unit 250. Some or all of the acquisition unit 220 and the processing unit 240 are realized, for example, by a hardware processor such as a CPU executing a program (software). Some or all of these components may also be realized by hardware (including circuitry) such as an LSI, ASIC, FPGA, GPU, or SOC, or by the cooperation of software and hardware. The program may be stored in advance in a storage device (non-transient storage medium) such as an HDD or flash memory, or it may be stored in a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM and installed when the storage medium is mounted in the drive device of the information processing device 200. The information processing device 200 may be implemented by the service operator of this embodiment installing a program on a cloud server, in which case the owner of the hardware of the information processing device 200 and the service operator may be different.

[0038] The memory unit 250 may be implemented by the various storage devices described above, or by an SSD, EEPROM, ROM, RAM, etc. The memory unit 250 stores, for example, a trained model 252, breeder's rights information 254, management information 256, a program, and various other information. At least a portion of the information contained in the memory unit 250 may be stored in an external device (for example, a database server) that can communicate with the information processing device 200. The trained model 252 is, for example, a model that, when given a plant image as input, outputs the variety of plant contained in the input image. The trained model 252 may be trained by the learning unit 242, or it may be acquired from an external device connected via a network NW.

[0039] Figure 4 shows an example of the contents of breeder's rights information 254. Breeder's rights information 254 includes information about registered varieties. A registered variety is a variety of plant for which breeder's rights have been established. Breeder's rights information 254 includes information that associates the registered variety name with the variety registration information. The registered variety name is not limited to the variety name, but may also be other identifying information that identifies the registered variety (e.g., variety number). The variety registration information includes information about rights, such as the name of the breeder, the rights period, and usage rights information. However, the types of variety registration information stored in breeder's rights information 254 are not limited to these, and some information may not be stored.

[0040] The breeder's rights holder name is identification information used to identify the person or organization that holds the breeder's rights for the registered variety corresponding to the registered variety name, such as a name, company name, or identification number. The rights term is, for example, the duration of the breeder's rights for the corresponding registered variety. The usage rights information is, for example, whether there are restrictions on the production area of ​​the corresponding registered variety, whether there are restrictions on export, and the name of the person or organization that holds the usage rights. However, the usage rights information is not limited to these, and may include any information regarding the scope of use of the breeder's rights as defined by the breeder. The breeder's rights information 254 is updated with the latest information at predetermined times or intervals. In other words, in the breeder's rights information 254, information on expired rights is deleted, and information on newly arising rights is registered.

[0041] Management information 256 consists of information transmitted by the terminal device 100 and various information processed by the processing unit 240 (for example, the results of identifying the plant variety included in the plant image and the results of determining infringement of breeder's rights).

[0042] Returning to Figure 3, the communication unit 210 communicates with the terminal device 100 and other external devices via a network NW or the like.

[0043] The acquisition unit 220 acquires various information from the terminal device 100 and external devices connected via the network NW, based on the information received by the communication unit 210. The acquisition unit 220 acquires plant images from the terminal device 100. The acquisition unit 220 may also acquire auxiliary information from the terminal device 100.

[0044] The output unit 230 includes, for example, a display unit 232. The output unit 230 outputs predetermined information to the display unit 232. The display unit 232 is, for example, an LCD or an organic EL display. The display unit 232 displays various information according to the embodiment. The display unit 132 may be an interface for connecting an image display device to the information processing device 200. In this case, the display unit 132 generates a video signal for displaying image data and outputs the video signal to the image display device connected to it. The display unit 232 may also be a touch panel equipped with the function of an input unit that accepts input operations from a user of the information processing device 200 (for example, a system administrator). The output unit 230 outputs an image corresponding to the information provided by the information processing device 200. The output unit 230 also outputs information to be input to the user of the information processing device 200 and information input by the user.

[0045] The processing unit 240 includes, for example, a specific unit 241, a learning unit 242, a first determination unit 243, an information output unit 244, a second determination unit 245, an information provision unit 246, and a management unit 247.

[0046] The identification unit 241 identifies the plant species contained in a plant image by inputting the information including the plant image acquired by the acquisition unit 220 into a trained model 252 that takes information including a plant image as input and outputs the plant species contained in the plant image.

[0047] The types of plant organs included in a plant image may be selected by the user, or the identification function included in the identification unit 241 may automatically identify them from characteristic information of the plant image (e.g., color, shape). The characteristic information is obtained by performing edge extraction, color extraction, brightness extraction, or shape extraction using pattern matching processing using existing image analysis processes. Furthermore, if there are multiple types of plant organs included in a plant image, the type of plant organ used to identify the plant variety may be one or multiple. If there are multiple types of plant organs selected by the user and multiple types of plant organs automatically identified, the identification unit 241 may identify the variety for each type of plant organ, or it may comprehensively identify the variety from the identification results for each type using a predetermined method. The predetermined method may be, for example, a majority vote, or a method that sets priorities in advance and uses the variety identified by the plant organ type with the highest priority. However, the predetermined method is not limited to the above; any method that can select one identification result from multiple identification results, or a method that can prioritize and display multiple identification results, is acceptable.

[0048] The learning unit 242 takes information including plant images as input and generates a trained model 252 that outputs the plant species contained in the plant images. The generated trained model 252 is stored in the storage unit 250. Details of the functions of the learning unit 242 will be described later.

[0049] The first determination unit 243 refers to the breeder's rights information 254 and determines whether the variety identified by the identification unit 241 is a registered variety. For example, the first determination unit 243 determines that the variety is a registered variety if the identified variety is included in the registered variety name in the breeder's rights information 254, and determines that it is not a registered variety if it is not included in the registered variety name. Whether or not to perform a determination by the first determination unit 243 may be specified by the user.

[0050] The information output unit 244, if the variety identified by the identification unit 241 is a registered variety, refers to the breeder's rights information 254 and outputs variety registration information for the plant included in the plant image. The output variety registration information may be all or part of the variety registration information registered in the breeder's rights information 254. The content of the output variety registration information may be specified by the user.

[0051] The second determination unit 245 determines whether or not there is an infringement of the breeder's rights for the variety identified by the identification unit 241, based on the auxiliary information acquired by the acquisition unit 220 and the variety registration information output by the information output unit 244. For example, the second determination unit 245 compares the name of a person who intends to cultivate (propagate, etc.), sell, import, or export the plant, which is included in the auxiliary information associated with the plant image for which the variety has been identified by the identification unit 241, with the name of the breeder in the variety registration information corresponding to the registered variety name of the identified variety. If the name of the person is included in the name of the breeder, the second determination unit 245 determines that the breeder's rights have not been infringed (the breeder possesses the rights). Furthermore, even if the name of the person is not included in the name of the breeder, the second determination unit 245 determines that the breeder's rights have not been infringed if the date and time of photography included in the auxiliary information is not included in the rights period of the variety registration information. Furthermore, the second determination unit 245 determines that breeder's rights have not been infringed even if the person's name is not included in the breeder's name and the date and time of filming are within the rights period, if the person's name included in the auxiliary information or the filming location is recognized for use according to the usage rights information. Conversely, the second determination unit 245 determines that breeder's rights have been infringed if the person's name is not included in the breeder's name, the date and time of filming are within the rights period, and the person's name included in the auxiliary information or the filming location is not recognized for use according to the usage rights information. Whether or not to perform a determination by the second determination unit 245 may be specified by the user.

[0052] The information provision unit 246 generates plant information contained in the plant image and provides the generated information to the terminal device 100 that transmitted the captured image via the communication unit 210. The generated information includes, for example, the plant image acquired by the acquisition unit 220, the name of the variety identified by the identification unit 241, and at least one of the determinations made by the second determination unit 245 regarding whether or not there is an infringement of breeder's rights. For example, the information provision unit 246 generates an image of the name of the variety identified by the identification unit 241 and the determination of whether or not there is an infringement of breeder's rights made by the second determination unit 245, and provides it to the terminal device 100. The plant information contained in the plant image is not limited to this and can be changed as appropriate as long as it relates to the variety identified by the identification unit 241. The information provision unit 246 may also output the generated information to the output unit 230.

[0053] The management unit 247 manages information related to plant images. For example, the management unit 247 stores and manages plant images acquired by the acquisition unit 220 and auxiliary information as management information 256 in the storage unit 250. The management unit 247 may also have the learning unit 242 update the trained model 252, or perform processing to acquire the trained model 252 from an external device. Furthermore, the management unit 247 may store and manage variety information identified by the identification unit 241, information on whether or not it is a registered variety determined by the first determination unit 243, variety registration information output by the information output unit 244, and information on whether or not there is an infringement of breeder's rights determined by the second determination unit 245, together with the plant images in the management information 256. This makes it possible to store all the information used when determining whether or not there is an infringement of breeder's rights in a corresponding manner. The information stored together with the plant images is not limited to the above example and can be changed as appropriate.

[0054] [Learning Department] Next, the functions of the learning unit 242 will be described in detail. Figure 5 is a diagram illustrating the method for generating the trained model 252 learned by the learning unit 242. In the example in Figure 5, the learning unit 242 takes information including images of plant leaves (mature leaves) taken in the past as input and generates a plant variety identification model that outputs the plant variety contained in the leaf images. Specifically, the learning unit 242 takes leaf images as input, performs training using deep learning, and generates a trained model 252 for leaf variety identification that outputs the plant variety contained in the leaf images. For training using deep learning, for example, DNN (Deep Neural Network) is used, and for the image classification model, for example, VGG16, ResNet50, ViT (Vision Transformer) is used, but other known learning methods and learning models may also be used.

[0055] Furthermore, the learning unit 242 inputs leaf images into the generated pre-trained model 252, compares the output result (inference result) with the actual variety (ground truth) associated with the input image to check the answer, and if there is a discrepancy between the inference result and the ground truth, changes (adjusts) the parameters of the classification model to reduce the discrepancy. After changing the parameters, the learning unit 242 again inputs leaf images into the pre-trained model 252, and compares the output result (inference result) with the actual variety (ground truth) included in the input image to check the answer. By repeating this comparison of results and parameter changes, the accuracy of the pre-trained model 252 can be improved.

[0056] In the example shown in Figure 5, an image of a plant leaf is used, but instead (or in addition to this), images of dormant branches and green branches, tubers, and other branches that are the vegetative reproductive organs of plants may be used, or images of fruits, seeds, tree shapes, etc. may be used. In this case, the learning unit 242 may generate different trained models 252 depending on the type of plant (or vegetative reproductive organ) described above.

[0057] Figure 6 shows an example of plant images used as input when generating the trained model 252. The example in Figure 6 shows images of plant leaves, fruit berries, dormant branches, and tree shapes. A dormant branch is a type of scion. Furthermore, S, R, and M in Figure 6 represent varieties. S is Shine Muscat, R is Rosario Bianco, and M is Muscat of Alexandria. Note that the varieties are not limited to these, and may include varieties that are not registered.

[0058] Figure 7 shows an example of an image of a scion including nodes. As shown in Figure 7, the learning unit 242 generates a trained model using an image that includes at least one node A. In this way, by including feature parts that clearly differentiate it from other varieties in the image of the vegetative reproductive organ input when generating the trained model 252, the accuracy of variety identification in the identification unit 241 described above can be improved. The learning unit 242 may generate different trained models 252 depending on the type of plant (e.g., grapes, melons, strawberries, pears) and the type of vegetative reproductive organ, or it may generate a trained model 252 that integrates them.

[0059] Furthermore, the learning unit 242 may also accept, in addition to plant images, location information associated with the plant images, and information on the date and time of shooting as input. By learning with information other than plant images, it becomes possible to perform detailed classifications even for the same variety produced in different regions, for example, thus generating a trained model that can produce more accurate inference results.

[0060] [Learning results and classification accuracy] Next, the learning results and classification accuracy of the embodiment will be described. Figure 8 shows the number of data points used when generating the trained model 252. In the example in Figure 8, the number of data points used when generating the trained model 252 for leaves, berries, dormant branches, and tree shape is shown. The numbers in parentheses indicate the number of data points used for each variety (Shine Muscat, Rosario Bianco, Muscat of Alexandria). There are three types of data used: training data, validation data, and test data. The training data is used in the learning unit 242 when generating the trained model 252. The validation data is used to verify the accuracy of the parameters of the trained model 252 generated using the training data. The test data is used to evaluate the output results of the trained model 252 whose parameters have finally been determined.

[0061] Figure 9 shows the output results when test data is input into the trained model 252. The total number of test data used when inputting the test data into the trained model 252 is shown in Figure 8. (A) in the figure is a table of output results when leaf images are used. (B) in the figure is a table of output results when fruit berry images are used. (C) in the figure is a table of output results when dormant branch images are used. (D) in the figure is a table of output results when tree shape images are used. The horizontal axis of the table shows the prediction, and the vertical axis shows the correct answer. The prediction is the output result (variety) identified (predicted) by the identification unit 241 when test data is input into the trained model 252, and the correct answer is the actual variety of the test data. The table shows the percentage (numerical value) of varieties identified using the trained model 252 of this embodiment relative to the total number of test data for each variety. As shown in Figure 9, varieties can be identified with high accuracy for leaves, fruit berry, dormant branch, and tree shape. Therefore, by using the pre-trained model 252 described above, the identification unit 241 can more accurately identify the variety of a plant even if it does not have fruits or leaves (for example, a plant in the state of vegetative reproduction organs).

[0062] [Screen example] Next, the information provided to the user by the information processing system 1 will be described. Figure 10 shows an example of image IM10 output to the display unit 132. Note that the items and layout displayed in image IM10 are not limited to this. The same applies to image IM20, which will be described later. Image IM10 shown in Figure 10 includes a captured image display area AR11 and an auxiliary information input area AR12. For example, the captured image display area AR11 displays a plant image captured by the shooting unit 120. For example, the auxiliary information input area AR12 displays auxiliary information acquired by the information acquisition unit 140. Image IM10 may also include icon IC1 and icon IC2. Icon IC1 is a GUI (Graphical User Interface) switch that accepts instructions to end the display of the captured plant image and return to a state where it can be captured again. When icon IC1 is selected by the user of the terminal device 100, the information processing application 172 ends the display of the captured plant image. Icon IC2 is a GUI switch that accepts transmission of the plant image captured by the shooting unit 120 to the information processing device 200. When the user of terminal device 100 selects icon IC2, the information processing application 172 transmits the plant image and auxiliary information captured by the imaging unit 120 to the information processing device 200 via the communication unit 110. As a result, the information processing device 200 performs variety identification by the identification unit 241 and various judgments by the first judgment unit 243 and the second judgment unit 245, and provides information regarding the execution results to terminal device 100.

[0063] Figure 11 shows an example of an image IM20 provided by the information provision unit 246 of the information processing device 200. The image IM20 shown in Figure 11 includes a plant image display area AR21 and a variety registration information display area AR22. The plant image display area AR21 displays, for example, the image entered when the variety was identified (i.e., the plant image displayed in the captured image display area AR11). The variety registration information display area AR22 displays, for example, information such as the name of the variety identified by the identification unit 241. The variety registration information display area AR22 may also display information indicating whether the variety of the plant image determined by the first determination unit 243 is a registered variety, or information such as whether or not there is an infringement of breeder's rights determined by the second determination unit 245. In addition, other images or GUI switches may be displayed in the image IM20.

[0064] [Processing flow] Next, an example of processing performed by the information processing device 200 in the embodiment will be described. Figure 12 is a flowchart of an example of processing performed by the information processing device 200 in the embodiment. First, the acquisition unit 220 acquires plant images and auxiliary information (step S102). Next, the management unit 247 manages the acquired plant images and auxiliary information (step S104). Next, the identification unit 241 uses the trained model 252 to identify the plant varieties included in the acquired plant images (step S106). In the processing of step S106, auxiliary information may be used in addition to plant images. Next, the first determination unit 243 determines whether the variety identified by the identification unit 241 is a registered variety (step S108). If it is determined that the identified variety is a registered variety, the information output unit 244 refers to the breeder's rights information 254 and outputs variety registration information regarding the plants included in the plant images (step S110). Next, the second determination unit 245 determines whether or not there is an infringement of breeder's rights for the variety identified by the identification unit 241, based on the auxiliary information acquired by the acquisition unit 220 and the variety registration information output by the information output unit 244 (step S112). If, after processing in step S112, or during processing in step S108, it is determined that the variety is not a registered variety, the information provision unit 246 generates plant information contained in the plant image and provides the generated information to the terminal device 100 that transmitted the captured image via the communication unit 210 (step S114). In this way, the information processing device 200 completes the process shown in Figure 12.

[0065] In this embodiment, steps S108 to S112 may not be executed in the process shown in Figure 12 described above, and information regarding the result of the determination process in step S108 may be provided without executing steps S110 and S112.

[0066] Next, we will describe an example of how the information processing system 1 in the embodiment can be applied. [First application example] By applying the information processing system 1 of this embodiment, even users who are not plant experts can accurately and quickly determine whether or not there is a breeder's rights infringement using the terminal device 100. For example, in situations such as when customs authorities are cracking down on the export of registered varieties abroad, even if the plants are distributed without fruits or leaves, the plant variety can be identified, and the terminal device 100 can confirm whether or not there is a breeder's rights infringement for the identified variety.

[0067] [Second application example] In the embodiment, the infringement determination of plant breeder's rights is mainly performed using plant images; therefore, to perform a more accurate infringement determination, it may be combined with conventional methods such as DNA analysis. In this case, the information processing system 1 can more efficiently detect infringement by first performing a plant breeder's rights infringement determination using plant images in order to screen for suspicious varieties before DNA analysis. Note that the application examples are not limited to the first and second examples.

[0068] [Differentiation] In the embodiment described above, the terminal device 100 and the information processing device 200 are configured as different devices, but a part of the configuration of the information processing device 200 may be provided in the terminal device 100, and a part of the configuration of the terminal device 100 may be provided in the information processing device 200. Figure 13 is a diagram showing an example of the functional configuration of the terminal device 100A. In the terminal device 100A shown in Figure 13, a part of the configuration of the information processing device 200 is provided in the terminal device 100A. More specifically, the terminal device 100A differs from the terminal device 100 shown in Figure 2 in that a processing unit 240A is provided in place of the application execution unit 160, and the storage unit 170 stores a trained model 174 and breeder rights information 176 in place of the information processing application 172. Therefore, the following explanation will mainly focus on the differences.

[0069] The trained model 174 and breeder's rights information 176 may be the same as the trained model 252 and breeder's rights information 254 described above, or they may be lightweight versions with less data. The processing unit 240A has the same functions as the processing unit 240 of the information processing unit 200. For example, the processing unit 240A uses the plant image taken by the terminal device 100A to perform variety identification processing and determination processing of whether or not there is an infringement of breeder's rights on the terminal device 100A side. This improves convenience because the user can complete everything from taking the image to identifying the variety and determining whether or not there is an infringement of breeder's rights on their own without communicating with the information processing unit 200. Note that the terminal device 100A shown in Figure 13 may be an example of an "information processing unit".

[0070] Furthermore, in the embodiment, the information processing device 200 and the terminal device 100A do not have to have at least one of the first determination unit 243 and the second determination unit 245. Also, in the embodiment, auxiliary information does not have to be acquired by the acquisition unit 220 or the information acquisition unit 140. Also, in the embodiment, the information processing device 200 does not have to have the information output unit 244. Also, in the embodiment, the information stored in the breeder's rights information 254 may consist only of the registered variety name, and the breeder's rights information 254 does not have to be stored in the storage unit 250. Also, in the embodiment, the information processing device 200 and the terminal device 100A do not have the learning unit 242. In this case, the learned model 252 (174) is acquired from an external device connected via the network NW. Also, in the embodiment, the terminal device 100 and the terminal device 100A do not have the imaging unit 120. In this case, the plant image may be acquired from an external device via the network NW, or it may be stored in the storage unit 170. Furthermore, in this embodiment, the plant images used for variety identification processing and determination of whether or not there is an infringement of breeder's rights, etc., performed by the information processing device 200 and the terminal device 100A, may be plant images stored in the storage unit 250. In this case, the acquisition unit 220 acquires the plant images from the storage unit 250.

[0071] Furthermore, in the embodiment, a method for improving the accuracy of the trained model 252 that outputs the plant variety may be provided. In this case, the learning unit 242 takes information including a plant image as input and generates a trained model 252 that outputs the plant variety included in the input plant image, based on information that associates a plant image, which is an image containing at least one of a dormant branch and a green branch of a plant, and / or an image containing one or more nodes of a plant, with the plant variety included in the plant image. This makes it possible to further improve the accuracy of the trained model 252 that outputs the plant variety. By inputting a plant image containing at least some of the plant's organs into the trained model 252 generated in this way, the variety included in the plant image can be identified, and the variety can be identified even if the plant does not contain fruit or leaves.

[0072] In this embodiment, for example, multiple trained models 252 may be generated that output the plant variety depending on the type of plant organ. In this case, the acquisition unit 220 acquires a plant image that includes at least some of the plant organs. The learning unit 242 takes information including the plant image as input and generates one of multiple trained models 252 that output the plant variety included in the plant image, depending on the type of plant organ included in the plant image. The multiple trained models 252 include, for example, a model that takes an image of a leaf as input and outputs the plant variety included in the input image, a model that takes an image of a fruit as input and outputs the plant variety included in the input image, a model that takes an image of a dormant branch as input and outputs the plant variety included in the input image, and a model that takes an image of the tree shape as input and outputs the plant variety included in the input image. The multiple trained models 252 are different models depending on the type of plant organ. Furthermore, the identification unit 241 identifies the plant species contained in the plant image by inputting the plant image acquired by the acquisition unit 220 into the trained model 252 generated by the learning unit 242. Thus, according to this embodiment, an appropriate trained model 252 can be generated according to the type of organ contained in the plant image, and the plant species contained in the plant image can be identified with high accuracy.

[0073] According to the embodiments described above, the information processing device 200 (terminal device 100A) includes an acquisition unit 220 that acquires a plant image including at least a part of the plant's organs, and an identification unit 241 that identifies the plant variety included in a plant image by inputting the information including the plant image acquired by the acquisition unit 220 into a trained model 252 that takes information including the plant image as input and outputs the plant variety included in the plant image. This makes it possible to identify the variety of a plant even if it does not have fruits or leaves.

[0074] The embodiments described above can be expressed as follows. A storage medium that stores computer-readable instructions, A processor connected to the storage medium, The processor executes the computer-readable instructions to: Obtain a plant image that includes at least some of the plant's organs, By inputting the acquired information including the plant image into a trained model that takes the information including the plant image as input and outputs the plant species contained in the plant image, the species contained in the plant image is identified. Information processing device.

[0075] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the present invention. [Explanation of Symbols]

[0076] 1…Information processing system, 100…Terminal device, 110, 210…Communication unit, 120…Shooting unit, 130, 230…Output unit, 132, 232…Display unit, 140…Information acquisition unit, 150…Control unit, 160…Application execution unit, 170, 250…Storage unit, 200…Information processing device, 240, 240A…Processing unit, 241…Identification unit, 242…Learning unit, 243…First determination unit, 244…Information output unit, 245…Second determination unit, 246…Information provision unit, 247…Management unit

Claims

1. An acquisition unit that acquires a plant image including at least a part of the plant's organs, The system includes a trained model that takes information including the aforementioned plant image as input and outputs the plant species contained in the plant image, and a specification unit that identifies the plant species contained in the plant image by inputting the information including the plant image acquired by the acquisition unit, Information processing device.

2. The aforementioned plant image is an image that includes at least one of the dormant branches and green branches of the plant. The information processing apparatus according to claim 1.

3. The aforementioned plant image is an image that includes one or more nodes of the plant. The information processing apparatus according to claim 1.

4. The system further includes a first determination unit that, by referring to pre-registered breeder's right information regarding registered varieties for which breeder's rights have been established, determines whether the variety identified by the identification unit is the registered variety. The information processing apparatus according to any one of claims 1 to 3.

5. The aforementioned breeder's rights information is information registered in association with the registered variety and the variety registration information relating to the breeder's rights covering the registered variety. If the variety identified by the identification unit is the registered variety, the system further includes an information output unit that refers to the breeder's rights information and outputs the variety registration information relating to the plant included in the plant image. The information processing apparatus according to claim 4.

6. The acquisition unit acquires auxiliary information which is information relating to the plant image, The system further includes a second determination unit that determines whether or not there is an infringement of breeder's rights with respect to the variety identified by the identification unit, based on the auxiliary information acquired by the acquisition unit and the variety registration information output by the information output unit. The information processing apparatus according to claim 5.

7. The system further comprises a learning unit that generates the trained model based on information relating the plant image and the variety, The identification unit identifies the variety included in the plant image by inputting the plant image acquired by the acquisition unit into the trained model trained by the learning unit. The information processing apparatus according to any one of claims 1 to 3.

8. An acquisition unit that acquires a plant image including at least a part of the plant's organs, A learning unit that generates one of several pre-trained models that take information including the plant image as input and output the plant variety included in the plant image, depending on the type of plant organ included in the plant image. The system includes an identification unit that identifies the plant species included in the plant image by inputting the plant image acquired by the acquisition unit into the trained model generated by the learning unit, Information processing device.

9. Computers Obtain a plant image that includes at least some of the plant's organs, By inputting the acquired information including the plant image into a trained model that takes the information including the plant image as input and outputs the plant species contained in the plant image, the species contained in the plant image is identified. Information processing methods.

10. Computers Based on information that associates a plant image, which is an image containing at least one of a dormant branch and a green branch of a plant, and / or an image containing one or more nodes of the plant, with the plant species included in the plant image, a trained model is generated that takes the information including the plant image as input and outputs the plant species. By inputting information including the plant image, which includes at least some of the plant's organs, into the generated trained model, the variety of the plant included in the plant image is identified. Information processing methods.

11. On the computer, Obtain a plant image that includes at least some of the plant's organs. By inputting the acquired information including the plant image into a trained model that takes the information including the plant image as input and outputs the plant variety contained in the plant image, the model identifies the plant variety contained in the plant image. program.

Images