Information processing device, information processing system, information processing method, control program, and recording medium

Abstract

To improve accuracy in extracting a plant leaf in an image, without depending on an optical environment when the image is captured.SOLUTION: An information processor (10b) comprises: a first extraction unit (23) that regards an image including a plant stock as an input and extracts edge information indicating an outline of an object included in the image and lightness information indicating lightness of the object; and a second extraction unit (25) that regards the edge information and the lightness information as inputs and extracts a region indicating a plant leaf in the image using a first already learned model for extracting a region indicating a plant leaf.SELECTED DRAWING: Figure 10

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing system, an information processing method, a control program, and a recording medium. [Background technology]

[0002] An information processing device that extracts an area showing a target plant strain when determining the presence or absence of an abnormality in a plant strain in an image is known as a conventional technology. Non-Patent Document 1 discloses a technology for processing plant images, in which only areas of leaves and fruits are extracted from an image and areas not extracted are removed as background. Non-Patent Document 2 discloses a technology for recognizing leaf areas in an image including multiple leaves in a greenhouse. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Yatomi, 2019 Trends and issues in automatic plant disease diagnosis technology Journal of the Japanese Society for Neural Networks, Vol. 26, No. 4, pp. 123-134 [Non-Patent Document 2] Suwa et al., 2019 Intrinsic difficulties of plant diagnosis from wide-angle image, IDDD Proc.Big Data 2019 workshop, URL(https: / / arxiv.org / pdf / 1910.11506.pdf) Summary of the Invention [Problem to be solved by the invention]

[0004] However, the conventional techniques described above have a problem in that the accuracy of extracting the leaves, etc. of a plant is easily affected by weather, etc., because the color of the plant in the image changes depending on the lighting environment when the image is captured.

[0005] One aspect of the present invention has been made in consideration of the above problems, and aims to improve the accuracy of extracting plant leaves in an image regardless of the lighting environment when the image was captured. [Means for solving the problem]

[0006] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes a first extraction unit that receives an image including a plant stock as input and extracts edge information indicating the contours of objects contained in the image and brightness information indicating the brightness of the object, and a second extraction unit that receives the edge information and the brightness information as input and extracts an area representing plant leaves in the image using a first trained model that extracts an area representing plant leaves.

[0007] In order to solve the above problems, an information processing device according to one embodiment of the present invention includes an acquisition unit that acquires edge information indicating the outline of an object included in an image including a plant stock, brightness information indicating the brightness of the object, and information indicating an area indicating the plant leaves, and a learning unit that uses the edge information and the brightness information as input and trains a learning model that extracts areas indicating plant leaves using a set of the edge information and the brightness information acquired by the acquisition unit and the information indicating the area indicating the plant leaves as teacher data.

[0008] In order to solve the above problems, an information processing method according to one embodiment of the present invention is an information processing method executed by an information processing device, and includes a first extraction step of taking an image including a plant stock as input and extracting edge information indicating the contour of an object included in the image and brightness information indicating the brightness of the object, and a second extraction step of taking the edge information and the brightness information as input and extracting an area indicating plant leaves in the image using a trained model that extracts areas indicating plant leaves. Effect of the Invention

[0009] According to one aspect of the present invention, it is possible to improve the accuracy of extracting plant leaves in an image, regardless of the lighting environment when the image is captured. [Brief description of the drawings]

[0010] [Figure 1] 1 is a functional block diagram of an information processing system according to a first embodiment. [Diagram 2] 1 is a flowchart showing a process flow according to the first embodiment; [Diagram 3] FIG. 2 shows the distribution of pixel values ​​corresponding to objects contained in an image in Lab color space showing a field of plant lines; [Figure 4] The distribution of the number of pixels with pixel values ​​corresponding to plant stocks and soil is shown on the ab plane of the Lab color space. [Diagram 5] 1 is an image showing a field of plant strains. [Figure 6] FIG. 11 is a functional block diagram of an information processing system according to a second embodiment. [Figure 7] 1 is an image showing a field of plant strains. [Figure 8] 10 is a flowchart showing a process flow according to the second embodiment. [Figure 9] 1 is an image showing a field of plant strains. [Figure 10] FIG. 11 is a functional block diagram of an information processing system according to a third embodiment. [Figure 11] 1 shows information relating to a first extraction unit. [Figure 12] 10 is a flowchart showing a process flow according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [Embodiment 1] An embodiment of the present invention will be described in detail below. In this embodiment, an example of an information processing device capable of classifying a plant stock portion and a soil portion with high accuracy even when an image of the plant stock captured on a sunny day is input will be described.

[0012] [1. Example of information processing system configuration] An information processing system 1 according to this embodiment will be described. FIG. 1 is a functional block diagram of the information processing system 1 according to this embodiment. As shown in FIG. 1, the information processing system 1 includes an information processing device 10, a display device 38, and a mobile object 42. Note that each component included in the information processing system 1 is not limited to being singular, and each may be multiple. Also, the function of a single component included in the information processing system 1 may be realized by multiple other components, and the functions of multiple components included in the information processing system 1 may be realized by a single other component.

[0013] The information processing device 10 includes a control unit 12, a storage unit 30, and a communication unit 34. The control unit 12 is a control device that controls the entire information processing device 10, and includes an acquisition unit 14, a preprocessing unit 16, an extraction unit 18, a classification unit 20, a learning unit 22, and a presentation unit 24.

[0014] The acquisition unit 14 acquires an image including a plant strain. The image may be one that has been converted to an image in the Lab color space by the preprocessing unit 16. Here, an image in the Lab color space is an image in which pixel values ​​related to color are defined by the value of L corresponding to lightness and the values ​​of a and b corresponding to complementary colors. The acquisition unit 14 also acquires information input from a moving object 42 or the like, and information regarding the presence or absence of abnormalities in the plant strain included in the image. Here, the information regarding the presence or absence of abnormalities may include, for example, information indicating the presence or absence of abnormalities, or information indicating what type of abnormality exists.

[0015] The pre-processing unit 16 performs a predetermined process on the image acquired by the acquisition unit 14. For example, the pre-processing unit 16 performs a process of converting the image into an image in the Lab color space, and a process of removing noise by applying a Gaussian filter to the image. Note that the configuration may also be such that the moving object 42 captures a moving image, and the pre-processing unit 16 extracts and uses a still image from the moving image.

[0016] The extraction unit 18 extracts an area representing a plant strain in the image by comparing a value corresponding to both the a-value and the b-value in the pixel values ​​of the image acquired by the acquisition unit 14 with a predetermined threshold value. In other words, the extraction unit 18 assigns a threshold value to a feature amount related to both the a-value and the b-value in the pixel values ​​of the image, and determines whether the target pixel represents a plant strain or an object other than a plant strain, such as soil, depending on the comparison result with the threshold value.

[0017] 4, for example, the extraction unit 18 extracts an area indicating a plant strain in accordance with the positional relationship between a position corresponding to the pixel value of a target pixel and a line or curve indicating a threshold value on the a-b plane in the Lab color space. From another perspective, the extraction unit 18 determines whether the target pixel indicates a plant strain or soil depending on which area defined by the line or curve the pixel value corresponds to.

[0018] The classification unit 20 receives an input of an image including a region indicating a plant line extracted by the extraction unit 18, and classifies the plant line as being healthy or abnormal using a trained model that outputs a determination result regarding the presence or absence of abnormality in the plant line. Here, the trained model refers to a learning model that has been trained a sufficient number of times.

[0019] The learning unit 22 inputs an image including a plant stock, and trains a learning model that outputs a determination result regarding the presence or absence of abnormality in the plant stock, using a pair of an image including the plant stock and information regarding the presence or absence of abnormality in the plant stock as teacher data. The learning unit 22 executes a process of training the learning model, for example, by performing the following procedure a sufficient number of times. The trained model trained by the following process is an example of a second trained model in the present disclosure. Note that when the second trained model targets plant leaves, "plant stock" in (1) below is read as "plant leaves".

[0020] (1) The acquisition unit 14 acquires, from the storage unit 30 or the like, teacher data including a plurality of pairs of an image including a plant line and information regarding the presence or absence of abnormality in the plant line included in the image.

[0021] (2) The learning unit 22 inputs the image into a learning model.

[0022] (3) The learning unit 22 updates the values ​​of the parameter set that defines the learning model in a gradient direction that reduces a loss function defined by the information and the output value of the learning model.

[0023] Moreover, an example of the first trained model in the present disclosure will be described later in embodiment 3.

[0024] In addition, the classification method performed by the classification unit 20 and the machine learning method performed by the learning unit 22 are not limited to a specific method, and may be, for example, a method using a CNN (Convolutional Neural Network) or an RNN (Recurrent Neural Network). In addition, either a classification model or a regression model may be used. When using a neural network such as a CNN or an RNN, input data may be processed in advance for input to the neural network. For such processing, in addition to two-dimensional or multidimensional arrangement of data, for example, various data augmentation methods, adjustment of brightness, color tone, image quality, angle of the object, object detection for extracting the object region, segmentation, etc. may be used.

[0025] In addition, when using CNN, a convolutional layer that performs convolutional calculations may be provided as one or more layers included in the neural network, and a filter calculation (product-sum calculation) may be performed on input data input to the layer. When performing the filter calculation, processing such as padding may be used in combination, or an appropriately set stride width may be adopted.

[0026] In addition, for example, any one of the following machine learning techniques or a combination thereof may be used.

[0027] Support Vector Machine (SVM) Clustering Inductive Logic Programming (ILP) · Genetic Algorithm (GP: Genetic Programming) ·Bayesian Network (BN) Autoencoder The presentation unit 24 presents information indicating the position of the plant strain classified as an abnormal strain by the classification unit 20. For example, the presentation unit 24 displays an image indicating the position of the plant strain on the screen of the display device 38. The presentation unit 24 may be configured to present the information indicating the position of the plant strain by sound. In the above configuration, the presentation unit 24 may present the information by, for example, outputting sound indicating the coordinates of the plant strain from a speaker (not shown) provided in the information processing device 10.

[0028] As one embodiment, the presenting unit 24 may present information indicating the location of a plant strain classified as an abnormal strain in real time in parallel with the movement of the mobile object 42. For example, in the above configuration, the presenting unit 24 may be configured to sound a notification sound while displaying information indicating the location of the abnormal strain on the screen of the display device 38 in response to the classification of a plant strain as an abnormal strain. In this way, even if the user is not looking at the screen, the user can easily understand that an abnormal strain has been found near the mobile object 42 when the notification sound is sounded, and can check the screen.

[0029] The memory unit 30 is a storage device that stores various information, for example, a parameter set that defines a learning model. The communication unit 34 performs communication processing with an external device such as a mobile object 42.

[0030] The display device 38 is a display that displays moving images, text, etc. For example, the display device 38 displays information instructed by the presentation unit 24 on the screen.

[0031] The moving body 42 is realized as a flying object such as a vehicle or a drone, and includes an imaging unit 44 and a communication unit 46. The imaging unit 44 is a camera that captures an image of the surroundings of the moving body 42. The communication unit 46 performs communication processing with an external device such as an information processing device 10. Note that the present disclosure also includes an embodiment in which the moving body 42 is realized by a person or an animal carrying a camera. The moving body 42 may also be capable of carrying a person, an information processing device 10, or the like. The moving body 42 transmits an image captured by the imaging unit 44 and information indicating the position of the moving body 42 when the image was captured to the information processing device 10, for example, by wireless communication. The configuration in which the moving body 42 acquires its own position may be realized by, for example, mounting a position information terminal using RTK (Real-Time Kinematic) on the moving body 42.

[0032] [2. Examples of processing in information processing systems] Next, an example of a process in which the information processing system 1 presents the position of an abnormal plant will be described. In this example, the information processing device 10 processes an image of a field in which a plant plant is planted as an image including the plant plant. This may also be the case in the following embodiments. FIG. 2 is a flowchart showing the flow of the process in this example.

[0033] In step S101, the moving object 42 captures one or more images including the plant plants around the moving object 42 with the imaging unit 44 while moving over the field of plant plants.

[0034] In step S102, the moving object 42 transmits the image captured by the imaging unit 44 and information indicating the position of the moving object 42 when the image was captured to the information processing device 10. The acquisition unit 14 of the information processing device 10 acquires each piece of information transmitted from the moving object 42 via the communication unit 34. The information processing device 10 associates the image with the information indicating the position of the moving object 42 and stores them in the storage unit 30. The process of step S102 is not limited to a configuration in which it is started after the imaging of the entire target farm field is completed, and may be a configuration in which it is executed whenever the imaging unit 44 captures an image.

[0035] In step S103, the preprocessing unit 16 converts the image acquired by the acquisition unit 14 into an image in the Lab color space, and performs processing to remove noise in the image, for example, by applying a Gaussian filter. The acquisition unit 14 also acquires the image that has been subjected to the processing.

[0036] In step S104, the extraction unit 18 extracts a region indicating a plant strain from the image acquired by the acquisition unit 14 and processed by the preprocessing unit 16.

[0037] Figure 3 shows an image in Lab color space, illustrating the distribution of pixel values ​​corresponding to objects contained in an image showing a field of plant stocks. In Figure 3, region 50 shows the union of the distribution of pixel values ​​corresponding to plant stocks imaged on a cloudy day and the distribution of pixel values ​​corresponding to plant stocks imaged on a sunny day. Region 52 shows the distribution of pixel values ​​corresponding to soil imaged on a cloudy day. Region 54 shows the distribution of pixel values ​​corresponding to soil imaged on a sunny day.

[0038] The explanatory diagram 58 of FIG. 4 shows the distribution of the number of pixels of pixel values ​​corresponding to plant stalks in the a-b plane when the horizontal axis is the a-axis and the vertical axis is the b-axis in the Lab color space. Here, the distribution of the number of pixels corresponds to the number of points overlapping in the L direction at each coordinate of the a-b plane when the graph of FIG. 3 is viewed from directly above. The explanatory diagram 60 shows the distribution of the number of pixels of pixel values ​​corresponding to soil captured on a cloudy day in the a-b plane. The explanatory diagram 62 shows the distribution of the number of pixels of pixel values ​​corresponding to soil captured on a sunny day in the a-b plane. The soil also includes soil parts that are in the shadow of plant stalks, etc. In each explanatory diagram, the distribution is expressed in exponential terms, and since the number of pixels in the peripheral parts of the distribution is very small compared to the central part of the distribution, there is almost no impact even if it is not taken into consideration. In each explanatory diagram, the straight line 66 is a straight line indicating a threshold value that is compared with only the a value, and the straight line 68 is a straight line indicating a threshold value that is compared with values ​​corresponding to both the a value and the b value.

[0039] As shown in diagrams 58 and 60, in an image captured on a cloudy day, it is possible to suitably classify pixel values ​​corresponding to plant stocks and pixel values ​​corresponding to soil using the line 66 as a boundary. However, as shown in diagrams 58 and 62, in an image captured on a sunny day, it is not possible to suitably classify pixel values ​​corresponding to plant stocks and pixel values ​​corresponding to soil using the line 66 as a boundary.

[0040] Therefore, by using the straight line 68 as the boundary, it becomes possible to suitably classify pixel values ​​corresponding to plant stocks and pixel values ​​corresponding to soil in an image captured on a sunny day. That is, in the example shown in each diagram of FIG. 4, when the pixel value of a pixel to be processed is located in the upper left region of the straight line 68 on the a-b plane, the extraction unit 18 determines that the pixel corresponds to a plant stock. Also, when the pixel value of a pixel to be processed is located in the lower right region of the straight line 68 on the a-b plane, the extraction unit 18 determines that the pixel corresponds to an object other than a plant stock, such as soil.

[0041] Moreover, the value of b / a indicating the gradient of the straight line 68 is not limited to a specific value, and may be 1.15, 1.5, or the like. From another perspective, generalizing the straight line 68 is equivalent to giving a threshold value to the value of (b-β) / a, where β is any real number. Moreover, at least one of the threshold values ​​may be represented by a curve. Moreover, in the a-b plane of the Lab color space, at least one of the straight lines or curves indicating the threshold values ​​may have a b-intercept value different from 0, like the straight line 68.

[0042] In addition, the equation of the straight line or curve indicating the threshold value may be configured to be updateable by the learning unit 22 through machine learning using a pair of an image including soil and plant stocks and information indicating the area of ​​the plant stock as training data.

[0043] Following or in parallel with the process of extracting the region showing the plant in the image, the extraction unit 18 performs a process of masking the background of the region. That is, the extraction unit 18 performs a process of deleting color information or filling the region other than the region showing the plant in the image with a single color such as black.

[0044] Image 72 in FIG. 5 is an image showing a field of plant stocks on a sunny day, captured by a drone, which is a moving body 42. Image 74 shows a detection result of a plant stock when pixel values ​​corresponding to the plant stocks and pixel values ​​corresponding to the soil are classified using a threshold value that is compared only with the a value. In image 74, the soil part on the right side is erroneously detected as a plant stock. Image 76 shows a detection result of a plant stock when pixel values ​​corresponding to the plant stocks and pixel values ​​corresponding to the soil are classified by comparing values ​​corresponding to both the a value and the b value in image 72 with a predetermined threshold value. In image 76, an area corresponding to the upper left of the above-mentioned straight line 68 is extracted as an area showing the plant stock. The detection result of the plant stock shown in image 76 has a higher detection accuracy than the detection result shown in image 74.

[0045] In step S105, the extraction unit 18 divides the image of the field into areas for each plant strain so as to include an area showing a single plant strain. The control unit 12 also calculates the position of each plant strain in the field by referring to the image of the field and information showing the position of the moving object 42 when the image was captured.

[0046] In step S106, the classification unit 20 inputs an image including an area showing a single plant line, and classifies the plant line as being healthy or abnormal using a trained model that outputs a determination result regarding the presence or absence of abnormalities in the plant line.

[0047] In step S107, the control unit 12 performs branching for the target plant strain depending on whether the classification result by the classification unit 20 is a healthy strain or an abnormal strain. If the plant strain is a healthy strain, the process of step 109 is subsequently executed, and if the plant strain is an abnormal strain, the process of step S108 is subsequently executed.

[0048] In step S108, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the position of the plant strain classified by the classification unit 20 as an abnormal strain.

[0049] In step S109, the control unit 12 determines whether or not the processes in steps S106 to S108 have been completed for all of the target plant lines. If the control unit 12 determines that the processes have been completed for all of the plant lines, the process based on the flowchart in Fig. 2 ends. If the control unit 12 determines that the processes have not been completed for any of the plant lines, the process from S106 is executed for the target plant line.

[0050] As described above, the information processing method executed by the information processing device 10 includes an acquisition step, an extraction step, etc. In the acquisition step, an image in Lab color space that includes a plant strain is acquired. In the extraction step, a region showing the plant strain in the image is extracted by comparing a value corresponding to both the a-value and the b-value in the pixel values ​​of the image acquired in the acquisition step with a predetermined threshold value.

[0051] According to the configuration of this example, by comparing the values ​​corresponding to both the a-value and the b-value with a predetermined threshold value, it is possible to improve the accuracy of extracting plant strains in an image regardless of the light environment when the image was captured. In addition, since the positions of abnormal strains can be grasped without the need for an expert to visually check, it is possible to reduce the labor required for, for example, removing abnormal strains.

[0052] [Embodiment 2] The second embodiment of the present invention will be described below. For convenience of explanation, the same reference numerals are attached to components having the same functions as those described in the above embodiment, and redundant explanations will not be repeated. This also applies to the following embodiments. In this embodiment, an example of an information processing system capable of detecting a plant strain having a predetermined difference compared to other plant strains will be described. Here, the plant strain having a predetermined difference may include a plant strain that is relatively small in size, a wilted plant strain, or a plant strain with a different color compared to nearby plant strains. In addition, the above-mentioned nearby plant strains may include adjacent plant strains, or one or more plant strains among a predetermined number of plant strains within a predetermined distance or adjacent to a certain plant strain.

[0053] [1. Example of information processing system configuration] An information processing system 1a according to this embodiment will be described. Fig. 6 is a functional block diagram of an information processing device 10a according to this embodiment. The information processing device 10a is configured to include a detection unit 21 instead of the classification unit 20, as compared with the information processing device 10 shown in Fig. 1.

[0054] The detection unit 21 receives an image including a plurality of plant strains as input, and detects a plant strain having a predetermined difference in a target image compared to nearby plant strains using a trained model for detecting a plant strain having the difference in the image compared to nearby plant strains. Here, the trained model is a trained model trained using a set of an image including a plurality of plant strains and information regarding a difference between any one of the plurality of plant strains and other plant strains as training data. Here, the information regarding the difference may include, for example, information indicating the presence or absence of the difference, or information indicating what kind of difference there is.

[0055] The acquisition unit 14 according to the present embodiment acquires an image including a plurality of plant strains, and acquires information regarding a difference between any one of the plurality of plant strains included in the image and other plant strains.

[0056] Here, the information on the difference between a certain plant strain and another plant strain includes: Information about the difference between the size of one plant strain and another Information about the difference between the color of one plant strain and another Information about the differences between the shape of one plant strain and the shape of another etc. may be included.

[0057] In addition, the learning unit 22 in this embodiment receives an image including a plurality of plant strains as input, and trains a learning model that detects plant strains in the image that have a predetermined difference compared to nearby plant strains, using a pair of the image acquired by the acquisition unit 14 and information regarding the difference as training data.

[0058] Furthermore, the detection method performed by the detection unit 21 and the machine learning method performed by the learning unit 22 may be, for example, a method using Faster R-CNN (Regions with Convolutional Neural Networks).

[0059] Each image in Figure 7 shows a field of plant lines. The portion of image 80 enclosed in a box 82 corresponds to an image that would be used as training data for a learning model in a configuration other than that of the present embodiment. As shown in box 82, the image contains a single plant line.

[0060] On the other hand, the image serving as the training data in this embodiment includes a plurality of plant strains. As an example, an image portion surrounded by a frame 86 in an image 84 corresponds to the image serving as the training data for the learning model according to this embodiment. Also, an image 88 in FIG. 7 shows an example of a search result by the detection unit 21.

[0061] Furthermore, the other components in the information processing system 1a have the function of performing the processes described below.

[0062] [2. Examples of processing in information processing systems] Next, an example of a process in which the information processing system 1a presents the position of a plant line that has a predetermined difference compared to other plant lines will be described. Fig. 8 is a flowchart showing the flow of the process in this example.

[0063] In step S201, the moving object 42 captures one or more images including a plurality of plant plants around the moving object 42 with the imaging unit 44 while moving over the field of plant plants.

[0064] In step S102, the same process as in the first embodiment is executed.

[0065] In step S203, the preprocessing unit 16 performs processing to remove noise in the image by applying, for example, a Gaussian filter to the image acquired by the acquisition unit 14. Furthermore, the acquisition unit 14 acquires the image that has been subjected to the processing.

[0066] In step S204, the extraction unit 18 performs a process of extracting a region indicating a plant strain in the image acquired by the acquisition unit 14 and processed by the pre-processing unit 16. Following or in parallel with the process of extracting a region indicating a plant strain in the image, the extraction unit 18 also performs a process of masking a background portion of the region.

[0067] In step S205, the detection unit 21 uses the trained model described above to detect plant strains that have a predetermined difference compared to nearby plant strains in the image processed by the extraction unit 18.

[0068] Images 94 and 98 in FIG. 9 are examples of images including multiple plant strains. Images 96 and 100 respectively show detection results when images 94 and 98 are input to a trained model. As exemplified by image 96 and image 88 in FIG. 7, when multiple plant strains are adjacent to a certain plant strain, some of the detection positions may overlap. When multiple plant strains having a predetermined difference are adjacent to each other, a configuration may be used to detect a range including the multiple plant strains and plant strains that do not have the predetermined difference.

[0069] In step S206, the control unit 12a calculates the position of each plant strain that has a predetermined difference by referring to the image processed by the extraction unit 18 and information indicating the position of the moving body 42 at the time the image was captured.

[0070] In step S207, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the position of the plant strain, as information indicating the position of the plant strain having a predetermined difference.

[0071] In step S208, the control unit 12a judges whether the processes of steps S206 and S207 have been completed for all of the detected adjacent plant lines. If the control unit 12a judges that the processes have been completed for all of the plant lines, the process based on the flowchart in Fig. 8 ends. If the control unit 12a judges that the processes have not been completed for any of the plant lines, the process from S206 is executed for the target plant line.

[0072] As described above, the information processing method executed by the information processing device 10a includes a detection step, etc. In the detection step, an image including a plurality of plant strains is input, and a trained model for detecting a plant strain having a predetermined difference in the image compared to nearby plant strains is used to detect the plant strain having the difference in the target image compared to nearby plant strains. The trained model is trained using a set of an image including a plurality of plant strains and information regarding the difference between any one of the plurality of plant strains and other plant strains as training data.

[0073] The process of training the learning model according to the present embodiment includes an acquisition step and a learning step. In the acquisition step, an image including a plurality of plant strains and information regarding differences between any of the plurality of plant strains and other plant strains are acquired. In the learning step, an image including a plurality of plant strains is input, and a learning model that detects a plant strain having a predetermined difference in the image compared with nearby plant strains is trained using a pair of the image acquired in the acquisition step and the information regarding the difference as training data.

[0074] According to the configuration of this example, by using an image including a plurality of plant strains as training data, it becomes possible to detect a plant strain that has a predetermined difference compared to other plant strains.

[0075] In the above-mentioned steps S203 and S204, similarly to the first embodiment, the extraction unit 18 may extract an area indicating a plant strain by comparing values ​​corresponding to both the a-values ​​and the b-values ​​in the pixel values ​​of the image converted by the pre-processing unit 16 into an image in the Lab color space with a predetermined threshold value.

[0076] [Embodiment 3] A third embodiment of the present invention will be described below. In this embodiment, an example of an information processing system capable of extracting a region showing a plant leaf using edge information and brightness information extracted from an image including a plant stock will be described. Here, edge information means information showing the outline of an object included in an image. Moreover, brightness information means information showing the brightness of an object included in an image. The brightness information corresponds to a pixel value showing the brightness of a pixel showing an object included in an image. For example, the brightness information corresponds to the value of L in the above-mentioned Lab color space, or the value of Y in the YCbCr color space represented by the luminance signal Y and the color difference signals Cb and Cr. Moreover, as will be described later in FIG. 11, the edge information and brightness information can be expressed in the form of an image.

[0077] [1. Example of information processing system configuration] An information processing system 1b according to this embodiment will be described. Fig. 10 is a functional block diagram of an information processing device 10b according to this embodiment. The information processing device 10b is configured such that the extraction unit 18 includes a first extraction unit 23 and a second extraction unit 25, as compared with the information processing device 10 shown in Fig. 1.

[0078] The first extraction unit 23 receives an image including a plant stock as an input, and extracts edge information and brightness information of an object included in the image. The first extraction unit 23 may be configured to extract edge information using a second-order differential filter such as a Laplacian filter, a Sobel filter, or a Canny filter as an edge filter.

[0079] The second extraction unit 25 receives edge information and brightness information as input, and extracts a region representing a plant leaf from the image using a first trained model that extracts a region representing a plant leaf.

[0080] Each image in Fig. 11 indicates information related to the first extraction unit 23. Image 104 indicates an image including a plant strain and is input to the first extraction unit 23. Images 106 and 108 indicate brightness information and edge information, respectively, extracted by the first extraction unit 23. Image 110 indicates a composite image in which image 106 indicating brightness information and image 108 indicating edge information are combined into a single image.

[0081] The first trained model may be configured to receive a composite image, as exemplified by image 110. The process of generating the composite image may be performed by first extraction unit 23 or second extraction unit 25.

[0082] In addition, the acquisition unit 14 in this embodiment acquires teacher data from a memory unit 30, etc., which includes multiple pairs of edge information indicating the contour of an object included in an image including plant leaves, brightness information indicating the brightness of the object, and information indicating an area showing the plant leaves.

[0083] The learning unit 22 according to the present embodiment receives edge information and brightness information as input, and performs a process of learning a learning model for extracting a region showing a plant leaf, using as teacher data a set of the edge information and brightness information acquired by the acquisition unit 14 and information showing the region showing the plant leaf. The trained model trained by the process is an example of a first trained model in the present disclosure.

[0084] Furthermore, the other components in the information processing system 1b have the function of performing the processes described below.

[0085] [2. Examples of processing in information processing systems] Next, an example of a process in which the information processing system 1b presents the position of an abnormal leaf will be described below. Fig. 12 is a flowchart showing the flow of the process in this example.

[0086] In steps S101 and S102, the same processes as those in the first embodiment are executed.

[0087] In steps S203 and S204, the same processes as those in the second embodiment are executed.

[0088] In step S304, the first extraction unit 23 extracts edge information and brightness information of an object contained in the image acquired by the acquisition unit 14 and processed by the preprocessing unit 16.

[0089] In step S305, the second extraction unit 25 inputs the edge information and brightness information extracted by the first extraction unit 23 into the first learned model, and extracts an area showing plant leaves in the image processed by the pre-processing unit 16.

[0090] In step S306, the extraction unit 18 divides the image of the farm field into areas each including a single plant leaf, and the control unit 12b calculates the position of each plant leaf in the farm field by referring to the image of the farm field and information indicating the position of the moving object 42 when the image was captured.

[0091] In step S307, the classification unit 20 inputs an image including a region indicating a plant leaf extracted by the second extraction unit 25, the image including a region indicating a single plant leaf, into a second trained model targeting plant leaves, thereby classifying the plant leaf as either a healthy leaf or an abnormal leaf.

[0092] In step S308, the control unit 12b performs branching for the target plant leaf depending on whether the classification result by the classification unit 20 is a healthy leaf or an abnormal leaf. If the plant leaf is a healthy leaf, the process of step S310 is subsequently executed, and if the plant leaf is an abnormal leaf, the process of step S309 is subsequently executed.

[0093] In step S309, the presentation unit 24 displays, on the screen of the display device 38, a map of the farm field and an icon indicating the position of the plant leaf classified by the classification unit 20 as an abnormal leaf.

[0094] In step S310, the control unit 12b determines whether the processes in steps S307 to S309 have been completed for all the target plant leaves. If the control unit 12b determines that the processes have been completed for all the plant leaves, the process based on the flowchart in Fig. 12 ends. If the control unit 12b determines that the processes have not been completed for any of the plant leaves, the process from S307 is executed for the target plant leaf.

[0095] As described above, the information processing method executed by the information processing device 10b includes a first extraction step and a second extraction step, etc. In the first extraction step, an image including a plant stock is input, and edge information indicating the outline of an object included in the image and brightness information indicating the brightness of the object are extracted. In addition, in the second extraction step, the edge information and brightness information are input, and a region indicating a plant leaf in the image is extracted using a trained model that extracts a region indicating a plant leaf.

[0096] According to the configuration of this example, by using the extracted edge information and brightness information to extract areas showing plant stems and leaves, it is possible to improve the accuracy of extracting plant stems and leaves in an image regardless of the lighting environment when the image was captured.

[0097] In addition, the present disclosure also includes a configuration in which the extraction unit 18b does not include the first extraction unit 23, and the second extraction unit 25 acquires edge information and brightness information extracted by an external device different from the information processing system 1b and inputs it into the first learned model to extract an area indicating a plant leaf.

[0098] [Modifications of the First to Third Embodiments] The configurations described above in the first to third embodiments are not limited to targeting plant stocks and leaves. In the first and third embodiments, for example, trash on the road, animals, or traffic signs may be the targets. Also, for example, a person carrying a terminal device having a camera and a GPS function may act as a mobile object 42, and the location of the trash or the like may be displayed on a map in real time on a display device 38 in a remote location. Also, in the second embodiment, for example, detection may be performed on industrial products and other products.

[0099] [Software implementation example] The functions of the information processing device (10, 10a, 10b) (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as the device, and a program for causing a computer to function as each control block of the device (particularly each part included in the control unit (12, 12a, 12b)).

[0100] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program to realize each function described in each of the above embodiments.

[0101] The program may be non-transitory and may be recorded in one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be provided to the device via any wired or wireless transmission medium.

[0102] In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of the present invention. In addition, the functions of each of the control blocks can be realized by, for example, a quantum computer.

[0103] Furthermore, each process described in each of the above embodiments may be executed by AI (Artificial Intelligence). In this case, the AI ​​may be executed by the control device or another device (for example, an edge computer or a cloud server).

[0104] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. [Explanation of symbols]

[0105] 1, 1a, 1b Information Processing System 10, 10a, 10b Information processing device 12, 12a, 12b Control section 14 Acquisition Department 16 Pretreatment section 18 Extraction part 20 Classification section 21 Detection unit 22 Learning Department 23 1st extraction part 24 Presentation section 25 Second extraction part 30 Storage section 34, 46 Communications Department 38 Display device 42 Mobile 44 Imaging unit

Claims

1. a first extraction unit that receives an image including a plant strain as input and extracts edge information indicating an outline of an object included in the image and brightness information indicating a brightness of the object; a second extraction unit that extracts a region indicating a plant leaf from the image using a first trained model that receives the edge information and the brightness information as input and extracts a region indicating a plant leaf; a classification unit that classifies the plant leaf as a healthy leaf or an abnormal leaf using a second trained model that receives an input of an image including a region indicating the plant leaf extracted by the second extraction unit and outputs a determination result regarding the presence or absence of an abnormality in the plant leaf; a presentation unit that presents information indicating the positions of plant leaves that have been classified as abnormal leaves by the classification unit; An information processing device comprising:

2. The present invention further includes a learning unit that uses a pair of an image including a plant leaf and information regarding the presence or absence of an abnormality in the plant leaf as training data to learn a learning model that receives an image including a plant leaf as an input and outputs a determination result regarding the presence or absence of an abnormality in the plant leaf.

2. The information processing apparatus according to claim 1,

3. The first extraction unit is The edge information is extracted using a second-order differential filter as an edge filter.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. an acquisition unit that acquires edge information indicating an outline of an object included in an image including a plant leaf, brightness information indicating the brightness of the object, and information indicating an area indicating the plant leaf; a learning unit that receives the edge information and the brightness information as input, and learns a learning model for extracting a region indicating a plant leaf, using a set of the edge information and the brightness information acquired by the acquisition unit, and information indicating the region indicating the plant leaf, as teacher data; The information processing apparatus according to claim 1 , further comprising:

5. A moving object including an imaging unit; The information processing device according to claim 1 , wherein the image captured by the imaging unit is input to the first extraction unit. An information processing system comprising:

6. An information processing method executed by an information processing device, a first extraction step of extracting edge information indicating an outline of an object included in the image and brightness information indicating a brightness of the object using an image including a plant strain as an input; a second extraction step of extracting a region indicating a plant leaf from the image using a first trained model that receives the edge information and the brightness information as input and extracts a region indicating a plant leaf; a classification step of classifying the plant leaf as a healthy leaf or an abnormal leaf using a second trained model that receives an image including a region indicating the plant leaf extracted in the second extraction step and outputs a determination result regarding the presence or absence of an abnormality in the plant leaf; a display step of displaying information indicating the positions of the plant leaves classified as abnormal leaves in the classification step; 13. An information processing method comprising:

7. A control program for causing a computer to function as the information processing device according to claim 1 , the control program causing a computer to function as the first extraction unit, the second extraction unit, the classification unit, and the presentation unit.

8. A computer-readable recording medium having the control program according to claim 7 recorded thereon.

Images