Information processing device and information processing method

Abstract

To provide a technique for enabling processing by a learning model according to a situation even if processing is difficult based on only information collected in the past, or even if there is no information collected in the past.SOLUTION: One or more learning models are selected, as candidate learning models, from a plurality of learning models learned under learning environments different from each other based on information concerning image capturing of an object. One or more candidate learning models are selected from the candidate learning models based on a result of object detection processing by the selected candidate learning models. The object detection processing is performed using a candidate learning model of at least one of the selected candidate learning models.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for prediction based on captured images. [Background technology]

[0002] In recent years, there have been many efforts to use IT to solve a variety of problems in agriculture, such as predicting yields, predicting optimal harvest times, controlling the amount of pesticide sprayed, and planning field restoration.

[0003] For example, Patent Document 1 discloses a method for early detection and response to abnormal growth conditions by appropriately referring to sensor information acquired from a field where agricultural crops are grown and a database in which that information is stored, thereby grasping growth conditions and harvest predictions at an early stage.

[0004] Furthermore, Patent Document 2 discloses a method for field management that reduces variation in the quality and yield of agricultural crops by making arbitrary inferences based on information acquired from a wide variety of agricultural crop-related sensors and referring to registered information. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2005-137209 A [Patent Document 2] JP 2016-49102 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, the methods proposed so far have been based on the premise that a sufficient number of previously acquired cases for the field for which predictions are to be performed are stored and that adjustments have been made so that predictions can be made with high accuracy based on information about those cases.

[0007] On the other hand, the success or failure of agricultural crops is generally greatly affected by environmental fluctuations such as weather and climate, and also varies greatly depending on the spraying of fertilizers and pesticides by workers. If all conditions due to external factors remained constant from year to year, there would be no need to predict yields or harvest times, but unlike industry, agriculture has many external factors that workers cannot control, making predictions extremely difficult. Also, when predicting yields when unprecedented weather conditions continue, it is difficult to make accurate predictions using the estimation system adjusted from the above-mentioned past cases.

[0008] The most difficult case to predict is when the above prediction system is newly introduced to a field. For example, consider the case of predicting yield in a specific field or detecting non-productive areas for the purpose of repairing poorly growing areas (dead branches, diseased areas). In such a task, images and parameters of crops collected in the past in the field are usually stored in a database. Then, when actually carrying out predictions on the field, images taken in the current field and data related to growth information obtained from other sensors are cross-referenced and adjusted to make a highly accurate prediction. However, as mentioned above, when these prediction systems and non-productive area detectors are introduced to a different new field, they cannot be applied immediately because the conditions (of the field) often do not match. In such a case, it was necessary to collect a sufficient amount of data in the new field and make adjustments.

[0009] In addition, when the above-mentioned prediction system or non-productive area detector is adjusted manually, it takes a lot of time because the parameters related to crop growth are high-dimensional. Even when deep learning or similar machine learning methods are used, manual labeling (annotation) is usually required to achieve good performance for new input, which results in high work costs.

[0010] Ideally, when introducing a new forecasting system or in the case of a natural disaster or weather that has never occurred before, it would be desirable to be able to make good forecasts and estimates with simple settings that place a low burden on the user.

[0011] The present invention provides technology that enables processing using a learning model appropriate to the situation, even when processing is difficult using only previously collected information or when no previously collected information is available. [Means for solving the problem]

[0012] One aspect of the present invention includes a first selection means for selecting one or more learning models as candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing an object; A second selection means for selecting one or more candidate learning models from the candidate learning models selected by the first selection means based on a result of the object detection process using the candidate learning models; a detection means for performing an object detection process on a photographed image of the object by using at least one of the candidate learning models selected by the second selection means; 、 A means for predicting a yield of a crop and detecting a repair area in a farm field based on an object detection area obtained as a result of the object detection process; The present invention is characterized by comprising: Effect of the Invention

[0013] According to the configuration of the present invention, even when processing is difficult using only previously collected information or when no previously collected information is available, processing can be performed using a learning model suited to the situation. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration. [Figure 2A] 4 is a flowchart of a process performed by the system. [Figure 2B] 10 is a flowchart showing details of the process in step S23. [Figure 2C] 10 is a flowchart showing details of the process in step S233. [Diagram 3] 1 is a diagram showing an example of a method for photographing a farm field using a camera 10. FIG. [Figure 4] Diagram showing a difficult case. [Diagram 5] FIG. 13 is a diagram showing the results of annotation work performed on a captured image. [Figure 6] FIG. 13 is a diagram showing an example of a GUI display. [Figure 7] FIG. 13 is a diagram showing an example of a GUI display. [Figure 8A] 4 is a flowchart of a process performed by the system. [Figure 8B] 10 is a flowchart showing details of the process in step S83. [Figure 8C] 10 is a flowchart showing details of the process in step S833. [Figure 9] FIG. 4 is a diagram showing a detection example of a detection region. [Figure 10] FIG. 13 is a diagram showing an example of a GUI display. [Figure 11] 1A is a diagram showing an example of the configuration of a query parameter, FIG. 1B is a diagram showing an example of the configuration of a parameter set of a learning model, and FIG. 1C is a diagram showing an example of the configuration of a query parameter. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0016] [First embodiment] In this embodiment, a system is described that performs analysis processing of a farm field, such as predicting the yield of agricultural crops in the field and detecting areas that need repair, based on an image of the field captured by a camera.

[0017] First, an example of the configuration of a system according to this embodiment will be described with reference to Fig. 1. As shown in Fig. 1, the system according to this embodiment includes a camera 10, a cloud server 12, and an information processing device 13.

[0018] First, the camera 10 will be described. The camera 10 captures a moving image of the field, and outputs the image of each frame in the moving image as a "captured image of the field". Alternatively, the camera 10 captures still images of the field periodically or irregularly, and outputs the captured still images as a "captured image of the field". In order to perform accurate predictions, which will be described later, from the captured images, it is desirable that images captured in the same field are captured in the same environment and conditions as much as possible. The captured images output from the camera 10 are transmitted to a cloud server 12 or an information processing device 13 via a communication network 11, such as a LAN or the Internet.

[0019] The method of photographing the field by the camera 10 is not limited to a specific photographing method. An example of the method of photographing the field by the camera 10 will be described with reference to FIG. 3(A). In FIG. 3(A), cameras 33 and 34 are used as the camera 10. In a typical field, rows of crop trees are planted by farmers in a planned manner. For example, as shown in FIG. 3(A), rows of crop trees are planted side by side, such as a row of crop trees 30 and a row of crop trees 31. The farm tractor 32 is provided with a camera 34 that photographs the row of crop trees 31 on the left side in the traveling direction indicated by the arrow, and a camera 33 that photographs the row of crop trees 30 on the right side. Therefore, when the farm tractor 32 moves between the rows 30 and 31 in the traveling direction indicated by the arrow, the camera 34 will photograph multiple images of the crop trees in the row 31, and the camera 33 will photograph multiple images of the crop trees in the row 30.

[0020] In many farm fields designed for agricultural tractors 32 to enter and work in, where agricultural trees are planted at equal intervals, it is relatively easy to photograph a larger number of agricultural trees at a constant height and at a constant distance from the agricultural trees by photographing the agricultural trees with cameras 33, 34 installed on the agricultural tractor 32 as shown in Fig. 3(A). This makes it possible to photograph the entire target farm field under almost the same conditions, and image capture under desirable conditions can be easily achieved.

[0021] However, other photographing methods may be used as long as the photographing of the field can be performed under roughly the same conditions. An example of a method of photographing a field using the camera 10 will be described with reference to FIG. 3(B). In FIG. 3(B), cameras 38 and 39 are used as the camera 10. As shown in FIG. 3(B), in a field or the like where the interval between the row 35 of the crop trees and the row 36 of the crop trees is narrow and travel by a tractor is impossible, photographing may be performed using the camera 38 and the camera 39 attached to the drone 37. The drone 37 is provided with a camera 39 that photographs the row 36 of the crop trees on the left side in the traveling direction indicated by the arrow, and a camera 38 that photographs the row 35 of the crop trees on the right side. Therefore, when the drone 37 moves between the rows 35 and 36 in the traveling direction indicated by the arrow, the camera 39 will take multiple photographed images of the crop trees in the row 36, and the camera 38 will take multiple photographed images of the crop trees in the row 35.

[0022] In addition, the images of the crop trees may be taken by a camera installed on the self-propelled robot. Although the number of cameras used for taking the images is two in Figures 3(A) and (B), the number is not limited to a specific number.

[0023] Regardless of the method used to capture an image of a crop tree, camera 10 outputs the captured image together with the shooting information at the time the image was captured (Exif information recording the shooting location (e.g., the shooting location measured by GPS), the shooting date and time, information related to camera 10, etc.).

[0024] Next, the cloud server 12 will be described. In the cloud server 12, the captured images and Exif information transmitted from the camera 10 are registered. In addition, in the cloud server 12, a plurality of learning models (detectors / settings) for detecting image areas related to agricultural products from the captured images are registered, and each learning model is a model that has been trained in a mutually different learning environment. In addition, the cloud server 12: Self From among the multiple learning models held by the image processing device 13, a candidate learning model to be used when detecting an image area relating to agricultural crops from a captured image is selected and presented to the information processing device 13.

[0025] CPU 191 executes various processes using computer programs and data stored in RAM 192 and ROM 193. In this way, CPU 191 controls the overall operation of cloud server 12, and executes or controls various processes that will be described as being performed by cloud server 12.

[0026] The RAM 192 has an area for storing computer programs and data loaded from the ROM 193 or the external storage device 196, and an area for storing data received from the outside via the I / F 197. The RAM 192 further has a work area used when the CPU 191 executes various processes. In this way, the RAM 192 can provide various areas as needed.

[0027] ROM 193 stores setting data for cloud server 12, computer programs and data related to the startup of cloud server 12, computer programs and data related to the basic operation of cloud server 12, and the like.

[0028] The operation unit 194 is a user interface such as a keyboard, a mouse, a touch panel, etc., and the user can input various instructions to the CPU 191 by operating it.

[0029] Display unit 195 has a screen such as a liquid crystal screen or a touch panel screen, and can display the results of processing by CPU 191 as images, characters, etc. Note that display unit 195 may be a projection device such as a projector that projects images and characters.

[0030] The external storage device 196 is a large-capacity information storage device such as a hard disk drive device. The external storage device 196 stores an OS (operating system) and computer programs and data for causing the CPU 191 to execute or control various processes described as being performed by the cloud server 12. The data stored in the external storage device 196 includes data related to the above-mentioned learning model. The computer programs and data stored in the external storage device 196 are loaded into the RAM 192 as appropriate under the control of the CPU 191, and become targets for processing by the CPU 191.

[0031] I / F 197 is a communication interface for performing data communication with the outside, and cloud server 12 transmits and receives data to and from the outside via I / F 197. CPU 191, RAM 192, ROM 193, operation unit 194, display unit 195, external storage device 196, and I / F 197 are all connected to system bus 198. Note that the configuration of cloud server 12 is not limited to the configuration shown in FIG.

[0032] In addition, the captured image and Exif information output from the camera 10 may be temporarily stored in the memory of another device, and the captured image and Exif information may be transferred from the memory to the cloud server 12 via the communication network 11.

[0033] Next, the information processing device 13 will be described. The information processing device 13 is a computer device such as a PC (personal computer), a smartphone, or a tablet terminal device. The information processing device 13 presents candidates for learning models presented by the cloud server 12 to the user, accepts the selection of a learning model from the user, and notifies the cloud server 12 of the learning model selected by the user. The cloud server 12 uses the learning model notified by the information processing device 13 (the learning model selected by the user from the candidates) to detect an image area related to agricultural products from the image captured by the camera 10 (object detection processing), and performs the above analysis processing.

[0034] The CPU 131 performs various processes using computer programs and data stored in the RAM 132 and the ROM 133. As a result, the CPU 131 controls the operation of the entire information processing device 13, and also executes or controls various processes that will be described as being performed by the information processing device 13.

[0035] The RAM 132 has an area for storing computer programs and data loaded from the ROM 133, and an area for storing data received from the camera 10 and the cloud server 12 via the input I / F 135. The RAM 132 further has a work area used when the CPU 131 executes various processes. In this way, the RAM 132 can provide various areas as needed.

[0036] The ROM 133 stores setting data for the information processing device 13, computer programs and data related to the startup of the information processing device 13, computer programs and data related to the basic operation of the information processing device 13, and the like.

[0037] The output I / F 134 is an interface used by the information processing device 13 to output / transmit various types of information to the outside.

[0038] The input I / F 135 is an interface used by the information processing device 13 to input / receive various types of information from the outside.

[0039] The display device 14 has a liquid crystal screen or a touch panel screen, and can display the results of processing by the CPU 131 as images, characters, etc. The display device 14 may be a projection device such as a projector that projects images and characters.

[0040] The user interface 15 includes a keyboard and a mouse, and can be operated by the user to input various instructions to the CPU 131. Note that the configuration of the information processing device 13 is not limited to the configuration shown in Fig. 1, and may include, for example, a large-capacity information storage device such as a hard disk drive device, in which computer programs such as a GUI (to be described later) and data are stored. The user interface 15 may also include a touch sensor such as a touch panel.

[0041] Next, a task flow for predicting the yield of agricultural products harvested in a farm field at an earlier stage than the harvest time from an image of the farm field captured by the camera 10 will be described. When predicting the yield by simply counting the fruits to be harvested at the harvest time, the purpose can be achieved by simply detecting the target fruits from the captured image using a classifier by a method called specific object detection. Since the fruits themselves have a very distinctive appearance, this method detects them using a classifier that has learned this distinctive appearance.

[0042] In this embodiment, when the crop is a fruit, the yield of the fruit is predicted at an earlier stage than the harvest time, in addition to counting the fruits after they have ripened. For example, the yield is predicted from the number of inflorescences that will later become fruits, the yield is predicted by detecting dead branches or diseased areas that are unlikely to bear fruit, or the yield is predicted from the state of leaf growth. In order to make such predictions, a prediction method that can handle the fact that the growth conditions of the crops differ depending on the shooting time and climate is required. In other words, it is necessary to select a prediction method with good prediction performance depending on the condition of the crops. In this case, it is expected that the above predictions will be made appropriately using a learning model that matches the field to be predicted.

[0043] Here, various objects captured in captured images are classified into classes such as the trunk class, branch class, dead branch class, and support class of agricultural crops, and the yield is predicted according to the class. Since the appearance of objects belonging to classes such as the trunk class and branch class changes depending on the time of shooting, it is difficult to make a universal prediction. Figure 4 shows such a difficult example.

[0044] 4(A) and 4(B) show examples of images captured by the camera 10. These captured images show crop trees at approximately equal intervals, but the fruits to be harvested have not yet borne fruit, so the task of detecting fruits cannot be performed from the captured images. The tree in the captured image of FIG. 4(A) is a crop tree captured at a relatively early stage of the season, and the tree in the captured image of FIG. 4(B) is a tree captured at a stage when the leaves have grown to a certain extent. In the captured image of FIG. 4(A), all the branches have the same amount of leaves, so it can be determined that there are no areas of poor growth, and all areas can be determined as harvestable. On the other hand, in the captured image of FIG. 4(B), the leaves of the branches near the central region 41 in the captured image are clearly different from the others, so it is easy to determine that they are poorly grown. However, the appearance of the central region 41 (an area with few leaves) can also be found as a similar pattern near the region 40 in the captured image of FIG. 4(A). These two examples show that abnormal regions of agricultural trees cannot be identified by local patterns. In other words, it is not possible to make a judgment based only on local patterns, as in the specific object detection method described above, and it is necessary to reflect the context obtained from the entire image.

[0045] In other words, unless the above specific object detection is performed using a learning model that has been trained using images of crops in similar growth conditions photographed in the past, sufficient performance will not be achieved.

[0046] In order to handle all kinds of cases, such as when an image is input that was taken in a new field where no images have been taken before, when an image is input that was taken under conditions different from those previously taken due to some external factor such as prolonged drought or extremely heavy rainfall, or when an image is input that was taken at a time convenient for the user, it is necessary to obtain a learning model that is trained under conditions close to those of the input image each time.

[0047] Here, we will explain what kind of annotation work is required when annotation work and deep learning are performed every time a farm field is photographed. For example, the results of annotation work performed on the photographed images in Figures 4(A) and 4(B) are shown in Figures 5(A) and 5(B), respectively.

[0048] Rectangular regions 500-504 in the captured image in FIG. 5(A) are image regions designated by annotation work. Rectangular region 500 is an image region designated as a region of normal branches, and rectangular regions 501-504 are image regions designated as a region of the tree trunk. Rectangular region 500 is an image region that represents a normal state of tree growth, and therefore this image region is an area that is highly relevant to yield prediction. Hereinafter, an area that represents a normal state of tree growth, such as rectangular region 500, and an area of ​​a part where fruit or the like can be harvested, will be referred to as a production area.

[0049] Rectangular regions 505-507 and 511-514 in the captured image in FIG. 5(B) are image regions designated by annotation work. Rectangular regions 505 and 507 are image regions designated as normal branch regions, and rectangular region 506 is an image region designated as an abnormal dead branch region. Regions that indicate abnormal conditions, such as rectangular region 506, and regions where fruit or the like cannot be harvested are referred to as non-productive regions. Rectangular regions 511-514 are image regions designated as tree trunk regions. The image regions that are determined to be regions where fruit or the like can be harvested (productive regions) are rectangular regions 505 and 507, and therefore rectangular regions 505 and 507 are regions that are closely related to yield prediction.

[0050] It is very costly to perform such annotation work on a large number of captured images (e.g., hundreds to thousands) each time a farm field is photographed. Therefore, in this embodiment, good prediction results are obtained without performing such more cumbersome annotation work. In this embodiment, a learning model is acquired by deep learning. However, the method of acquiring the learning model is not limited to a specific acquisition method. Also, various object detectors may be applied instead of the learning model.

[0051] Next, the processing performed by the system according to this embodiment to perform analytical processing such as predicting the yield in a field and calculating the non-productive rate for the entire field based on images of the field captured by the camera 10 will be described with reference to the flowchart in FIG. 2A.

[0052] In step S20, the camera 10 captures an image of the farm field while a mobile object such as a farm tractor 32 or a drone 37 is moving, thereby generating a captured image of the farm field.

[0053] In step S21, the camera 10 attaches the above-mentioned Exif information (shooting information) to the captured image generated in step S20, and transmits the captured image with the Exif information attached to the cloud server 12 and the information processing device 13 via the communication network 11.

[0054] In step S22, the CPU 131 of the information processing device 13 acquires information on the field and the crops photographed by the camera 10 (such as the variety and age of the crops, and the cultivation and pruning methods of the crops) as photographed field parameters. For example, the CPU 131 displays a GUI (Graphical User Interface) shown in Fig. 6(A) on the display device 14 and accepts input of the photographed field parameters from the user.

[0055] In the GUI of Fig. 6(A), a map of the entire farm field is displayed in area 600. The map of the farm field displayed in area 600 is divided into a number of sections, and each section is displayed with its own unique identifier (ID). The user operates user interface 15 to specify a location in area 600 that corresponds to the section photographed by camera 10 (i.e., the section on which the above-mentioned analysis processing is to be performed), or inputs the identifier of the section in area 601. When the user operates user interface 15 to specify a location in area 600 that corresponds to the section photographed by camera 10, the identifier of the section is displayed in area 601.

[0056] The user can operate the user interface 15 to input a crop name (the name of the agricultural product) in an area 602. The user can also operate the user interface 15 to input a variety of the agricultural product in an area 603. The user can also operate the user interface 15 to input a Trellis in an area 604. For example, if the agricultural product is grapes, a Trellis is a method of designing grape vines for growing grapes in a grape field. The user can also operate the user interface 15 to input a Planted Year in an area 605. For example, if the agricultural product is grapes, the Planted Year indicates the time when the grape vines were planted. Note that it is not essential to input the photography field parameters for all of these items.

[0057] When the user operates the user interface 15 to select a registration button 606, the CPU 131 of the information processing device 13 transmits the photographed farm field parameters of each item input in the GUI of Fig. 6(A) to the cloud server 12. The CPU 191 of the cloud server 12 stores (registers) the photographed farm field parameters transmitted from the information processing device 13 in the external storage device 196.

[0058] Furthermore, when the user operates the user interface 15 to instruct a correction button 607, the CPU 131 of the information processing device 13 enables correction of the photographed field parameters that have already been input in the GUI of FIG. 6(A).

[0059] The GUI in Fig. 6(A) is a GUI for inputting photographed field parameters, particularly assuming management of a grape field, but even if the purpose is the same, the photographed field parameters input by the user are not limited to those shown in Fig. 6(A). Similarly, even if the crop is not grapes, the photographed field parameters input by the user are not limited to those shown in Fig. 6(A). For example, when the crop name input in the area 602 is changed, the titles of the areas 603 to 605 and the photographed field parameters to be input may be changed.

[0060] Since the photographed field parameters entered in the GUI in Fig. 6(A) can basically be used as fixed parameters once they have been determined, for example, when photographing a field every year to predict yields, the already registered photographed field parameters can be called and used. If the photographed field parameters have already been registered for a desired division, the next time, as shown in Fig. 6(B), the photographed field parameters corresponding to the desired division can be displayed in regions 609 to 613 by pointing to a location in region 600 that corresponds to the desired division.

[0061] Here, it is desirable to input all correct photographed field parameters in order to select a learning model at a later stage, but even if there are photographed field parameters that could not be input because they are unknown to the user, subsequent processing can be carried out while they remain unknown.

[0062] In step S23, a process is performed to select candidates for a learning model to be used for detecting an object such as a crop from a captured image. Details of the process in step S23 will be described with reference to the flowchart in FIG. 2B.

[0063] In step S230, the CPU 191 of the cloud server 12 generates query parameters from the Exif information attached to each captured image obtained from the camera 10 and the captured field parameters (the captured field parameters of the section corresponding to the captured image) registered in the external storage device 196.

[0064] An example of the configuration of the query parameters is shown in Fig. 11(A). The query parameters in Fig. 11(A) are generated when the photographed farm field parameters in Fig. 6(B) are input.

[0065] The "query name" is set to "F5" input in the area 609. The "variety" is set to "Shiraz" input in the area 611. The "Trellis" is set to "Scott-Henry" input in the area 612. The "age" is set to the number of years elapsed from "2001" input in the area 613 to the shooting date and time (year) included in the Exif information as the age of the tree "19". The "shooting date" is set to "Oct 20", the shooting date and time (month and day) included in the Exif information. The "shooting time zone" is set to "12:00-14:00", the time zone between the oldest shooting date and time (time) and the most recent shooting date and time (time) among the shooting dates and times (times) in the Exif information attached to each captured image received from the camera 10. The "latitude, longitude" is set to the shooting position "35°28'S, 149°12'E" included in the Exif information.

[0066] The method of generating the query parameters is not limited to the above method. For example, data that a farmer of the crop has already used for field management may be read, and a set of parameters that match the above items may be used as the query parameters.

[0067] In some cases, information about some items may be unknown. For example, if information about the planted year or variety is unknown, it is not possible to fill in all items as shown in Figure 11(A). In this case, some of the query parameters will be left blank, as shown in Figure 11(C).

[0068] Next, in step S231, the CPU 191 of the cloud server 12 selects M (1 ≤ M < E) candidate learning models (candidate learning models) from among the E learning models (where E is an integer of 2 or more) stored in the external storage device 196. In this selection, a learning model trained based on an environment similar to the environment indicated by the query parameter is selected as a candidate learning model. In the external storage device 196, for each of the E learning models, a parameter set indicating the environment based on which the learning model was trained is stored. A configuration example of the parameter set for each learning model in the external storage device 196 is shown in FIG. 11(B).

[0069] "Model Name" is the name of the learning model, "Variety" is the variety of the agricultural crop on which the learning model was trained, "Trellis" is the "method for designing grapevines for growing grapes in a vineyard" on which the learning model was trained. "Tree Age" is the age of the agricultural crop on which the learning model was trained, and "Shooting Date" is the date and time of shooting of the captured image of the agricultural crop used for learning by the learning model. "Shooting Time Zone" is the period from the oldest shooting date and time to the most recent shooting date and time among the captured images of the agricultural crop used for learning by the learning model, and "Latitude, Longitude" is the shooting position "35°28’S, 149°12’E" of the captured image of the agricultural crop used for learning by the learning model.

[0070] Depending on the learning model, there are also those that learn by mixing datasets collected from blocks in multiple farms. Therefore, for example, there may be cases where the parameter set is set to include multiple settings (such as variety and tree age) like the learning models with model names "M004" and "M005".

[0071] Therefore, the CPU 191 of the cloud server 12 calculates the similarity between the query parameter and the parameter set for each learning model shown in FIG. 11(B), and selects the top M learning models in descending order of the similarity as candidate learning models.

[0072] The parameter sets of each learning model with model names M001, M002, ... are 1 ,M 2 , ..., the CPU 191 of the cloud server 12 receives a query parameter Q and a parameter set M x The similarity D(Q,M x ) is calculated using the following formula (1).

[0073]

number

[0074] Here, q k represents the k-th element from the top in the query parameter Q. In the case of FIG. 11(A), the query parameter Q includes six elements, namely, “variety”, “Trellis”, “age”, “photographed date”, “photographed time zone”, and “latitude, longitude”, so k=1 to 6.

[0075] m x、k is the parameter set M x 11B, the parameter set includes six elements: "variety", "Trellis", "age", "photographed date", "photographed time zone", and "latitude, longitude", so k=1 to 6.

[0076] f k (a k , b k ) is element a k and b k This is a function for calculating the distance between k (a k , b k ) may be carefully set in advance through experiments, but the distance definition in the above formula (1) should basically be set to a larger value for learning models with different characteristics, so it can be simply set as follows.

[0077] In other words, elements are basically divided into two types: classification elements (variety, Trellis) and continuous value elements (age of tree, date of photography, etc.). Therefore, the function that specifies the distance between classification elements is defined as in the following formula (2), and the function that specifies the distance between continuous value elements is defined as in the following formula (3).

[0078]

number

[0079]

number

[0080] The functions for all elements (k) are implemented in advance as a rule-based function. In addition, α is set according to the influence of each element on the final inter-model distance. k For example, the difference due to the "variety" (k=1) does not appear much in the image, so α 1 As α approaches 0, the difference in "Trellis" (k=2) has a large effect. 2 Adjust it in advance, for example, by setting it large.

[0081] In addition, in the case of a learning model in which multiple settings are registered for "variety" or "age of tree," such as the learning models with model names "M004" and "M005" in Fig. 11(B), for example, in the case of "variety," distances are calculated for each setting registered for "variety," and the average distance is regarded as the distance corresponding to "variety." Similarly, in the case of "age of tree," distances are calculated for each setting registered for "age of tree," and the average distance is regarded as the distance corresponding to "age of tree."

[0082] In addition, the selection method is not limited to a specific method as long as the CPU 191 of the cloud server 12 selects M learning models as candidate learning models based on the above similarity. For example, the CPU 191 of the cloud server 12 may select M learning models having a similarity equal to or greater than a threshold.

[0083] However, if all elements in the query parameters are empty, the process in step S231 is not performed, and as a result, the subsequent processes are performed with all learning models as candidate learning models.

[0084] The effects of selecting candidate learning models are manifold. First, by eliminating learning models with low probability as prior knowledge in this step, the processing time for subsequent steps such as ranking by scoring learning models can be significantly reduced. Also, even in rule-based scoring of learning models, if learning models that do not need to be compared are included as candidates, there is a possibility that the accuracy of the learning model selection will decrease, but this possibility can be minimized.

[0085] Next, in step S232, the CPU 191 of the cloud server 12 selects P (P is an integer equal to or greater than 2) captured images from the captured images received from the camera 10 as model selection target images. The method of selecting P captured images from the captured images received from the camera 10 is not limited to a specific selection method. For example, the CPU 191 may randomly select P captured images from the captured images received from the camera 10, or may select them according to some criteria.

[0086] Next, in step S233, a process is performed to select one of the M candidate learning models as a selected learning model using the P captured images selected in step S232. Details of the process in step S233 will be described with reference to the flowchart in FIG. 2C.

[0087] In step S2330, the CPU 191 of the cloud server 12 performs, for each of the M candidate learning models, "an object detection process that is a process of detecting an object from each of the P captured images using the candidate learning model."

[0088] As a result, for each of the P captured images, the "result of the object detection process for the captured image" of each of the M candidate learning models is obtained. In this embodiment, the "result of the object detection process for the captured image" is position information of the image area (rectangular area, detection area) of the object detected from the captured image.

[0089] In step S2331, the CPU 191 obtains a score for the "result of the object detection process for each of the P captured images" for each of the M candidate learning models. Then, the CPU 191 ranks the M candidate learning models based on the scores, and selects N (N≦M) candidate learning models from the M candidate learning models.

[0090] In this case, since the captured image does not have annotation information, accurate evaluation of detection accuracy is not possible. However, for targets that are systematically designed and maintained, such as farms, it is possible to predict and evaluate the accuracy of the object detection process using the following rules. The score for the result of the object detection process by the candidate learning model is calculated, for example, as follows.

[0091] In a typical farm field, crops are planted at equal intervals as shown in Figures 3(A) and 3(B). Therefore, when detecting an object such as the annotation (rectangular area) shown in Figures 5(A) and 5(B), the detection is normal if the rectangular area is always detected continuously and evenly from the left edge to the right edge of the image.

[0092] For example, in the case where the entire area from the left end to the right end of the photographed image is detected as an area where fruits or the like can be harvested, as in FIG. 5(A), a production area should be detected as rectangular area 500. Also, in the case where a rectangular area 506 that is a non-production area is present in the photographed image, as in FIG. 5(B), rectangular areas 505, 506, and 507 should be detected from the left end to the right end of the photographed image. If an object detection process is performed on the photographed image using a learning model that does not match the conditions of the photographed image, there is a possibility that some of the above rectangular areas will not be detected. The more the learning model corresponds to conditions that are farther from the conditions of the photographed image, the higher the possibility of this happening. Therefore, the simplest scoring method for evaluating candidate learning models can be, for example, the following method.

[0093] The target candidate learning model detects detection areas of a plurality of objects from the target photographed image. Therefore, the target photographed image is searched for a detection area in the vertical direction to count the number of pixels Cp in the area where the detection area is not present, and the ratio of the number of pixels Cp to the number of pixels in the width of the target photographed image is set as the penalty score of the target photographed image. In this way, a penalty score is obtained for each of the P photographed images on which the target candidate learning model has performed object detection processing, and the total value of the obtained penalty scores is set as the score of the target candidate learning model. By performing such processing for each of the M candidate learning models, the score of each candidate learning model is determined. Then, the M candidate learning models are ranked in order of decreasing score, and the top N candidate learning models are selected in order of decreasing score. When making this selection, a condition that "the score is less than a threshold value" may be added.

[0094] Also, as the score of the candidate learning model, a score inferred from the detection area of ​​the trunk of a tree, which is usually planted at equal intervals, may be obtained. As shown in Fig. 5(A), tree trunks should be detected at approximately equal intervals, such as rectangular areas 501, 502, 503, and 504, so the expected number of "detected tree trunk areas" for the width of the captured image is fixed. Images with fewer / more than the expected number are likely to have detection errors, so this detection number may be reflected in the score.

[0095] Then, the CPU 191 transmits the P captured images, the "results of the object detection process on the P captured images" of each of the N candidate learning models selected from the M candidate learning models, and information on the N candidate learning models (such as model names) to the information processing device 13. As described above, in this embodiment, the "results of the object detection process on the captured images" are position information of the image area (rectangular area, detection area) of the object detected from the captured image, and such position information is transmitted to the information processing device 13 as data in a file format such as json format or txt format.

[0096] Next, the user is asked to select one of the selected N candidate learning models. At the end of the process in step S2331, N candidate learning models still remain, and the output on which the performance comparison is based is the result of the object detection process on the P captured images. Therefore, the user must compare the results of the object detection process on the N×P captured images. In such a situation, it is difficult to appropriately select one candidate learning model as the selected learning model (narrow it down to one).

[0097] Therefore, in step S2332, the CPU 131 of the information processing device 13 performs scoring (display image scoring) for the P captured images to present information that is easy for the user to subjectively compare. In the display image scoring, a larger score is determined for each of the P captured images as the arrangement patterns of the detection regions differ more significantly among the N candidate learning models. Such a score can be obtained, for example, by calculating the following formula (4).

[0098]

number

[0099] Here, Score(z) is the score for the captured image Iz. Iz(Ma, Mb) is a function for calculating a score based on the difference between the result of the object detection process (arrangement pattern of the detection area) performed by the candidate learning model Ma on the photographed image Iz and the result of the object detection process (arrangement pattern of the detection area) performed by the candidate learning model Mb on the photographed image Iz. Various functions can be applied to such a function, and it is not limited to a specific function. For example, for each detection area Ra detected by the candidate learning model Ma from the photographed image Iz, a function is defined as T that calculates the difference between the position (e.g., the position of the upper left corner and the position of the lower right corner) of the detection area Rb detected by the candidate learning model Mb from the photographed image Iz that is closest to the detection area Ra and the position of the detection area Ra (e.g., the position of the upper left corner and the position of the lower right corner), and returns the sum of the calculated differences. Iz (Ma, Mb) may also be used.

[0100] The results of object detection processing using the top N candidate learning models are often similar, so there is often little difference when comparing randomly selected images, and this does not provide a basis for selecting a learning model. Therefore, by looking at only the top captured images scored using the above formula (4), it becomes easier to judge the quality of the learning model.

[0101] In step S2333, the CPU 131 of the information processing device 13 controls the display device 14 to display (display control) the top F (a prescribed number from the top) captured images in descending order of score among the P captured images received from the cloud server 12 for each of the above N candidate learning models, and the results of the object detection process for the captured images received from the cloud server 12. At this time, the F captured images are displayed in descending order of score from the left.

[0102] An example of a GUI displaying the captured images and the object detection processing results for each candidate learning model is shown in Fig. 7(A). Fig. 7(A) shows the case of N=3 and F=4.

[0103] The top row displays the model name "M002" of the candidate learning model with the highest score along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in descending order from left to right. A frame indicating the detection area of ​​the object detected from the captured image by the candidate learning model with the model name "M002" is superimposed on the captured image.

[0104] In the middle row, the model name "M011" of the candidate learning model with the second highest score is displayed along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area of ​​the object detected from the captured image by the candidate learning model with the model name "M011" is superimposed on the captured image.

[0105] The bottom row displays the model name "M009" of the candidate learning model with the third highest score along with a radio button 70, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area of ​​the object detected from the captured image by the candidate learning model with the model name "M009" is superimposed on the captured image.

[0106] In this GUI, captured images arranged in the same row are displayed as the same captured images, so that the results of the object detection processing using each candidate learning model can be easily compared at a glance.

[0107] Then, the user visually checks the difference in the results of the object detection process for the F captured images using the N candidate learning models, and selects one of the N candidate learning models using the user interface 15.

[0108] In step S2334, the CPU 131 of the information processing device 13 accepts a selection operation (user operation, user input) of a candidate learning model by the user. In step S2335, the CPU 131 of the information processing device 13 determines whether or not a selection operation (user input) of a candidate learning model has been performed by the user.

[0109] In the case of FIG. 7(A), when the user selects a candidate learning model with the model name "M002", the user selects the radio button 70 in the top row using the user interface 15. When the user selects a candidate learning model with the model name "M011", the user selects the radio button 70 in the middle row using the user interface 15. When the user selects a candidate learning model with the model name "M009", the user selects the radio button 70 in the bottom row using the user interface 15. In FIG. 7(A), since the radio button 70 corresponding to the model name "M002" is selected, a frame 74 is displayed indicating that the candidate learning model with the model name "M002" has been selected.

[0110] When the user operates the user interface 15 to select the decision button 71, the CPU 131 determines that "the user has performed a candidate learning model selection operation (user input)," and selects the candidate learning model corresponding to the selected radio button 70 as the selected learning model.

[0111] As a result of this determination, if the user has performed a selection operation (user input) of a candidate learning model, processing proceeds to step S2336, and if the user has not performed a selection operation (user input) of a candidate learning model, processing proceeds to step S2334.

[0112] In step S2336, the CPU 131 of the information processing device 13 checks whether only one learning model is finally selected. If only one learning model is finally selected, the process proceeds to step S24, and if not, the process proceeds to step S2332.

[0113] Here, if the user is unable to narrow down to one just by looking at the display in FIG. 7(A), multiple radio buttons 70 may be selected to select multiple candidate learning models. For example, when the user operates the user interface 15 to select the radio button 70 corresponding to the model name "M002" and the radio button 70 corresponding to the model name "M011" in FIG. 7(A) and presses the decision button 71, the number of selected radio buttons 70 is set to "2" as N, and the process proceeds to step S2332 via step S2336. In this case, in and after step S2332, the same process is performed for N=2 and F=4. In this manner, the process is repeated until the number of learning models finally selected is "1".

[0114] Also, the user may select a learning model using the GUI of FIG. 7(B) instead of the GUI of FIG. 7(A). The GUI of FIG. 7(A) is a GUI that allows the user to directly select which learning model is better. In contrast, in the GUI of FIG. 7(B), a check box 72 is provided for each captured image, and the user operates the user interface 15 to specify and turn on (put a check mark on) the check box 72 of the captured image that is determined to have a better result of the object detection process among the captured images in the captured image sequence, for each captured image sequence that is lined up in a vertical line. Then, when the user operates the user interface 15 to instruct the decision button 75, the CPU 131 of the information processing device 13 selects, as the selected learning model, the candidate learning model with the largest number of captured images with the check box 72 turned on, from among the candidate learning models with the model names "M002", "M011", and "M009". In the example of FIG. 7(B), three check boxes 72 are turned on among the four captured images of the candidate learning model with the model name "M002", one check box 72 is turned on among the four captured images of the candidate learning model with the model name "M011", and none of the check boxes 72 are turned on among the four captured images of the candidate learning model with the model name "M009". In this case, the candidate learning model with the model name "M002" is selected as the selected learning model. This method of selecting the selected learning model using a GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is best.

[0115] If there are candidate learning models with the same or a small difference in the "number of captured images with check boxes 72 turned on," it is determined in step S2336 that "finally, only one learning model has not been selected," and the process proceeds to step S2332. Then, from step S2332 onward, the process targets candidate learning models with the same or a small difference in the "number of captured images with check boxes 72 turned on." Even in such a case, the process is repeated until the number of learning models finally selected becomes "1."

[0116] In addition, since the captured image displayed further to the left is the captured image with a larger difference in the results of the object detection process between the candidate learning models, a larger weight value may be assigned to the captured image displayed further to the left. In this case, for each candidate learning model, the sum of the weight values ​​of the captured images with the check boxes 72 turned on may be calculated, and the candidate learning model with the largest calculated sum may be selected as the selected learning model.

[0117] Then, regardless of the method used to select the selected learning model, the CPU 131 of the information processing device 13 notifies the cloud server 12 of information indicating the selected learning model (for example, the model name of the selected learning model).

[0118] In step S24, the CPU 191 of the cloud server 12 performs object detection processing on the captured image (the captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using the selected learning model identified by the information notified from the information processing device 13.

[0119] In step S25, the CPU 191 of the cloud server 12 performs analysis processing such as predicting the yield of the target field and calculating the non-productive rate for the entire field from the detection area obtained as a result of the object detection processing in step S24. This calculation is performed taking into account both the production area rectangle detected from all captured images and the non-productive area determined to be a dead branch area, a diseased area, etc.

[0120] Note that the learning model in this embodiment is a model learned by deep learning, but various object detection techniques such as a rule-based detector defined by various parameters, fuzzy inference, genetic algorithms, etc. may also be used as the learning model.

[0121] [Second embodiment] In the following embodiments, differences from the first embodiment will be described, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the present embodiment, a system for performing visual inspection in a factory production line will be described as an example. The system according to the present embodiment detects abnormal areas in industrial products that are the objects of inspection.

[0122] Traditionally, in visual inspections on factory production lines, the imaging conditions of the inspection equipment (equipment that photographs and inspects the appearance of products) were carefully adjusted for each production line, and it was common to spend time adjusting the settings of the inspection equipment each time a production line was started up. However, in recent years, manufacturing sites are expected to respond immediately to diversifying customer needs and changes in the market. There is an increasing need for speedy responses, such as setting up a line in a short period of time even for small lots, producing a quantity that matches demand, and then immediately dismantling the line once sufficient supply has been completed to prepare for the next production line.

[0123] In this case, if the settings for visual inspection are set every time based on the experience and intuition of a manufacturing site expert, as in the past, it will not be possible to respond to a rapid start-up.If similar products have been inspected in the past, and the related setting parameters can be stored and the past setting parameters can be called up when a similar inspection is to be performed, anyone can set up the inspection device without relying on the experience of an expert.

[0124] As in the first embodiment, the above object can be achieved by assigning the already-held learning model to the inspection target image of the new product. Therefore, the above information processing device 13 can be applied to the second embodiment as well.

[0125] The setting process for the inspection device by the system according to this embodiment (setting process for visual inspection) will be described with reference to the flowchart in Fig. 8A. Note that the setting process for visual inspection is assumed to be performed at the start of an inspection step in a manufacturing line.

[0126] The external storage device 196 of the cloud server 12 has multiple learning models (appearance inspection models / settings) registered for performing appearance inspection on captured images, and each learning model is a model that has been learned in a mutually different learning environment.

[0127] The camera 10 is a camera for photographing a product that is the subject of visual inspection (product to be inspected). As in the first embodiment, the camera 10 may be a camera that photographs periodically or irregularly, or may be a camera that photographs moving images. In order to accurately detect an abnormal area in the product to be inspected from the photographed image, when the product to be inspected that includes an abnormal area enters the inspection process, it is desirable to photograph the product under conditions that emphasize the abnormal area as much as possible. The camera 10 may be a multi-camera if the product to be inspected is photographed under multiple conditions.

[0128] In step S80, the camera 10 captures an image of the product to be inspected to generate a captured image of the product to be inspected. 80 The captured image generated in step 1 is transmitted to the cloud server 12 and the information processing device 13 via the communication network 11.

[0129] In step S82, the CPU 131 of the information processing device 13 acquires information relating to the product to be inspected photographed by the camera 10 (part name and material of the product to be inspected, manufacturing date, imaging system parameters at the time of photographing, lot number, temperature, humidity, etc.) as parameters of the product to be inspected. For example, the CPU 131 displays a GUI on the display device 14 to accept input of the parameters of the product to be inspected from the user. Then, when the user operates the user interface 15 to input a registration instruction, the CPU 131 of the information processing device 13 transmits the parameters of the product to be inspected for each of the above items input in the GUI to the cloud server 12. The CPU 191 of the cloud server 12 saves (registers) the parameters of the product to be inspected transmitted from the information processing device 13 in the external storage device 196.

[0130] In step S83, a process is performed to select a learning model to be used for detecting the above-mentioned inspection target product from the captured image. Details of the process in step S83 will be described with reference to the flowchart in FIG. 8B.

[0131] In step S831, the CPU 191 of the cloud server 12 selects M candidate learning models (candidate learning models) from the E learning models stored in the external storage device 196. The CPU 191 generates query parameters from the inspection target product parameters registered in the external storage device 196 in the same manner as in the first embodiment, and selects a learning model that has learned about an environment similar to the environment indicated by the query parameters (a learning model used in a similar past inspection).

[0132] If the query parameters include "circuit board" as the "part name," a learning model used in past circuit board inspections is more likely to be selected, and if the query parameters include "glass epoxy" as the "material," a learning model used in the inspection of glass epoxy circuit boards is more likely to be selected.

[0133] In step S831, as in the first embodiment, M candidate learning models are selected using the parameter set of the learning model and the query parameters, and in this case, the above formula (1) is used as in the first embodiment.

[0134] Next, in step S832, the CPU 191 of the cloud server 12 selects P captured images from the captured images received from the camera 10 as model selection target images. For example, products flowing through the main inspection process of the main production line are randomly selected, and P captured images are acquired from the captured images captured by the camera 10 with the same settings as those during actual operation. Since the number of abnormal products that occur on a production line is usually small, the processing in the following steps does not function well when the number of products captured in the process is small. Therefore, as a guideline, it is desirable to capture images of several hundred or more products.

[0135] Next, in step S833, a process is performed to select one of the M candidate learning models as a selected learning model using the P captured images selected in step S832. Details of the process in step S833 will be described with reference to the flowchart in FIG. 8C.

[0136] In step S8330, the CPU 191 of the cloud server 12 performs "object detection processing, which is processing for detecting an object from each of the P photographed images using the candidate learning model for each of the M candidate learning models" for each of the M photographed images. In this embodiment, too, the result of the object detection processing for the photographed image is position information of the image area (rectangular area, detection area) of the object detected from the photographed image.

[0137] In step S8331, the CPU 191 obtains a score for the "result of the object detection process for each of the P captured images" for each of the M candidate learning models. The CPU 191 then ranks the M candidate learning models based on the scores, and selects N candidate learning models from the M candidate learning models. The score for the result of the object detection process using the candidate learning models is obtained, for example, as follows.

[0138] For example, in a task of detecting anomalies on a printed circuit board, object detection processing is performed on various specific local patterns on a fixed print pattern. Here, it is assumed that detection areas 901 to 906 as shown in FIG. 9(A) are obtained from a captured image of a normal product in a specific learning model. Since the occurrence frequency of anomalies in products produced on a manufacturing line is extremely low, a good learning model for performing the above task is one that can output stable results against expected variations in captured images. For example, the appearance of an image of a product may change slightly due to environmental fluctuations on the area sensor side, making it impossible to detect detection area 906 among detection areas 901 to 906 as shown in FIG. 9(B). In such a case, a penalty should be imposed on the evaluation score of a learning model in which the detection area changes for inputs with only slight differences.

[0139] Therefore, for example, the CPU 191 of the cloud server 12 determines a score for each of the M candidate learning models, the greater the difference in the arrangement pattern of the detection area by the candidate learning model between the P captured images. Such a score can be obtained, for example, by calculating the above formula (4). Then, the M candidate learning models are ranked in ascending order of score, and the top N candidate learning models are selected in descending order of score. When making this selection, a condition that "the score is less than a threshold value" may be added.

[0140] In step S8332, the CPU 131 of the information processing device 13 performs scoring (display image scoring) for presenting information that allows easy subjective comparison by the user for the P captured images in the same manner as in the first embodiment (step S2332).

[0141] In step S8333, the CPU 131 of the information processing device 13 causes the display device 14 to display, for each of the N candidate learning models selected in step S8331, the top F captured images in descending order of score among the P captured images received from the cloud server 12, and the results of the object detection process for the captured images received from the cloud server 12. At this time, the F captured images are displayed in descending order of score from the left.

[0142] An example of a GUI displaying the captured images and the object detection processing results for each candidate learning model is shown in Fig. 10(A). Fig. 10(A) shows the case of N=3 and F=4.

[0143] The top row displays the model name "M005" of the candidate learning model with the highest score along with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in descending order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with the model name "M005" is superimposed on the captured image.

[0144] In the middle row, the model name "M023" of the candidate learning model with the second highest score is displayed together with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with the model name "M023" is superimposed on the captured image.

[0145] In the bottom row, the model name "M014" of the candidate learning model with the third highest score is displayed together with a radio button 100, and to the right of that, the top four captured images with the highest scores are displayed in order from left to right. A frame indicating the detection area detected from the captured image by the candidate learning model with the model name "M014" is superimposed on the captured image.

[0146] In this GUI, captured images arranged in the same row are displayed as the same captured images, so that the results of the object detection processing using each candidate learning model can be easily compared at a glance.

[0147] In this case, the difference in the arrangement pattern of the detection area is as shown in Fig. 10(A) because the product appearance is almost fixed and most of the products are normal. The F captured images are displayed in order from the left to the right in descending order of score, but the images with higher scores tend to be those in which the shooting conditions at the time of individual shooting were significantly different or those containing abnormal areas. Therefore, compared to the conventional method in which the user had to annotate the abnormal areas in the captured images of the product in advance and manually search for defective products from a large number of products before setting up the inspection device, the user can preferentially present the captured images of products that may contain abnormal areas just by looking at this GUI without having to perform any of the above tasks, which saves labor. The user can compare the results of the object detection process in the GUI in Fig. 10(A) and select the learning model that correctly detects abnormal areas.

[0148] Then, the user visually checks the difference in the results of the object detection process for the F captured images using the N candidate learning models, and selects one of the N candidate learning models using the user interface 15.

[0149] In step S8334, the CPU 131 of the information processing device 13 accepts a selection operation (user input) of a candidate learning model by the user. In step S8335, the CPU 131 of the information processing device 13 determines whether or not a selection operation (user input) of a candidate learning model has been performed by the user.

[0150] In the case of FIG. 10(A), when the user selects a candidate learning model with the model name "M005", the user selects the radio button 100 in the top row using the user interface 15. When the user selects a candidate learning model with the model name "M023", the user selects the radio button 100 in the middle row using the user interface 15. When the user selects a candidate learning model with the model name "M014", the user selects the radio button 100 in the bottom row using the user interface 15. In FIG. 10(A), since the radio button 100 corresponding to the model name "M005" is selected, a frame 104 is displayed indicating that the candidate learning model with the model name "M005" has been selected.

[0151] When the user operates the user interface 15 to select the decision button 101, the CPU 131 determines that "the user has performed a candidate learning model selection operation (user input)," and selects the candidate learning model corresponding to the selected radio button 100 as the selected learning model.

[0152] As a result of this determination, if the user has performed a candidate learning model selection operation (user input), processing proceeds to step S8336, and if the user has not performed a candidate learning model selection operation (user input), processing proceeds to step S8334.

[0153] In step S8336, the CPU 131 of the information processing device 13 checks whether the "number of learning models desired by the user" has been selected. If the "number of learning models desired by the user" has been selected, the process proceeds to step S84, and if the "number of learning models desired by the user" has not been selected, the process proceeds to step S8332.

[0154] Here, the "number desired by the user" is determined mainly according to the time (takt time) that can be spent on visual inspection. For example, if the "number desired by the user" is two, it may be possible to detect a wide range of defects by changing the tendency of the detection target, such as detecting low-frequency abnormal areas with one learning model and detecting high-frequency defects with the other learning model.

[0155] Here, if the user is unable to narrow down the number to "the number desired by the user" just by looking at the display in Fig. 10(A), multiple candidate learning models may be selected by selecting multiple radio buttons 100. For example, if the "number desired by the user" is "1" and the number of selected radio buttons 100 is 2, N=2 and the process proceeds to step S8332 via step S8336. In this case, from step S8332 onwards, the same process is performed for N=2 and F=4. In this way, the process is repeated until the number of learning models finally selected becomes "the number desired by the user".

[0156] Also, the user may select a learning model using the GUI of FIG. 10(B) instead of the GUI of FIG. 10(A). The GUI of FIG. 10(A) is a GUI that allows the user to directly select which learning model is better. In contrast, in the GUI of FIG. 10(B), a check box 102 is provided for each captured image, and the user operates the user interface 15 to specify and turn on (put a check mark on) the check box 102 of the captured image that is determined to have a better result of the object detection process among the captured images in the captured image sequence, for each captured image sequence that is lined up vertically. Then, when the user operates the user interface 15 to specify the decision button 1015, the CPU 131 of the information processing device 13 selects, as the selected learning model, the candidate learning model with the largest number of captured images with the check box 102 turned on, from among the candidate learning models with the model names "M005", "M023", and "M014". In the example of FIG. 10(B), two check boxes 102 are turned on among the four captured images of the candidate learning model with the model name "M005", one check box 102 is turned on among the four captured images of the candidate learning model with the model name "M023", and one check box 102 is turned on among the four captured images of the candidate learning model with the model name "M014". In this case, the candidate learning model with the model name "M005" is selected as the selected learning model. This method of selecting the selected learning model using a GUI is effective, for example, when the value of F increases and it is difficult for the user to determine which candidate learning model is best.

[0157] The easiest way to narrow down the learning models to the "number desired by the user" in the GUI of Figure 10(B) is to select the "number desired by the user" from the top in order of the number of check boxes that are checked.

[0158] In addition, "Check box 102If there are candidate learning models with the same or a small difference in the number of captured images with check boxes 102 turned on, it is determined in step S8336 that "the number of learning models selected is not the 'number desired by the user'," and the process proceeds to step S8332. Then, from step S8332 onwards, the process targets candidate learning models with the same or a small difference in the number of captured images with check boxes 102 turned on. Even in such a case, the process is repeated until the number of learning models finally selected is the "number desired by the user."

[0159] Then, regardless of the method used to select the selected learning model, the CPU 131 of the information processing device 13 notifies the cloud server 12 of information indicating the selected learning model (for example, the model name of the selected learning model).

[0160] In step S84, the CPU 191 of the cloud server 12 performs an object detection process on the captured image (the captured image transmitted by the camera 10 to the cloud server 12 and the information processing device 13) using the selected learning model specified by the information notified from the information processing device 13. Then, the CPU 191 of the cloud server 12 performs a final setting of the inspection device from the detection area obtained as a result of the object detection process. Inspection is performed at the stage when the production line is actually started up, using the learning model and various parameters set here.

[0161] Note that the learning model in this embodiment is a model learned by deep learning, but various object detection techniques such as a rule-based detector defined by various parameters, fuzzy inference, genetic algorithms, etc. may also be used as the learning model.

[0162] <Modification> Each of the above embodiments is an example of a technology for reducing the cost of learning a learning model and adjusting the settings each time a detection / identification process is performed on a new target in a task for performing a target detection / identification process. Therefore, the technology described in each of the above embodiments is not limited to application to prediction of agricultural crop yields, detection of repair areas, detection of abnormal areas in industrial products to be inspected, etc. The target is applied to agriculture, industry, fisheries, and a wide range of other fields.

[0163] Moreover, the above radio buttons and check boxes are displayed as an example of a selection section for the user to select a target, and other display items may be displayed instead as long as they have a similar effect. In the above embodiment, a configuration in which a learning model to be used in the object detection process is selected based on a user operation has been described (S24). However, this is not limited to this, and a learning model to be used in the object detection process may be automatically selected. For example, a candidate learning model with the highest score may be automatically selected as the learning model to be used in the object detection process.

[0164] Also, the subject of each process in the above description is just an example. For example, a part or all of the processes described as being performed by the CPU 191 of the cloud server 12 may be performed by the CPU 131 of the information processing device 13. Also, a part or all of the processes described as being performed by the CPU 131 of the information processing device 13 may be performed by the CPU 191 of the cloud server 12.

[0165] In the above description, the system of each embodiment is described as performing the analysis process. However, the subject of the analysis process is not limited to the system of each embodiment, and for example, the analysis process may be performed by another device / system.

[0166] Furthermore, the numerical values, processing timing, processing order, processing subject, data (information) structure / destination / source, etc. used in each of the above embodiments and variant examples are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0167] In addition, some or all of the embodiments and modified examples described above may be used in appropriate combination. In addition, some or all of the embodiments and modified examples described above may be used selectively.

[0168] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0169] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0170] 10: Camera 11: Communication network 12: Cloud server 13: Information processing device 14: Display device 15: User interface 131: CPU 132: RAM 133: ROM 134: Output I / F 135: Input I / F 191: CPU 192: RAM 193: ROM 194: Operation unit 195: Display unit 196: External storage device 197: I / F 198: System bus

Claims

1. a first selection means for selecting one or more learning models as candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing of the object; a second selection means for selecting one or more candidate learning models from the candidate learning models selected by the first selection means based on a result of the object detection process using the candidate learning models; a detection means for performing an object detection process on a photographed image of the object using at least one of the candidate learning models selected by the second selection means; a means for predicting crop yields and detecting repair areas in a farm field based on object detection areas obtained as a result of the object detection process; An information processing device comprising:

2. The information processing device according to claim 1, characterized in that the first selection means generates query parameters based on the information and selects one or more learning models from the plurality of learning models that have been trained in an environment similar to the environment indicated by the query parameters as candidate learning models.

3. The information processing device described in claim 1 or 2, characterized in that the second selection means calculates a score based on the results of the object detection processing using each candidate learning model selected by the first selection means, and selects one or more candidate learning models from the candidate learning models selected by the first selection means based on the score.

4. Furthermore, 4. The information processing apparatus according to claim 1, further comprising a display control unit that displays a result of the object detection process using the candidate learning model selected by the second selection unit.

5. The information processing device described in claim 4, characterized in that the display control means determines a higher score for each of a plurality of captured images for which each candidate learning model selected by the second selection means has performed object detection processing, the greater the difference in the results of the object detection processing between the candidate learning models, and displays a GUI including the results of the object detection processing by the candidate learning model for a specified number of captured images, starting from the highest score for each candidate learning model selected by the second selection means.

6. the GUI includes a selection portion for selecting a candidate learning model; The information processing device according to claim 5, characterized in that the detection means selects a candidate learning model corresponding to a selection section selected in accordance with user operation in the GUI as a selected learning model, and performs object detection processing using the selected learning model.

7. The information processing device described in claim 5, characterized in that the detection means selects the candidate learning model that has the most object detection processing results selected in accordance with user operation from the object detection processing results displayed for each candidate learning model by the display control means as a selected learning model, and performs object detection processing using the selected learning model.

8. 8. The information processing device according to claim 1, wherein the information includes Exif information of the captured image, information relating to the field in which the captured image was taken, and information relating to objects included in the captured image.

9. 8. The information processing apparatus according to claim 1, wherein the information includes information relating to an object included in the photographed image.

10. The information processing device described in any one of claims 1 to 9, characterized in that the detection means performs the object detection processing on the captured image of the object using a candidate learning model selected from the candidate learning models selected by the second selection means based on user operation.

11. An information processing method performed by an information processing device, a first selection step in which a first selection means of the information processing device selects one or more learning models as candidate learning models from a plurality of learning models trained in different learning environments based on information related to photographing of an object; a second selection step in which a second selection means of the information processing device selects one or more candidate learning models from the candidate learning models selected in the first selection step based on a result of an object detection process using the candidate learning models; a first detection step in which a first detection means of the information processing device performs an object detection process on a photographed image of the object using at least one of the candidate learning models selected in the second selection step; a second detection step in which a second detection means of the information processing device predicts a yield of agricultural crops and detects repair areas in the field based on a detection area of ​​the object obtained as a result of the object detection process; An information processing method comprising:

12. A computer program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 10.

Images