Model generation device, image recognition device, model generation method, and image recognition method

Abstract

To improve image recognition accuracy by machine learning in an imaging device difference case.SOLUTION: In a learning device (11), a learning-time image processing unit (114) (i) performs image processing according to each of a plurality of prescribed image processing settings to a plurality of learning images captured by a first imaging device to generate a plurality of processed learning images, and (ii) performs the image processing to a plurality of evaluation images captured by a second imaging device different from the first imaging device to generate a plurality of processed evaluation images, a learning model generation unit (115) executes a prescribed machine learning algorithm to generate a plurality of learning models by using the plurality of processed learning images, and the learning model generation unit (115) (i) selects the operation-time image processing setting from the plurality of prescribed image processing settings and (ii) selects the operation-time learning model from the plurality of learning models on the basis of an index value obtained by evaluating each of the plurality of learning models using the plurality of processed evaluation images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] One aspect of the present invention relates to a model generation device that generates a learning model for performing image recognition in an image recognition device. [Background technology]

[0002] In recent years, various techniques have been proposed for image recognition using machine learning. For example, Patent Document 1 discloses a learning method for pattern recognition that appropriately extracts local features useful for detecting and recognizing objects of different categories from a collection of images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-215988 Summary of the Invention [Problem to be solved by the invention]

[0004] As will be described later, there may be cases where the imaging device (first imaging device) used to capture the learning images is different from the imaging device (second imaging device) used to capture the operational images. In this specification, this case is referred to as an imaging device difference case. In the imaging device difference case, the features of the learning images and the features of the operational images may differ due to the difference between the first imaging device and the second imaging device. As a result, when image recognition using machine learning is applied to the operational images, the image recognition accuracy may decrease.

[0005] However, the conventional technology (e.g., the technology of Patent Document 1) is based on the premise that the first and second imaging devices are the same. In other words, the conventional technology does not consider specific measures to improve the accuracy of image recognition by machine learning in the case where the imaging devices are different.

[0006] One aspect of the present invention has been made in consideration of the above-mentioned problems, and aims to improve the accuracy of image recognition by machine learning in cases where the imaging device is different. [Means for solving the problem]

[0007] In order to solve the above problem, a model generation device according to one aspect of the present invention is a model generation device that generates, based on a plurality of training images captured by a first imaging device, a learning model for performing image recognition in an image recognition device based on operational images captured by a second imaging device different from a first imaging device, the model generation device including: (i) a training image processing unit that applies image processing according to each of a plurality of predetermined image processing settings to the plurality of training images to generate a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings, and (ii) a training image processing unit that applies the image processing to a plurality of evaluation images captured by the second imaging device to generate a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; and a predetermined machine learning algorithm. and a learning unit that generates a plurality of learning models using the plurality of post-processing learning images corresponding to each of the plurality of predetermined image processing settings by executing a algorithm, and the learning unit (i) selects, from each of the plurality of predetermined image processing settings, an operational image processing setting to be applied when the image recognition device is operated, and (ii) selects, from each of the plurality of learning models, a learning model corresponding to the operational image processing setting as an operational learning model to be applied when the image recognition device is operated, based on a plurality of index values ​​that indicate the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of predetermined image processing settings.

[0008] Furthermore, an image recognition device according to one aspect of the present invention is an image recognition device that performs image recognition based on operational images captured by a second imaging device different from the first imaging device, using a learning model that is pre-generated based on a plurality of training images captured by a first imaging device, wherein (i) image processing according to each of a plurality of predetermined image processing settings is performed on the plurality of training images to generate a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings, and (ii) the image processing is performed on a plurality of evaluation images captured by the second imaging device to generate a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings, and a predetermined machine learning algorithm is executed to generate a plurality of learning models using the plurality of processed training images corresponding to each of the plurality of predetermined image processing settings, and (i) an operational image processing setting to be applied when the image recognition device is operated is pre-selected from among each of the plurality of predetermined image processing settings, and (ii) a learning model corresponding to the operational image processing setting is pre-selected from among each of the plurality of learning models as an operational learning model to be applied when the image recognition device is operated, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to the plurality of post-processing evaluation images. The image recognition device comprises an operational image processing unit that generates a processed operational image corresponding to the operational image processing setting by applying the image processing to the operational image in accordance with the operational image processing setting, and an image recognition unit that performs image recognition on the processed operational image based on the output of the operational learning model obtained by inputting the processed operational image into the operational learning model.

[0009] Furthermore, a model generation method according to one aspect of the present invention is a model generation method for generating, based on a plurality of training images captured by a first imaging device, a learning model for performing image recognition in an image recognition phase based on operational images captured by a second imaging device different from the first imaging device, the model generation method including: (i) applying image processing according to each of a plurality of predetermined image processing settings to the plurality of training images to generate a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings; and (ii) applying the image processing to a plurality of evaluation images captured by the second imaging device to generate a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; and executing a predetermined machine learning algorithm. and a learning process for generating a plurality of learning models using the plurality of post-processing learning images corresponding to each of the plurality of predetermined image processing settings, the learning process further including the steps of (i) selecting an operational image processing setting to be applied in the image recognition phase from among the plurality of predetermined image processing settings, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of predetermined image processing settings, and (ii) selecting a learning model corresponding to the operational image processing setting from among the plurality of learning models as an operational learning model to be applied in the image recognition phase.

[0010] Furthermore, an image recognition method according to one aspect of the present invention is an image recognition method for performing image recognition based on an operational image captured by a second imaging device different from the first imaging device, using a learning model generated in advance based on a plurality of training images captured by a first imaging device, wherein (i) a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings are generated in advance by applying image processing according to each of a plurality of predetermined image processing settings to the plurality of training images, and (ii) a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings are generated in advance by applying the image processing to a plurality of evaluation images captured by the second imaging device, and a plurality of learning models are generated in advance using the plurality of processed training images corresponding to each of the plurality of predetermined image processing settings by executing a predetermined machine learning algorithm, and (i) an operational image processing setting to be applied in the image recognition method is pre-selected from among each of the plurality of predetermined image processing settings, and (ii) a learning model corresponding to the operational image processing setting is pre-selected from among each of the plurality of learning models as an operational learning model to be applied in the image recognition method, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to the plurality of post-processing evaluation images. The image recognition method includes an operational image processing step of generating a processed operational image corresponding to the operational image processing setting by applying the image processing to the operational image in accordance with the operational image processing setting, and an image recognition step of performing image recognition on the processed operational image based on the output of the operational learning model obtained by inputting the processed operational image into the operational learning model. [Effects of the Invention]

[0011] According to one aspect of the present invention, it is possible to improve the accuracy of image recognition by machine learning in cases where the imaging device is different. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a block diagram showing the configuration of a main part of an information processing system according to a first embodiment. [Figure 2] FIG. 10 is a diagram illustrating an example of an image processing setting list. [Figure 3] FIG. 10 is a diagram illustrating an example of a hyperparameter setting list. [Figure 4] FIG. 2 is a diagram illustrating an example of an overall processing flow in the learning device of the first embodiment. [Figure 5] FIG. 10 is a diagram illustrating an example of the flow of image processing for learning images. [Figure 6] FIG. 10 is a diagram illustrating an example of the flow of image processing for an evaluation image. [Figure 7] FIG. 10 is a diagram illustrating a processing flow in hyperparameter setting. [Figure 8] FIG. 10 is a diagram illustrating an example of the processing flow for learning, evaluation of a learning model, and determination of a final accuracy candidate value. [Figure 9] FIG. 10 is a diagram illustrating an example of image processing during operation. [Figure 10] FIG. 10 is a diagram illustrating another example of a hyperparameter setting list. [Figure 11] FIG. 10 is a diagram illustrating yet another example of a hyperparameter setting list. [Figure 12] FIG. 10 is a block diagram showing the configuration of the main part of an information processing system according to a second embodiment. [Figure 13] FIG. 10 is a diagram illustrating an example of a schematic flow of main processes in the learning device of the second embodiment. [Figure 14] FIG. 10 is a diagram illustrating a processing flow for acquiring evaluation accuracy. [Figure 15] This is a diagram for explaining the concept of the Nelder-Mead method. [Figure 16] FIG. 10 is a diagram illustrating a process flow for performing optimization. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Embodiment 1] The information processing system 100 of the first embodiment will be described below. For convenience of explanation, components having the same functions as those described in the first embodiment will be denoted by the same reference numerals in the following embodiments, and the description thereof will not be repeated. Explanations of matters similar to those in known technologies will also be omitted as appropriate. Note that the numerical values ​​described below in this specification are merely examples.

[0014] (Overview of information processing system 100) 1 is a block diagram showing the configuration of the main parts of an information processing system 100. The information processing system 100 includes an information processing device 1, a learning image DB (Database) 910, an evaluation image DB 920, and an operation image DB 930.

[0015] The information processing device 1 includes a control device 10, an input unit 71, a display unit 72, and a storage unit 80. The control device 10 includes a learning device 11 and an image recognition device 12. The control device 10 comprehensively controls each unit of the information processing device 1. The storage unit 80 stores various data and programs used in processing by the control device 10. As described below, the control device 10 (more specifically, the image recognition device 12) performs image recognition using machine learning.

[0016] The input unit 71 accepts user operations (user operations). The display unit 72 displays various types of data. As one example, the display unit 72 may display data indicating the results of image recognition search performed by the image recognition device 12. As another example, the display unit 72 may display data indicating the results of machine learning performed by the learning device 11. Unlike the example of FIG. 1, the input unit 71 and the display unit 72 may be provided as a single unit. For example, by using a touch panel, the input unit 71 and the display unit 72 can be realized as a single functional unit.

[0017] The information processing device 1 only needs to be communicably connected to the learning image DB 910, the evaluation image DB 920, and the operational image DB 930. Therefore, unlike the example of Fig. 1, at least one of the learning image DB 910, the evaluation image DB 920, and the operational image DB 930 may be provided inside the information processing device 1.

[0018] (Regarding training images, evaluation images, and operational images) The training image DB 910 stores a plurality of training images (e.g., IMGL, described later). Training images are images used as training data (data for generating a training model) in the machine learning process in the training device 11 (in other words, in the training phase). Training images according to one aspect of the present invention are images captured by the first imaging device. The training image DB 910 also stores label data (correct answer data in image recognition) corresponding to each training image. In this specification, the label data corresponding to the training images is referred to as training label data.

[0019] The evaluation image DB 920 stores a plurality of evaluation images (e.g., IMGT, described later). A learning image is an image used for evaluating a learning model in the learning phase. An evaluation image according to one aspect of the present invention is an image captured by a second imaging device different from the first imaging device. The evaluation image DB 920 also stores label data corresponding to each evaluation image. In this specification, the label data corresponding to the evaluation image is referred to as evaluation label data.

[0020] The operation image DB 930 stores a plurality of operation images (e.g., IMGO, which will be described later). An operation image is an image that is input to the image recognition device 12 as a target for image recognition in the image recognition process in the image recognition device 12 (in other words, in the operation phase that follows the learning phase). An operation image according to one aspect of the present invention is an image captured by the second imaging device.

[0021] 1 illustrates a case where the training image DB 910, the evaluation image DB 920, and the operational image DB 930 are separate functional units for the sake of convenience. However, as would be clear to one skilled in the art, the training image DB 910 and the evaluation image DB 920 may be provided as an integrated functional unit. Furthermore, the training image DB 910, the evaluation image DB 920, and the operational image DB 930 may be provided as an integrated functional unit.

[0022] In the first embodiment, a case where the first imaging device and the second imaging device have different light receiving characteristics (more specifically, different wavelength sensitivity characteristics) is illustrated. As an example, in the first embodiment, (i) the first imaging device is a visible light camera having light receiving characteristics in the visible light wavelength region, and (ii) the second imaging device is a non-visible light camera having light receiving characteristics in the non-visible light wavelength region is illustrated. More specifically, in the first embodiment, (i) the first imaging device is an RGB (Red, Green, Blue) camera, and (ii) the second imaging device is an IR camera (near-infrared light camera) having light receiving characteristics in the IR (Infrared, near-infrared light) wavelength region is illustrated. The RGB camera is an example of a color camera.

[0023] For this reason, in the first embodiment, the learning images have a different amount of information than the evaluation images and the operation images. For example, the learning images have a different number of image channels (hereinafter abbreviated as the number of channels) than the evaluation images and the operation images. In the example of the first embodiment, the learning images are RGB images, so the number of channels of the learning images is 3. In contrast, the evaluation images and the operation images are grayscale images (monochrome images), so the number of channels of the evaluation images and the operation images is 1.

[0024] However, as will be apparent to those skilled in the art, the first and second imaging devices are not limited to the above examples. For example, the first imaging device may be a different type of imaging device from the second imaging device. In one aspect of the present invention, the first and second imaging devices may be selected so that images captured by the first imaging device (e.g., learning images) have a different amount of information than images captured by the second imaging device (e.g., evaluation images and operational images).

[0025] For example, if the first imaging device is a visible light camera, the second imaging device as a non-visible light camera may be an FIR camera (far-IR camera) that has light receiving characteristics in the FIR (Far-IR) wavelength range. Alternatively, the second imaging device may be an MWIR camera (mid-wavelength infrared camera) that has light receiving characteristics in the MWIR (Mid-Wavelength-IR) wavelength range. In these examples, the evaluation image and the operational image are grayscale images, so the number of channels for the evaluation image and the operational image is one.

[0026] (Learning Device 11) The learning device 11 (model generation device) includes a learning image acquisition unit 111, an evaluation image acquisition unit 112, a setting item acquisition unit 113, a learning image processing unit 114, and a learning model generation unit 115 (learning unit). The learning device 11 generates a learning model for image recognition by the image recognition device 12 based on the learning images. An example of the processing of the learning device 11 will be described below.

[0027] (Obtaining training and evaluation image sets) The training image acquisition unit 111 acquires at least some of the multiple training images stored in the training image DB 910 as a training image set. The training image acquisition unit 111 also acquires training label data corresponding to the training image set from the training image DB 910. The training image acquisition unit 111 supplies the acquired training image set and training label data to the training image processing unit 114.

[0028] The evaluation image acquisition unit 112 acquires, as an evaluation image set, at least a portion of the multiple evaluation images stored in the evaluation image DB 920. Furthermore, the training image acquisition unit 111 acquires evaluation label data corresponding to the evaluation image set from the evaluation image DB 920. The evaluation image acquisition unit 112 supplies the acquired evaluation image set and evaluation label data to the training image processing unit 114.

[0029] (Getting the image processing settings list) The setting item acquisition unit 113 acquires an image processing setting list stored in advance in the storage unit 80. The image processing setting list is a list showing a list of settings for image processing to be executed by the learning-time image processing unit 114. In other words, the image processing setting list is a list showing candidates for the final image processing setting, which will be described later.

[0030] Examples of image processing executed by the learning image processing unit 114 include: ·Standardization ·Histogram equalization Blur Adaptive thresholding Canny edge detector Sobel edge detector Laplacian edge detector Band-pass filter Affine transformation Homography transformation Therefore, the image processing according to an aspect of the present invention may include at least one of these. Therefore, the setting image processing according to an aspect of the present invention may include settings related to the image processing.

[0031] However, as will be apparent to those skilled in the art, the image processing according to one embodiment of the present invention is not limited to the above examples. The image processing according to one embodiment of the present invention may be any processing useful for reducing the difference between the features of the training image and the features of the evaluation image. In other words, the image processing according to one embodiment of the present invention may be any processing useful for improving the consistency between the features of the processed training image (e.g., IMGLP described below) and the features of the processed evaluation image (e.g., IMGTP described below).

[0032] Fig. 2 illustrates an example of an image processing setting list (LIST_IMP) in embodiment 1. Fig. 2 shows some items in LIST_IMP. As shown in Fig. 2, in this specification, for example, "edge detection using the Canny method" is also abbreviated as "Canny method." In LIST_IMP, (i) candidates for parameters (e.g., minVal and maxVal) used in the Canny method, and (ii) candidates for parameters (ksize) used in the Laplacian method are pre-recorded.

[0033] (Getting a list of hyperparameter settings) The setting item acquisition unit 113 acquires a hyperparameter setting list stored in advance in the storage unit 80. The hyperparameter setting list is a list showing a list of hyperparameters in a predetermined machine learning algorithm executed in the learning model generation unit 115. In other words, the hyperparameter setting list is a list showing candidates for hyperparameter settings during operation, which will be described later.

[0034] Examples of prescribed machine learning algorithms include: Neural Network (NN) Convolutional Neural Network (CNN) Vision Transformer (ViT) Support Vector Machine (SVM) Random Forest (RF) Examples include:

[0035] However, as will be apparent to those skilled in the art, the machine learning algorithm according to an aspect of the present invention is not limited to the above examples, and may be any algorithm that is applicable to image recognition.

[0036] Fig. 3 illustrates an example of a hyperparameter setting list (LIST_HP) in embodiment 1. Some items in LIST_HP are shown in Fig. 3. Examples of machine learning algorithms that are highly effective for image recognition include CNN and ViT.

[0037] Therefore, in the first embodiment, as an example, a case where CNN is selected as the predetermined machine learning algorithm will be described. For this reason, a list of hyperparameter settings in CNN is shown in LIST_HP in Fig. 3. Candidates for multiple hyperparameters (e.g., ksize, kchannel, and stride) in CNN are recorded in advance in LIST_HP.

[0038] (Example of processing flow in learning device 11) 4 to 8, the processing flow in the learning device 11 will be described in more detail. Fig. 4 is a flowchart that schematically shows the overall processing flow of the learning device 11V (particularly, the learning-time image processing unit 114 and the learning model generation unit 115).

[0039] As shown in FIG. 4, first, the learning model generation unit 115 sets the initial value of the final accuracy candidate value to 0 (S1). In other words, the learning model generation unit 115 initializes the final accuracy candidate value. Next, the learning image processing unit 114 initializes all flags in the image processing setting list (S2). Then, the learning image processing unit 114 performs image processing on each learning image in the learning image set (S3). The processing content of S3 will be described below.

[0040] 5 is a flowchart showing the flow of image processing for learning images. FIG. 5 is a flowchart showing the processing content of S3 described above in more detail. First, the learning image processing unit 114 selects one image processing setting from the image processing setting list (e.g., the setting shown in the first item in LIST_IMP in FIG. 2) and assigns flag A (a flag related to the image processing setting of the learning image set) to that image processing setting (S11). Flag A may also be referred to as the first flag.

[0041] Next, the training image processing unit 114 performs the image processing for which flag A was assigned in S11 on each training image in the training image set (S12). Then, the training image processing unit 114 temporarily stores (temporarily stores) the processed training images (training images after image processing) in the storage unit 80 (S13). Thereafter, the training image processing unit 114 performs image processing on each evaluation image in the evaluation image set (S14). The processing content of S14 will be described later.

[0042] Next, the learning-time image processing unit 114 determines whether flag A has been assigned to all items in the image processing setting list (S15). If flag A has not been assigned to all items in the image processing setting list (NO in S15), the learning-time image processing unit 114 selects another image processing setting that has not yet been selected from the image processing setting list (e.g., the setting shown as the second item in LIST_IMP in FIG. 2) and assigns flag A to that image processing setting (S16). Then, the process returns to S12, and the above-described processes in FIG. 5 are repeated.

[0043] On the other hand, if flag A is attached to all items in the image processing setting list (YES in S15), learning-time image processing unit 114 completes image processing for the learning images. In this way, S12 to S16 are repeated until flag A is attached to all items in the image processing setting list (in other words, until all patterns of image processing are completed for all learning images).

[0044] Fig. 6 is a flowchart showing the flow of image processing for evaluation images. Fig. 6 is a flowchart showing the processing content of S14 described above in more detail. First, the learning-time image processing unit 114 initializes flag B (a flag related to the image processing setting of the evaluation image set) for all items in the image processing setting list (S21). Flag B may be referred to as a second flag.

[0045] Next, the learning-time image processing unit 114 selects one image processing setting from the image processing setting list (for example, the setting shown in the first item in LIST_IMP in FIG. 2) and marks that image processing setting with flag B (S22).The learning-time image processing unit 114 then saves the image processing setting to which flag B was marked in S22 in the memory unit 80 as a temporary image processing setting (temporary image processing setting) (S23).

[0046] Next, the training image processing unit 114 performs the image processing for which flag B was assigned in S22 on each evaluation image in the evaluation image set (S24). Then, the training image processing unit 114 temporarily stores the processed evaluation images (evaluation images after image processing) in the storage unit 80 (S25). Thereafter, the learning model generation unit 115 sets hyperparameters (S26). The processing content of S26 will be described later.

[0047] Next, the learning-time image processing unit 114 determines whether flag B has been added to all items in the image processing setting list (S27). If flag B has not been added to all items in the image processing setting list (NO in S27), the learning-time image processing unit 114 selects another image processing setting that has not yet been selected from the image processing setting list (e.g., the setting shown as the second item in LIST_IMP in FIG. 2) and adds flag B to that image processing setting (S28). Then, the process returns to S23, and the above-described processes in FIG. 6 are repeated.

[0048] On the other hand, if flag B is set for all items in the image processing setting list (YES in S27), learning-time image processing unit 114 completes image processing for the evaluation images. In this way, S23 to S28 are repeated until flag B is set for all items in the image processing setting list (in other words, until image processing for all patterns is completed for all evaluation images).

[0049] 7 is a flowchart showing the flow of processing in hyperparameter setting. Fig. 7 is a flowchart showing the processing content of S26 described above in more detail. First, the learning model generation unit 115 initializes all flags in the hyperparameter setting list (S31).

[0050] Next, the learning model generation unit 115 selects one hyperparameter setting from the hyperparameter setting list (e.g., the setting shown in the first item in LIST_HP in FIG. 3) and assigns a flag HP to the selected hyperparameter setting (a flag related to the hyperparameter setting of a predetermined machine learning algorithm) (S32). The flag HP may also be referred to as a third flag.

[0051] Next, the learning model generation unit 115 performs initial machine learning settings using the hyperparameter settings flagged with HP in S32 (S33). In the example of embodiment 1, the learning model generation unit 115 performs initial CNN settings using the hyperparameter settings flagged with HP in S32. Thereafter, the learning model generation unit 115 performs learning (generation of a learning model), evaluation of the learning model, and determination of a final accuracy candidate value (S34). The processing content of S34 will be described later.

[0052] Next, the learning model generation unit 115 determines whether or not the flag HP has been added to all items in the hyperparameter setting list (S35). If the flag HP has not been added to all items in the hyperparameter setting list (NO in S35), the learning model generation unit 115 selects another hyperparameter setting that has not yet been selected from the hyperparameter setting list (e.g., the setting shown in the second item in LIST_HP in FIG. 3) and adds the flag HP to that hyperparameter setting (S36). Then, the process returns to S33, and the above-described processes in FIG. 7 are repeated.

[0053] On the other hand, if the flag HP is set to all items in the hyperparameter setting list (YES in S35), the learning model generation unit 115 completes the hyperparameter setting. In this way, S33 to S36 are repeated until the flag HP is set to all items in the hyperparameter setting list (in other words, until all patterns of hyperparameter setting have been applied to the predetermined machine learning algorithm).

[0054] 8 is a flowchart showing the process flow for learning, evaluation of the learning model, and determination of the final accuracy candidate value. FIG. 8 is a flowchart showing the process content of S34 described above in more detail.

[0055] First, the learning model generation unit 115 generates (constructs) a learning model by performing learning using a predetermined machine learning algorithm using the processed learning image (S41). In the example of embodiment 1, the learning model generation unit 115 generates the learning model by executing CNN. The CNN in the example of embodiment 1 is an example of a supervised learning algorithm that uses the processed learning image and learning label data. Then, the learning model generation unit 115 temporarily stores the learning model generated in S41 in the storage unit 80 (S42).

[0056] Next, the learning model generation unit 115 evaluates (verifies) the learning model temporarily saved in S42 using the processed evaluation image (S43). In the example of embodiment 1, the learning model generation unit 115 inputs the processed evaluation image into the learning model temporarily saved in S42 and acquires an index value indicating the evaluation accuracy (e.g., classification accuracy) of the learning model. The index value may be derived by comparing the output of the learning model (e.g., a value indicating the class of the object shown in the processed evaluation image classified by the learning model) with the evaluation label data. The index value can also be expressed as a value indicating the quality of the learning model.

[0057] In the first embodiment, the learning model generation unit 115 acquires the accuracy rate (Accuracy) as the evaluation accuracy (index value). However, as will be apparent to those skilled in the art, the evaluation accuracy according to one aspect of the present invention is not limited to the above example, and other evaluation accuracies known in the field of machine learning may be used. Therefore, for example, the learning model generation unit 115 may acquire the precision rate (Precision) or the recall rate (Recall) as the evaluation accuracy.

[0058] Alternatively, the learning model generation unit 115 may acquire an F-score as the evaluation accuracy. As is well known, the F-score is the harmonic mean of the precision and recall. As is clear from the above explanation, the learning model generation unit 115 may acquire a value (e.g., a statistical value) derived from any combination of the accuracy rate, precision, and recall as the evaluation accuracy.

[0059] The learning model generation unit 115 temporarily stores the evaluation accuracy acquired in S43 (the evaluation result of the learning model temporarily stored in S42) (S44). Then, the learning model generation unit 115 compares the evaluation accuracy temporarily stored in S44 with the final accuracy candidate value (S45). Specifically, the learning model generation unit 115 determines whether the evaluation accuracy is equal to or greater than the final accuracy candidate value.

[0060] If the evaluation accuracy is not greater than or equal to the final accuracy candidate value (NO in S45), that is, if the evaluation accuracy is less than the final accuracy candidate value, the learning model generation unit 115 completes the processing of learning, evaluation of the learning model, and determination regarding the final accuracy candidate value.

[0061] On the other hand, if the evaluation accuracy is equal to or greater than the final accuracy candidate value (YES in S45), the learning model generation unit 115 updates the final accuracy candidate value to the value of the evaluation accuracy temporarily saved in S44 (S46).

[0062] Next, the training image processing unit 114 updates the final image processing settings to the temporary image processing settings temporarily saved in S23 (S47). Then, the learning model generation unit 115 temporarily saves the learning model temporarily saved in S42 as the final learning model (S48). Then, the learning model generation unit 115 completes the processes of learning, evaluation of the learning model, and determination of the final accuracy candidate value.

[0063] (Summary of the processing of the learning device 11) As described above, the learning device 11 can comprehensively (by exhaustively trying) all image processing settings (all patterns of image processing settings) defined in the image processing setting list by performing the series of processes shown in Figures 4 to 8. In addition, the learning device 11 can also comprehensively apply all hyperparameter settings (all patterns of hyperparameter settings) defined in the hyperparameter setting list.

[0064] Therefore, in the first embodiment, the learning model generation unit 115 selects the final image processing settings stored in the storage unit 80 as the in-operation image processing settings (image processing settings to be applied when the image recognition device 12 is in operation) at the completion of the learning phase (at the completion of the series of processes in Figures 4 to 8). Then, the learning model generation unit 115 selects the final learning model stored in the storage unit 80 as the in-operation learning model (learning model to be applied when the image recognition device 12 is in operation) at the completion of the learning phase.

[0065] The in-operation image processing settings are image processing settings that correspond to the final accuracy candidate value stored in the memory unit 80 at the completion of the learning phase (i.e., the highest evaluation accuracy throughout the entire learning phase). For this reason, the in-operation image processing settings are expected to be the image processing settings that are most suitable for the in-operation learning model. For this reason, the in-operation image processing settings may be referred to as the best image processing settings. Furthermore, the in-operation learning model is a learning model that corresponds to the in-operation image processing settings and is expected to be the learning model that is most suitable for the in-operation image processing settings. For this reason, the in-operation learning model may be referred to as the best learning model. Furthermore, the hyperparameter settings that correspond to the in-operation learning model may be referred to as the in-operation hyperparameter settings (hyperparameter settings that are applied when the image recognition device 12 is in operation). The in-operation hyperparameter settings may be referred to as the best hyperparameter settings.

[0066] As described above, the learning device 11 can find out the hyperparameter settings during operation by searching for the hyperparameter settings, and can also find out the learning model during operation corresponding to the hyperparameter settings during operation.

[0067] According to the learning device 11, in the learning phase, it is possible to find an optimal pair of image processing settings and optimal hyperparameters by exhaustively searching (grid search) all predefined patterns of image processing settings and all predefined patterns of hyperparameter settings. In other words, according to the learning device 11, it is possible to find an optimal pair of image processing settings and an optimal learning model (i.e., an in-operation image processing setting and an in-operation learning model) in the learning phase. Below, as an example, a case will be described in which (i) the in-operation image processing setting is the setting shown in the second item in LIST_IMP of FIG. 2, and (ii) the in-operation learning model is a learning model generated with the setting shown in the first item in LIST_HP of FIG. 3.

[0068] (Image Recognition Device 12) Next, referring back to FIG. 1 described above, image recognition device 12 will be described. Image recognition device 12 includes an operational image acquisition unit 121, an operational image processing unit 122, and an image recognition unit 123. As described below, image recognition device 12 performs image processing on operational images using operational image processing settings preselected in learning device 11. Then, image recognition device 12 performs image recognition on processed operational images (operational images after image processing) using an operational learning model preselected in learning device 11. A series of processes (operation phase) in image recognition device 12 will be described below. The operation phase may also be referred to as the image recognition phase.

[0069] (Acquisition of operational images) The operation image acquisition unit 121 acquires an operation image from the operation image DB 930, for example, when the input unit 71 receives a predetermined user operation. The operation image acquisition unit 121 supplies the acquired operation image to the operation image processing unit 122.

[0070] 1 illustrates, for convenience of explanation, a case in which the learning image acquisition unit 111, the evaluation image acquisition unit 112, and the operational image acquisition unit 121 are separate functional units. However, as would be clear to one skilled in the art, the learning image acquisition unit 111 and the evaluation image acquisition unit 112 may be provided as an integrated functional unit. Furthermore, the learning image acquisition unit 111, the evaluation image acquisition unit 112, and the operational image acquisition unit 121 may be provided as an integrated functional unit.

[0071] (Image processing for operational images) The operational image processing unit 122 acquires the operational image processing settings from the learning image processing unit 114. Then, the operational image processing unit 122 generates a processed operational image by performing image processing on the operational image in accordance with the operational image processing settings (e.g., the operational image processing settings are the settings shown in the second item in LIST_IMP in FIG. 2). Hereinafter, image processing in accordance with the operational image processing settings will be referred to as operational image processing. The operational image processing may also be referred to as best image processing.

[0072] 1, for convenience of explanation, the example illustrates a case where the learning-time image processing unit 114 and the operation-time image processing unit 122 are separate functional units. However, as will be apparent to those skilled in the art, the learning-time image processing unit 114 and the operation-time image processing unit 122 may be provided as an integrated functional unit.

[0073] Fig. 9 is a diagram illustrating an example of image processing during operation. To aid understanding, Fig. 9 shows an example in which image processing during operation is performed on each of a learning image IMGL, an evaluation image IMGT, and an operational image IMGO. As can be understood from the above explanations, the image processing during operation in the example of Fig. 9 is the Canny algorithm with minVal=100 and maxVal=200.

[0074] 9 is an example of an image (RGB image) captured by an RGB camera serving as a first imaging device. In contrast, IMGT and IMGO are each an example of an image (IR image) captured by an IR camera serving as a second imaging device. As is clear from the above explanations, an IR image is an example of a grayscale image.

[0075] The processed learning image IMGLP in FIG. 9 is a binary image obtained by applying in-operation image processing to IMGL. Note that the binary image in the example of FIG. 9 may also be referred to as an edge image. Similarly, the processed evaluation image IMGTP is a binary image obtained by applying in-operation image processing to IMGT. Furthermore, the processed operational image IMGOP is a binary image obtained by applying in-operation image processing to IMGO.

[0076] Because IMGLP and IMGOP are both binary images, IMGLP and IMGOP have more in common than IMGL and IMGO. For example, IMGLP and IMGOP have in common the fact that they both have one channel. IMGLP and IMGOP also have in common the fact that they only have non-zero pixel values ​​at edge positions. In this way, best image processing can reduce the differences that existed between the features of IMGL and IMGO.

[0077] (Image recognition for processed operational images) The image recognition unit 123 acquires the processed operation image from the operation image processing unit 122, and also acquires the operation learning model from the learning model generation unit 115. Then, the image recognition unit 123 inputs the processed operation image into the operation learning model, thereby acquiring an output corresponding to the processed operation image from the operation learning model.

[0078] The image recognition unit 123 performs image recognition on the processed operational image based on the output of the in-operation learning model. As an example, the image recognition unit 123 may identify the class of an object appearing in the processed operational image based on the output of the in-operation learning model. Note that the image recognition unit 123 may further detect the position of the object in the processed operational image based on the output of the in-operation learning model.

[0079] As an example, image recognition unit 123 may supply the results of image recognition on the processed operation image to display unit 72 and cause the results to be displayed on display unit 72. This allows the results to be visually presented to the user. Image recognition unit 123 may also cause display unit 72 to display each piece of data related to the results. For example, display unit 72 may further display (i) the processed operation image that was the subject of image recognition processing in image recognition unit 123, and (i) an operation image corresponding to the processed operation image.

[0080] (effect) Conventionally, the case where the first imaging device and the second imaging device are the same (for convenience, referred to as the imaging device matching case) has been mainly assumed. In other words, the case where the learning image and the operational image have the same amount of information has been mainly assumed. However, depending on the situation in the operational phase, the above-mentioned imaging device mismatch case may occur.

[0081] As an example, consider a case where image recognition is performed using a learning model on an image captured by an imaging device at night. Generally, at night, the amount of ambient light (surrounding light) is low, making it difficult for an RGB camera to capture an image that clearly captures objects within the capture range. At night, it is expected that an IR camera can be used as the imaging device to capture an image that clearly captures objects within the capture range.

[0082] Therefore, in the above case, it is ideal to generate a learning model using images captured by an IR camera (IR images) in the learning phase prior to the operation phase. The image capture device matching case that has been mainly assumed in the past is based on this ideal concept.

[0083] However, in practice, it is not always easy to prepare a sufficient number of IR images for training. Generally, in the field of image recognition, although there are many open datasets for RGB images, there are almost no open datasets for IR images.

[0084] In such a case, from the viewpoint of ensuring the quantity of training images in the training phase, one option is to use RGB images as training images to generate a training model. However, as described above, in the case of a different imaging device, the difference between the features of the training images and the features of the operational images may result in a decrease in image recognition accuracy in the operational phase.

[0085] In view of the above-mentioned problems in the prior art, the inventors of the present application have newly created an information processing device 1 (particularly, a learning device 11 and an image recognition device 12). The learning device 11 generates a learning model using processed learning images. Therefore, a sufficient number of processed learning images can be used as learning data to generate multiple learning models.

[0086] Additionally, the learning device 11 evaluates each learning model (more specifically, the hyperparameter settings of each learning model) using the post-processing evaluation image. The learning device 11 can also evaluate each image processing setting using the post-processing evaluation image. Therefore, the learning device 11 can (i) select the above-mentioned in-operation learning model (best learning model) from among the multiple generated learning models, and (ii) select the above-mentioned in-operation image processing setting (best image processing setting) from among multiple predefined image processing settings.

[0087] Next, image recognition device 12 performs image processing on the operational image in accordance with the operational image processing settings preselected in learning device 11. That is, image recognition device 12 performs operational image processing on the operational image to obtain a processed operational image. Then, image recognition device 12 performs image recognition on the processed operational image using the operational learning model preselected in learning device 11.

[0088] As described above, image processing during operation can reduce the differences that exist between the features of the training images and the features of the operational images. In other words, image processing can be performed on the operational images to improve the consistency between the features of the processed training images and the features of the processed operational images.

[0089] As a result, by performing image recognition on processed operational images using the in-operation learning model, it is possible to prevent a decrease in image recognition accuracy in the operational phase in cases where the imaging device is different. As described above, the information processing device 1 makes it possible to improve image recognition accuracy through machine learning in cases where the imaging device is different.

[0090] [Modification] In the first embodiment, an example is given in which an in-operation learning model is selected from among a plurality of learning models by selecting an in-operation hyperparameter setting from among a plurality of hyperparameter settings in the learning device 11. However, a learning device according to one aspect of the present invention is only required to be able to select an in-operation learning model corresponding to an in-operation image processing setting from among a plurality of learning models.

[0091] Therefore, a learning device according to one aspect of the present invention does not necessarily need to select an operational hyperparameter setting from among multiple hyperparameter settings. Therefore, for example, the hyperparameter setting may be fixed throughout multiple learning phases. However, from the perspective of obtaining a higher-quality operational learning model, as described in the first embodiment, it is preferable to select an operational learning model from among multiple learning models by selecting an operational hyperparameter setting from among multiple hyperparameter settings.

[0092] [Modification] In the first embodiment, the case where the predetermined machine learning algorithm is CNN is exemplified. Therefore, in the first embodiment, a hyperparameter setting list (LIST_HP) corresponding to CNN is exemplified. However, as described above, the machine learning algorithm according to one aspect of the present invention is not limited to CNN. Therefore, it is sufficient to prepare a hyperparameter setting list corresponding to a pre-selected machine learning algorithm.

[0093] Fig. 10 is a diagram showing another example of a hyperparameter setting list. Fig. 10 shows an example of a hyperparameter setting list (LIST_HPV1) corresponding to SVM. Fig. 11 is a diagram showing yet another example of a hyperparameter setting list. Fig. 11 shows an example of a hyperparameter setting list (LIST_HPV2) corresponding to RF.

[0094] [Modification] In the first embodiment, a case where all patterns of image processing settings and all patterns of hyperparameter settings are comprehensively searched is illustrated. However, the learning device 11 does not necessarily have to search all patterns of image processing settings and all patterns of hyperparameter settings.

[0095] As an example, in the learning device 11, the initial value of the final accuracy candidate value may be set to a predetermined threshold value (a non-zero value). Then, the learning device 11 may end the search for image processing settings and hyperparameter settings when the evaluation accuracy becomes equal to or greater than the final accuracy candidate value for the first time in the process of FIG. 8 described above. In other words, the learning device 11 may end the search when the final accuracy candidate value is updated for the first time.

[0096] In this modified example, the training image processing unit 114 also selects the final image processing settings stored in the storage unit 80 as the operational image processing settings upon completion of the training phase. The training model generation unit 115 then selects the final training model stored in the storage unit 80 as the operational training model upon completion of the training phase.

[0097] As described above, the learning device 11 in this modified example terminates the search for image processing settings and hyperparameter settings when the evaluation accuracy reaches or exceeds the final accuracy candidate value. Therefore, according to this modified example, it is possible to find in-operation image processing settings and in-operation learning models without exhaustively searching all patterns of image processing settings and all patterns of hyperparameter settings.

[0098] Therefore, according to this modification, even if at least one of the total number of image processing setting patterns and the total number of hyperparameter setting patterns is large, it is possible to find an in-operation image processing setting and an in-operation learning model with a relatively low computational cost. Therefore, for example, the learning phase can be completed in a shorter time than in the first embodiment. Furthermore, it is also possible to reduce the computational resources (e.g., memory capacity) required to execute the learning phase compared to the first embodiment.

[0099] [Embodiment 2] FIG. 12 is a block diagram showing the configuration of the main parts of an information processing system 100V according to the second embodiment. The information processing device and control device according to the second embodiment are referred to as an information processing device 1V and a control device 10V, respectively. The control device 10V includes a learning device 11V (model generation device) instead of the learning device 11. The learning device 11V includes a learning-time image processing unit 114V and a learning model generation unit 115V (learning unit). The learning model generation unit 115V includes an optimization execution unit 116. In the second embodiment, some of the processing in the learning phase differs from that in the first embodiment.

[0100] (Example of processing flow in learning device 11V) The flow of processing in learning device 11V will be described with reference to Figures 13 to 14 and 16 in this order. Figure 13 is a flowchart that schematically shows the flow of main processing in learning device 11V.

[0101] First, the optimization execution unit 116 reads out an optimization list stored in advance in the storage unit 80 and acquires the number of items in the optimization list (total number of items in the list N) (S51). The optimization list lists a list of image processing settings and hyperparameter settings to be optimized.

[0102] The optimization list items include (i) an item for image processing settings for the training image set (for convenience, referred to as image processing setting A), (ii) an item for image processing settings for the evaluation image set (for convenience, referred to as image processing setting B), and (iii) an item for hyperparameter settings.

[0103] Therefore, N is given as the sum of (i) the number of items in image processing setting A, (ii) the number of items in image processing setting, and (iii) the number of items in hyperparameter setting. In the second embodiment, a case where the number of each item is 1 is exemplified. Therefore, the following description will be given for the case where N=3.

[0104] Next, the optimization execution unit 116 sorts the image processing settings A and B in the optimization list based on the item contents of the image processing settings A and the item contents of the image processing settings B. Then, the learning model generation unit 115V sorts the hyperparameter settings in the optimization list based on the item contents of the hyperparameter settings (S52).

[0105] Next, the optimization execution unit 116 initializes the flags of each lattice point in the N-dimensional space (S53). The term "lattice point" as used herein refers to a point that forms a spatial lattice. In the second embodiment, a certain lattice point P is given by a combination of the indexes (subscripts) of the image processing setting A, image processing setting B, and hyperparameter items in the optimization list.

[0106] That is, the lattice point P in the second embodiment is P=(SA,SB,SH) Here, SA is an index pointing to an item in image processing setting A, SB is an index pointing to an item in image processing setting B, and SH is an index pointing to an item in hyperparameter setting.

[0107] Next, the optimization execution unit 116 sets N+1 combinations of random indexes from the items in the optimization list. Then, the optimization execution unit 116 generates N+1 lattice points corresponding to the combinations as initial points (S54). Next, the learning model generation unit 115V attaches a flag to each initial point. In the second embodiment, since N+1=4, the optimization execution unit 116 P1=(SA1,SB1,SH1) P2=(SA2,SB2,SH2) P3=(SA3,SB3,SH3) P4=(SA4,SB4,SH4) These four grid points are generated as initial points.

[0108] Next, the learning model generation unit 115V calculates the function f(P j) to obtain the evaluation accuracy corresponding to each initial point (for convenience, also referred to as "evaluation accuracy of each initial point") (S55). The processing content of S55 will be described later. In the example of the second embodiment, the learning model generation unit 115V obtains the evaluation accuracy of each of the four initial points.

[0109] Next, the optimization execution unit 116 ranks the initial points based on the evaluation accuracy (S56). For example, the optimization execution unit 116 may rank the point to which the best evaluation accuracy (highest evaluation accuracy) is assigned among the four initial points as the best point B. (m) Here, m is the number of times optimization is repeated (number of optimization iterations) in S57 described below. m is an integer that satisfies 1≦m≦q. q is the upper limit of the number of optimization iterations. q is set in advance. Then, the optimization execution unit 116 determines the point assigned with the worst evaluation accuracy (lowest evaluation accuracy) among the four initial points as the worst point W (m) Next, the learning model generation unit 115V determines the point among the four initial points to which the second worst evaluation accuracy (second lowest evaluation accuracy) is assigned as the second worst point BW (m) It is determined as follows.

[0110] Next, the optimization execution unit 116 executes optimization using a predetermined optimization method (optimization algorithm) based on the ranked points (S57). The processing content of S57 will be described later.

[0111] 14 is a flowchart showing the flow of the process for acquiring the evaluation accuracy. FIG. 14 is a flowchart showing the process content of the above-mentioned S55 in more detail. First, the learning-time image processing unit 114V calculates the accuracy of the point P j Index SA included in j is applied to each training image in the training image set (S61), where j is an integer that satisfies 1≦j≦N+1. In the example of the second embodiment, 1≦j≦4.

[0112] Next, the learning model generation unit 115V calculates the point P j Index SH included in jThen, the learning model generation unit 115V uses the hyperparameter settings to generate a learning model according to a predetermined machine learning algorithm using each processed learning image in the processed learning image set (set of processed learning images) obtained in S61 (S62).

[0113] Next, the learning image processing unit 114V calculates the point P j Index SB included in j The learning model generation unit 115V performs image processing corresponding to the above on each evaluation image in the evaluation image set (S63). Then, the learning model generation unit 115V evaluates the learning model obtained in S62 using each processed evaluation image in the processed evaluation images (set of processed evaluation images) obtained in S63, and acquires an index value indicating the evaluation result of the learning model as the evaluation accuracy (S64).

[0114] As an example, the learning model generation unit 115V may acquire the micro-average recall of the learning model as an index value indicating the evaluation result of the learning model. In this case, the learning model generation unit 115V may calculate the micro-average recall at the point P j However, as will be clear to those skilled in the art, the index value is not limited to the micro-average recall.

[0115] In the second embodiment, an example is shown in which the Nelder-Mead method is used as the optimization method. The Nelder-Mead method is also called the downhill simplex method, the hyperpolyhedron method, or the amoeba method. The Nelder-Mead method can solve multidimensional nonlinear optimization problems without relying on differential calculus. For example, the Nelder-Mead method can derive a locally optimal solution of an objective function without relying on differential calculus. Therefore, the Nelder-Mead method can obtain an extreme value (maximum or minimum value) of an objective function without relying on differential calculus. In the second embodiment, f(P j) is an example of an objective function. Therefore, in the second embodiment, by using the Nelder-Mead method, it is possible to obtain a local maximum value (ideally the maximum value) of the evaluation accuracy.

[0116] Figure 15 is a diagram for explaining the concept of the Nelder-Mead method. For simplicity, Figure 15 illustrates the case of N=2 (two-dimensional case). Figure 15 shows the relationship between B and m for m=1, 2, q-1, and q. (m) , W (m) , and B.W. (m) The dotted lines in the example of FIG. 15 indicate the contours of the evaluation accuracy.

[0117] As shown in Figure 15, according to the Nelder-Mead method, as the optimization iteration process progresses (increases in m), the combination of each parameter (B (m) , W (m) , and B.W. (m) In other words, the Nelder-Mead method allows each parameter to be optimized to reach a local optimum (maximum value of evaluation accuracy).

[0118] Fig. 16 is a flowchart showing the flow of the process for executing optimization. Fig. 16 is a flowchart showing the process content of S57 described above in more detail. Fig. 16 illustrates optimization using the Nelder-Mead method.

[0119] First, in S71, the optimization execution unit 116 calculates the worst point W (m) Each point other than (collectively, each point P j (m) ) and each point P j (m) The center of gravity of the hyperpolyhedron G (m)’ Specifically, in the example of the second embodiment, the optimization execution unit 116 calculates:

number

[0120] Next, in S72, the optimization execution unit 116 calculates the line segment W (m) G (m) External division point R that divides (m)’ Specifically, the optimization execution unit 116 calculates the following: R (m)’ =2G (m) -W (m) …(2) As shown above, the external division point R (m)’ Next, the optimization execution unit 116 calculates the external division point R (m)’ nearest unflagged point R to (m) Then, the optimization execution unit 116 identifies the point R (m) Flag the .

[0121] Next, in S73, the optimization execution unit 116 calculates f(R (m) ) is derived. Then, the optimization execution unit 116 calculates f(R (m) )≦f(B (m) ) …(3) It is determined whether or not the following condition (for convenience, referred to as the first determination condition) is satisfied. If the first determination condition is satisfied, the process proceeds to S74. If the first determination condition is not satisfied, the process proceeds to S77.

[0122] If the first judgment condition is satisfied, in S74, the optimization execution unit 116 calculates the line segment W (m) R (m) External division point E, which divides (m)’ Specifically, the optimization execution unit 116 calculates the following: E (m)’ =3G (m) -2W (m) …(4) As shown, the external division point E (m)’ Next, the optimization execution unit 116 calculates the external division point E (m)’ nearest unflagged point E to (m) Then, the optimization execution unit 116 identifies the point E (m) Flag the .

[0123] Next, in S75, the optimization execution unit 116 uses the function f to calculate f(E (m) ) is derived. Then, the optimization execution unit 116 calculates f(E (m) )≦f(R (m) ) …(5) It is determined whether or not the following condition (for convenience, referred to as the second determination condition) is satisfied. If the second determination condition is satisfied, the process proceeds to S76. If the second determination condition is not satisfied, the process proceeds to S78.

[0124] If the second judgment condition is satisfied, in S76, the optimization execution unit 116 calculates the worst point W (m) is removed from the optimization target, and point E (m) Then, the process proceeds to step S83, which will be described later.

[0125] If the first judgment condition is not satisfied, in S77, the optimization execution unit 116 f(R (m) )≦f(BW (m) ) …(6) It is determined whether or not the following condition (for convenience, referred to as the third determination condition) is satisfied. If the third determination condition is satisfied, the process proceeds to S78. If the third determination condition is not satisfied, the process proceeds to S79.

[0126] If the third judgment condition is satisfied, in S78, the optimization execution unit 116 calculates the worst point W (m) is removed from the optimization target, and point R (m) Then, the process proceeds to S83.

[0127] If the third judgment condition is not satisfied, in S79, the optimization execution unit 116(m) G (m) Midpoint S of (m)’ Specifically, the optimization execution unit 116 calculates the following: S (m)’ =(1 / 2)×(G (m) +W (m) ) …(7) As shown, the midpoint S (m)’ Next, the optimization execution unit 116 calculates the midpoint S (m)’ nearest unflagged point S to (m) Then, the optimization execution unit 116 identifies the point S (m) Flag the .

[0128] Next, in S80, the optimization execution unit 116 uses the function f to calculate f(S (m) ) is derived. Then, the optimization execution unit 116 calculates f(S (m) )≦f(W (m) ) …(8) It is determined whether or not the following condition (for convenience, referred to as the fourth determination condition) is satisfied. If the fourth determination condition is satisfied, the process proceeds to S81. If the fourth determination condition is not satisfied, the process proceeds to S82.

[0129] If the fourth judgment condition is satisfied, in S81, the optimization execution unit 116 calculates the worst point W (m) is removed from the optimization target, and point S (m) Then, the process proceeds to S83.

[0130] If the fourth judgment condition is not satisfied, in S82, the optimization execution unit 116 calculates the line segment B (m) P j (m) Midpoint P of j (m’)’ Specifically, the optimization execution unit 116 calculates the following: P j (m’)’ =(1 / 2)×(P j (m) +B (m) ) …(9) As shown, the midpoint Pj (m’)’ Next, the optimization execution unit 116 calculates the midpoint P j (m’)’ The nearest unflagged point P to j (m)’ Then, the optimization execution unit 116 identifies each point P j (m) At each point P j (m)’ Next, the optimization execution unit 116 updates each point P j (m)’ Then, the process proceeds to S83.

[0131] In S83, the optimization execution unit 116 performs the same procedure as in S55 described above, using the function f to calculate the j (m) Next, the optimization execution unit 116 calculates the evaluation accuracy of each point P based on the evaluation accuracy in the same manner as in S56 described above. j (m) Rank the following.

[0132] Next, in S84, the optimization execution unit 116 determines whether a predetermined setting judgment condition is satisfied. If the setting judgment condition is satisfied, the process proceeds to S85. If the setting judgment condition is not satisfied, the process returns to S71, and the above-mentioned processes are repeated. In this way, the optimization execution unit 116 repeats the above-mentioned processes until the setting judgment condition is satisfied.

[0133] The set determination conditions in the second embodiment may include, for example, at least one of the following conditions 1 to 7: j (m) is the volume of the hyperpolyhedron formed by the initial points, and Vs is the volume of the hyperpolyhedron formed by each initial point.

number

[0134] (Condition 1) Vl / Vs (the ratio of Vl to Vs) is equal to or less than a predetermined threshold (volume ratio threshold); (Condition 2) m reaches the upper limit q; (Condition 3) The calculation time for the optimization process has reached a predetermined time; (Condition 4) Point B (m) was not changed for more than a given number of iterations; (Condition 5) Point B (m) The evaluation accuracy of is equal to or greater than a predetermined threshold (evaluation accuracy threshold); (Condition 6) α is equal to or less than a predetermined threshold (gradient threshold); (Condition 7) All grid points are flagged.

[0135] However, as will be apparent to those skilled in the art, the set judgment condition is not limited to the above example. The set judgment condition according to one aspect of the present invention may be set to represent at least one of "a state in which a local optimum solution of the function f (i.e., the evaluation accuracy in the example of the second embodiment) has been found" and "a condition under which further search for the local optimum solution should be terminated."

[0136] If the set judgment condition is satisfied, in S85, the optimization execution unit 116 selects the best point B (m) As an example, the best point B in S85 is used. (m) The coordinates of B (m) =(SA best ,SB best ,SH best ) In this case, the optimization execution unit 116 performs the following: (i) SA best The item of image processing setting A corresponding to (ii) SB is set as the best image processing setting A (the optimal image processing setting for the training image set), best The image processing setting B corresponding to (iii) SH is set as the best image processing setting B (the optimal image processing setting for the evaluation image set), best The hyperparameter setting items corresponding to the above are selected as the best hyperparameter settings.

[0137] Then, the optimization execution unit 116 selects the best image processing setting B as the image processing setting during operation in embodiment 2. Furthermore, the optimization execution unit 116 selects, from among the multiple learning models generated by the learning model generation unit 115V through the above-mentioned processes in embodiment 2, the learning model corresponding to the best hyperparameter setting as the best learning model in embodiment 2. Then, the optimization execution unit 116 selects the best learning model as the learning model during operation in embodiment 2. The subsequent processes in the information processing system 100V are the same as those in embodiment 1.

[0138] (Effects of the 100V Information Processing System) According to the learning device 11V of embodiment 2, it is possible to find in-operation image processing settings and in-operation learning models by executing a predetermined optimization method (e.g., the Nelder-Mead method). Unlike the learning device 11 of embodiment 1, it is possible to find in-operation image processing settings and in-operation learning models without exhaustively searching all patterns of image processing settings and all patterns of hyperparameter settings.

[0139] In this way, similar to the modified example of embodiment 1, the learning device 11V can reduce the computational cost for finding in-operation image processing settings and in-operation learning models compared to the learning device 11. In addition, the learning device 11V can find more effective in-operation image processing settings and in-operation learning models through the execution process of the optimization method compared to the modified example of embodiment 1.

[0140] [Modification] In the second embodiment, the Nelder-Mead method is used as an optimization method, but as will be apparent to those skilled in the art, other methods may be used as the optimization method according to one aspect of the present invention.

[0141] Another example of an optimization method is the Powell method. Like the Nelder-Mead method, the Powell method can solve multidimensional nonlinear optimization problems without relying on differential calculus. Therefore, the Powell method can be used instead of the Nelder-Mead method to derive the maximum value of evaluation accuracy.

[0142] In addition, when solving one-dimensional nonlinear optimization problems, the golden section method can also be used as an optimization method. The golden section method can solve one-dimensional nonlinear optimization problems without relying on differential calculus. The golden section method can derive the maximum value of evaluation accuracy expressed as a one-dimensional nonlinear function without relying on differential calculus.

[0143] As will be apparent to those skilled in the art, the optimization method according to an embodiment of the present invention is not particularly limited. However, from the viewpoint of facilitating implementation of the algorithm, the optimization method according to an embodiment of the present invention is preferably an optimization method that does not rely on differential calculus, i.e., derivative-free optimization (DFO). The Nelder-Mead algorithm, the Powell algorithm, and the golden section method are all examples of DFO.

[0144] [Software implementation example] The functions of information processing system 100 / 100V (hereinafter referred to as "device" for convenience) can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (particularly each part included in control device 10 / 10V).

[0145] In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The control device and storage device execute the program, thereby realizing the functions described in each of the above embodiments.

[0146] The program may be non-transitory and may be recorded on one or more computer-readable recording media. The recording media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

[0147] In addition, some or all of the functions of the control blocks can be realized by logic circuits. For example, an integrated circuit in which a logic circuit that functions as each of the control blocks is formed is also included in the scope of one aspect of the present invention. In addition, the functions of the control blocks can be realized by, for example, a quantum computer.

[0148] As is clear from the description of each of the above embodiments, each process described in each of the above embodiments can be executed by AI (Artificial Intelligence). In this case, the AI ​​may run on the control device or on another device (for example, an edge computer or a cloud server).

[0149] [Additional Notes] One aspect of the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of one aspect of the present invention. [Explanation of symbols]

[0150] 1. 1V information processing equipment 10, 10V control device 11, 11V Learning device (model generation device) 12 Image recognition device 100, 100V Information Processing System 111 Learning image acquisition unit 112 Evaluation image acquisition unit 113 Setting item acquisition section 114, 114V Image processing unit during learning 115, 115V Learning model generation unit (learning unit) 116 Optimization Execution Unit 121 Operational Image Acquisition Unit 122 Operational Image Processing Unit 123 Image Recognition Unit 910 Learning Image DB 920 Evaluation Image DB 930 Operational Image DB IMGL learning images IMGLP processed training image IMGT evaluation images IMGTP processed evaluation image IMGO operational image IMGOP processed operational image

Claims

1. A model generation device that generates a learning model for an image recognition device to perform image recognition in an image recognition phase based on operational images captured by a second imaging device different from a first imaging device, based on a plurality of learning images captured by the first imaging device, in a learning phase preceding the image recognition phase, The model generating device (i) a learning image processing unit that applies image processing according to each of a plurality of predetermined image processing settings to the plurality of learning images to generate a plurality of processed learning images corresponding to each of the plurality of predetermined image processing settings, and (ii) applies the image processing to a plurality of evaluation images captured by the second imaging device to generate a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; a learning unit that executes a predetermined machine learning algorithm to generate a plurality of learning models using the plurality of processed learning images corresponding to each of the plurality of predetermined image processing settings, The learning unit (i) selects an operational image processing setting to be applied in the image recognition phase from among the plurality of specified image processing settings based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of specified image processing settings, and (ii) selects a learning model corresponding to the operational image processing setting from among the plurality of learning models as an operational learning model to be applied in the image recognition phase.

2. The learning section is generating the plurality of learning models using the plurality of processed learning images corresponding to the plurality of predetermined image processing settings by executing the predetermined machine learning algorithm using each of a plurality of predetermined hyperparameter settings; selecting, based on the plurality of index values, from among the plurality of predetermined hyperparameter settings, an operational hyperparameter setting to be applied during operation of the image recognition device; The model generation device according to claim 1 , wherein a learning model corresponding to the hyperparameter setting during operation is selected as the during operation learning model from among the plurality of learning models.

3. 3. The model generating device according to claim 1, wherein the image processing includes at least one of standardization, histogram equalization, blurring, adaptive binarization, edge detection, band-pass filtering, affine transformation, and homography transformation.

4. The model generating device according to claim 1 , wherein the predetermined machine learning algorithm is a convolutional neural network or a vision transformer.

5. The model generating device according to claim 1 , wherein the first imaging device and the second imaging device have different light receiving characteristics.

6. the first imaging device is a visible light camera having light receiving characteristics in a visible light wavelength region, 6. The model generating device according to claim 5, wherein the second imaging device is a non-visible light camera having a light receiving characteristic in a non-visible light wavelength region.

7. the visible light camera is an RGB camera, 7. The model generating device according to claim 6, wherein the non-visible light camera is a near-infrared light camera having light receiving characteristics in the near-infrared light wavelength region.

8. The model generating device according to claim 1 , wherein the plurality of training images and the plurality of evaluation images have different numbers of image channels.

9. A model generation device as described in any one of claims 1 to 8, wherein the learning unit (i) finds the operational image processing setting by comprehensively searching each of the plurality of specified image processing settings based on the plurality of index values, and (ii) finds the operational learning model by comprehensively searching each of the plurality of learning models.

10. 9. The model generation device according to claim 1, wherein the learning unit terminates the search for each of the plurality of predetermined image processing settings and the search for each of the plurality of learning models when the maximum value among the plurality of index values ​​becomes equal to or greater than a predetermined threshold.

11. A model generation device as described in any one of claims 1 to 8, wherein the learning unit (i) finds the operational image processing setting by searching each of the plurality of specified image processing settings based on the plurality of index values ​​in accordance with a predetermined optimization method, and (ii) finds the operational learning model by searching each of the plurality of learning models.

12. The model generating device according to claim 11, wherein the predetermined optimization method is derivative-free optimization (DFO).

13. 13. The model generating apparatus according to claim 12, wherein the predetermined optimization method is the Nelder-Mead method or the Powell method.

14. An image recognition device that performs image recognition based on operational images captured by a second imaging device different from the first imaging device in an image recognition phase subsequent to the learning phase, using a learning model that is generated in advance by a model generation device in a learning phase based on a plurality of learning images captured by the first imaging device, the model generation device (i) applies image processing according to each of a plurality of predetermined image processing settings to the plurality of training images to generate in advance a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings, and (ii) applies the image processing to a plurality of evaluation images captured by the second imaging device to generate in advance a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; the model generation device executes a predetermined machine learning algorithm to generate a plurality of learning models in advance using the plurality of processed learning images corresponding to the plurality of predetermined image processing settings, respectively; The model generation device (i) pre-selects an operational image processing setting to be applied in the image recognition phase from among the plurality of predetermined image processing settings, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of predetermined image processing settings, and (ii) pre-selects a learning model corresponding to the operational image processing setting from among the plurality of learning models as an operational learning model to be applied in the image recognition phase, The image recognition device an operational image processing unit that performs the image processing according to the operational image processing settings on the operational image in the image recognition phase to generate a processed operational image corresponding to the operational image processing settings; An image recognition device comprising: an image recognition unit that performs image recognition on the processed operational image based on the output of the operational learning model obtained by inputting the processed operational image into the operational learning model in the image recognition phase.

15. A model generation method in which a model generation device generates a learning model in a learning phase prior to the image recognition phase, based on a plurality of learning images captured by a first imaging device, for an image recognition device to perform image recognition in an image recognition phase based on operational images captured by a second imaging device different from a first imaging device, the method comprising: The above model generation method is as follows: the model generation device performs an image processing step during learning in which (i) image processing according to each of a plurality of predetermined image processing settings is performed on the plurality of learning images to generate a plurality of processed learning images corresponding to each of the plurality of predetermined image processing settings, and (ii) the image processing is performed on a plurality of evaluation images captured by the second imaging device to generate a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; a learning step in which the model generation device executes a predetermined machine learning algorithm to generate a plurality of learning models using the plurality of processed learning images corresponding to each of the plurality of predetermined image processing settings, The learning process further includes the steps of (i) selecting an operational image processing setting to be applied in the image recognition phase from among the plurality of specified image processing settings, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by the model generation device evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of specified image processing settings, and (ii) selecting a learning model corresponding to the operational image processing setting from among the plurality of learning models as the operational learning model to be applied in the image recognition phase.

16. An image recognition method in which an image recognition device performs image recognition based on operational images captured by a second imaging device different from the first imaging device in an image recognition phase subsequent to the learning phase, using a learning model previously generated by a model generation device in a learning phase based on a plurality of learning images captured by the first imaging device, the model generation device (i) applies image processing according to each of a plurality of predetermined image processing settings to the plurality of training images to generate in advance a plurality of processed training images corresponding to each of the plurality of predetermined image processing settings, and (ii) applies the image processing to a plurality of evaluation images captured by the second imaging device to generate in advance a plurality of processed evaluation images corresponding to each of the plurality of predetermined image processing settings; the model generation device executes a predetermined machine learning algorithm to generate a plurality of learning models in advance using the plurality of processed learning images corresponding to the plurality of predetermined image processing settings, respectively; The model generation device (i) pre-selects an operational image processing setting to be applied in the image recognition phase from among the plurality of predetermined image processing settings, based on a plurality of index values ​​indicating the quality of each of the plurality of learning models obtained by evaluating each of the plurality of learning models using the plurality of post-processing evaluation images corresponding to each of the plurality of predetermined image processing settings, and (ii) pre-selects a learning model corresponding to the operational image processing setting from among the plurality of learning models as an operational learning model to be applied in the image recognition phase, The image recognition method is as follows: an operational image processing step in which, in the image recognition phase, the image recognition device performs the image processing according to the operational image processing settings on the operational image to generate a processed operational image corresponding to the operational image processing settings; An image recognition method including an image recognition step in which, in the image recognition phase, the image recognition device performs image recognition on the processed operational image based on the output of the operational learning model obtained by inputting the processed operational image into the operational learning model.

Images