Model evaluation method and model evaluation system

Abstract

To improve the accuracy of model evaluation performed by a detection AI.SOLUTION: There is provided a model evaluation method for an object model outputting output data according to input data, and includes the steps of: acquiring a plurality of properties related to the input data of the object model; acquiring an evaluation index for evaluating the object model and acquiring an evaluation index related to the plurality of properties; and outputting index specification data for specifying an evaluation index corresponding to a predetermined condition based on the plurality of properties.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a model evaluation method and a model evaluation system.

Background Art

[0002] It is known to "apply sensor data obtained in the manufacturing process of various electronic devices such as display devices to a machine learning model to detect or predict a defective state or defect" (see, for example, Patent Document 1). Patent Document 1 Japanese Patent Application Laid-Open No. 2020-126601

Summary of the Invention

Problems to be Solved by the Invention

[0003] It is preferable to improve the model evaluation accuracy of the detection AI.

Means for Solving the Problems

[0004] In a first aspect of the present invention, there is provided a model evaluation method for a target model that outputs output data according to input data, the method including: obtaining a plurality of properties related to the input data of the target model; obtaining an evaluation index for evaluating the target model, the step of obtaining an evaluation index related to the plurality of properties; and outputting index specification data for specifying an evaluation index that meets a predetermined condition based on the plurality of properties.

[0005] The index specification data may include sample data of the input data corresponding to the evaluation index that meets the predetermined condition.

[0006] The index specification data may include a range of values of the property corresponding to the evaluation index that meets the predetermined condition.

[0007] The model evaluation method may include obtaining a distribution of evaluation indices according to changes in the plurality of properties.

[0008] The evaluation index that meets the predetermined conditions may be an extreme value of the distribution of the evaluation index or an evaluation index within a predetermined range from the extreme value.

[0009] The index specific data may be information for specifying an evaluation index that does not meet the predetermined criteria and should be improved.

[0010] The model evaluation method may include a step of outputting an evaluation index for each arbitrary property of a plurality of properties.

[0011] The plurality of properties may include a continuously changing property whose value changes continuously.

[0012] The continuously changing property may indicate at least one of the position of the detection target, the shape of the detection target, the color of the detection target, the light source position, or the light source intensity in the input data of the target model.

[0013] The plurality of properties may include a non - continuously changing property whose value does not change continuously.

[0014] The non - continuously changing property may indicate at least one of the texture of the detection target, the input image, the category, or the class in the input data of the target model.

[0015] The model evaluation method may include a step of generating complementary data for complementing between non - continuously changing properties using a predetermined extended model. The model evaluation method may include a step of causing the target model to execute a learning process using the complementary data.

[0016] The model evaluation method may include a step of specifying non - evaluation target properties that are not evaluation targets of the target model from a plurality of properties. The model evaluation method may include a step of obtaining an evaluation index corresponding to the combination of a plurality of properties based on the evaluation target properties obtained by excluding the non - evaluation target properties from the plurality of properties.

[0017] The target model may be a model learned based on original data and extended data extended from the original data.

[0018] The model evaluation method may include a step of generating additional extended data based on an evaluation index that meets predetermined conditions. The model evaluation method may include a step of executing the learning process of the target model based on the additional extended data.

[0019] In a second aspect of the present invention, a program for causing a computer to execute the model evaluation method described in the first aspect of the present invention is provided.

[0020] In a third aspect of the present invention, there is provided a model evaluation system for a target model that outputs output data corresponding to input data, the model evaluation system including a property acquisition unit that acquires a plurality of properties related to the input data of the target model, an index acquisition unit that acquires an evaluation index for evaluating the target model, the evaluation index being related to the plurality of properties, and an output unit that outputs index specification data for specifying an evaluation index that meets predetermined conditions based on the plurality of properties.

[0021] Note that the above summary of the invention does not enumerate all the features of the present invention. Also, sub-combinations of these feature groups can also be inventions.

Brief Description of the Drawings

[0022]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 3

Figure 4A

Figure 4B

Figure 4C

Figure 4D

Figure 4E

Figure 5A

Figure 5B

Figure 6

Embodiments for Carrying Out the Invention

[0023] Hereinafter, the present invention will be described through embodiments of the invention. However, the following embodiments do not limit the invention according to the claims. Also, not all combinations of features described in the embodiments are essential for the solution means of the invention.

[0024] FIG. 1 shows an overview of the configuration of the model evaluation system 100. The model evaluation system 100 includes an input unit 10, an index acquisition unit 20, and an output unit 30. The input unit 10 in this example has a target model acquisition unit 12, a property acquisition unit 14, and an original data acquisition unit 16. The model evaluation system 100 evaluates a predetermined target model Mt and outputs an evaluation result Er.

[0025] The target model acquisition unit 12 acquires the output data of the target model Mt. The target model acquisition unit 12 in this example has a learned target model Mt and acquires the output data. The target model acquisition unit 12 may execute the learning process of the target model Mt. The target model acquisition unit 12 may not have the target model Mt and may acquire the output data from the target model Mt provided externally.

[0026] The target model Mt outputs a detection result of detecting a detection target according to predetermined input data. The target model Mt may be a detection AI that can be generated by learning processing. The type of the detection model of the target model Mt is not particularly limited. The target model Mt may be an object detection model, a segmentation model, or a label estimation model.

[0027] The detection model of the target model Mt may be any one of an image detection model, an audio detection model, a motion detection model, a text determination model, a movie analysis model, or a plant anomaly detection model. In the case of a face detection model, the target model Mt may acquire the coordinates of a rectangular area including a face in the input image. In the case of a defect detection model, the target model Mt may detect the position or shape of a defective part of the detection target.

[0028] The property acquisition unit 14 acquires property information Ip regarding the input data of the target model Mt. The property information Ip includes information regarding the type of the specified property P and the value of the property P. The model evaluation system 100 may evaluate the target model Mt based on the output data of the property P specified by the property acquisition unit 14.

[0029] The property information Ip may include evaluation target properties specified as evaluation targets from a plurality of properties P, or may include evaluation non-target properties that are not evaluation targets of the target model Mt. The property acquisition unit 14 can simplify the calculation and reduce the cost by designating evaluation non-target properties. The property acquisition unit 14 may also specify the property P to be evaluated according to an input from the user.

[0030] By specifying the type or value of property P, a specific range can be specified to evaluate the target model Mt. The property information Ip may include any information such as a property to be detected, a background property, or an environmental property. Also, the property information Ip may include information regarding the texture of the detection target and may include information regarding the mode (e.g., the type of defect).

[0031] The property to be detected may include information such as the personality, gender, race, age, or face angle of the detection target. The background property may include information indicating the location where the detection target exists. The environmental property may include information regarding the lighting condition, the presence or absence of obstacles in the face region (e.g., hands, arms, glasses, masks, hair, or hats), the face resolution, or noise addition. The face resolution may be calculated according to the number of pixels between the eyes.

[0032] The property acquisition unit 14 acquires the property P for evaluating the target model Mt. The property acquisition unit 14 may specify a sample point for evaluating the target model Mt from the acquired property P. The method of specifying the sample point may be a grid search or may be a Gaussian process regression. When using the grid search, the property acquisition unit 14 may specify sample points at a specified interval within a predetermined range of each property P. Also, when using Gaussian process regression, the sample points can be reduced compared to the grid search. Thereby, comprehensiveness and continuity can be improved more efficiently. Also, a method of specifying sample points using a learned model may be used, not limited to the grid search or Gaussian process regression.

[0033] The original data acquisition unit 16 acquires the original data Do for inputting to the target model Mt. The original data Do may be used for learning the target model Mt. The original data Do may be an image including the detection target. The original data acquisition unit 16 inputs the acquired original data Do to the index acquisition unit 20.

[0034] The index acquisition unit 20 acquires an evaluation index Ei for evaluating the target model Mt. For example, the index acquisition unit 20 acquires, as the evaluation index Ei, the detection rate, false detection rate, or omission rate of the detection AI. The index acquisition unit 20 in this example acquires evaluation indices Ei related to a plurality of properties P. The index acquisition unit 20 may calculate the evaluation index Ei within a range corresponding to the sample points of the property P specified by the property acquisition unit 14.

[0035] The index acquisition unit 20 may acquire evaluation indices Ei corresponding to the plurality of properties P based on the property to be evaluated from among the plurality of properties P. In this case, the index acquisition unit 20 does not need to evaluate the target model Mt within a range corresponding to the property not to be evaluated. As a result, there is no need to acquire unnecessary evaluation indices Ei, and the calculation can be simplified.

[0036] The output unit 30 outputs the evaluation result Er of the target model Mt. The evaluation result Er may include the relationship between the specified property P and the evaluation index Ei. The output unit 30 may have a display unit such as a display for displaying the evaluation result Er.

[0037] The output unit 30 in this example outputs index specification data Di for specifying an evaluation index Ei that meets a predetermined condition based on a plurality of properties P. The evaluation index Ei that meets the predetermined condition may be, for example, an evaluation index Ei with a detection rate worse than the standard required for the target model Mt when the evaluation index Ei is the detection rate.

[0038] The index specification data Di may be input data corresponding to the evaluation index Ei that meets the predetermined condition. In one example, the index specification data Di is information for specifying an evaluation index Ei with a low detection rate. For example, the index specification data Di includes sample data such as input image data of the target model Mt. By outputting the input image data as sample data, the output unit 30 can intuitively inform the user of the index that meets the predetermined condition.

[0039] Further, the index identification data Di may include the range of the property P corresponding to the evaluation index Ei that meets the predetermined conditions. The index identification data Di may be the numerical range of the property P corresponding to the evaluation index Ei equal to or higher than the predetermined false detection rate. For example, the output unit 30 can visualize the range that the target model Mt is not good at by outputting the range of the property P with a low detection rate. Note that the index identification data Di may be an image sample of the evaluation index Ei with a high detection rate, the contribution rate of the property P to the evaluation index Ei, or the like.

[0040] FIG. 2A is an example of an evaluation method using the model evaluation system 100. The model evaluation system 100 in this example evaluates the target model Mt, which is a person detection model, and displays the relationship between the evaluation index Ei and the value of the property P. The original data Do in this example is an image of a person to be detected and is input to the target model Mt. The output unit 30 in this example outputs an evaluation result Er for evaluating the target model Mt based on the evaluation index Ei of the target model Mt.

[0041] The display format of the evaluation result Er is not limited as long as it is the result of the model evaluation system 100 evaluating the target model Mt. The evaluation result Er in this example is the distribution of the evaluation index Ei according to the change of the property P. The vertical axis of the evaluation result Er indicates the evaluation index Ei, and the horizontal axis indicates the value of the property P. For example, the evaluation result Er is the distribution of the false detection rate according to age. In the evaluation result Er of this example, the area where the evaluation index Ei should be improved is marked by a broken-line circle.

[0042] FIG. 2B is an example of the evaluation result Er using the model evaluation system 100. The model evaluation system 100 outputs index identification data Di that meets the predetermined conditions using the extreme value Ve of the distribution of the evaluation index Ei.

[0043] The range Rve is the value of the property P within a predetermined range from the extreme value Ve. For example, the range Rve is the range of the property P where the evaluation index Ei becomes half of the extreme value Ve. The evaluation index Ei corresponding to the predetermined condition may be the evaluation index Ei of the extreme value Ve or the range Rve. The model evaluation system 100 of this example can visualize the range with a low detection rate based on the extreme value Ve.

[0044] The model evaluation system 100 may output the index identification data Di according to the extreme value Ve or the range Rve. The model evaluation system 100 of this example visualizes the original data Do with a low detection rate by displaying the input image of the extreme value Ve as the index identification data Di. The model evaluation system 100 can intuitively inform the user of the original data Do with a low detection rate by displaying the input image to the user as the index identification data Di.

[0045] FIG. 2C is a modification of the evaluation result Er using the model evaluation system 100. The model evaluation system 100 shows the relationship between each of the plurality of properties P and the evaluation index Ei.

[0046] The output unit 30 outputs the evaluation index Ei for each property P of the plurality of properties P. In this example, as the plurality of properties P, the age of the detection target and the face angle are specified. The output unit 30 outputs the evaluation index Ei independently for each specified property P.

[0047] For example, when there is a large amount of original data Do in the age range of 20 to 30 years old and the detection rates for the 10s and over 40s are low, the model evaluation system 100 may increase the data of the age with a low detection rate and execute learning processing on the target model Mt. Thereby, the detection accuracy of the target model Mt can be improved.

[0048] The output unit 30 of this example outputs the distribution of the evaluation index Ei for one property P, but may also output the distribution of the evaluation index Ei for each property P of a plurality of properties P. The output unit 30 may output the distribution of the evaluation index Ei according to the combination of a plurality of properties P. For example, the output unit 30 displays the distribution of the evaluation index Ei in a three-dimensional space formed by combining a plurality of properties P. Further, the output unit 30 may divide the specified property P into an arbitrary number of parts, and display, as the value of the range, the average of the indexes for all the data included in the divided range. Further, the output unit 30 may display the confidence interval of the evaluation index Ei.

[0049] The model evaluation system 100 can efficiently train the target model Mt by improving the continuity and comprehensiveness of the property P by visualizing the data with poor evaluation index Ei. By improving the comprehensiveness, the model evaluation system 100 can detect the evaluation index Ei corresponding to various properties P, and can improve the accuracy of model evaluation compared to the case of evaluating within a limited data range. The model evaluation system 100 can improve the evaluation accuracy by improving the continuity of the property P, rather than obtaining the evaluation index Ei discretely.

[0050] FIG. 3 shows a modified example of the model evaluation system 100. The model evaluation system 100 is different from the embodiment of FIG. 1 in that it includes a data expansion unit 22 and a storage unit 40. Other points may be the same as those of the embodiment of FIG. 1.

[0051] The data expansion unit 22 has a data expansion model Me and generates expanded data De obtained by expanding the original data Do. The data expansion unit 22 of this example expands the original data Do based on the evaluation result Er or the evaluation index Ei output by the output unit 30. The data expansion unit 22 may generate the expanded data De so as to complement the range with poor evaluation index Ei. The data expansion unit 22 outputs the generated expanded data De to the index acquisition unit 20.

[0052] The index acquisition unit 20 executes the learning process of the target model Mt using the original data Do and the extended data De. In this way, by expanding and learning the original data Do, it is possible to avoid the relationship between the property P and the evaluation index Ei from becoming discrete, and improve comprehensiveness and continuity.

[0053] The storage unit 40 may store input data such as the property P or the original data Do acquired by the input unit 10. The storage unit 40 may also store the extended data De generated by the data expansion unit 22. The data expansion unit 22 may read out the information stored in the storage unit 40 during data expansion.

[0054] Note that the data expansion means by the data expansion unit 22 is not particularly limited. The data expansion means by the data expansion unit 22 may be data expansion using an image conversion model or data expansion by image processing. In data expansion by an image conversion model, data may be expanded using methods such as age conversion, face angle conversion, or illumination change. In data expansion by image processing, data may be expanded using methods such as changing the resolution by resizing, applying blur, applying noise, or pasting obstacles. The data expansion unit 22 may expand the personality of the original data Do by the data expansion model Me. In the expansion of personality, a person between any person A and another person B may be generated.

[0055] FIG. 4A is an example of an evaluation method using the model evaluation system 100. The model evaluation system 100 in this example generates extended data De expanded from the original data Do and evaluates the target model Mt learned with the original data Do and the extended data De. The vertical axis of the evaluation result Er shows the false detection rate as an example of the evaluation index Ei.

[0056] The original data Do includes a good product actual image Dog and a defective product actual image Dob. The data augmentation model Me generates an augmented defective product image Deb according to the good product actual image Dog and the defective product actual image Dob. The defective product augmented image Deb is an example of the augmented data De. Thereby, even when the number of defective product actual images Dob is small, the defective product image can be supplemented with the defective product augmented image Deb and the target model Mt can be made to learn it.

[0057] The target model Mt executes a learning process using the good product actual image Dog, the defective product actual image Dob, and the defective product augmented image Deb as learning data. The evaluation index Ei is generated by evaluating the target model Mt learned with the defective product augmented image Deb using an arbitrary evaluation model. The evaluation result Er shows the relationship between the evaluation index Ei of the target model Mt learned with the defective product augmented image Deb and the value of the property P.

[0058] The data augmentation unit 22 may augment data of the property P in which the original data Do for learning is small and the evaluation index Ei is worse than a predetermined standard. The lower limit of the data amount of the augmented data De may be determined so that the evaluation index Ei is improved compared to a predetermined standard. The upper limit of the augmented data De may be determined so as to avoid an increase in the data generation cost for generating the evaluation result Er. In one example, the data augmentation unit 22 determines in advance the data amount of the augmented data De to be generated, and preferentially augments the data with a high uncertainty of detection accuracy.

[0059] The dashed line of the evaluation result Er indicates the evaluation result Er before data augmentation. The solid line of the evaluation result Er indicates the evaluation result Er after data augmentation. In the evaluation index Ei after data augmentation, the overall false detection rate is reduced by using the augmented data De. Also, in the evaluation index Ei after data augmentation, a data range that was not evaluated before data augmentation is evaluated.

[0060] In this way, the model evaluation system 100 of this example can perform data augmentation on unknown images that are not included in the original data Do of the real images. Thereby, the detection accuracy of the target model Mt can be improved, and the preparation period until the actual line introduction can be shortened.

[0061] FIG. 4B shows an example of a method for generating the augmented data De. The detection target in this example is the cap of a plastic bottle. In this example, the defective position of the cap of the plastic bottle is specified using the label information L1, and the augmented data De of the defective product is generated. The good product real image Dog is an image of the cap of a plastic bottle without defects.

[0062] The label information L1 includes information specifying the data application area R1 at an arbitrary position of the good product real image Dog. The label information L1 may include information regarding at least one of the position, shape, or texture of the detection target. The label information L1 may be any information that can specify the position or the like of the detection target, and may be numerical data such as image data, coordinates, or a combination thereof.

[0063] The data application area R1 specifies an arbitrary position and shape on the cap of the plastic bottle. The data augmentation model Me applies data of an arbitrary defective part to the data application area R1 to generate the augmented data De of the defective product. In this example, the defect is a scratch on the side of the cap, but the type of the defect mode is not limited to this. The data augmentation unit 22 may perform data augmentation by applying a plurality of types of scratches to the data application area R1.

[0064] FIG. 4C shows an example of a method for generating the augmented data De. In this example, the augmented data De in which different types of defective parts from the example of FIG. 4B are augmented is generated. The data application area R1 in this example indicates a chip on the cap. The data application area R1 is specified as a triangular area at the lower part of the cap. The shape of the data application area R1 may be other shapes, or a plurality of data application areas R1 may be provided. The position, size, etc. of the data application area R1 may be appropriately changed according to the distribution of the evaluation index Ei output by specifying the property P to the position and size of the defective part.

[0065] Figure 4D shows an example of a method for generating the extended data De. In this example, an image of a lid having a defective portion is extended to change the position of the defective portion. The model evaluation system 100 in this example generates n pieces of extended data De, i.e., extended data De_1 to extended data De_n, by changing the position of the defective portion.

[0066] Here, depending on which position in the horizontal axis direction of the image the position of the defective portion belongs to, a distribution occurs in the detection accuracy of the target model Mt. For example, the detection accuracy may deteriorate more when the defective portion is located at the end of the lid than when the defective portion is located at the center of the lid. In such a case, the model evaluation system 100 can improve the detection accuracy of the target model Mt by designating a data application area R1 at the end of the lid and generating the extended data De.

[0067] Figure 4E shows an example of an evaluation result Er including the discontinuous change property Pd. The plurality of properties P in this example include a continuous change property Pc and a discontinuous change property Pd. The vertical axis shows the false detection rate as the evaluation index Ei. The horizontal axis shows the coordinates of the defective position as an example of the continuous change property Pc and the texture as an example of the discontinuous change property Pd, respectively.

[0068] The continuous change property Pc is a property P whose value changes continuously. For example, the continuous change property Pc is the position, shape, or color of the detection target. The continuous change property Pc may include the light source position or light source intensity in the space where the detection target exists. The continuous change property Pc may be used by normalizing the value of the property P to a range from 0 to 1, for example. When the continuous change property Pc is age, the age value may be used as it is, or the age value may be divided into a predetermined number (e.g., 20) of sample points and set.

[0069] The discontinuous change property Pd is a property P whose value does not change continuously. The texture is an example of the discontinuous change property Pd and may include textures such as scratches or stains in defective areas. The discontinuous change property Pd may be at least one of a texture, an input image, a category, or a class to be detected in the input data of the target model Mt.

[0070] Here, the property acquisition unit 14 may generate complementary data that complements between the discontinuous change properties Pd using a predetermined extended model Me. When the discontinuous change property Pd is a texture, values of a number of properties P corresponding to the type of texture may be specified, and values of the properties P obtained by complementing the values of the respective properties P may be generated. The target model acquisition unit 12 may cause the target model Mt to execute learning processing using the complementary data.

[0071] For example, the model evaluation system 100 can obtain the distribution of the evaluation index Ei even for a property P whose value is not continuous like a texture by generating a texture such as an intermediate between a stain and a scratch on the lid of a plastic bottle. Therefore, the model evaluation system 100 can improve the continuity and comprehensiveness of the discontinuous change property Pd by complementing the values of the property P.

[0072] FIG. 5A shows a modification of the model evaluation system 100. The data expansion unit 22 in this example differs from the example of FIG. 3 in that it generates additional expansion data Da generated based on the evaluation result Er. Other points may be the same as those in the example of FIG. 3.

[0073] The data expansion unit 22 generates additional expansion data Da based on the evaluation index Ei that meets a predetermined condition. The data expansion unit 22 can complement with the additional expansion data Da a portion where the number of sample data is insufficient in the original data Do or the expansion data De. The data expansion unit 22 may acquire the original data Do and the expansion data De stored in the storage unit 40 when generating the additional expansion data Da. The additional expansion data Da may be stored in the storage unit 40.

[0074] The data volume of the additional extended data Da may be determined according to the data volumes of the original data Do and the extended data De. If the data volume of the additional extended data Da is less than the data volumes of the original data Do and the extended data De, the effect of the additional extended data Da cannot be fully obtained. If the data volume of the additional extended data Da is too much more than the data volumes of the original data Do and the extended data De, the detection accuracy of the target model Mt may deteriorate.

[0075] The index acquisition unit 20 executes the learning process of the target model Mt based on the additional extended data Da. Thereby, according to the evaluation result Er, the comprehensiveness and continuity of the property P can be improved, and the detection accuracy of the target model Mt can be improved.

[0076] FIG. 5B is an example of an evaluation method using the model evaluation system 100. The model evaluation system 100 in this example generates a defective product additional extended image Dab and evaluates a target model Mt learned with the defective product additional extended image Dab. The defective product additional extended image Dab is an example of the additional extended data Da. The vertical axis of the evaluation result Er shows the false detection rate as an example of the evaluation index Ei. The data expansion unit 22 in this example additionally expands a data range with a large false detection rate in the evaluation result Er. Thereby, in the evaluation index Ei after data expansion, improvement is centered on the data range with a high false detection rate, and the bias of the distribution of the evaluation index Ei can be corrected.

[0077] The model evaluation system 100 in this example generates the additional extended data Da based on the evaluation result Er, so that data can be expanded more efficiently. In addition, the model evaluation system 100 can consistently realize both the evaluation of the target model Mt and the improvement of the detection accuracy of the target model Mt.

[0078] The model evaluation system 100 can evaluate and improve any target model Mt without being limited to the type of detection AI model. For example, the model evaluation system 100 is used for evaluating an object detection model for autonomous driving. In this case, the model evaluation system 100 may specify, as the property P, weather, object category, size, color, texture, angle, person's pose or clothing. Also, the model evaluation system 100 may be used for evaluating a segmentation model. The model evaluation system 100 can prevent accidents in advance by visualizing an evaluation result Er such as missing a pedestrian in a specific situation. The model evaluation system 100 may also be used for detecting human behavior by a surveillance camera.

[0079] FIG. 6 shows an example of a computer 2200 in which multiple aspects of the present invention may be embodied in whole or in part. Programs installed in the computer 2200 can cause the computer 2200 to function as an operation associated with the apparatus according to an embodiment of the present invention or as one or more sections of the apparatus, or to execute the operation or the one or more sections, and / or can cause the computer 2200 to execute a process according to an embodiment of the present invention or a stage of the process. Such a program may be executed by the CPU 2212 to cause the computer 2200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

[0080] The computer 2200 according to this embodiment includes a CPU 2212, a RAM 2214, a graphic controller 2216, and a display device 2218, which are mutually connected by a host controller 2210. The computer 2200 also includes an input / output unit such as a communication interface 2222, a hard disk drive 2224, a DVD-ROM drive 2226, and an IC card drive, which are connected to the host controller 2210 via an input / output controller 2220. The computer also includes legacy input / output units such as a ROM 2230 and a keyboard 2242, which are connected to the input / output controller 2220 via an input / output chip 2240.

[0081] The CPU 2212 operates according to programs stored in the ROM 2230 and the RAM 2214, thereby controlling each unit. The graphic controller 2216 acquires image data provided in the frame buffer or the like in the RAM 2214 or generated by the CPU 2212 in itself, and causes the image data to be displayed on the display device 2218.

[0082] The communication interface 2222 communicates with other electronic devices via a network. The hard disk drive 2224 stores programs and data used by the CPU 2212 in the computer 2200. The DVD-ROM drive 2226 reads a program or data from the DVD-ROM 2201 and provides the program or data to the hard disk drive 2224 via the RAM 2214. The IC card drive reads programs and data from an IC card and / or writes programs and data to the IC card.

[0083] The ROM 2230 stores therein a boot program or the like executed by the computer 2200 at activation and / or a program dependent on the hardware of the computer 2200. The input / output chip 2240 may also connect various input / output units to the input / output controller 2220 via a parallel port, a serial port, a keyboard port, a mouse port, etc.

[0084] The program is provided by a computer-readable medium such as a DVD-ROM 2201 or an IC card. The program is read from the computer-readable medium, installed in the hard disk drive 2224, the RAM 2214, or the ROM 2230 which is also an example of a computer-readable medium, and executed by the CPU 2212. The information processing described in these programs is read by the computer 2200, bringing about cooperation between the programs and the various types of hardware resources described above. The apparatus or method may be configured by realizing the operation or processing of information according to the use of the computer 2200.

[0085] For example, when communication is executed between the computer 2200 and an external device, the CPU 2212 may execute a communication program loaded in the RAM 2214 and instruct the communication interface 2222 to perform communication processing based on the processing described in the communication program. The communication interface 2222 reads the transmission data stored in the transmission buffer processing area provided in a recording medium such as the RAM 2214, the hard disk drive 2224, the DVD-ROM 2201, or the IC card under the control of the CPU 2212, transmits the read transmission data to the network, or writes the received data received from the network to the reception buffer processing area or the like provided on the recording medium.

[0086] In addition, the CPU 2212 may cause all or necessary parts of files or databases stored in external recording media such as a hard disk drive 2224, a DVD-ROM drive 2226 (DVD-ROM 2201), and an IC card to be read into the RAM 2214, and may execute various types of processing on the data on the RAM 2214. Next, the CPU 2212 writes back the processed data to the external recording media.

[0087] Various types of information such as various types of programs, data, tables, and databases may be stored in a recording medium and may be subjected to information processing. The CPU 2212 may execute various types of processing on the data read from the RAM 2214, including various types of operations, information processing, condition determination, conditional branching, unconditional branching, information search / replacement, etc. described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 2214. Also, the CPU 2212 may search for information in files, databases, etc. in the recording medium. For example, when a plurality of entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 2212 searches for an entry that matches the condition where the attribute value of the first attribute is specified from among the plurality of entries, reads the attribute value of the second attribute stored in the entry, and thereby may obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

[0088] The programs or software modules described above may be stored on a computer-readable medium on or near the computer 2200. Also, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable medium, and thereby provide the program to the computer 2200 via the network.

[0089] As described above, the present invention has been explained using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments. It is obvious to those skilled in the art that various changes or improvements can be made to the above embodiments. It is clear from the description of the claims that forms with such changes or improvements can also be included in the technical scope of the present invention.

[0090] It should be noted that the execution order of each process such as operations, procedures, steps, and stages in the apparatuses, systems, programs, and methods shown in the claims, the specification, and the drawings is not explicitly indicated as "before" or "preceding" etc., and can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flows in the claims, the specification, and the drawings, even if "first," "next," etc. are used for convenience of explanation, it does not mean that it is essential to implement in this order.

Explanation of Reference Numerals

[0091] 10... Input unit, 12... Target model acquisition unit, 14... Property acquisition unit, 16... Original data acquisition unit, 20... Index acquisition unit, 22... Data expansion unit, 30... Output unit, 40... Storage unit, 100... Model evaluation system, 2200... Computer, 2201... DVD-ROM, 2210... Host controller, 2212... CPU, 2214... RAM, 2216... Graphics controller, 2218... Display device, 2220... Input / output controller, 2222... Communication interface, 2224... Hard disk drive, 2226... DVD-ROM drive, 2230... ROM, 2240... Input / output chip, 2242... Keyboard

Claims

1. A method for evaluating a target model that outputs output data according to input data, comprising: obtaining a plurality of properties related to the input data of the target model; obtaining an evaluation index for evaluating the target model, the step of obtaining an evaluation index related to the plurality of properties; outputting index identification data for identifying the evaluation index that meets a predetermined condition based on the plurality of properties; and the index identification data includes a range of values of properties corresponding to the evaluation index that meets the predetermined condition, a model evaluation method executed by a computer.

2. The index identification data includes sample data of the input data corresponding to the evaluation index that meets the predetermined condition. The model evaluation method according to claim 1.

3. A method for evaluating a target model that outputs output data according to input data, comprising: obtaining a plurality of properties related to the input data of the target model; obtaining an evaluation index for evaluating the target model, the step of obtaining an evaluation index related to the plurality of properties; outputting index identification data for identifying the evaluation index that meets a predetermined condition based on the plurality of properties; obtaining a distribution of the evaluation index according to changes in the plurality of properties; and a model evaluation method executed by a computer.

4. The evaluation index that meets the predetermined condition is an extreme value of the distribution of the evaluation index or an evaluation index within a predetermined range from the extreme value. The model evaluation method according to claim 3.

5. The index identification data is information for identifying the evaluation index to be improved that does not meet a predetermined criterion. The model evaluation method according to any one of claims 1 to 4.

6. comprising the step of outputting the evaluation index for each arbitrary property of the plurality of properties. The model evaluation method according to any one of claims 1 to 5.

7. The plurality of properties include continuously changing properties whose values change continuously. The model evaluation method according to any one of claims 1 to 6.

8. The continuously changing property indicates at least one of a position of a detection target, a shape of the detection target, a color of the detection target, a light source position, or a light source intensity in the input data of the target model. The model evaluation method according to claim 7.

9. The plurality of properties include discontinuous change properties whose values do not change continuously. The model evaluation method according to any one of claims 1 to 8.

10. A model evaluation method for a target model that outputs output data according to input data, A step of obtaining a plurality of properties related to the input data of the target model; A step of obtaining an evaluation index for evaluating the target model, the step of obtaining an evaluation index related to the plurality of properties; A step of outputting index specification data for specifying the evaluation index corresponding to a predetermined condition based on the plurality of properties; Comprising The plurality of properties include discontinuous change properties whose values do not change continuously, The discontinuous change property indicates at least one of the texture, input image, category, or class of the detection target in the input data of the target model. A model evaluation method executed by a computer.

11. A model evaluation method for a target model that outputs output data according to input data, A step of obtaining a plurality of properties related to the input data of the target model; A step of obtaining an evaluation index for evaluating the target model, the step of obtaining an evaluation index related to the plurality of properties; A step of outputting index specification data for specifying the evaluation index corresponding to a predetermined condition based on the plurality of properties; Comprising The plurality of properties include discontinuous change properties whose values do not change continuously, The model evaluation method A step of generating complementary data that complements between the discontinuous change properties using a predetermined extended model; A step of causing the target model to execute a learning process using the complementary data A model evaluation method executed by a computer, comprising.

12. A step of designating evaluation non-target properties that are not evaluation targets of the target model from the plurality of properties; A step of obtaining an evaluation index corresponding to the plurality of properties based on the evaluation target properties obtained by excluding the evaluation non-target properties from the plurality of properties; The model evaluation method according to any one of claims 1 to 11, comprising.

13. The target model is a model learned based on original data and extended data extended from the original data. The model evaluation method according to any one of claims 1 to 12.

14. A step of generating additional extended data based on the evaluation index corresponding to the predetermined condition; A step of executing learning processing of the target model based on the additional extended data comprising The model evaluation method according to any one of claims 1 to 13.

15. A program for causing a computer to execute the model evaluation method according to any one of claims 1 to 14.

16. A model evaluation system for a target model that outputs output data corresponding to input data, A property acquisition unit that acquires a plurality of properties related to the input data of the target model; An index acquisition unit that acquires an evaluation index for evaluating the target model, the evaluation index being related to the plurality of properties; An output unit that outputs index specification data for specifying the evaluation index corresponding to the predetermined condition based on the plurality of properties comprising The index specification data includes a range of values of properties corresponding to the evaluation index that meets the predetermined condition. A model evaluation system.

17. A model evaluation system for a target model that outputs output data corresponding to input data, A property acquisition unit that acquires a plurality of properties related to the input data of the target model; An index acquisition unit that acquires an evaluation index for evaluating the target model, the evaluation index being related to the plurality of properties; An output unit that outputs index specification data for specifying the evaluation index corresponding to the predetermined condition based on the plurality of properties comprising acquiring a distribution of the evaluation index according to changes in the plurality of properties A model evaluation system.

Images