Training data generation device, training data generation method, and training data generation program

Abstract

To reduce human cost required for re-learning for adapting a learned inference model to monitoring in a specific monitoring space.SOLUTION: A learning data generation device 10 includes a learned object inference model 12 for outputting a first inference result to a first output value from an object sensor 3 for monitoring a monitoring space, a learned reference inference model 13 for outputting a second inference result to a second output value from a reference sensor 4 for monitoring the monitoring space, a contribution degree calculation part 14a for calculating a contribution degree of the second inference result in accordance with the first output value and the second output value, and a learning data generation part 14 for generating learning data for re-learning the object inference model on the basis of the contribution degree calculated by the contribution degree calculation part, the first output value and the second inference result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a training data generation device, a training data generation method, and a training data generation program. [Background technology]

[0002] In recent years, techniques have been proposed for automatically acquiring knowledge, rules, etc. from various data using inference models such as classifiers trained by machine learning, etc. For example, Patent Document 1 describes a classifier that estimates a person's gender from a facial image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-167834 Summary of the Invention [Problem to be solved by the invention]

[0004] When trying to obtain inference results by inputting the output signals of a sensor monitoring a specific monitored space into a pre-trained inference model, inference results with sufficient accuracy may not be obtained depending on the monitored space. In such cases, annotation can be performed by manually assigning labels to the sensor output signals, and the inference model can be retrained using the labeled training data to adapt it to the specific monitored space. However, annotation is a manual process, which means that the labor costs are high. The present invention has been made in consideration of the above problems, and aims to reduce the human cost required to re-train a trained inference model to adapt it to a specific monitored space. [Means for solving the problem]

[0005] According to one form of the present invention, a learning data generation device is provided, comprising: a trained target inference model that outputs a first inference result for a first output value from a target sensor monitoring a monitored space; a trained reference inference model that outputs a second inference result for a second output value from a reference sensor monitoring the monitored space; a contribution calculation unit that calculates a contribution degree, which is the degree to which the second inference result contributes to re-learning of the target inference model, based on the second output value; and a learning data generation unit that generates learning data for re-learning the target inference model based on the contribution degree calculated by the contribution calculation unit, the first output value, and the second inference result.

[0006] According to another aspect of the present invention, there is provided a training data generation method for generating training data for retraining a trained target inference model that outputs a first inference result in response to a first output value from a target sensor monitoring a monitored space. The training data generation method causes a computer to execute the following processes: acquiring a second inference result using a trained reference inference model that outputs a second inference result in response to a second output value from a reference sensor monitoring the monitored space; calculating a contribution rate, which is the degree to which the second inference result contributes to the retraining of the target inference model, based on the second output value; and generating training data based on the calculated contribution rate, the first output value, and the second inference result.

[0007] According to yet another aspect of the present invention, there is provided a training data generation program for generating training data for retraining a trained target inference model that outputs a first inference result in response to a first output value from a target sensor monitoring a monitored space. The training data generation program causes a computer to execute the following processes: a process of acquiring a second inference result using a trained reference inference model that outputs a second inference result in response to a second output value from a reference sensor monitoring the monitored space; a process of calculating a contribution rate, which is the degree to which the second inference result contributes to the retraining of the target inference model, based on the second output value; and a process of generating training data based on the calculated contribution rate, the first output value, and the second inference result. [Effects of the Invention]

[0008] According to the present invention, the human cost required for re-training a trained inference model to adapt it to a specific monitored space can be reduced. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the schematic configuration of an inference model learning system of an embodiment. [Figure 2] FIG. 10 is a schematic explanatory diagram of the resolution in an inference target region in an output image. [Figure 3] 10A and 10B are schematic explanatory diagrams of the dynamic range of the inference target region in the output image. [Figure 4] 10A and 10B are schematic explanatory diagrams of the luminance resolution of the inference target region in the output image. [Figure 5] FIG. 2 is a schematic explanatory diagram of the detection directions of a target sensor and a reference sensor relative to an inference target. [Figure 6] 4 is a flowchart illustrating an example of a learning data generation method according to the first embodiment. [Figure 7] 10 is a flowchart illustrating an example of a re-learning process of a target artificial intelligence (AI) unit. [Figure 8] FIG. 10 is a schematic configuration diagram of an example of a training data generation device according to a second embodiment. [Figure 9] FIG. 10 is a schematic diagram of an example of a screen on which a user inputs the correct answer of an inference result. [Figure 10] FIG. 10 is a schematic diagram of another example of a screen on which a user inputs the correct answer of an inference result. [Figure 11] 10 is a flowchart illustrating an example of a learning data generation method according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the embodiments of the present invention shown below are merely examples of devices and methods for embodying the technical concept of the present invention, and the technical concept of the present invention does not limit the structure, arrangement, etc. of the components to those described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims.

[0011] (First embodiment) (composition) 1 is an explanatory diagram of an example of the schematic configuration of an inference model learning system according to an embodiment. The inference model learning system according to an embodiment is a learning system that retrains an inference model that has already been trained to output inference results from sensor output values ​​so that it can be adapted to monitoring in a specific monitored space. Now, let us assume that in order to identify an inference target (a person in the example of Figure 1) 2 present in a specific monitored space 1, a sensor 3 for monitoring the monitored space 1 and a trained inference model that outputs inference results from the output value of the sensor 3 are introduced. In the following description, the monitored space 1 in which the sensor 3 and the inference model are introduced may be referred to as the "site."

[0012] When introducing such a trained inference model to a new site, the pre-trained inference model may not be able to achieve sufficient recognition accuracy depending on the installation environment of the site. For this reason, the inference model training system of the embodiment re-trains the inference model to adapt it to the site where it is introduced. In the following description, the sensor 3 that outputs data to be input into the inference model to be re-trained will be referred to as the "target sensor 3."

[0013] The inference model learning system of the embodiment includes a learning data generation device 10, a storage unit 20, and a learning device 30. The training data generation device 10 is a device that generates training data for retraining a target inference model. The training data generation device 10 includes a control unit 11. The control unit 11 includes a control unit 11 configured with a calculation device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), and an MCU (Micro Control Unit). The control unit 11 is connected to a storage device (not shown) provided separately from the training data generation device 10, and operates as a target artificial intelligence unit 12, a reference artificial intelligence unit 13, and a training data generation unit 14 by reading and executing computer programs from the storage device. Note that artificial intelligence is abbreviated as AI (Artificial Intelligence) in the figure.

[0014] The target artificial intelligence unit 12 is an artificial intelligence that is introduced into the monitored space 1 together with the target sensor 3 in order to identify the inference target 2 present in the monitored space 1. The target artificial intelligence unit 12 receives as input a first output value output from the target sensor 3 that monitors the monitored space 1, determines the class of the inference target 2, and outputs the determination result as a first inference result.

[0015] For example, the target artificial intelligence unit 12 determines whether the inference target 2 is a "male" or a "female." In this case, since it is a two-class problem, the correct class may be set to "male" and the incorrect class may be set to "female." The target artificial intelligence unit 12 can also be applied to "person detection," and "attribute estimation," "posture estimation," and "behavior recognition" other than gender. The target artificial intelligence unit 12 includes an inference model 12a and a threshold determination unit 12b. The targets of re-learning by the inference model learning system of the embodiment are the inference model 12a and the threshold determination unit 12b.

[0016] The inference model 12a is a classifier that is trained to receive the first output value output from the target sensor 3 as input and to output an inference score of the inference target 2 obtained from the first output value. This inference score is an evaluation value that indicates the degree to which the feature value of the inference target 2 is similar to the correct class. As the inference model 12a, various classifiers such as a convolutional neural network (CNN) or a rule-based (tree structure) inference model can be used.

[0017] The threshold determination unit 12b compares the inference score output from the inference model 12a with a determination threshold to determine whether the class of the inference target 2 is the correct class or the incorrect class. The determination result of the threshold determination unit 12b becomes the first inference result of the inference target 2 by the target artificial intelligence unit 12. The more similar the feature is to the feature of the correct class, the higher the inference score; the less similar it is, the lower the inference score. For example, the inference score of the correct class may be set to a positive value, and the inference score of the incorrect class may be set to a negative value. Here, the discrimination boundary between the correct class and the incorrect class is a hyperplane that separates the classes in feature space. The closer the absolute value of the inference score is to 0, the closer the feature is to the discrimination boundary, and the lower the certainty of the class estimation. Furthermore, the larger the absolute value of the inference score, the higher the certainty of the class estimation. In other words, the absolute value of the inference score is equal to the "estimated likelihood," which means the certainty of the class estimation. The estimated likelihood is an example of the "reliability of the inference result" in the claims.

[0018] Furthermore, when the attribute to be identified has three or more classes (for example, age estimation), the inference model outputs a class likelihood distribution that expresses the likelihood of each class. The judgment result of the threshold judgment unit 12b may be the class with the highest estimated likelihood exceeding the judgment threshold and its estimated likelihood (absolute value of the inference score). Furthermore, when considering a class likelihood distribution such as age estimation, the more normal the distribution is centered on the correct age class, the higher the reliability of the estimated likelihood distribution. Therefore, the "class likelihood distribution" may also be referred to as the "reliability of the estimation result" in the claims.

[0019] On the other hand, the reference artificial intelligence unit 13 is an artificial intelligence that identifies the class of the inference target 2 in order to perform annotation to generate learning data for re-learning of the target artificial intelligence unit 12. By using the reference artificial intelligence unit 13 to generate labeled learning data instead of manual annotation, the human cost required for re-learning can be reduced. The reference artificial intelligence unit 13 and the reference sensor 4 are temporarily installed in the monitored space 1 for relearning of the target artificial intelligence unit 12. The reference artificial intelligence unit 13 receives as input the second output value output from the reference sensor 4 monitoring the monitored space 1, determines whether the class of the inference target 2 is a correct class or an incorrect class, and outputs the determination result as a second inference result.

[0020] It should be noted that various sensors can be used as the target sensor 3 and reference sensor 4 as long as they output values ​​that can extract the feature quantities of the inference target 2. For example, the target sensor 3 and the reference sensor 4 may be sensors that output a set of component values ​​at multiple coordinates in one-dimensional space as a detection signal. For example, the target sensor 3 and the reference sensor 4 may be line sensors.

[0021] Furthermore, for example, the target sensor 3 and the reference sensor 4 may be sensors that output a set of component values ​​at multiple coordinates in two-dimensional space as detection signals. For example, the target sensor 3 and the reference sensor 4 may be cameras. For example, they may be visible image cameras, near-infrared cameras, or thermal image cameras. Furthermore, for example, the target sensor 3 and the reference sensor 4 may be sensors that output a collection of component values ​​at multiple coordinates in three-dimensional space as a detection signal. For example, the target sensor 3 and the reference sensor 4 may be a laser sensor, a millimeter wave sensor, an ultrasonic sensor, or a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging).

[0022] The target sensor 3 and the reference sensor 4 may be the same type of sensor or different types of sensors. When the same type of sensor is used as the target sensor 3 and the reference sensor 4, the target sensor 3 and the reference sensor 4 may be sensors with the same performance. However, it is preferable that the reference sensor 4 is a sensor with higher performance (for example, a sensor with higher resolution performance) than the target sensor 3. This is because the second output value of the reference sensor 4 is used to perform annotation for re-learning of the target artificial intelligence unit 12, so it is necessary to more accurately estimate the class of the inference target 2. The following describes an embodiment in which the target sensor 3 and the reference sensor 4 are visible image cameras. In the following description, the first output value output from the target sensor 3 and the second output value output from the reference sensor 4 may be referred to as the "first output image" and the "second output image," respectively.

[0023] The reference artificial intelligence unit 13 includes an inference model 13a and a threshold determination unit 13b. The inference model 13a is a classifier that receives the second output image output from the reference sensor 4 as input and is trained to output an inference score of the inference target 2 obtained from the second output image. Various classifiers can be used as the inference model 13a, such as a convolutional neural network or a rule-based (tree structure) inference model. The type and configuration of the inference model 13a of the reference artificial intelligence unit 13 do not necessarily have to be the same as those of the target artificial intelligence unit 12.

[0024] It is preferable to use, as the inference model 13a of the reference artificial intelligence unit 13, a model whose accuracy has been guaranteed by evaluation using various environmental data collected by the development department. The threshold determination unit 13b compares the inference score output from the inference model 13a with a determination threshold to determine whether the class of the inference target 2 is a correct class or an incorrect class.

[0025] The reference artificial intelligence unit 13 outputs the judgment result of the threshold judgment unit 13b to the learning data generation unit 14 as a second inference result. In addition, the reference artificial intelligence unit 13 outputs to the learning data generation unit 14 the second output image output from the reference sensor 4 and area information indicating the image area in the second output image in which the inference target 2 is detected. In the following description, the range in which the inference target 2 is detected from the image may be referred to as the "inference target area." The inference target area in the second output image of the reference sensor 4 is an example of the "detection range, which is the range of second output values ​​in which the inference target is detected" in the claims.

[0026] On the other hand, the target artificial intelligence unit 12 outputs the first output image output from the target sensor 3 and the first inference result to the learning data generation unit 14. The learning data generation unit 14 generates a label indicating the class of the inference object 2 based on the second inference result input from the reference artificial intelligence unit 13. Furthermore, the learning data generation unit 14 has a coordinate transformation matrix (calibration data) for transforming the coordinate system of the second output image of the reference sensor 4 into the coordinate system of the first output image of the target sensor 3 as the installation conditions of the reference sensor 4 and the target sensor 3. Such a coordinate transformation matrix can be obtained according to the installation positions, optical axis directions, angle of view, etc. of the reference sensor 4 and the target sensor 3. Information on the installation positions, optical axis directions, angle of view, and calibration may be stored in a storage device (not shown) provided in the learning data generation device 10.

[0027] The learning data generation unit 14 identifies an image region of the first output image of the target sensor 3 that corresponds to an inference target region of the second output image of the reference sensor 4 as the inference target region of the first output image. For example, based on the calibration data, the image region of the first output image of the target sensor 3 that corresponds to the inference target region of the second output image of the reference sensor 4 may be calculated as the inference target region of the first output image, or a correspondence relationship between each coordinate of the second output image of the reference sensor 4 and each coordinate of the first output image of the target sensor 3 may be stored in advance in a storage device (not shown). The image region of the first output image that corresponds to the inference target region of the second output image is an example of "a corresponding range that is a range of first output values ​​that corresponds to a detection range that is a range of second output values ​​in which the inference target is detected" or "coordinates of the first output values ​​that correspond to detection coordinates that are coordinates of the second output values ​​in which the inference target is detected" in the claims. The learning data generation unit 14 generates learning data including the first output image, information on the inference target area of ​​the first output image, and a label indicating the class of the inference target 2 as learning data for re-learning of the target artificial intelligence unit 12.

[0028] However, if the second inference result input from the reference artificial intelligence unit 13 is incorrect, the second inference result does not contribute to the re-learning of the target artificial intelligence unit 12. In this case, it is not desirable to train the target artificial intelligence unit 12 using training data labeled with a label indicating the second inference result. Therefore, the learning data generation unit 14 calculates the contribution degree, which is the degree to which the second inference result contributes to the re-learning of the target artificial intelligence unit 12. If the contribution degree satisfies a predetermined condition, the learning data generation unit 14 generates learning data labeled with the second inference result, and if the predetermined condition is not satisfied, the learning data generation unit 14 does not generate learning data.

[0029] The learning data generation unit 14 includes a contribution calculation unit 14a and a correct label determination unit 14b. The contribution degree calculation unit 14a calculates the contribution degree, which is the degree to which the second inference result input from the reference artificial intelligence unit 13 contributes to the re-learning of the target artificial intelligence unit 12. For example, the contribution degree may be calculated depending on how accurate the second inference result is estimated to be. For example, the contribution degree calculation unit 14a may calculate the contribution degree based on the resolution of the second output image input when the reference artificial intelligence unit 13 obtains the second inference result. The contribution degree calculated based on the resolution of the second output image is referred to as the "first contribution degree P1."

[0030] For example, the contribution calculation unit 14a may calculate the first contribution P1 based on the resolution of the reference sensor 4 and the resolution of the target sensor 3. For example, the contribution calculation unit 14a may calculate the ratio of the resolution of the target sensor 3 to the resolution of the reference sensor 4 as the first contribution P1=(resolution of the target sensor 3 / resolution of the reference sensor 4). In this embodiment, an index value is calculated as the contribution of the second inference result, which becomes smaller the greater the degree to which the second inference result contributes to the re-learning of the target artificial intelligence unit 12, but it is also possible to adopt an index value that becomes larger the greater the degree of contribution.

[0031] For example, the contribution degree calculation unit 14a may calculate the first contribution degree P1 based on the spatial resolution (resolution, pixel resolution) of the target sensor 3 and the reference sensor 4. The first contribution P1 may be calculated based on the luminance resolution as the resolution of the target sensor 3 and the reference sensor 4. The luminance resolution of a sensor is the minimum value of the change in luminance value that the sensor can distinguish, and indicates how finely the pixel values ​​that make up an image change in response to a difference in luminance.

[0032] Furthermore, for example, the contribution calculation unit 14a may calculate the first contribution P1 based on the dynamic ranges of the target sensor 3 and the reference sensor 4. The dynamic range of a sensor is the range between the minimum and maximum brightness values ​​that the sensor can detect. Furthermore, the contribution degree calculation unit 14a may calculate the first contribution degree P1 based on the frame rate (that is, the time resolution) as the resolution of the target sensor 3 and the reference sensor 4.

[0033] Furthermore, even if the spatial resolution of the target sensor 3 and the reference sensor 4 themselves is the same, the spatial resolution in the inference target area of ​​the first output image output by the target sensor 3 and the second output image output by the reference sensor 4 may differ depending on camera parameters such as the distance from the inference target 2 to the target sensor 3 and the reference sensor 4, and the angle of view of the target sensor 3 and the reference sensor 4. FIG. 2 is a diagram showing a schematic diagram of how the spatial resolution (resolution) in the inference target area changes depending on the angle of view of the target sensor 3 and the reference sensor 4. In FIG.

[0034] Dashed line 3a indicates the angle of view of the target sensor 3, and dashed line 4a indicates the angle of view of the reference sensor 4. In the example of Figure 2, the angle of view of the target sensor 3 is wider than the angle of view of the reference sensor 4. Therefore, when the maximum number of pixels of the target sensor 3 and the reference sensor 4 themselves are approximately equal, and the distances from the inference target 2 to the target sensor 3 and the reference sensor 4 are approximately equal, the spatial resolution in the inference target region of the second output image of the reference sensor 4 will be higher than the spatial resolution in the inference target region of the first output image of the target sensor 3. Similarly, the closer the distance from the inference object 2 to the sensor, the higher the spatial resolution in the inference object area.

[0035] Therefore, the contribution calculation unit 14a may calculate the spatial resolution in the inference target area of ​​the second output image based on the second output image, the area information (i.e., information on the inference target area in the second output image) output from the reference artificial intelligence unit 13, and the spatial resolution of the reference sensor 4 itself. The spatial resolution in the inference target area can be calculated based on the maximum number of pixels of the sensor and the size of the inference target area relative to the size of the sensor's detection surface. In addition, the image area of ​​the first output image of the target sensor 3 corresponding to the inference target area of ​​the second output image of the reference sensor 4 may be determined as the inference target area of ​​the first output image, and the spatial resolution in the inference target area of ​​the first output image may be calculated based on the inference target area of ​​the first output image and the spatial resolution of the target sensor 3 itself. Then, the first contribution P1 may be calculated based on the spatial resolution of the inference target region of the first output image and the inference target region of the second output image.

[0036] Similarly, even if the dynamic range and luminance resolution of the sensor itself are the same, the dynamic range and luminance resolution in the inference target region of the first output image and the second output image may differ. Here, the dynamic range in the inference target region refers to the degree of distribution of pixel component values ​​included in the inference target region. 3(a) and 3(b) are luminance distributions (histograms) with the luminance (pixel component value) of each pixel in the inference target area on the horizontal axis and the number of pixels on the vertical axis. For example, the dynamic range in the inference target area may be expressed as the width of the luminance distribution. The luminance distribution in FIG. 3(b) is wider than the luminance distribution in FIG. 3(a), indicating a wider dynamic range.

[0037] Furthermore, the luminance resolution in the inference target region indicates how finely the pixel component values ​​contained in the inference target region change in response to a change in luminance. For example, the pixel component values ​​in FIG. 4(b) change more finely with respect to the change in luminance than the pixel component values ​​in FIG. 4(a), indicating that the more finely the pixel component values ​​change, the higher the luminance resolution.

[0038] The dynamic range and brightness resolution in these inference target regions vary depending on the lighting conditions of the inference target 2 and the gains of the target sensor 3 and reference sensor 4. They also vary depending on image correction such as gamma correction that is applied to improve the characteristics of the first output image and the second output image. Therefore, the contribution degree calculation unit 14a may calculate the first contribution degree P1 based on the dynamic range and brightness resolution in the inference target region of the first output image and the inference target region of the second output image. Furthermore, the contribution calculation unit 14a may calculate the first contribution P1 based on a combination of two or more of the contributions based on the spatial resolution, luminance resolution, dynamic range, and frame rate. For example, the first contribution P1 may be calculated as a weighted sum of these combinations.

[0039] Additionally, the accuracy of the inference results from the inference model also varies depending on how well the direction of the inference target 2 detected by the sensor is suited to inference by the inference model. For example, when performing "person detection" or "attribute estimation" such as gender, more accurate inference results can be obtained by inputting an image taken from a direction closer to the front of the inference target 2. Therefore, the contribution degree calculation unit 14a may calculate the contribution degree based on the detection direction in which the target sensor 3 detects the inference target 2 and the detection direction in which the reference sensor 4 detects the inference target 2. The contribution degree calculated based on the detection direction is referred to as the "second contribution degree P2."

[0040] See Figure 5. The dashed line d0 indicates the shooting direction suitable for inference by the inference model 12a of the target artificial intelligence unit 12 and the inference model 13a of the reference artificial intelligence unit 13, and the dashed lines d1 and d2 indicate the shooting direction in which the target sensor 3 shoots the inference target 2 and the shooting direction in which the reference sensor 4 shoots the inference target 2, respectively. For example, the contribution calculation unit 14a may calculate the shooting direction d2 of the reference sensor 4 based on the coordinates of the pixel at the detection position where the inference target 2 is detected in the second output image of the reference sensor 4. The coordinates of the pixel at the detection position may be found by calculating the coordinates of the center or center of gravity of the inference target area.

[0041] Based on the calibration data, the contribution calculation unit 14a may convert the coordinates of the detection position of the inference target 2 on the second output image into coordinates on the first output image of the target sensor 3 based on the detection position and detection size, and calculate the shooting direction d1 of the target sensor 3 based on the converted coordinates. Furthermore, the contribution degree calculation unit 14a may track the inference target 2 detected in the first output image of the target sensor 3 or the second output image of the reference sensor 4, estimate the traveling direction of the trajectory of the inference target 2, and determine the shooting direction d0 suitable for inference. Alternatively, the contribution degree calculation unit 14a may obtain an inference result of the facial direction and posture of the person who is the inference target 2 from the first output image or the second output image using a separately prepared inference model, and dynamically determine the shooting direction d0 suitable for inference based on the obtained inference result. Alternatively, if the orientation of the inference object 2 can be estimated to some extent based on the direction of the passageway or the position of the entrance door, as shown in Figure 5, the shooting direction d0 suitable for inference can be estimated based on the direction of the passageway or the position of the entrance door, etc. The contribution calculation unit 14a may calculate the second contribution P2 = (α / β) as the ratio of the angle α between the shooting direction d2 of the reference sensor 4 and the shooting direction d0 suitable for inference to the angle β between the shooting direction d1 of the target sensor 3 and the shooting direction d0 suitable for inference.

[0042] Furthermore, the contribution calculation unit 14a may calculate the contribution based on the estimated likelihood (the certainty or reliability of the class estimation) when the inference model performs class estimation using the second output image output from the reference sensor 4 as input. The contribution calculated based on the estimated likelihood is referred to as the "third contribution P3." This allows the contribution to be calculated based on how suitable the second output image is for class estimation by the inference model.

[0043] For example, the contribution calculation unit 14a may calculate the third contribution P3 according to the estimated likelihood of class estimation based on the second output image of the reference sensor 4 and the estimated likelihood of class estimation based on the first output image of the target sensor 3. For example, the contribution calculation unit 14a may calculate the ratio of the estimated likelihood of class estimation based on the first output image to the estimated likelihood of class estimation based on the second output image as the third contribution P3 = (estimated likelihood of class estimation based on the first output image / estimated likelihood of class estimation based on the second output image).

[0044] If the attribute to be identified has three or more classes (e.g., age estimation), the inference model outputs a class likelihood distribution that expresses the likelihood of each class. In this case, for example, the contribution calculation unit 14a may calculate the contribution based on the "degree of agreement" between the class likelihood distribution based on the second output image and the class estimation likelihood distribution based on the first output image. A common method such as KL divergence may be used to measure the distance between the likelihood distributions.

[0045] For example, the inference model used to calculate the third contribution P3 may be an inference model that estimates the same type of class as the inference model 12a of the target artificial intelligence unit 12 that is the subject of re-learning, or the inference model 13a of the reference artificial intelligence unit 13. For example, the inference model 12a and the inference model 13a may output the estimated likelihoods obtained when the first inference result and the second inference result are obtained, respectively, to the contribution calculation unit 14a, which may then calculate the third contribution P3 based on these estimated likelihoods.

[0046] Also, for example, the inference model used to calculate the third contribution P3 may be an inference model that infers a different type of class from the inference model 12a and the inference model 13a. In addition, the third contribution P3 may be calculated by combining the estimated likelihood when the inference model 12a or the inference model 13a obtains the inference result with the estimated likelihood when the inference result is obtained using an inference model other than the inference model 12a or the inference model 13a. Furthermore, the inference model that performs class estimation based on the second output image of the reference sensor 4 and the inference model that performs class estimation based on the first output image of the target sensor 3 may be the same inference model or different inference models.

[0047] For example, when inference model 12a and inference model 13a perform attribute estimation (e.g., gender estimation) of inference target 2, the estimated likelihoods are "L11" and "L12," respectively, and when an inference model different from inference model 12a or inference model 13a performs face detection based on the first output image and the second output image, the estimated likelihoods are "L21" and "L22," respectively. The ratio of the estimated likelihoods for attribute estimation (L11 / L12) and the ratio of the estimated likelihoods for face detection (L21 / L22) may be multiplied to calculate the third contribution P3 = (L11 / L12) × (L21 / L22).

[0048] The contribution calculation unit 14a calculates the contribution P of the second inference result based on at least one of the first contribution P1, the second contribution P2, and the third contribution P3. When calculating the contribution P based on two or more combinations of the first contribution P1, the second contribution P2, and the third contribution P3, the contribution P may be calculated as a weighted sum of two or more combinations of the first contribution P1, the second contribution P2, and the third contribution P3. For example, when calculating the contribution P based on all combinations of the first contribution P1, the second contribution P2, and the third contribution P3, the weighted sum of the first contribution P1, the second contribution P2, and the third contribution P3 may be calculated as contribution P = w1 × P1 + w2 × P2 + w3 × P3, where w1, w2, and w3 are weighting coefficients.

[0049] Furthermore, the contribution calculation unit 14a may prepare a plurality of inference models 13a that receive the second output image output from the reference sensor 4 as input, and calculate the contribution based on the degree of agreement between the estimation results output by each inference model. The contribution calculated based on the degree of agreement between the plurality of inference results is referred to as the "fourth contribution P4." As the plurality of inference models, for example, inference models with different learning configurations may be prepared. For example, inference models with different classifier structures (neural networks or support vector machines) may be used. Furthermore, for example, inference models with different sets of training data may be used. Inference models with different learning configurations will produce slightly different inference results even when the same image is input. For this reason, if the inference results of multiple inference models are similar, the reliability of the inference result can be said to be high, and if the inference results are different, the reliability of the inference result can be said to be low. This makes it possible to calculate the contribution based on how suitable the second output image is for class estimation by multiple inference models.

[0050] Consider gender estimation, for example. A method for calculating the contribution based on the estimated class of multiple inference models will be described. The contribution calculation unit 14a may calculate the fourth contribution P4 according to the degree of variation in the estimated class (male or female) output by the multiple inference models. For example, the fourth contribution P4 may be calculated as P4 = 1 - (number of inference models that estimated the male class / total number of inference models). The final estimated class using multiple inference models may be calculated by majority vote. Consider age estimation, for example. A method for calculating the contribution based on the class likelihood distributions output by multiple inference models is shown below. The contribution calculation unit 14a may calculate the fourth contribution P4 according to the degree of agreement between the age class likelihood distributions output by the multiple inference models. For example, the fourth contribution P4 may be calculated using the KL divergence according to the following equation (1):

[0051]

number

[0052] Here, when the number of inference models is n, M refers to the number of combination pairs used to measure the distance between distributions. For example, if the number of inference models is n = 3, then M = 6 (3P2). Furthermore, likelihood distribution Pi refers to the likelihood distribution of the i-th inference model, and likelihood distribution Pj refers to the likelihood distribution of the j-th inference model. Set S refers to the set of likelihood distribution pairs i and j, the number of combinations used to measure the distance between distributions. If the number of inference models is n = 3, then M = 6, and set S is (i, j) = {(1, 2), (1, 3), (2, 1), (2, 3), (3, 1), (3, 2)}. The fourth contribution P4, calculated using KL divergence, represents the average distance between distributions; the closer the average distance is to 0, the more similar the likelihood distributions output by multiple inference models are. Furthermore, the final estimated age using multiple inference models may be calculated by, for example, calculating the age expectation value from the age class likelihood distribution output by each inference model, and then averaging the age expectation values ​​calculated from each inference model. The age expectation value is calculated using the formula {1 / K}×Σ(v k ×p k ) where K represents the number of age classes, k represents the age class number, and vk is the age value, v k represents the likelihood.

[0053] Referring to Figure 1, the correct label determination unit 14b determines whether the second inference result by the inference model 13a of the reference artificial intelligence unit 13 contributes to the re-learning of the target artificial intelligence unit 12, based on the contribution P calculated by the contribution calculation unit 14a. For example, if the contribution degree P is equal to or less than a predetermined reference value, the correct label determination unit 14b determines that the second inference result contributes to re-learning and generates learning data. If the contribution degree P exceeds the predetermined reference value, the correct label determination unit 14b determines that the second inference result does not contribute to re-learning and does not generate learning data.

[0054] In addition, as a criterion value for determining whether the second inference result contributes to the re-learning of the target artificial intelligence unit 12, lower limit values ​​may be set for each of the spatial resolution, luminance resolution, dynamic range, and frame rate of the reference sensor 4, the spatial resolution, luminance resolution, and dynamic range in the inference target area of ​​the second output image, and the estimated likelihood (absolute value of the inference score) of the inference model 13a of the reference artificial intelligence unit 13. If the spatial resolution, luminance resolution, dynamic range, frame rate, or estimated likelihood falls below the criterion value, it may be determined that the second inference result does not contribute to the re-learning. Furthermore, as a criterion value, an upper limit value for the angle α between the shooting direction d2 of the reference sensor 4 and the shooting direction d0 suitable for inference may be set. If the angle α exceeds the criterion value, it may be determined that the second inference result does not contribute to the re-learning. In addition, lower limits may be set for the spatial resolution, brightness resolution, dynamic range, and frame rate of the target sensor 3, and the spatial resolution, brightness resolution, and dynamic range in the inference target area of ​​the first output image, and it may be determined that no learning data will be generated if the lower limits are not met.

[0055] The learning data generated by the correct label determining unit 14b is stored in the storage unit 20 as learning data 23. The learning data 23 may include, for example, the first output image output by the target sensor 3, the time the first output image was captured, a label indicating the class of the inference target 2 estimated by the inference model 13a of the reference artificial intelligence unit 13, the inference target area of ​​the first output image, and the features of the inference target 2 obtained when the class was estimated by the inference model 13a. The memory unit 20 also stores discriminant function parameters 21 (labeled as "inference model 21" in Figure 1) of the inference model 12a of the target artificial intelligence unit 12, and threshold parameters 22 used by the threshold judgment unit 12b as a judgment threshold to be compared with the inference score output from the inference model 12a.

[0056] The learning device 30 re-learns the inference model 12 a of the target artificial intelligence unit 12 using the learning data 23 stored in the memory unit 20 . The learning device 30 includes an initial setting unit 32, a learning data selection unit 33, and an inference model learning unit . When relearning the inference model 12a, the initial setting unit 32 acquires learning parameters related to the learning of the inference model 12a from the relearning start input device 31. The learning parameters are setting items related to the learning method, and include, for example, the number of learning data to be used and the conditions for ending the relearning. Furthermore, for example, the learning parameters may specify whether the target of relearning is the inference model 12a or the threshold parameters of the threshold determination unit 12b.

[0057] Next, the initial setting unit 32 reads the discriminant function parameters 21 and threshold parameters 22 of the inference model stored in the memory unit 20 and sets them as the initial state before learning. As described above, the discriminant function parameters 21 and threshold parameters 22 stored in the memory unit 20 are the same as those used in the inference model 12a and threshold determination unit 12b of the target artificial intelligence unit 12 on-site.

[0058] The learning data selection unit 33 selects learning data to be used for re-learning from the learning data 23 stored in the storage unit 20. It is preferable to select as diverse data as possible as the training data to be used for training. For example, the training data selection unit 33 may cluster the training data by the features contained in the training data and randomly select training data from each cluster. Alternatively, for example, the training data selection unit 33 may cluster the training data based on factors other than the features (e.g., shooting time, contribution, resolution), and randomly select training data from each cluster. Alternatively, for example, the data may be selected so that each attribute class (e.g., gender) is distributed evenly.

[0059] The inference model learning unit 34 uses the learning data selected by the learning data selection unit 33 to learn the discriminant function parameters 21 and threshold parameters 22 read from the storage unit 20. Specifically, the discriminant function parameters 21 and threshold parameters 22 are updated so as to increase the accuracy rate of the inference result when the first output image included in the learning data is input. The discriminant function parameters 21 may be updated so as to approach the feature amount of the inference target 2 obtained when estimating the inference model 13a. For example, a known algorithm such as a support vector machine (SVM) may be used to update the discriminant function parameters 21. Furthermore, if the inference model is a convolutional neural network, a known method such as a gradient method may be used to update the discriminant function parameters 21.

[0060] When the learning of the discriminant function parameters 21 and the threshold parameters 22 is completed, the inference model learning unit 34 stores the learned discriminant function parameters 21 and threshold parameters 22 in the memory unit 20. In addition, the learned discriminant function parameters 21 and threshold parameters 22 are used to update the inference model 12a and threshold determination unit 12b of the on-site target artificial intelligence unit 12.

[0061] The re-learning of the target artificial intelligence unit 12 by the learning device 30 may be performed periodically while new learning data is being collected from the learning data generation device 10. For example, the re-learning may be performed periodically (once a day, for example), or may be performed when the number of newly collected learning data reaches a predetermined number. If it is determined that the target artificial intelligence unit 12 has adapted to the site where it has been introduced, the re-learning of the target artificial intelligence unit 12 may be terminated and the reference sensor 4 may be removed. For example, if the number of times that the accuracy rate of the inference model before re-learning has not been exceeded reaches a predetermined number, it may be determined that further re-learning will be of little effect and the re-learning may be terminated.

[0062] (operation) FIG. 6 is a flowchart of an example of the training data generation method according to the first embodiment. In step S1, the target artificial intelligence unit 12 receives an input of a first output image output from the target sensor 3. The reference artificial intelligence unit 13 receives an input of a second output image output from the reference sensor 4.

[0063] In step S2, the inference model 12a and threshold determination unit 12b of the target artificial intelligence unit 12 obtain a first inference result from the first output image. The inference model 13a and threshold determination unit 13b of the reference artificial intelligence unit 13 obtain a second inference result from the second output image. The reference artificial intelligence unit 13 determines whether or not the inference target 2 has been detected from the second output image of the reference sensor 4. If the inference target 2 has not been detected (step S2: N), the process returns to step S1.

[0064] If the inference target 2 is detected (step S2: Y), the reference artificial intelligence unit 13 outputs the second inference result, the second output image, region information of the inference target region in the second output image where the inference target 2 is detected, and the estimated likelihood to the training data generation unit 14. In addition, the target artificial intelligence unit 12 outputs the first output image, the first inference result, and the estimated likelihood to the training data generation unit 14. Thereafter, the processing proceeds to step S3.

[0065] In step S3, the contribution calculation unit 14a of the learning data generation unit 14 calculates the inference target area of ​​the first output image of the target sensor 3 that corresponds to the inference target area of ​​the second output image of the reference sensor 4. In step S4, the contribution calculation unit 14a calculates the contribution P of the second inference result based on the spatial resolution, brightness resolution, dynamic range, and frame rate of the target sensor 3 and the reference sensor 4, the spatial resolution, brightness resolution, and dynamic range in the inference target area, the detection direction in which the target sensor 3 and the reference sensor 4 detect the inference target 2, the estimated likelihood, etc.

[0066] In step S5, the correct label determination unit 14b determines whether the contribution degree P is equal to or less than the determination reference value. If the contribution degree P is equal to or less than the determination reference value (step S5: Y), the process proceeds to step S6. If the contribution degree P is not equal to or less than the determination reference value (step S5: N), the process proceeds to step S7. In step S6, the correct label determining unit 14b generates learning data and stores it in the storage unit 20.

[0067] In step S7, the learning data generation unit 14 determines whether or not the termination condition for the learning data generation has been met. For example, it may be determined that the termination condition has been met when a predetermined time (e.g., one day) has elapsed. Alternatively, it may be determined that the termination condition has been met when a predetermined number of pieces of learning data have been generated. The predetermined time and the number of pieces of learning data may be set according to the frequency with which the inference target 2 is detected. If the termination condition for the learning data generation is not met (step S7: N), the process returns to step S1. If the termination condition for the learning data generation is met (step S7: Y), the process ends.

[0068] FIG. 7 is a flowchart showing an example of the re-learning process of the target artificial intelligence unit 12. In step S11 , the initial setting unit 32 of the learning device 30 acquires learning parameters from the relearning start input device 31 . In step S12, the initial setting unit 32 reads the discriminant function parameters 21 and threshold parameters 22 of the inference model stored in the storage unit 20 and sets them as the initial state before learning.

[0069] In step S13, the learning data selection unit 33 selects learning data to be used for re-learning from the learning data 23 stored in the storage unit 20. In step S14, the inference model learning unit 34 updates the discriminant function parameters 21 and threshold parameters 22 so as to increase the accuracy rate of the inference results when the first output image included in the learning data selected by the learning data selection unit 33 is input.

[0070] In step S15, the inference model learning unit 34 determines whether the termination condition for relearning has been met. For example, the inference model learning unit 34 determines that the termination condition has been met if the highest value of the accuracy rate has not been updated a predetermined number of times. Also, the inference model learning unit 34 determines that the termination condition has been met when the number of learning attempts reaches a predetermined value. If the termination condition has not been met (step S15: N), the process returns to step S14. If the termination condition has been met (step S15: Y), the process proceeds to step S16. In step S16, the inference model learning unit 34 stores the learned discriminant function parameters 21 and threshold parameters 22 in the memory unit 20. The inference model 12a and threshold determination unit 12b of the on-site target artificial intelligence unit 12 are updated using the learned discriminant function parameters 21 and threshold parameters 22. The process then ends.

[0071] (Effects of the first embodiment) (1) The learning data generation device 10 comprises a trained target artificial intelligence unit 12 that outputs a first inference result for a first output value from a target sensor 3 that monitors the monitored space, a trained reference artificial intelligence unit 13 that outputs a second inference result for a second output value from a reference sensor 4 that monitors the monitored space, a contribution calculation unit 14a that calculates a contribution degree, which is the degree to which the second inference result contributes to the re-learning of the target inference model 12a of the target artificial intelligence unit 12, based on the second output value, and a learning data generation unit 14 that generates learning data for re-learning the target artificial intelligence unit 12 based on the contribution degree calculated by the contribution calculation unit 14a, the first output value, and the second inference result.

[0072] This allows the reference artificial intelligence unit 13 to generate labeled training data for retraining the target artificial intelligence unit 12, thereby reducing the human costs required for retraining. Furthermore, since training data is generated according to the contribution of the second inference result, even when the reference artificial intelligence unit 13 is used instead of human labor, it is possible to prevent the generation of inappropriate training data.

[0073] (2) The contribution degree calculation unit 14a may calculate the contribution degree of the second inference result according to the first output value and the second output value, thereby making it possible to more accurately calculate the contribution degree of the second inference result. (3) The contribution may be calculated based on the respective resolutions of the target sensor 3 and the reference sensor 4. This makes it possible to determine the contribution of the second inference result according to the resolution of the sensor. (4) Each of the first output value and the second output value of the target sensor 3 and the reference sensor 4 may include a set of component values ​​at multiple coordinates. The contribution calculation unit 14a may calculate the contribution based on the spatial resolution of the first output value in a corresponding range, which is a range of first output values ​​corresponding to a detection range, which is a range of second output values ​​in which the inference target is detected, and the spatial resolution of the second output value in the detection range. This makes it possible to determine the contribution of the second inference result to the output values ​​of the sensor based on the spatial resolution in the inference target area where the inference target is detected.

[0074] (5) Each of the first output value and the second output value of the target sensor 3 and the reference sensor 4 may include a set of component values ​​at a plurality of coordinates. The contribution calculation unit 14a may calculate the contribution based on the detection direction d1 of the target sensor 3 that detects the component values ​​of the corresponding coordinates, which are the coordinates of the first output value corresponding to the detection coordinates, which are the coordinates of the second output value at which the inference target is detected, and the detection direction d2 of the reference sensor 4 that detects the component values ​​of the detection coordinates. This allows the contribution of the second inference result to be determined depending on how suitable the sensor's detection direction is for inference using the inference model.

[0075] (6) The target sensor 3 and the reference sensor 4 may each be a camera. The contribution may be calculated based on at least one of the dynamic range, the luminance resolution, and the frame rate of each of the target sensor 3 and the reference sensor 4. This makes it possible to determine the contribution of the second inference result according to the dynamic range, luminance resolution, and frame rate of the sensor.

[0076] (7) The target sensor 3 and the reference sensor 4 may each be a camera. The contribution calculation unit 14a may calculate the contribution based on the distribution of pixel component values ​​in the detection range, which is a range of first output values ​​corresponding to the detection range, which is a range of second output values ​​in which the inference target is detected. This makes it possible to determine the contribution of the second inference result based on the distribution of pixel component values ​​in the inference target area where the inference target is detected, among the output values ​​of the sensor. (8) The learning data generation device 10 may include a storage unit that stores at least the installation conditions of the target sensor 3 and the reference sensor 4. The contribution degree calculation unit 14a may calculate the corresponding range and the corresponding coordinates based on the installation conditions.

[0077] (9) The contribution calculation unit 14a may calculate the contribution based on the reliability of the inference result output by a predetermined inference model that has been trained for the first output value and the reliability of the inference result output by a predetermined inference model that is the same as or different from the predetermined inference model for the second output value. This allows the contribution of the second inference result to be determined depending on how suitable the sensor output value is for the inference of the inference model. (10) The contribution calculation unit 14a may calculate the contribution according to the degree of agreement between the inference results output by multiple predetermined reference inference models 14a that have been trained for the second output value. This allows the contribution of the second inference result to be determined according to how well the sensor output value is suited to the inference of the inference model.

[0078] (Second embodiment) As described above, the training data generation device 10 compares the contribution P of the second inference result with a predetermined reference value. Alternatively, the device 10 compares the resolution of the reference sensor 4, the resolution of the second output image of the reference sensor 4, the detection direction d2 of the reference sensor 4, and the estimated likelihood of the second inference result with predetermined reference values. Then, based on the comparison result, the device 10 determines whether the second inference result will contribute to re-learning of the target artificial intelligence unit 12, and generates training data if it is determined that the second inference result will contribute to re-learning. Therefore, in order to efficiently generate learning data while preventing the generation of learning data that does not contribute to the re-learning of the target artificial intelligence unit 12, it is preferable that the judgment reference value be set appropriately. Therefore, in the training data generation device 10 of the second embodiment, the criterion value for determining whether the second inference result contributes to the re-training of the target artificial intelligence unit 12 is set to an appropriate value by annotation by the user.

[0079] Below, we will explain an example of annotating a judgment reference value that is compared with the contribution P of the second inference result, but the judgment reference value can also be annotated in the same way when it is compared with the resolution of the reference sensor 4, the resolution of the second output image of the reference sensor 4, the detection direction d2 of the reference sensor 4, or the estimated likelihood of the second inference result. 8 is a schematic diagram of an example of the training data generation device 10 of the second embodiment. The training data generation device 10 of the second embodiment has a configuration similar to that of the training data generation device 10 of the first embodiment, and the same reference numerals are assigned to the same or similar components.

[0080] The training data generation device 10 of the second embodiment includes a display unit 15 and an input unit 16. The display unit 15 may include a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display. The display unit 15 is connected to the control unit 11 and displays the second inference result by the reference artificial intelligence unit 13 to the user. The input unit 16 includes a user interface such as a keyboard, a mouse, etc. that is operated by the user to input data, etc. The input unit 16 is connected to the control unit 11, converts the user's input into an operation signal, and outputs it to the control unit 11.

[0081] If the contribution degree P of the second inference result is less than a predetermined judgment reference value, the correct label determination unit 14b of the learning data generation unit 14 determines that the second inference result will contribute to the re-learning of the target artificial intelligence unit 12 and generates learning data. On the other hand, if the contribution degree P exceeds a predetermined judgment reference value, the correct label judgment unit 14b of the second embodiment displays on the display unit 15 an input screen for the user to input the correct answer to the inference result of the inference target 2.

[0082] 9 is a schematic diagram of an example of an input screen on which a user inputs the correct answer of the inference result. An input screen 40 displays a partial image (face image) 41 of the inference target area in the second output image, a radio button 42 indicating the class of the inference target 2 (gender attribute in the example of FIG. 9), and a registration button 43 instructing registration. The correct label determination unit 14b may present the second inference result to the user by reflecting the second inference result on the input screen 40. For example, the second inference result can be presented to the user by setting the default value of the radio button 42 to the second inference result ("Male" in the example of FIG. 9).

[0083] The user determines the class ("male" or "female" in the example of FIG. 9) for the partial image (face image) 41. If the user determines that the default value of the radio button 42 is correct (i.e., if the user determines that the second inference result is correct), the user operates a mouse or the like provided as the input unit 16 to click the registration button 43. If it is determined that the default value of radio button 42 is incorrect (i.e., if it is determined that the second inference result is incorrect), the inference result is corrected by operating radio button 42 using a mouse or the like, and then the registration button 43 is clicked.

[0084] The correct label determination unit 14b determines, in accordance with the user's input result, whether the second inference result contributes to the relearning of the target artificial intelligence unit 12. For example, the correct label determination unit 14b determines, in accordance with the user's input result, whether the second inference result is correct, and determines, in accordance with whether the second inference result is correct, whether it contributes to the relearning. The correct label determination unit 14b updates the determination criterion value in accordance with whether the second inference result contributes to the relearning. For example, if the second inference result contributes to relearning (i.e., if the user's input result is equal to the second inference result), the correct label determination unit 14b increases the determination reference value by a predetermined step width. As described above, the contribution degree P of the second inference result is calculated as an index value that becomes smaller the more the second inference result contributes to relearning of the target artificial intelligence unit 12. Therefore, by increasing the determination reference value, the criterion for determining that the second inference result will contribute to relearning is relaxed.

[0085] The step width for correcting the judgment reference value may be set according to an average value calculated from the contribution degree P of the second inference results that were correct. Furthermore, the user's input may be received for a plurality of inference subjects 2 (for example, 10), and if the percentage of correct answers is equal to or greater than a predetermined value (for example, 80%), the judgment reference value may be increased by a predetermined step width. Furthermore, if the second inference result does not contribute to re-learning (that is, if the user's input result differs from the second inference result), the correct label determining unit 14b decreases the determination reference value by a predetermined step width.

[0086] FIG. 10 is a schematic diagram of another example of the input screen 40 on which the user inputs the correct answer of the inference result. Even if appropriately labeled training data is generated, if the resolution (e.g., resolution, dynamic range, or luminance resolution) of the first output image included in the training data is low, the training data may not be suitable as training data. Therefore, in the example of the input screen 40 shown in Figure 10, in addition to the input screen 40 of Figure 9, a partial image (face image) 44 of the inference target area in the first output image from the target sensor 3 and a non-registration button 45 that instructs not to register the learning data are displayed.

[0087] This allows the user to refer to the partial image 44 of the first output image and determine whether the first output image is appropriate as learning data. If the user determines that the first output image is not appropriate as learning data, the user can avoid registering learning data that includes the inappropriate first output image by clicking the non-registration button 45 using a mouse or the like.

[0088] FIG. 11 is a flowchart of an example of a training data generating method according to the second embodiment. The processing in steps S21 to S24 is the same as the processing in steps S1 to S4 described with reference to FIG. In step S25, the correct label determination unit 14b determines whether the contribution degree P is equal to or less than the determination reference value. If the contribution degree P is equal to or less than the determination reference value (step S25: Y), the process proceeds to step S28. If the contribution degree P is not equal to or less than the determination reference value (step S25: N), the process proceeds to step S26.

[0089] In step S26, the correct label determining unit 14b displays the input screen 40 on which the user inputs the correct answer of the inference result, and receives the user's input. In step S27, the correct label determination unit 14b updates the determination criterion value for determining the correct label depending on whether the correct answer of the estimation result input by the user is equal to the second inference result (i.e., whether the second inference result contributes to re-learning of the target artificial intelligence unit 12). Thereafter, the process proceeds to step S28.

[0090] In step S28, the correct label determination unit 14b generates learning data and stores it in the storage unit 20. At this time, if the contribution degree P is equal to or less than the determination reference value and the correct answer of the estimation result input by the user is different from the second inference result (i.e., if the second inference result is incorrect and does not contribute to the re-learning of the target artificial intelligence unit 12), a label indicating the inference result input by the user instead of the second inference result is assigned to the learning data. In step S29, the learning data generation unit 14 determines whether the termination condition for learning data generation is satisfied. If the termination condition for learning data generation is not satisfied (step S29: N), the process returns to step S21. If the termination condition for learning data generation is satisfied (step S29: Y), the process ends.

[0091] (Effects of the second embodiment) (1) The training data generation device 10 may include a display unit 15 that displays the inference result to the user. The training data generation unit 14 may display the second inference result on the display unit 15 when the contribution degree does not satisfy a predetermined criterion. This allows the user to confirm whether the second inference result is correct or not when the contribution of the second inference result does not satisfy a predetermined criterion.

[0092] (2) The training data generation device 10 may include an input unit 16 that accepts input from a user. The training data generation unit 14 may update the judgment criterion when the input unit 16 accepts a user input that determines whether the second inference result output from the display unit 15 is correct. This allows the judgment criterion to be set to an appropriate value based on the user input. [Explanation of symbols]

[0093] 1...Monitored space, 2...Inference target (person), 3...Target sensor, 4...Reference sensor, 10...Learning data generation device, 11...Control unit, 12...Target artificial intelligence unit, 12a...Inference model, 12b...Threshold judgment unit, 13...Reference artificial intelligence unit, 13a...Inference model, 13b...Threshold judgment unit, 14...Learning data generation unit, 14a...Contribution calculation unit, 14b...Correct label judgment unit, 15...Display unit, 16...Input unit, 20...Memory unit, 21...Discrimination function parameters (inference model), 22...Threshold parameters, 23...Learning data, 30...Learning device, 31...Re-learning start input device, 32...Initial setting unit, 33...Learning data selection unit, 34...Inference model learning unit

Claims

1. a trained target inference model that outputs a first inference result for a first output value from a target sensor that monitors the monitored space; a trained reference inference model that outputs a second inference result for a second output value from a reference sensor that monitors the monitored space; a contribution calculation unit that calculates a contribution, which is the degree to which the second inference result contributes to the re-learning of the target inference model, according to the second output value; a learning data generation unit that generates learning data for re-learning the target inference model based on the contribution calculated by the contribution calculation unit, the first output value, and the second inference result; A training data generation device comprising:

2. The training data generating device according to claim 1 , wherein the contribution calculation unit calculates the contribution in accordance with the first output value and the second output value.

3. 3. The training data generating device according to claim 2, wherein the contribution is calculated based on the respective resolutions of the target sensor and the reference sensor.

4. each of the first output value and the second output value includes a set of component values ​​at multiple coordinates; The learning data generation device according to claim 2 or 3, characterized in that the contribution calculation unit determines a corresponding range, which is the range of the first output values ​​corresponding to a detection range, which is the range of the second output values ​​in which an inference target present in the monitored space is detected, and calculates the contribution based on the spatial resolution of the first output values ​​in the corresponding range and the spatial resolution of the second output values ​​in the detection range.

5. each of the first output value and the second output value includes a set of component values ​​at multiple coordinates; The learning data generation device according to claim 2 or 3, characterized in that the contribution calculation unit determines corresponding coordinates, which are coordinates of the first output value corresponding to detection coordinates, which are coordinates of the second output value at which an inference target present in the monitored space is detected, and calculates the contribution based on the detection direction of the target sensor that detects component values ​​of the corresponding coordinates and the detection direction of the reference sensor that detects component values ​​of the detection coordinates.

6. the target sensor and the reference sensor are each a camera; The training data generation device according to any one of claims 2 to 5, characterized in that the contribution is calculated based on at least one of the dynamic range, luminance resolution, and frame rate of each of the target sensor and the reference sensor.

7. the target sensor and the reference sensor are each a camera; The learning data generation device according to claim 2 or 3, characterized in that the contribution calculation unit obtains a corresponding range, which is the range of the first output values ​​corresponding to a detection range, which is the range of the second output values ​​in which an inference target present in the monitored space is detected, and calculates the contribution based on a distribution of pixel component values ​​in the corresponding range and a distribution of pixel component values ​​in the detection range.

8. a storage unit that stores at least installation conditions of the target sensor and the reference sensor; 8. The learning data generating device according to claim 4, wherein the contribution calculation unit calculates the corresponding range based on the installation conditions.

9. a storage unit that stores at least installation conditions of the target sensor and the reference sensor; The training data generating device according to claim 5 , wherein the contribution calculation unit calculates the corresponding coordinates based on the installation conditions.

10. The learning data generation device described in any one of claims 2 to 9, characterized in that the contribution calculation unit calculates the contribution based on the reliability of the inference result output by a trained predetermined inference model for the first output value and the reliability of the inference result output by a trained inference model that is the same as or different from the predetermined inference model for the second output value.

11. A learning data generation device described in any one of claims 1 to 10, characterized in that the contribution calculation unit calculates the contribution based on the degree of agreement between the inference results output by multiple specified reference inference models that have been learned for the second output value.

12. an output unit that outputs an inference result to a user; The learning data generation device according to any one of claims 1 to 11, characterized in that the learning data generation unit outputs the second inference result from the output unit when the contribution does not satisfy a predetermined judgment criterion.

13. an input unit that accepts input from a user; The learning data generation device according to claim 12, characterized in that the learning data generation unit updates the judgment criteria when the input unit receives input from the user that determines whether the second inference result output from the output unit is correct.

14. A learning data generation method for generating learning data for retraining a trained target inference model that outputs a first inference result in response to a first output value from a target sensor that monitors a monitored space, comprising: A process of obtaining the second inference result using a trained reference inference model that outputs a second inference result in response to a second output value from a reference sensor that monitors the monitored space; A process of calculating a contribution rate, which is the degree to which the second inference result contributes to re-learning of the target inference model, according to the second output value; generating the learning data based on the calculated contribution, the first output value, and the second inference result; A learning data generation method characterized by causing a computer to execute the above.

15. A learning data generation program for generating learning data for re-learning a trained target inference model that outputs a first inference result in response to a first output value from a target sensor that monitors a monitored space, A process of obtaining the second inference result using a trained reference inference model that outputs a second inference result in response to a second output value from a reference sensor that monitors the monitored space; A process of calculating a contribution rate, which is the degree to which the second inference result contributes to re-learning of the target inference model, according to the second output value; generating the learning data based on the calculated contribution, the first output value, and the second inference result; A learning data generation program that causes a computer to execute the above.

Images