Learning device, learning method, program, and pre-trained model

Abstract

To provide a learning device capable of efficiently retraining a pre-trained model to improve the inference performance, a learning method, a program, and a pre-trained model.SOLUTION: A disclosed embodiment of a learning device includes: an inference unit that performs inference on each training data to detect a target using a pre-trained model and outputs the likelihood of the inference; a feature extraction unit that extracts the features of the multiple detection targets for which inference has been performed using the pre-trained model; an identification unit that identifies a first detection target from the plurality of detection targets based on the likelihood; a target extraction unit that extracts a second detection target from multiple detection targets whose distance in feature space from the first detection target is within a predetermined distance based on the feature value; and a learning unit that performs retraining the pre-trained model using the training data containing the second detection target.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a learning device, a learning method, a program, and a trained model. [Background technology]

[0002] Object detection technologies that use trained models generated by machine learning such as deep learning (DL) are known. In machine learning, when generating a trained model, the countless parameters of the model are determined inductively from each piece of training data contained in a dataset, making it difficult for developers to grasp in advance the dataset required to generate a trained model with the desired inference performance. In the case of a deep learning model, examples of the countless parameters that the model has include weights set between each node and bias term values.

[0003] For this reason, in existing technologies, when generating a trained model, developers and others repeat learning by adding training data based on empirical rules. However, if the added training data does not lead to an improvement in the inference performance of the trained model, this will result in a decrease in the efficiency of generating the trained model.

[0004] For example, Patent Document 1 discloses a technology for improving the accuracy of a trained model for image analysis by using the evaluation results (likelihood and reliability) of the trained model for the training data to present data to be added to the training data. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent Publication No. 2021-128415 Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if a trained model is retrained using training data selected using likelihood or reliability, as in the technology disclosed in Patent Document 1, the inference performance of the trained model does not necessarily improve, and may result in a decrease in the efficiency of generating the trained model.

[0007] The present disclosure has been made in consideration of the above circumstances, and contributes to providing a learning device, a learning method, a program, and a trained model that are capable of efficiently performing re-learning to improve the inference performance of a trained model. [Means for solving the problem]

[0008] One aspect of the learning device of the present disclosure includes an inference unit that uses a trained model to perform inference to detect a detection target for each training data and outputs the likelihood of the inference; a feature extraction unit that uses the trained model to extract features of the multiple detection targets for which the inference has been performed; an identification unit that identifies a first detection target from the multiple detection targets based on the likelihood; a target extraction unit that extracts a second detection target from the multiple detection targets based on the features, the second detection target being within a predetermined distance in feature space from the first detection target; and a learning unit that re-trains the trained model using training data including the second detection target.

[0009] In one aspect of the learning method of the present disclosure, an inference unit uses a trained model to make an inference to detect a detection target for each training data and output the likelihood of the inference, a feature extraction unit uses the trained model to extract features of the multiple detection targets for which the inference was made, an identification unit identifies a first detection target from the multiple detection targets based on the likelihood, a target extraction unit extracts a second detection target from the multiple detection targets based on the features, the second detection target being within a predetermined distance in feature space from the first detection target, and a learning unit re-trains the trained model using training data including the second detection target.

[0010] One aspect of the program of the present disclosure is for causing a computer to execute the following steps: using a trained model to perform inference to detect a detection target for each training data and outputting the likelihood of the inference; using the trained model to extract features of the multiple detection targets for which the inference was performed; identifying a first detection target from the multiple detection targets based on the likelihood; extracting a second detection target from the multiple detection targets based on the features, the second detection target being within a predetermined distance in feature space from the multiple detection targets; and re-training the trained model using training data including the second detection target.

[0011] One aspect of the trained model of the present disclosure is generated by the device described above, the method described above, or the program described above. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to provide a learning device, a learning method, a program, and a trained model that can efficiently perform re-learning to improve the inference performance of a trained model. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of a learning device according to this embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the functional configuration of the learning device of this embodiment. [Figure 3] FIG. 3 is a diagram showing an example of an image for learning in this embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of information stored in the inference result storage unit of this embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a deep learning model according to the present embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of information stored in the feature amount storage unit of this embodiment. [Figure 7]FIG. 7 is a diagram showing an example of priority information stored in the priority information storage unit of this embodiment. [Figure 8] FIG. 8 is a diagram showing an example of an image of the feature space displayed by the display control unit of this embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a list screen of images including the second detection target displayed by the display control unit of this embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an attribute tally result screen displayed by the display control unit of this embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a distribution screen of feature amounts by attribute displayed by the display control unit of this embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of the learning process performed by the learning device of this embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, an embodiment of the present disclosure (hereinafter simply referred to as "the present embodiment") will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the following embodiment.

[0015] In the following, an example will be described in which the trained model obtained after training by the learning device of this embodiment is adopted in an on-board camera for functions such as AEB (Autonomous Emergency Braking) and detects obstacles such as pedestrians from images captured by the on-board camera. However, the trained model of this embodiment is not limited to this, and may be, for example, a model for checking the presence of protective equipment at a manufacturing site or a model for inspecting the appearance of products at a manufacturing site. Note that the trained model of this embodiment can be any model generated by machine learning, such as an object recognition model or a person recognition model. Furthermore, in the following, an example will be described in which the training data of this embodiment is images for training, but this is not limited to this.

[0016] First, the configuration of the learning device of this embodiment will be described.

[0017] Fig. 1 is a block diagram showing an example of the hardware configuration of a learning device 100 of this embodiment. As shown in Fig. 1, learning device 100 includes a control device 11, a main memory device 13, an auxiliary memory device 15, a display device 17, an input device 19, a communication device 21, and various buses 23. Control device 11, main memory device 13, auxiliary memory device 15, display device 17, input device 19, and communication device 21 are connected via various buses 23. As such, learning device 100 of this embodiment has an existing hardware configuration that utilizes an existing computer.

[0018] The control device 11 controls the overall operation of the learning device 100. Examples of the control device 11 include at least one of a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit), but are not limited to these. There may be any number of CPUs or GPUs as long as they are one or more, and they may be single-core or multi-core.

[0019] Examples of the main memory device 13 include, but are not limited to, a ROM (Read Only Memory) and a RAM (Random Access Memory). The ROM stores various programs, such as a program for controlling the learning device 100 and the machine learning program of this embodiment. The RAM is used as a work area when the control device 11 performs various controls based on the programs stored in the ROM.

[0020] The auxiliary storage device 15 stores various data, such as the various programs described above and data for machine learning in this embodiment. The various programs described above may be stored in at least one of the main storage device 13 and the auxiliary storage device 15. Examples of the auxiliary storage device 15 include at least one of existing storage devices capable of magnetic, electrical, or optical storage, such as a hard disk drive (HDD), a solid state drive (SSD), and a digital versatile disc (DVD). However, the auxiliary storage device 15 is not limited to these. The auxiliary storage device 15 may be built into the learning device 100 or externally connected to the learning device 100 via an interface such as a universal serial bus (USB). The auxiliary storage device 15 may also be a network-attached storage (NAS) connected via a network, such as a local area network (LAN) or a wide area network (WAN).

[0021] Display device 17 displays various screens used during machine learning by learning device 100, and serves as a user interface with the user (operator). Examples of display device 17 include, but are not limited to, various displays such as a liquid crystal display, an organic electroluminescence (EL) display, and a touch panel display. Display device 17 may be an internal display built into learning device 100, or an external display connected to learning device 100 via a display interface such as HDMI (registered trademark).

[0022] Input device 19 is used for various inputs, selections, and specifications used during machine learning by learning device 100, and serves as a user interface with the user (operator). Examples of input device 19 include, but are not limited to, a keyboard, a mouse, and a touch panel. Input device 19 may be built into learning device 100 or may be externally attached to learning device 100 via an interface such as USB.

[0023] Examples of the communication device 21 include, but are not limited to, a communication device for a wired LAN and a wireless communication device for a wireless LAN. The communication device 21 may be used to externally acquire the machine learning program and data of this embodiment, or to externally output the trained model generated by the learning device 100.

[0024] In addition to the above configuration, the learning device 100 may further include hardwired circuits such as an IC (Integrated Circuit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field-Programmable Gate Array) specific to the learning device 100 in order to realize the learning function.

[0025] 2 is a block diagram showing an example of the functional configuration of the learning device 100 of this embodiment. As shown in FIG. 2, the learning device 100 includes a learning data storage unit 101, a learning unit 103, a trained model storage unit 105, an inference unit 107, an inference result storage unit 109, a feature extraction unit 111, a feature storage unit 113, a priority information storage unit 115, an identification unit 117, a target extraction unit 119, and a display control unit 121.

[0026] The training data storage unit 101, the trained model storage unit 105, the inference result storage unit 109, the feature storage unit 113, and the priority information storage unit 115 can be realized, for example, by at least one of the main storage unit 13 and the auxiliary storage unit 15 described in Figure 1.

[0027] The learning unit 103, the inference unit 107, the feature extraction unit 111, the identification unit 117, the target extraction unit 119, and the display control unit 121 can be realized by, for example, the control device 11 and the main memory device 13 described in FIG.

[0028] For example, the control device 11 reads a machine learning program stored in the main memory device 13 (ROM) or the auxiliary memory device 15, or a machine learning program acquired from an external device via the communication device 21, and loads the program into the main memory device 13 (RAM). The control device 11 executes various processes in accordance with the loaded program, thereby realizing each of the above-described functional units. Here, the description has been given taking an example in which each of the above-described functional units is realized as software, but at least a portion of each of the above-described functional units may also be realized as hardware. In this case, the functional units realized as hardware may be realized, for example, by the above-described hardwired circuit. Furthermore, any of the above-described functional units may be realized by a combination of software and hardware.

[0029] The learning data storage unit 101 stores learning data. The learning data storage unit 101 stores, for example, a plurality of learning images as learning data. Examples of learning images include, but are not limited to, images showing a detection target such as a pedestrian. The learning images (learning data) may include an image ID (identification) that identifies the image, or the image ID may be added to the image.

[0030] Furthermore, the training images (training data) may include attribute information indicating attributes (e.g., labels) of the image or the detection target, such as a pedestrian, appearing in the image, or the attribute information may be added as an annotation. Image attributes include, but are not limited to, information such as low contrast. Attributes of the detection target (pedestrian) include, but are not limited to, information such as type, position, posture, physique, and occlusion. Types include, but are not limited to, the type of detection target appearing in the image, such as a pedestrian. Positions include, but are not limited to, the relative distance from the camera that captured the image. Postures include, but are not limited to, the posture of the pedestrian, such as facing forward or facing sideways. Physical types include, but are not limited to, adult or child. Occlusions include, but are not limited to, objects that partially obscure the pedestrian, such as an umbrella or a bag.

[0031] The learning unit 103 learns the learning data stored in the learning data storage unit 101 to generate a trained model and outputs it to the trained model storage unit 105. The learning unit 103, for example, acquires some of the multiple learning images stored in the learning data storage unit 101 to construct a dataset. For each image included in the constructed dataset, the learning unit 103 generates a trained model by performing supervised learning, for example, using the image as input and attribute information included in or added to the image as a correct answer for a detection target (pedestrian, etc.) appearing in the image.

[0032] In this embodiment, the learning unit 103 learns each image included in the data set using deep learning (DL), but the learning method is not limited to this. The learning method of the trained model may be any existing machine learning method.

[0033] For example, assume that the trained model of this embodiment is a deep learning model (network) consisting of an input layer, one or more intermediate layers, and an output layer. In this case, when an image (e.g., pixel values ​​of each pixel constituting the image) is input to the deep learning model, each of the input layer and one or more intermediate layers multiplies the input value by a weight and outputs the result to the next layer, and finally, the output layer outputs a likelihood. The likelihood is a value indicating the accuracy of the inference performed to detect a detection target in the image. The learning unit 103 generates the trained model by adjusting (learning) the value of each weight so as to reduce the error between the output likelihood and a correct value. The correct value can be identified from an attribute (e.g., type) assigned to the image. The trained model is not limited to the above-mentioned deep learning model, and may be any model or network used in machine learning.

[0034] The trained model storage unit 105 stores the trained model generated by the training unit 103. The trained model performs inference on input data and outputs an inference result and a likelihood indicating the accuracy of the inference. For example, when an image is input, the trained model performs inference to detect a detection target in the image and outputs an inference result and the likelihood of the inference.

[0035] The inference unit 107 uses the trained model stored in the trained model memory unit 105 to perform inference for detecting the detection target for each piece of training data stored in the training data memory unit 101, and outputs the inference result and the likelihood of the inference.

[0036] The inference unit 107 acquires, for example, a plurality of learning images stored in the learning data storage unit 101. The plurality of learning images acquired by the inference unit 107 from the learning data storage unit 101 preferably includes at least learning images that have not been used to construct a dataset by the learning unit 103, but this is not limitative. In this embodiment, a case will be described in which the inference unit 107 acquires all images stored in the learning data storage unit 101, but this is not limitative.

[0037] The inference unit 107 inputs each acquired training image into the trained model stored in the trained model storage unit 105, and obtains the inference result and likelihood of the inference for detecting the detection target performed on each training image. In this way, the inference unit 107 obtains the likelihood for each detection target detected by inference.

[0038] Fig. 3 is a diagram showing an example of a training image of this embodiment. A training image 201 shown in Fig. 3 shows a pedestrian 203 and a pedestrian 205. For example, assume that the image 201 is input to a trained model, and both the pedestrian 203 and the pedestrian 205 are detected by inference. In this case, likelihoods are obtained for the pedestrian 203 and the pedestrian 205, respectively.

[0039] The inference unit 107 associates the likelihood of each detection target detected by inference with attribute information of the detection target and outputs the associated data to the inference result storage unit 109. The inference result storage unit 109 stores the likelihood of each detection target detected by inference with attribute information of the detection target in association with each other.

[0040] FIG. 4 is a diagram showing an example of information stored in the inference result storage unit 109 of this embodiment. In the example shown in FIG. 4, the inference result storage unit 109 stores a detection target ID, a likelihood, and attribute information in association with each other. The detection target ID is an ID indicating the detection target. The detection target ID is assigned by a trained model to a detection target detected by inference, but is not limited to this. The likelihood is the likelihood of the detection target indicated by the detection target ID. The likelihood is expressed in the range of 0 to 1, with a larger value indicating a higher probability of inference. The attribute information is assigned to an image in which the detection target indicated by the detection target ID is detected.

[0041] The feature extraction unit 111 uses the trained model stored in the trained model storage unit 105 to extract feature amounts of a plurality of detection targets for which inference has been performed by the inference unit 107.

[0042] The feature extraction unit 111 acquires, for example, a plurality of learning images stored in the learning data storage unit 101. In this embodiment, the feature extraction unit 111 acquires, from the learning data storage unit 101, the learning images acquired from the learning data storage unit 101 by the inference unit 107. For example, the feature extraction unit 111 acquires all images stored in the learning data storage unit 101.

[0043] The feature extraction unit 111 inputs each acquired training image into the trained model stored in the trained model storage unit 105, and extracts the features of the detection target detected by the inference performed on each training image.

[0044] Examples of feature quantities include values ​​related to parameters of a trained model, or values ​​obtained by reducing the dimensionality of such values. Examples of feature quantities include, but are not limited to, parameters of a deep learning model. Examples of parameters of a deep learning model include, but are not limited to, multidimensional values ​​formed from the values ​​of each node constituting the deep learning model, multidimensional values ​​formed from combinations of the values ​​of each node and each weight, and multidimensional values ​​formed from the values ​​of nodes constituting a specific layer of the deep learning model. Furthermore, the feature quantities of the detection target may be values ​​obtained by reducing the dimensionality of the above-mentioned multidimensional values ​​using techniques such as principal component analysis (PCA) or topic modeling (Latent Dirichlet Allocation (LDA)).

[0045] 5 is a diagram illustrating an example of a deep learning model according to this embodiment. In the example illustrated in FIG. 5, deep learning model 301 is configured with input layer 311, one or more intermediate layers 313, and output layer 315. However, the configuration of deep learning model 301, such as the number of intermediate layers and the number of nodes constituting each layer, is not limited to this. In this embodiment, for example, a case will be described in which, of the one or more intermediate layers 313, the values ​​of nodes 321 and 323 constituting intermediate layer 314, which is located before output layer 315, are used as feature quantities of the detection target; however, this is not limiting.

[0046] For each detection target detected by inference, the feature extraction unit 111 associates the feature of the detection target with the image ID of an image in which the detection target appears, and outputs the associated feature to the feature storage unit 113. The feature storage unit 113 stores, for each detection target detected by inference, the feature of the detection target with the image ID of an image in which the detection target appears, in association with each other.

[0047] Fig. 6 is a diagram showing an example of information stored in the feature amount storage unit 113 of this embodiment. In the example shown in Fig. 6, the feature amount storage unit 113 stores a detection target ID, a feature amount, and an image ID in association with each other. The detection target ID is the same as in Fig. 4. The feature amount is the value of node 321 and the value of node 323 described above. The image ID is the ID of an image in which the detection target indicated by the detection target ID appears, and is assigned to that image.

[0048] The priority information storage unit 115 stores priority information in which priorities are associated with attributes. Fig. 7 is a diagram showing an example of priority information stored in the priority information storage unit 115 of this embodiment. In the example shown in Fig. 7, the attribute "concealment" is associated with the highest priority "1", and the attribute "low contrast" is associated with the second highest priority "2".

[0049] The identification unit 117 identifies a first detection target from the multiple detection targets based on the likelihood inferred by the inference unit 107. Specifically, the identification unit 117 identifies, from the multiple detection targets detected by inference by the inference unit 107, a detection target whose likelihood is less than a threshold and to which an attribute associated with a predetermined priority in the priority information stored in the priority information storage unit 115 is assigned, as the first detection target.

[0050] For example, a case will be described where the threshold value is 0.3 and the predetermined priority is priority "1". In this case, from the information shown in FIG. 4, the detection targets whose likelihood is less than the threshold value "0.3" are the detection targets with detection target IDs "DT003" and "DT004" whose likelihood is 0.2. Furthermore, from the information shown in FIG. 7, the attribute associated with priority "1" is the attribute "concealment". Of the detection targets with detection target IDs "DT003" and "DT004", the detection target with detection target ID "DT003" is the one to which the attribute "concealment" is assigned. Therefore, the identification unit 117 identifies the detection target with detection target ID "DT003" as the first detection target.

[0051] Based on the features extracted by the feature extraction unit 111, the target extraction unit 119 extracts a second detection target from the multiple detection targets detected by inference by the inference unit 107, the second detection target being within a predetermined distance in the feature space from the first detection target identified by the identification unit 117.

[0052] Specifically, the target extraction unit 119 plots multiple feature quantities stored in the feature quantity storage unit 113 on a feature quantity space expressed by the number of dimensions of the feature quantities. The target extraction unit 119 also identifies the detection target ID of the first detection target from the information shown in FIG. 4 and identifies the feature quantities of the first detection target from the information shown in FIG. 6. As a result, the target extraction unit 119 identifies the feature quantities of the first detection target from the feature quantities plotted on the feature quantity space. The target extraction unit 119 extracts feature quantities that are within a predetermined distance from the feature quantities of the first detection target on the feature quantity space. The target extraction unit 119 identifies the detection target indicated by the detection target ID of the extracted feature quantity as the second detection target from the information shown in FIG. 6. As a result, the target extraction unit 119 extracts the second detection target. The predetermined distance can be set to any distance by a developer or the like, but is not limited to this.

[0053] The distance in the feature space may be measured on any scale, for example, Euclidean distance or Mahalanobis distance that takes data variance into account.

[0054] The learning unit 103 retrains the trained model stored in the trained model storage unit 105 using training data including the second detection target extracted by the target extraction unit 119. For example, the target extraction unit 119 can identify an image ID including the second detection target from the information shown in FIG. 6, so the learning unit 103 acquires the training image indicated by this image ID from the training data storage unit 101. The learning unit 103 adds the acquired training image to the dataset and reconstructs the dataset. The learning unit 103 retrains the trained model stored in the trained model storage unit 105 using the reconstructed dataset.

[0055] Note that the retraining method for the trained model may be another method similar to the training method for the trained model. For example, if the trained model has been trained using a supervised learning method such as deep learning (DL), the trained model may be trained using another similar supervised learning method.

[0056] The display control unit 121 displays an image of the feature space in which the target extraction unit 119 plots the multiple feature amounts on the display device 17. Specifically, the display control unit 121 displays, in the feature space, the distribution of the feature amounts of the multiple detection targets, the feature amounts of the first detection target, the feature amounts of the second detection target, and a boundary that identifies whether the multiple detection targets have been detected. The boundary may be represented by a straight line or a curve. Note that when the dimension of the feature amounts is reduced, the boundary is represented by a distorted curve.

[0057] FIG. 8 is a diagram showing an example of an image of a feature space displayed by the display control unit 121 of this embodiment. The example shown in FIG. 8 shows a two-dimensional feature space with the value of node 321 on the x-axis and the value of node 323 on the y-axis. Dots plotted in the feature space represent features. Feature 401 represents the feature of the first detection object, and feature values ​​located within a circle 403 centered on feature 401 represent the feature of the second detection object. Boundary 405 represents a boundary that distinguishes whether the likelihood is less than or equal to a threshold. Boundary 405 represents, for example, a boundary that indicates whether the trained model has been able to detect the detection object in the image by inference. The area in the direction of arrow 407 based on boundary 405 (the area above boundary 405) indicates that the likelihood is greater than or equal to a threshold (for example, that the detection object has been detected in the image). On the other hand, the area in the direction of the arrow 409 with the boundary 405 as the reference (area below the boundary 405) indicates that the likelihood is below the threshold (for example, the detection target was difficult to detect on the image).

[0058] The boundary 405 can be calculated, for example, as follows: Here, the value of node 321 is A, the weight between node 321 and the output layer is w1, the value of node 323 is B, the weight between node 323 and the output layer is w2, the weight between a bias term (not shown) and the output layer is b, and the boundary of the output value of the output layer is c. The value c is a hyperparameter specified by a developer or the like based on experience or evaluation results on other datasets.

[0059] y=Aw1+Bw2+b…(1)

[0060] c=sigmoid(y)=1 / (1+e -y )…(2)

[0061] By transforming the formulas (1) and (2) into the form B=A, the boundary 405 can be obtained.

[0062] From formula (1), B = -Aw1 / w2 - b / w2 + y...(3)

[0063] From equation (2), 1+e -y =1 / c…(4)

[0064] From equation (4), -y=log(1 / c-1)...(5)

[0065] Substituting formula (5) into formula (3), B=-Aw1 / w2-b / w2-log(1 / c-1)…(6)

[0066] Here, if the value c is set to, for example, 0.5, B=-Aw1 / w2-b / w2-log(1 / 0.5-1)…(7)

[0067] In this way, the boundary 405 can be expressed by, for example, the linear function of equation (7).

[0068] The display control unit 121 also displays a list of learning data including the second detection target on the display device 17. In the list of learning data (images), the second detection target is displayed surrounded by a detection frame on the image. The list of learning data (images) also includes false detection images in which objects other than the second detection target are displayed surrounded by a detection frame.

[0069] Fig. 9 is a diagram showing an example of a list screen of images including the second detection target displayed by the display control unit 121 of this embodiment. In the example shown in Fig. 9, the list screen of images including the second detection target is displayed as a list screen of images for relearning. Note that the list screen shown in Fig. 9 is displayed by the display control unit 121 when, for example, a developer or the like uses the input device 19 to perform an operation input or the like to select a feature located within the circle 403.

[0070] In image 421, a pedestrian 423, which is the second detection target, is displayed surrounded by a detection frame 425 with a likelihood. Note that, because the pedestrian 423 has been detected, the feature amount of the pedestrian 423 is located in the area in the direction of the arrow 407. In image 431, a pedestrian 433, which is the second detection target, is displayed without being surrounded by a detection frame. This is because the pedestrian 433 has not been detected, and the feature amount of the pedestrian 433 is located in the area in the direction of the arrow 409. In image 441, an object other than the second detection target is displayed surrounded by a detection frame 443 with a likelihood. This is because an object other than the detection target has been erroneously detected as the detection target, and the feature amount of this object is located in the area in the direction of the arrow 409.

[0071] The display control unit 121 also displays the results of tallying the attributes assigned to the second detection targets. FIG. 10 is a diagram illustrating an example of an attribute tallying result screen displayed by the display control unit 121 of this embodiment. The example illustrated in FIG. 10 shows the results of tallying the attributes assigned to the second detection targets having features located within the circle 403, sorted by attribute type. The number of attributes assigned to the attributes "occlusion" is 10, and the number of attributes "low contrast" is 5. This allows the user to see at a glance which attributes have been assigned to images that have been evaluated as having a high probability of contributing to improving the inference performance of the trained model and have been adopted for re-training. The tallying result screen illustrated in FIG. 10 may be displayed together with the list screen illustrated in FIG. 9 or separately.

[0072] Furthermore, the display control unit 121 can display the distribution of feature amounts by attribute. When displaying a first distribution, which is a distribution of feature amounts of a detection target to which a first attribute has been assigned, the display control unit 121 may identify a second attribute having a second distribution whose distribution of feature amounts is similar to the first distribution, and display the second distribution of the second attribute. Note that the similarity of the distributions can be calculated using a method such as KL divergence, for example.

[0073] FIG. 11 is a diagram showing an example of a distribution screen of feature quantities by attribute displayed by the display control unit 121 of this embodiment. The distribution screen 451 shown in FIG. 11 displays, for example, the distribution of feature quantities for the attribute "concealment." In this case, the display control unit 121 identifies an attribute whose distribution of feature quantities is similar to that of the distribution screen 451, and displays a distribution screen 461 of that distribution. The identified attribute may be, for example, "low contrast," but is not limited to this. Note that the distribution screen can be switched by, for example, a developer or the like using the input device 19 to input an operation to instruct switching of the distribution screen.

[0074] Next, the flow of processing by the learning device of this embodiment will be described.

[0075] FIG. 12 is a flowchart showing an example of the re-learning process performed by the learning device 100 of this embodiment.

[0076] First, the learning unit 103 acquires some of the multiple learning images stored in the learning data storage unit 101 and constructs a data set (step S101).

[0077] Next, the learning unit 103 learns each image included in the constructed dataset to generate a trained model (step S103).

[0078] Next, the inference unit 107 uses the trained model stored in the trained model memory unit 105 to perform inference to detect the target object for each image stored in the training data memory unit 101, and outputs the inference result and the likelihood of the inference (step S105).

[0079] Next, the feature extraction unit 111 uses the trained model stored in the trained model storage unit 105 to extract feature amounts of the multiple detection targets for which inference has been performed by the inference unit 107 (step S107).

[0080] Next, the identifying unit 117 checks whether there is a target for countermeasures among the multiple detection targets based on the likelihood inferred by the inference unit 107 (step S109). Specifically, if there is a detection target whose likelihood is less than a threshold, the identifying unit 117 determines that there is a target for countermeasures (Yes in step S109).

[0081] Next, the identification unit 117 identifies, from among the detection targets whose likelihood is less than the threshold, a detection target that has been assigned an attribute associated with a predetermined priority in the priority information stored in the priority information storage unit 115, as a first detection target that is the target of countermeasures (step S111).

[0082] Next, based on the features extracted by the feature extraction unit 111, the target extraction unit 119 extracts a second detection target from the multiple detection targets detected by inference by the inference unit 107, whose distance in the feature space to the first detection target, which is the target of countermeasures identified by the identification unit 117, is within a predetermined distance (step S113), and then returns to step S101.

[0083] In step S109, if there is no detection target whose likelihood is less than the threshold, the identifying unit 117 determines that there is no target for countermeasures (No in step S109), and the process ends.

[0084] The display process by the display control unit 121 may be performed, for example, every time step S113 is completed, or may be performed after step S109 is determined as No and the process is completed.

[0085] As described above, according to this embodiment, the trained model is retrained using training data including detection targets having features that are close in distance in feature space to detection targets whose likelihood is less than the threshold, which makes it possible to efficiently perform retraining to improve the inference performance of the trained model. As a result, it is expected that detection targets that were difficult to detect because their likelihood was less than the threshold will have a likelihood equal to or greater than the threshold after retraining and become detectable.

[0086] Furthermore, in this embodiment, a feature space is displayed, and for detection targets whose likelihood is less than a threshold, the type of detection target included in the training data extracted for re-training and the attributes of the extracted training data are visually displayed. Therefore, this embodiment makes it possible to objectively grasp which detection targets are included in the training data used for re-training that are likely to contribute to improving the inference performance of the trained model. This makes it possible to provide a well-founded explanation of the training process of the trained model to people who are not familiar with the trained model, such as users of the trained model.

[0087] (program) The programs executed by the learning device of the above embodiments are provided as files in an installable or executable format stored on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD, or flexible disk (FD).

[0088] The program executed by the learning device of the above embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The program executed by the learning device of the above embodiment may be provided or distributed via a network such as the Internet. The program executed by the learning device of the above embodiment may be provided by being pre-installed in a ROM or the like.

[0089] The program executed by the learning device of the above embodiment has a modular configuration for implementing the above-mentioned units on a computer. In terms of actual hardware, for example, the CPU reads the learning program from the HDD onto the RAM and executes it, thereby implementing the above-mentioned units on the computer.

[0090] As described above, according to the above embodiment, it is possible to efficiently perform re-learning to improve the inference performance of a trained model.

[0091] The above-described embodiments merely illustrate examples of specific embodiments for implementing the present disclosure, and the technical scope of the present disclosure should not be construed as being limited by these embodiments. For example, the present disclosure can be implemented in various forms without departing from the gist or main features thereof. For example, the above-described embodiments and each of the above-described modifications may be appropriately combined in their respective constituent units. Furthermore, for example, some components may be deleted from all components in the above-described embodiments and each of the above-described modifications.

[0092] In the above description, the notation "... part" used for each component may be replaced with other notations such as "... assembly," "... circuit," "... device," "... unit," or "... module."

[0093] The present disclosure includes the following aspects.

[0094] (1) an inference unit that uses a trained model to perform inference for detecting a detection target for each training data and outputs the likelihood of the inference; a feature extraction unit that extracts feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; an identification unit that identifies a first detection target from the plurality of detection targets based on the likelihood; an object extraction unit that extracts, from the plurality of detection objects, a second detection object whose distance in a feature amount space to the first detection object is within a predetermined distance based on the feature amount; a learning unit that re-learns the trained model using training data including the second detection target; A learning device comprising:

[0095] (2) The system further includes a display control unit that displays a list of the learning data including the second detection target. The learning device according to (1) above.

[0096] (3) an attribute is assigned to the detection target; the display control unit further displays a result of aggregating the attributes assigned to the second detection targets. The learning device according to (2) above.

[0097] (4) The display control unit displays, in the feature amount space, a distribution of the feature amounts of the plurality of detection targets, the feature amount of the first detection target, the feature amount of the second detection target, and a boundary that identifies whether the plurality of detection targets have been detected. The learning device according to (3) above.

[0098] (5) The display control unit is capable of displaying the distribution of the feature amount in units of the attribute. The learning device according to (4) above.

[0099] (6) When the display control unit displays a first distribution, which is a distribution of the feature amounts of the detection target to which a first attribute has been assigned, the display control unit identifies a second attribute having a second distribution in which the distribution of the feature amounts is similar to the first distribution, and displays the second distribution of the second attribute. The learning device according to (5) above.

[0100] (7) The learning data is an image, In the list, the second detection target is displayed on the image surrounded by a detection frame. The learning device according to (2) above.

[0101] (8) The list includes a false detection image in which an object other than the second detection object is surrounded by a detection frame. The learning device according to (7) above.

[0102] (9) The boundary is represented by a straight line or a curved line. The learning device according to (4) above.

[0103] (10) The feature is a value related to a parameter of the trained model or a value obtained by reducing the number of dimensions of the value. The learning device according to (1) above.

[0104] (11) An attribute is assigned to the detection target, a priority information storage unit that stores priority information in which a priority is associated with the attribute; the identification unit identifies, from the plurality of detection targets, a detection target to which the likelihood is less than a threshold and to which an attribute associated with a predetermined priority among the priorities in the priority information is assigned, as the first detection target; The learning device according to (1) above.

[0105] (12) The inference unit uses the trained model to perform inference for detecting the detection target for each training data, and outputs the likelihood of the inference; a feature extraction unit extracts feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; an identification unit identifying a first detection target from the plurality of detection targets based on the likelihood; an object extraction unit extracts, from the plurality of detection objects, a second detection object whose distance in the feature amount space to the first detection object is within a predetermined distance, based on the feature amount; a learning unit re-learning the learned model using learning data including the second detection target; How to learn.

[0106] (13) The display control unit further displays a list of the learning data including the second detection target. The learning method described in (12) above.

[0107] (14) An attribute is assigned to the detection target, the display control unit further displays a result of aggregating the attributes assigned to the second detection targets. The learning method described in (13) above.

[0108] (15) The display control unit displays, in the feature amount space, a distribution of the feature amounts of the plurality of detection targets, the feature amount of the first detection target, the feature amount of the second detection target, and a boundary that identifies whether the plurality of detection targets have been detected. The learning method described in (14) above.

[0109] (16) The display control unit is capable of displaying the distribution of the feature amount in units of the attribute. The learning method described in (15) above.

[0110] (17) When the display control unit displays a first distribution, which is a distribution of the feature amounts of the detection target to which a first attribute is assigned, the display control unit identifies a second attribute having a second distribution whose distribution of the feature amounts is similar to the first distribution, and displays the second distribution of the second attribute. The learning method according to (16) above.

[0111] (18) The learning data is an image, In the list, the second detection target is displayed on the image surrounded by a detection frame. The learning method described in (13) above.

[0112] (19) performing an inference for detecting a detection target for each training data using the trained model, and outputting the likelihood of the inference; extracting feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; identifying a first detection target from the plurality of detection targets based on the likelihood; extracting, from the plurality of detection targets, a second detection target whose distance in a feature amount space to the first detection target is within a predetermined distance based on the feature amount; Re-learning the trained model using training data including the second detection target; A program that causes a computer to execute the following.

[0113] (20) A trained model generated by the device according to any one of (1) to (11), the method according to any one of (12) to (18), or the program according to (19). [Explanation of symbols]

[0114] 11 Control device 13 Main memory 15 Auxiliary storage 17 Display device 19 Input Devices 21 Communication equipment 23 Various buses 100 Learning Device 101 Learning data storage unit 103 Learning Department 105 Trained model memory 107 Reasoning part 109 Inference result storage unit 111 Feature Extraction Unit 113 Feature memory unit 115 Priority information storage unit 117 Specific section 119 Target Extraction Unit 121 Display control unit

Claims

1. an inference unit that uses the trained model to perform inference for detecting a detection target for each training data and outputs the likelihood of the inference; a feature extraction unit that extracts feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; an identification unit that identifies a first detection target from the plurality of detection targets based on the likelihood; an object extraction unit that extracts, from the plurality of detection objects, a second detection object whose distance in a feature amount space to the first detection object is within a predetermined distance based on the feature amount; a learning unit that re-learns the trained model using training data including the second detection target; A learning device comprising:

2. a display control unit that displays a list of the learning data including the second detection target; The learning device according to claim 1 .

3. An attribute is assigned to the detection target, the display control unit further displays a result of aggregating the attributes assigned to the second detection targets. The learning device according to claim 2 .

4. the display control unit displays, in the feature amount space, a distribution of the feature amounts of the plurality of detection targets, the feature amount of the first detection target, the feature amount of the second detection target, and a boundary that identifies whether the plurality of detection targets have been detected. The learning device according to claim 3 .

5. the display control unit is capable of displaying the distribution of the feature amounts in units of the attributes. The learning device according to claim 4 .

6. when displaying a first distribution which is a distribution of the feature amounts of the detection target to which a first attribute has been assigned, the display control unit identifies a second attribute having a second distribution in which the distribution of the feature amounts is similar to the first distribution, and displays the second distribution of the second attribute. The learning device according to claim 5 .

7. the training data is an image, In the list, the second detection target is displayed on the image surrounded by a detection frame. The learning device according to claim 2 .

8. The list includes a false detection image in which an object other than the second detection object is surrounded by a detection frame. The learning device according to claim 7 .

9. The boundary is represented by a straight line or a curved line. The learning device according to claim 4 .

10. The feature is a value related to a parameter of the trained model or a value obtained by reducing the number of dimensions of the value. The learning device according to claim 1 .

11. An attribute is assigned to the detection target, a priority information storage unit that stores priority information in which a priority is associated with the attribute; the identification unit identifies, from the plurality of detection targets, a detection target to which the likelihood is less than a threshold and to which an attribute associated with a predetermined priority among the priorities in the priority information is assigned, as the first detection target; The learning device according to claim 1 .

12. an inference unit, using the trained model, performing inference for detecting the detection target for each piece of training data, and outputting the likelihood of the inference; a feature extraction unit extracts feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; an identification unit identifying a first detection target from the plurality of detection targets based on the likelihood; an object extraction unit extracts, from the plurality of detection objects, a second detection object whose distance in a feature amount space to the first detection object is within a predetermined distance, based on the feature amount; a learning unit that re-learns the learned model using learning data including the second detection target; How to learn.

13. Furthermore, the display control unit displays a list of the learning data including the second detection target. The learning method according to claim 12.

14. An attribute is assigned to the detection target, the display control unit further displays a result of aggregating the attributes assigned to the second detection targets. The learning method according to claim 13.

15. the display control unit displays, in the feature amount space, a distribution of the feature amounts of the plurality of detection targets, the feature amount of the first detection target, the feature amount of the second detection target, and a boundary that identifies whether the plurality of detection targets have been detected. The learning method according to claim 14.

16. the display control unit is capable of displaying the distribution of the feature amounts in units of the attributes; The learning method according to claim 15.

17. When the display control unit displays a first distribution which is a distribution of the feature amounts of the detection target to which a first attribute has been assigned, the display control unit identifies a second attribute having a second distribution in which the distribution of the feature amounts is similar to the first distribution, and displays the second distribution of the second attribute. The learning method according to claim 16.

18. the training data is an image, In the list, the second detection target is displayed on the image surrounded by a detection frame. The learning method according to claim 13.

19. a step of performing an inference for detecting a detection target for each piece of training data using the trained model and outputting the likelihood of the inference; extracting feature amounts of the plurality of detection targets for which the inference has been performed using the trained model; identifying a first detection target from the plurality of detection targets based on the likelihood; extracting, from the plurality of detection targets, a second detection target whose distance in a feature amount space from the first detection target is within a predetermined distance based on the feature amount; Re-learning the trained model using training data including the second detection target; A program that causes a computer to execute the following.

20. A trained model generated by the device according to any one of claims 1 to 11, the method according to any one of claims 12 to 18, or the program according to claim 19.

Images