Learning Device

Abstract

Provided is a learning recognition device capable of learning such that a fixed recognition performance can be maintained, even if there is bias in the characteristics of each label.　The learning recognition device is provided with: a training data input unit 131 for acquiring teacher data and a plurality of corresponding teacher labels; a control unit 130 for controlling a neural network to calculate an inference error for each teacher label, with respect to the teacher data; and a weighted value assigning unit for calculating a corrected inference error for each teacher label, by correcting the inference errors calculated for each teacher label by means of a weighted value set on the basis of the learning characteristics of the teacher label. The control unit 130 repeatedly causes the neural network to act in such a way that the corrected inference error for each teacher label decreases.

Description

[Technical field]

[0001] The present disclosure relates to a technology for training a neural network and a technology for performing recognition using a neural network, and in particular to a learning technology in the case where multiple teacher labels correspond to one teacher data and a recognition technology in the case where multiple teacher labels correspond to one teacher data. [Background technology]

[0002] When training a neural network, it is common to use teacher data with teacher labels. For example, training data including image data (teacher data) such as photos of cars and the models of cars shown in the image data (teacher labels) is used to train the network in advance, and the learning results are used to recognize labels from new photos.

[0003] According to Patent Document 1, a learning device trains a first supervised learner, which receives features generated from input data having data items of a matched label and a non-matched label and outputs a first estimation result, so that the first estimation result matches the matched label. The learning device trains a second supervised learner, which receives features and outputs a second estimation result, so that the second estimation result matches the non-matched label. The learning device causes a feature extractor, which generates features from input data, to perform learning that promotes recognition of matched labels and suppresses recognition of non-matched labels.

[0004] According to Patent Document 2, the problem to be solved is to perform learning using learning data to which exclusive labels are assigned, and a learning device sets a label vector having one or more labels as elements for each piece of data to be learned. For example, the learning device generates a label vector based on the correlation between the labels to be set for each piece of data to be learned. Then, the learning device performs learning on a learning model including a neural network using the data to be learned and the label vector set for each piece of data to be learned.

[0005] As described above, Patent Documents 1 and 2 disclose learning techniques in which a plurality of teacher labels are associated with one teacher data. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2019-159824 A [Patent Document 2] JP 2019-159612 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, it is difficult to prepare well-balanced teacher data for each label. For example, if vehicle catalog data is used as teacher data, the variation of vehicle models and colors can be increased, but the catalog data is expected to have few images of vehicle driving scenes. On the other hand, images of vehicle driving scenes as teacher data are expected to have few variations of vehicle models and colors.

[0008] If the amount of training data for each label is unbalanced, the level of proficiency for each label will also be unbalanced, resulting in a decrease in recognition performance in the recognition device.

[0009] In addition to the balance in the amount of training data for each label, if there is an imbalance in the probability of occurrence of training data for each label, if there is an imbalance in the difficulty of recognition for each label, or if there is an imbalance in the learning proficiency for each label, these will also result in a decrease in the recognition performance of the recognition device.

[0010] The present disclosure solves such problems and provides a learning device that can learn to maintain a certain level of recognition performance even when there is a bias in the learning characteristics for each label, as described above. Place The purpose is to provide. [Means for solving the problem]

[0011] In order to achieve the above object, one aspect of the present disclosure is a learning device for causing a neural network to learn, the learning device including: an acquisition means for acquiring teacher data and a plurality of corresponding teacher labels; a control means for controlling the neural network to calculate an inference error for each teacher label for the teacher data; and a correction means for correcting the inference error calculated for each teacher label by a weighting value set based on the learning characteristics of the teacher label for each teacher label to calculate a corrected inference error. 、 The control means repeatedly applies the corrected inference error for each teacher label to the neural network so as to reduce the corrected inference error. The acquisition means further acquires missing labels that cannot be recognized from the teacher data in the neural network, and the control means further determines whether any of the multiple teacher labels matches the missing label, and if it is determined that there is a match, suppresses the repeated action of the missing label on the neural network. Effect of the Invention

[0013] According to the above aspect, even when there is a bias in the learning characteristics for each label, as described above, it is possible to learn so as to maintain a certain level of recognition performance, and it is also possible to achieve the excellent effect of maintaining a certain level of recognition performance. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing a configuration of a learning and recognition device 10 according to an embodiment. [Diagram 2] FIG. 1 is a block diagram showing the configuration of a typical neural network 50. [Diagram 3] 1A is a schematic diagram showing one neuron U of the neural network 50. FIG. 1B is a neuron setting table 490 including each neuron weight and neuron threshold value set in the neural network 50. [Figure 4] FIG. 2 is a diagram illustrating a data propagation model during pre-learning (training) in a neural network 50. [Diagram 5]FIG. 2 is a diagram illustrating a data propagation model during field inference in a neural network 50. [Figure 6] 2 shows examples of multiple categories that indicate characteristics of image data 201. [Figure 7] 1 shows an example of training image data stored in the training data DB 151. [Figure 8] 2 shows the data structure of a training data table 230 stored in the training data DB 151. [Figure 9] 2 shows the data structure of a frequency table 250 stored in a parameter DB 152. [Figure 10] 1 shows the data structure of a difficulty level table 260 stored in the parameter DB 152. [Figure 11] 1 shows the data structure of a skill level table 270 stored in the parameter DB 152. [Figure 12] 1 shows the data structure of a missing information table 290 stored in the parameter DB 152. [Figure 13] 1 shows an example of a label weight LW calculated for each label in a category. [Figure 14] 1 shows an example of an inference threshold T calculated for each label in a category. [Figure 15] 2 is a block diagram showing configurations of a training recognition processing unit 124 and an actual recognition processing unit 125. FIG. [Figure 16] FIG. 2 is a block diagram showing a configuration of a feature learning unit 133. [Figure 17] The data structure of the inference result 280 is shown. [Figure 18] The process of multiplying the inference error d by the label weight LW to obtain the corrected inference error d' is shown. [Figure 19] FIG. 2 is a block diagram showing a configuration of an inference unit 143. [Figure 20] The process of obtaining an inference error R' from an inference result R based on an inference threshold is shown below. [Figure 21] An example of the final recognition result 440 is shown. [Figure 22]13 is a flowchart showing the operation of the training recognition processing unit 124. [Diagram 23] 13 is a flowchart showing the operation of the actual recognition processing unit 125. [Figure 24] This shows the pattern of categories that should be set for each inference object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] 1. Embodiment A learning and recognition device 10 according to an embodiment of the present disclosure will be described with reference to the drawings.

[0016] 1.1 Learning Recognition Device 10 1, the learning recognition device 10 is composed of a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random access memory) 103, a storage unit 104, and a network communication circuit 111 connected to a bus B1, a GPU (Graphics Processing Unit) 105, a ROM 106, and a RAM 107 connected to a bus B2, and a GPU 108, a ROM 109, and a RAM 110 connected to a bus B3. The buses B1 and B2 are connected to each other, and the buses B1 and B3 are also connected to each other.

[0017] (CPU 101, ROM 102, RAM 103) The RAM 103 is made up of a non-volatile semiconductor memory, and provides a work area when the CPU 101 executes a program.

[0018] The ROM 102 stores a control program, which is a computer program for causing the learning and recognition device 10 to execute processes.

[0019] The CPU 101 operates according to a control program stored in the ROM 102 .

[0020] The CPU 101 operates according to a control program stored in the ROM 102 using the RAM 103 as a work area, whereby the CPU 101, the ROM 102 and the RAM 103 constitute a main control unit 121.

[0021] The main control unit 121 includes a general control unit 122, a parameter acquisition unit 123, and an inference result output unit 126.

[0022] The overall control unit 122 controls the entire learning recognition device 10 .

[0023] The parameter acquisition unit 123 and the inference result output unit 126 will be described later.

[0024] (Network communication circuit 111) The network communication circuit 111 is connected to an external information terminal via a network, and relays transmission and reception of information between the network communication circuit 111 and the external information terminal via the network.

[0025] (GPU105, ROM106, RAM107) The RAM 107 is made up of a non-volatile semiconductor memory, and provides a work area when the GPU 105 executes a program.

[0026] The ROM 106 stores a control program, which is a computer program for executing processing in the training recognition processing unit 124, which will be described later. A part of this control program constitutes a learning program.

[0027] The GPU 105 operates according to a control program stored in the ROM 106 .

[0028] The GPU 105 operates according to a control program stored in the ROM 106 using the RAM 107 as a work area, whereby the GPU 105, the ROM 106 and the RAM 107 constitute a training recognition processing unit 124.

[0029] The training recognition processor 124 incorporates a neural network.

[0030] The neural network incorporated in the training recognition processing unit 124 performs its functions by the GPU 105 operating in accordance with a control program stored in the ROM 106 .

[0031] (GPU108, ROM109, RAM110) The RAM 110 is made up of non-volatile semiconductor memory, and provides a work area when the GPU 108 executes a program.

[0032] The ROM 109 stores a control program, which is a computer program for executing processing in the on-site recognition processing unit 125 (described later), etc. A part of this control program constitutes a recognition program.

[0033] The GPU 108 operates according to a control program stored in the ROM 109 .

[0034] The GPU 108 operates according to a control program stored in the ROM 109 using the RAM 110 as a work area, whereby the GPU 108 , the ROM 109 and the RAM 110 constitute a real-world recognition processing unit 125 .

[0035] The actual recognition processing unit 125 incorporates a neural network.

[0036] The neural network incorporated in the real-world recognition processing unit 125 performs its functions by the GPU 108 operating in accordance with a control program stored in the ROM 109 .

[0037] (Storage unit 104) The storage unit 104 is, for example, configured from a hard disk, but may also be configured from a non-volatile semiconductor memory.

[0038] As shown in FIG. 1, the storage unit 104 has a training data DB 151, a parameter DB 152, a learning result DB 153, and an actual field data DB 154.

[0039] The training data DB 151, the parameter DB 152, the learning result DB 153, and the field data DB 154 will be described later.

[0040] Here, the main control unit 121, the storage unit 104, the training recognition processing unit 124, and the network communication circuit 111 constitute a learning device. Also, the main control unit 121, the storage unit 104, the field recognition processing unit 125, and the network communication circuit 111 constitute a recognition device.

[0041] 1.2 A typical neural network As an example of a typical neural network, a neural network 50 shown in FIG. 2 will be described.

[0042] (1) Structure of Neural Network 50 As shown in this figure, the neural network 50 is a hierarchical neural network having an input layer 50a, a feature extraction layer 50b, and a recognition layer 50c.

[0043] Here, the neural network is an information processing system that mimics the human neural network. In the neural network 50, the engineered neuron model that corresponds to a nerve cell is called a neuron U here. The input layer 50a, the feature extraction layer 50b, and the recognition layer 50c are each configured with a plurality of neurons U.

[0044] The input layer 50a usually consists of one layer. Each neuron U of the input layer 50a receives, for example, the pixel value of each pixel constituting an image. The received image value is output as is from each neuron U of the input layer 50a to the feature extraction layer 50b.

[0045] The feature extraction layer 50b extracts features from the data received from the input layer 50a (all pixel values ​​constituting one image) and outputs them to the recognition layer 50c. This feature extraction layer 50b extracts, for example, an area showing a vehicle from the received image by calculation in each neuron U.

[0046] The recognition layer 50c performs classification using the features extracted by the feature extraction layer 50b. The recognition layer 50c identifies labels such as the vehicle model, manufacturer name, and color from the vehicle area extracted by the feature extraction layer 50b through calculations in each neuron U.

[0047] As shown in Figure 3(a), a multi-input, single-output element is usually used as neuron U. Signals are transmitted in only one direction, and the input signal xi (i = 1, 2, ..., n) is multiplied by a certain neuron weight (SUwi) and input to neuron U. This neuron weight represents the strength of the connection between neurons U arranged hierarchically. The neuron weight can be changed by learning. Neuron U outputs a value X, which is calculated by subtracting a neuron threshold θU from the sum of each input value (SUwi × xi) multiplied by the neuron weight SUwi, after being transformed by a response function f(X). In other words, the output value y of neuron U is expressed by the following formula.

[0048] y=f(X) Where: X = Σ(SUwi × xi)-θU In addition, Σ indicates the sum for i=1, 2, 3, . . . , n. As the response function, for example, a sigmoid function can be used.

[0049] Each neuron U in the input layer 50a does not usually have a sigmoid characteristic or a neuron threshold. Therefore, the input value appears as it is in the output. On the other hand, each neuron U in the final layer (output layer) of the recognition layer 50c outputs the classification result (data corresponding to the classified label) in the recognition layer 50c.

[0050] As a learning algorithm for the neural network 50, for example, the error backpropagation method is used, in which the neuron weights of the recognition layer 50c and the neuron weights of the feature extraction layer 50b are sequentially changed using the steepest descent method so that the squared error between the value (data) indicating the correct answer and the output value (data) from the recognition layer 50c is minimized.

[0051] (2) Training process The training process in the neural network 50 will now be described.

[0052] The training step is a step of performing pre-learning of the neural network 50. In the training step, pre-learning of the neural network 50 is performed using image data with correct answers (supervised and annotated) that is obtained in advance.

[0053] Figure 4 shows a schematic diagram of the data propagation model during pre-learning.

[0054] Image data is input to the input layer 50a of the neural network 50 for each image, and is output from the input layer 50a to the feature extraction layer 50b. Each neuron U in the feature extraction layer 50b performs a calculation with a neuron weight on the input data. Through this calculation, the feature extraction layer 50b extracts features (e.g., the area of ​​the vehicle) from the input data, and outputs data indicating the extracted features to the recognition layer 50c (step S51).

[0055] Each neuron U in the recognition layer 50c performs a calculation with a neuron weight on the input data (step S52). This allows classification based on the above features (for example, classification of the vehicle type). Data indicating the classification result is output from the recognition layer 50c.

[0056] The output value (data) of the recognition layer 50c is compared with a value indicating the correct answer, and the error (loss) between them is calculated (step S53). To reduce this error, the neuron weights of the recognition layer 50c and the neuron weights of the feature extraction layer 50b are sequentially changed (backpropagation) (step S54). This allows the recognition layer 50c and the feature extraction layer 50b to learn.

[0057] (3) On-site recognition process The on-site recognition process in the neural network 50 will now be described.

[0058] FIG. 5 shows a data propagation model in which neural network 50 learned by the above-mentioned training process is used to actually perform recognition (for example, recognition of the vehicle model) using data obtained in the field as input.

[0059] In the practical recognition process in the neural network 50, feature extraction and recognition are performed using the trained feature extraction layer 50b and the trained recognition layer 50c (step S55).

[0060] 1.3 Examples of image data to be recognized and their categories 6 shows, as an example, image data 201 that is a recognition target handled by the learning recognition device 10. The image data 201 represents a single image, and the image is configured by arranging a plurality of pixels in a matrix. The image represented by the image data 201 includes, as an example, a vehicle object and a background object. Note that the background object is not shown in the figure.

[0061] As shown in this figure, the labels recognized from the image data 201 are classified into categories 211, 212, 213, ..., 218. The categories 211, 212, 213, ..., 218 are "car model", "maker", "color", ..., and "stable driving", respectively.

[0062] Here, the labels 211a, 211b, 211c, and 211d belong to the category 211 of "vehicle type", and the labels 211a, 211b, 211c, and 211d are "passenger car", "truck", "bus", and "two-wheeled vehicle", respectively.

[0063] Moreover, the labels 212a, 212b, 212c, and 212d belong to the "manufacturer" category 212, and the labels 212a, 212b, 212c, and 212d are "Company A", "Company B", "Company C", and "Company D", respectively.

[0064] In addition, categories 213, 214, . . . , 218 each have multiple labels.

[0065] In this embodiment, the image contains vehicle objects, and is classified into the categories shown in FIG. 6, with the labels shown in the same figure belonging to each category.

[0066] 1.4 Training data DB151 The training data DB 151 holds a training data table 230 and a plurality of image data 221, 222, 223, . . .

[0067] (Image data 221, 222, 223, ...) The image data 221, 222, 223, . . . are image data serving as training data (teacher data).

[0068] 7, each of the image data 221, 222, 223, etc. includes, for example, a vehicle object and a background object. Note that the background objects are not shown. Specifically, the image data 221 includes an automobile object, etc., the image data 222 includes a truck object, etc., the image data 223 includes an automobile object, etc., and the image data 224 includes a bus object, etc.

[0069] Moreover, the image data 221, 222, 223, etc. are each identified by an image data identifier. For example, the identifier of the image data 221 is "A001.jpg", the identifier of the image data 222 is "A002.jpg", and the identifier of the image data 223 is "A003.jpg".

[0070] (Training data table 230) The training data table 230 is a data table for indicating preset training labels (teacher labels) for the image data 221, 222, 223, . . . held in the training data DB 151.

[0071] 8, the training data table 230 includes a plurality of training data 231. Each training data 231 corresponds to the image data held in the training data DB 151 on a one-to-one basis.

[0072] Each training data 231 includes an image data identifier 232 and training labels 233, 234, . . . , 239.

[0073] The image data identifier 232 is an identifier for identifying the image data corresponding to the training data 231 .

[0074] The training labels 233, 234, ..., 239 are labels for vehicle objects included in an image represented by image data corresponding to the training data 231. Among the training labels 233, 234, ..., 239, there are some that have no label set (indicated by "-").

[0075] The training label 233 indicates a label for a "vehicle type", which is one of the vehicle categories. The training label 233 for the "vehicle type" includes, for example, a "passenger car", a "truck", a "bus", a "two-wheeled vehicle", and the like.

[0076] The training labels 234 indicate labels for "manufacturer", which is one of the categories for vehicles. The training labels 234 for "manufacturer" include, for example, "Company A", "Company B", "Company C", "Company D", and the like.

[0077] The training labels 235 indicate labels for the "color" of a vehicle body, which is one of the categories for a vehicle. The training labels 235 for "color" include, for example, "white," "black," "blue," and "red."

[0078] The training label 236 indicates a label for "number of passengers", which is one of the categories for a vehicle. The training label 236 for "number of passengers" includes, for example, "one person" and "multiple people".

[0079] The training label 237 indicates a label for "cargo load", which is one of the categories for a vehicle. The training label 237 for "cargo load" includes, for example, "none", "small load", "large load", and the like.

[0080] The training label 238 indicates a label for "distance from another vehicle", which is one of the categories for vehicles. The training label 238 for "distance from another vehicle" includes, for example, "tends to be narrow", "appropriate", etc.

[0081] The training labels 239 indicate labels for "driving speed", which is one of the categories for vehicles. The training labels 239 for "driving speed" include, for example, "too slow", "within the legal speed limit", "too fast", and the like.

[0082] 1.5 Parameter DB152 The parameter DB 152 holds a frequency table 250 , a difficulty table 260 , a mastery table 270 , and a missing information table 290 .

[0083] (1) Frequency Table 250 The frequency table 250 is a data table for storing the frequency (the number of items held) of training data held in the training data DB 151 for each category and label shown in FIG.

[0084] As shown in FIG. 9, the frequency table 250 includes a plurality of frequency information 251 .

[0085] Each piece of frequency information 251 corresponds to a category and a label belonging to that category shown in Fig. 6. Each piece of frequency information 251 includes a category 252, a label 253, and a frequency 254.

[0086] Category 252 corresponds to one category shown in FIG.

[0087] Label 253 corresponds to one label belonging to category 252 .

[0088] The frequency 254 is the frequency of occurrence of the combination of the category 252 and the label 253 in the training data table 230 .

[0089] (2) Difficulty Table 260 The difficulty level table 260 is a data table for storing the difficulty level of the training data held in the training data DB 151 for each category and label shown in FIG.

[0090] The difficulty level table 260 includes a plurality of pieces of difficulty level information 261, as shown in FIG.

[0091] Each piece of difficulty level information 261 corresponds to a category and a label belonging to that category shown in Fig. 6. Each piece of difficulty level information 261 includes a category 262, a label 263, and a difficulty level 264.

[0092] Category 262 corresponds to one category shown in FIG.

[0093] Label 263 corresponds to one label belonging to category 262 .

[0094] The difficulty level 264 is the difficulty level of recognition for the combination of the category 262 and the label 263 .

[0095] For example, the difficulty level of the category "car model" is generally low since the car model can be recognized based on the shape of the car in the image.

[0096] For the category "maker", for example, if the shape of the emblem attached to the vehicle in the image can be clearly recognized, it is possible to recognize the "maker". However, for example, when a moving vehicle is photographed from a distance, from above the vehicle, in bad weather, or at night, it is often difficult to recognize the shape of the emblem. For this reason, the difficulty level for the category "maker" may be set to medium.

[0097] In addition, the difficulty level for the category "color" is generally low since recognition is possible based on the color of the vehicle in the image.

[0098] On the other hand, the label "Lots of wobble" in the category "Stable driving" is generally considered to be of high difficulty because it is obtained by recognizing the wobble of the vehicle from the image. Also, the label "Driving within lane" in the category "Stable driving" is obtained by recognizing whether the vehicle is driving within the lane from the image, so the difficulty is considered to be medium.

[0099] In this way, the difficulty level varies depending on the label. Here, as an example, the difficulty level is classified into three levels: "low", "medium", and "high", and the difficulty levels are set to "3", "2", and "1" for "low", "medium", and "high", respectively.

[0100] Here, the difficulty level 264 for each combination of the category 262 and the label 263 may be set by the user of the learning recognition device 10, for example.

[0101] (3) Proficiency Table 270 The mastery level table 270 is a data table for storing the mastery level of the training data held in the training data DB 151 for each category and label shown in FIG.

[0102] As shown in FIG. 11, the skill table 270 includes a plurality of pieces of skill information 271.

[0103] Each piece of skill level information 271 corresponds to a category and a label belonging to the category shown in Fig. 6. Each piece of skill level information 271 includes a category 272, a label 273, and a skill level 274.

[0104] Category 272 corresponds to one category shown in FIG.

[0105] Label 273 corresponds to one label belonging to category 272 .

[0106] The proficiency 274 is the proficiency of learning about the combination of the category 272 and the label 273 .

[0107] The proficiency level varies depending on the label. As an example, the proficiency level is classified into three levels: "low", "medium", and "high", and the proficiency levels are set to "1", "2", and "3" for "low", "medium", and "high", respectively.

[0108] Here, the level of proficiency 274 for each combination of the category 272 and the label 273 may be set by the user of the learning recognition device 10, for example.

[0109] (4) Missing Information Table 290 Some image data held in the training data DB 151 does not include images for inferring any of the categories shown in FIG.

[0110] For example, when analyzing an image showing the exterior of a stationary vehicle, it is possible to infer the model and color of the vehicle, but it is not possible to obtain information about the vehicle's speed or whether it is driving stably.

[0111] Also, if an image shows the speedometer of a moving vehicle, it is not possible to infer the model or color of the vehicle, but it is possible to obtain information about the speed at which the vehicle is moving.

[0112] In this way, image data held in the training data DB 151 may include image data from which it is impossible to infer that the image data belongs to any one of the multiple categories shown in FIG.

[0113] The missing information table 290 thus indicates the indeterminable category for image data held in the training data DB 151 from which it is impossible to infer that the image data belongs to any of the multiple categories shown in FIG. 6.

[0114] The missing information table 290 includes a plurality of pieces of missing information 291 as shown in FIG.

[0115] As described above, each piece of missing information corresponds to image data stored in training data DB151 from which it is impossible to infer that the image data belongs to any of the multiple categories shown in Figure 6.

[0116] Each piece of missing information 291 includes an image data identifier 292 and a number of categories 293, 294, 295, . . .

[0117] The image data identifier 292 is an identifier for identifying image data stored in the training data DB 151 from which it is impossible to infer that the image data belongs to any of the multiple categories shown in FIG.

[0118] Categories 293, 294, 295, ... indicate categories for which it is impossible to infer that image data belongs to any of the multiple categories shown in Fig. 6. Each piece of missing information 291 includes one or more categories.

[0119] As described above, the missing information table 290 includes missing information on a category basis. However, this is not limiting. The missing information table 290 may include missing information on a label basis within a category.

[0120] For example, if it is possible to infer from image data the label “driving within lane” in the category “stable driving”, but it is impossible to infer from image data the label “frequent wobbling”, then missing information table 290 may include the label “frequent wobbling” in the category “stable driving”.

[0121] Here, each piece of missing information in the missing information table 290 may be set by the user of the learning recognition device 10, for example.

[0122] 1.6 Field Data DB154 The field data DB 154 holds image data 289 as shown in FIG.

[0123] Image data 289 is image data that is to be subjected to recognition processing in the learning recognition device 10 .

[0124] 1.7 Learning result DB153 The learning result DB 153 holds a neuron setting table 490 as shown in FIG.

[0125] As shown in Fig. 3(b), the neuron setting table 490 is composed of a plurality of pieces of neuron information 491. Each piece of neuron information 491 corresponds to each neuron U of the feature extraction unit 132 (Fig. 15) of the training recognition processing unit 124 (described later) and the label inference unit 134 (Fig. 16) of the feature learning unit 133 (described later).

[0126] Each neuron information 491 includes a neuron number 492 , a neuron weight value 493 , and a neuron threshold value 494 .

[0127] The neuron number 492 is a number for identifying each neuron U in the feature extraction unit 132 and the label inference unit 134 .

[0128] Neuron weights 493 and neuron thresholds 494 are the neuron weights and neuron thresholds described above in "1.2 Neural Nets."

[0129] 1.8 Parameter Acquisition Unit 123 The parameter acquiring unit 123 reads frequency information from the frequency table 250, difficulty information from the difficulty table 260, and skill information from the skill table 270 for each of all the labels in all the categories shown in FIG.

[0130] In addition, the parameter acquisition unit 123 reads out missing information (missing label) from the missing information table 290.

[0131] (Calculation of label weight LW) The parameter acquisition unit 123 (calculation means) uses the read frequency information, difficulty information and proficiency information, that is, calculates a label weighting value LW for each label in each category based on the learning characteristics of the teacher label.

[0132] For example, the parameter acquiring unit 123 may calculate a lower label weighting value LW if the frequency is high, and may calculate a higher label weighting value LW if the frequency is low. Also, the parameter acquiring unit 123 may calculate a lower label weighting value LW if the difficulty level is low, and may calculate a higher label weighting value LW if the difficulty level is high. Furthermore, the parameter acquiring unit 123 may calculate a lower label weighting value LW if the proficiency level is high, and may calculate a higher label weighting value LW if the proficiency level is low.

[0133] Also, for example, the parameter acquiring unit 123 may calculate the label weight LW by the following formula.

[0134] Label weight LW = G(α × (frequency) + β × (difficulty) + γ × (proficiency)) Here, G(x) is, for example, G(x)=1 / x, and α, β, and γ are coefficients set according to frequency, difficulty, and proficiency, respectively. α, β, and γ are set so that 0<label weight value LW<1.

[0135] FIG. 13 shows an example of the label weights LW calculated for each category and each label.

[0136] As shown in this figure, label weights LW311a, 312a, 313a, ..., 314a are calculated for label 311 "passenger car", label 312 "truck", label 313 "bus", ..., label 314 "two-wheeled vehicle" in category 301 "vehicle type".

[0137] In addition, for the label 331 “black”, the label 332 “white”, the label 333 “blue”, . . . , and the label 334 “pink” in the category 321 “color”, the label weights LW331a, 332a, 333a, . . . , and 334a are calculated.

[0138] In addition, the parameter acquiring unit 123 may use only the frequency information, may use only the difficulty information, or may use only the proficiency information in calculating the label weight value LW. In addition, the parameter acquiring unit 123 may use a combination of any two of the frequency information, the difficulty information, and the proficiency information in calculating the label weight value LW.

[0139] The parameter acquisition unit 123 outputs the label weighting value LW calculated for each category and each label to a feature learning unit 133 of the training recognition processing unit 124, which will be described later.

[0140] (Calculation of inference threshold) The parameter acquisition unit 123 (threshold acquisition means) uses the read frequency information, difficulty information and proficiency information, that is, based on the learning characteristics of the teacher label, to calculate an inference threshold T for each label in each category.

[0141] For example, if the frequency is high, the parameter acquiring unit 123 may calculate a lower inference threshold T, and if the frequency is low, the parameter acquiring unit 123 may calculate a higher inference threshold T. Also, if the difficulty level is low, the parameter acquiring unit 123 may calculate a lower inference threshold T, and if the difficulty level is high, the parameter acquiring unit 123 may calculate a lower inference threshold T, and if the proficiency level is low, the parameter acquiring unit 123 may calculate a higher inference threshold T. Furthermore, if the proficiency level is high, the parameter acquiring unit 123 may calculate a lower inference threshold T, and if the proficiency level is low, the parameter acquiring unit 123 may calculate a higher inference threshold T.

[0142] Also, for example, the parameter acquiring unit 123 may calculate the inference threshold T by the following formula.

[0143] Inference threshold T = G(δ × (frequency) + ε × (difficulty) + ζ × (proficiency)) Here, G(x) is, for example, G(x)=1 / x as described above, and δ, ε, and ζ are coefficients set according to frequency, difficulty, and proficiency, respectively. δ, ε, and ζ are set so that 0<inference threshold T<1.

[0144] FIG. 14 shows an example of the inference threshold T calculated for each category and each label.

[0145] As shown in this figure, inference thresholds T371a, 372a, 373a, ..., 374a are calculated for label 371 "passenger car", label 372 "truck", label 373 "bus", ..., label 374 "two-wheeled vehicle" in category 361 "vehicle type".

[0146] Furthermore, inference thresholds T391a, 392a, 393a, . . . , 394a are calculated for label 391 “black”, label 392 “white”, label 393 “blue”, . . . , label 394 “pink” in category 381 “color”.

[0147] As shown in category 361 in Fig. 14, the inference threshold T may be different for each label in category 361. Also, the inference threshold T may be the same for all labels in a category. Furthermore, the inference threshold T may be the same for all labels in the same category, and the inference threshold T may be different for each category.

[0148] The parameter acquisition unit 123 outputs the inference threshold T calculated for each category and each label to an inference unit 143 (to be described later) of the actual recognition processing unit 125.

[0149] 1.9 Training Recognition Processor 124 As shown in FIG. 15, the training recognition processing unit 124 is made up of a control unit 130, a training data input unit 131, a feature extraction unit 132, and a feature learning unit 133.

[0150] (1) Training Data Input Unit 131 The training data input unit 131 (acquisition means) corresponds to the input layer 50a of the neural network 50 shown in Fig. 2. The training data input unit 131 reads out one image data stored in the training data DB 151. In addition, the training data corresponding to the read image data is read out from the training data table 230.

[0151] The training data input unit 131 outputs the read image data to the feature extraction unit 132. In addition, the training data input unit 131 outputs the read training data to the feature learning unit 133.

[0152] (2) Feature extraction unit 132 The feature extraction unit 132 corresponds to the feature extraction layer 50b of the neural network 50 shown in FIG. 2. The feature extraction unit 132 receives image data from the training data input unit 131. The feature extraction unit 132 extracts features from the received image data. For example, the feature extraction unit 132 extracts an area in which a vehicle is shown as an object from within an image constituting the received image data. The feature extraction unit 132 outputs the extracted features to the feature learning unit 133. For example, the feature extraction unit 132 outputs the area in which a vehicle is shown to the feature learning unit 133.

[0153] (3) Feature learning unit 133 As shown in FIG. 16, the feature learning unit 133 is made up of a label inference unit 134, a weighting unit 135, and an inference error transfer unit 136.

[0154] (a) Label inference unit 134 The label inference unit 134 includes a function corresponding to the recognition layer 50c of the neural network 50 shown in Fig. 2. The label inference unit 134 receives, for example, an area in which a vehicle is shown from the feature extraction unit 132. Upon receiving the area in which a vehicle is shown, the label inference unit 134 calculates the occurrence probability for all categories and labels shown in Fig. 6 as an inference result from the area in which the vehicle is shown.

[0155] 17 shows an inference result 280 as an example of an inference result generated by the label inference unit 134. As shown in this figure, the inference result 280 includes multiple pieces of inference result information 281. Each piece of inference result information 281 corresponds to the category shown in Fig. 6. Each piece of inference result information 281 includes a category 282 and one or more pieces of label information 283.

[0156] Here, category 282 is the category shown in FIG.

[0157] Label information 283 included in each piece of inference result information 281 corresponds to a label belonging to one category shown in Fig. 6. The label information 283 includes a label 284 and an occurrence probability 285. The label 284 is a label belonging to one category, and the occurrence probability 285 is an occurrence probability inferred for the label 284.

[0158] Next, the label inference unit 134 receives the training data from the training data input unit 131, and calculates an inference error d between the calculated occurrence probability and the label included in the training data for all categories and labels. The label inference unit 134 outputs the calculated inference error d to the weighting unit 135.

[0159] (b) Weighting unit 135 The weighting unit 135 (correction means) receives the inference error d from the label inference unit 134. In addition, the weighting unit 135 receives the label weight LW from the parameter acquisition unit 123.

[0160] Next, the weighting unit 135 multiplies the inference error d by the label weighting LW to calculate a corrected inference error d'.

[0161] Corrected inference error d' = inference error d × label weight LW In this way, the weighting unit 135 corrects the inference error calculated for each teacher label using the label weighting value to calculate a corrected inference error.

[0162] As described above, for example, if the frequency is high, the parameter acquiring unit 123 calculates a lower label weighting value LW, and if the frequency is low, the parameter acquiring unit 123 calculates a higher label weighting value LW. Also, if the degree of difficulty is low, the parameter acquiring unit 123 calculates a lower label weighting value LW, and if the degree of difficulty is high, the parameter acquiring unit 123 calculates a lower label weighting value LW, and if the degree of proficiency is low, the parameter acquiring unit 123 calculates a higher label weighting value LW.

[0163] Therefore, for example, if the frequency is high, the weighting unit 135 calculates a smaller corrected inference error, and if the frequency is low, the weighting unit 135 calculates a larger corrected inference error. Therefore, if the frequency is high, the corrected inference error converges with fewer backpropagation iterations. On the other hand, if the frequency is low, the corrected inference error converges with more backpropagation iterations. In this way, the learning degree in the neural network can be maintained constant according to the frequency of the label, and as a result, the recognition performance can be maintained constant.

[0164] Furthermore, the weighting unit 135 calculates a smaller corrected inference error if the level of difficulty is low, and calculates a larger corrected inference error if the level of difficulty is high. Furthermore, the weighting unit 135 calculates a smaller corrected inference error if the level of proficiency is high, and calculates a larger corrected inference error if the level of proficiency is low. In these cases, as in the case of the frequency of labels, the learning degree in the neural network can be maintained constant according to the level of difficulty or proficiency of the labels, and as a result, the recognition performance can be maintained constant.

[0165] In this way, by correcting the inference error according to the characteristics of the labels (frequency, difficulty, proficiency, etc.), it is possible to learn to maintain a certain level of recognition performance even when there is a bias in the characteristics of each label.

[0166] 18, for one label 351d, the inference error d (351a) is multiplied by the label weighting value LW (351b) to obtain a corrected inference error d' (351c). Similarly, for the other labels 352d, 353d, ..., 354d, the inference errors d are multiplied by the label weighting value LW to obtain corrected inference errors d' (352c), (353c), ..., (354c), respectively.

[0167] Next, the weighting unit 135 outputs the corrected inference error d′ calculated for each category and label to the inference error transfer unit 136 .

[0168] (c) Inference error transfer unit 136 The inference error transfer unit 136 receives the corrected inference error d′ from the weighting unit 135 .

[0169] The inference error transfer unit 136 outputs the received corrected inference error d′ to the feature extraction unit 132 and the label inference unit 134 .

[0170] The inference error transfer unit 136 repeatedly outputs the received corrected inference error d' to the feature extraction unit 132 and the label inference unit 134 until the received corrected inference error d' becomes smaller than a predetermined convergence threshold.

[0171] (d) Repeated correction of neuron weights and inference As described above, the inference error and the corrected inference error are calculated for each label, and the neuron weights and the neuron threshold are corrected and the inference is repeated for all labels until the corrected inference error is smaller than a predetermined convergence threshold. The predetermined convergence threshold is a small value, preferably zero. That is, the neuron weights and the neuron threshold are corrected and the inference is repeated until the corrected inference error becomes zero. In other words, the neuron weights and the neuron threshold are corrected and the inference is repeated so that the corrected inference error becomes smaller.

[0172] The feature learning unit 133 compares the corrected inference error d' with a predetermined convergence threshold for each label, and determines whether the corrected inference error d' is greater than the predetermined convergence threshold.

[0173] If it is determined that the corrected inference error d' is greater than the predetermined convergence threshold, the label inference unit 134 corrects the neuron weights and the neuron threshold. Next, the feature learning unit 133 repeats the process of inferring the labels and assigning the label weights.

[0174] On the other hand, if it is determined that the corrected inference error d' is not greater than the predetermined convergence threshold, feature learning unit 133 ends the process.

[0175] For example, as shown in Figure 6, assume that there are multiple labels in each of the eight categories, totaling 40 labels. In this case, learning is repeated so that the corrected inference error d' becomes small for all 40 labels.

[0176] The speed of convergence of learning differs for each label. For example, assume that there are four labels in the category "vehicle type" - "passenger car," "truck," "bus," and "two-wheeled vehicle" - and the correct label is "passenger car." According to the inference result shown in FIG. 17, the occurrence probability of the correct answer, "passenger car," is "0.8," so the inference error is 1-0.8=0.2. On the other hand, "two-wheeled vehicle" is an incorrect answer, so the inference error is 0-0.05=-0.05. Learning is repeated for each label until the inference error becomes zero (as close to zero as possible).

[0177] (4) Control unit 130 The control unit 130 (control means) collectively controls the training data input unit 131, the feature extraction unit 132, and the feature learning unit 133. The control unit 130 also controls the training data input unit 131, the feature extraction unit 132, and the feature learning unit 133, respectively.

[0178] In addition, the control unit 130 controls the feature learning unit 133, which is a part of the neural network, to calculate an inference error for each teacher label.

[0179] Furthermore, the control unit 130 repeatedly operates the neural network so that the corrected inference error for each teacher label becomes smaller than a predetermined threshold value.

[0180] In addition, the control unit 130 judges whether any of the multiple teacher labels matches the missing information (missing label) read from the missing information table 290. If it is judged that there is a match, the control unit 130 suppresses the iterative action on the neural network.

[0181] 1.10 Handling Missing Information The process for the missing information included in the missing information table 290 shown in FIG. 12 will be described.

[0182] For example, when the image data identifier is "A001.jpg" among the missing information included in the missing information table 290, the missing category includes "stable driving." On the other hand, the category "car model" and the category "maker" are not included.

[0183] As mentioned above, the inference error (corrected inference error) is calculated for each label, so the inference error is calculated for the four labels in the category "Vehicle type": "Car", "Truck", "Bus", and "Motorcycle". In addition, the inference error is calculated for the four labels in the category "Manufacturer": "Company A", "Company B", "Company C", and "Company D".

[0184] On the other hand, if the category “stable driving” contains two labels, “frequent wobbling” and “driving within lane”, and these two labels are missing information, the feature learning unit 133 stops propagating inference errors for these two labels and does not propagate them.

[0185] In this way, when the two labels "many wobble" and "driving within lane" are missing information in the category "stable driving", the category "stable driving" ends up being missing information.

[0186] In addition, if the amount of training data is unbalanced, a weighted value for processing missing data can be assigned to each label. For example, if there are many labels "white" for the category "color" in the training data, the propagation of inference errors may be stopped and not propagated for one image data out of, for example, 10 images of multiple image data showing a vehicle body whose color is "white."

[0187] 1.11 Field Recognition Processing Unit 125 As shown in FIG. 15, the site recognition processing unit 125 includes a control unit 140, a site data input unit 141, a feature extraction unit 142, and an inference unit 143.

[0188] (1) Field Data Entry Section 141 The field data input unit 141 (acquisition means) corresponds to the input layer 50a of the neural network 50 shown in Fig. 2. The field data input unit 141 reads out image data 289 stored in the field data DB 154. The field data input unit 141 outputs the read out image data 289 to the feature extraction unit 142.

[0189] (2) Feature extraction unit 142 The feature extraction unit 142 corresponds to the feature extraction layer 50b of the neural network 50 shown in Fig. 2. Prior to extracting features, the feature extraction unit 142 acquires a neuron weight value and a neuron threshold value corresponding to the feature extraction unit 142 from the neuron setting table 490 held in the learning result DB 153, and sets the acquired neuron weight value and neuron threshold value therein.

[0190] Furthermore, the feature extraction unit 142 receives image data from the field data input unit 141. The feature extraction unit 142 extracts features from the received image data. For example, the feature extraction unit 142 extracts an area in which a vehicle is shown as an object from within an image constituting the received image data. The feature extraction unit 142 outputs the extracted features to the inference unit 143. For example, the feature extraction unit 142 outputs an area in which a vehicle is shown to the inference unit 143.

[0191] (3) Reasoning section 143 As shown in FIG. 19, the inference unit 143 is made up of a label inference unit 144, an inference result adjustment unit 145, and an inference result transfer unit 146.

[0192] (a) Label inference unit 144 The label inference unit 144 includes a function corresponding to the recognition layer 50c of the neural network 50 shown in FIG.

[0193] Prior to inferring a label, the label inference unit 144 obtains a neuron weight value and a neuron threshold value corresponding to the label inference unit 144 from the neuron setting table 490 stored in the learning result DB 153, and sets the obtained neuron weight value and neuron threshold value internally.

[0194] Furthermore, the label inference unit 144 receives, for example, an area in which a vehicle is shown from the feature extraction unit 142. Upon receiving the area in which a vehicle is shown, the label inference unit 144 calculates occurrence probabilities for all categories and labels as an inference result R from the area in which the vehicle is shown.

[0195] An example of an inference result R is shown in Fig. 20. As shown in this figure, the inference result R includes occurrence probabilities 402a, 403a, 404a, ... for all labels 402, 403, 404, ... in a category 401, occurrence probabilities 412a, 413a, 414a, ... for all labels 412, 413, 414, ... in a category 411, and occurrence probabilities 422a, ... for all labels 422, ... in a category 421.

[0196] The label inference unit 144 outputs the inference result R to the inference result adjustment unit 145 .

[0197] (b) Inference result adjustment unit 145 The inference result adjustment unit 145 receives the inference result R from the label inference unit 144. In addition, the inference result adjustment unit 145 receives the inference threshold T for all labels in all categories from the parameter acquisition unit 123.

[0198] The inference result adjustment unit 145 (judgment means) compares the occurrence probability in the inference result R with the inference threshold T for each label of each category.

[0199] If the occurrence probability is smaller than the inference threshold T, the inference result adjustment unit 145 rejects this occurrence probability. For example, in Fig. 20, for label 402 of category 401, occurrence probability 402a is smaller than inference threshold Ta 405, so it is rejected. Also, for label 404 of category 401, occurrence probability 404a is smaller than inference threshold Tb 415, so it is rejected.

[0200] On the other hand, if the occurrence probability is equal to the inference threshold T or is greater than the inference threshold T, the inference result adjustment unit 145 adopts this occurrence probability. For example, in Fig. 20, for the label 403 of the category 401, the occurrence probability 403a is greater than the inference threshold Ta 405, and is therefore adopted.

[0201] As described above, for example, if the frequency of a label is high, the parameter acquiring unit 123 calculates a lower inference threshold T, and if the frequency is low, the parameter acquiring unit 123 calculates a higher inference threshold T. Therefore, for a label with a high frequency, the occurrence probability is more likely to be adopted even when compared with a lower calculated inference threshold. On the other hand, for a label with a low frequency, the occurrence probability is more likely to be rejected when compared with a lower calculated inference threshold.

[0202] Furthermore, if the difficulty level is low, the parameter acquiring unit 123 calculates a lower inference threshold T, and if the difficulty level is high, the parameter acquiring unit 123 calculates a higher inference threshold T. Furthermore, if the proficiency level is high, the parameter acquiring unit 123 calculates a lower inference threshold T, and if the proficiency level is low, the parameter acquiring unit 123 calculates a higher inference threshold T. In these cases, the same applies as in the case of frequency.

[0203] In this way, the inference result adjustment unit 145 determines, for each category, the occurrence probability to be adopted and the occurrence probability to be rejected, and adopts the label corresponding to the occurrence probability to be adopted.

[0204] In this way, by correcting the inference threshold according to the characteristics of the labels (frequency, difficulty, proficiency, etc.), a certain level of recognition performance can be maintained even when there is a bias in the characteristics of each label.

[0205] In addition, when all occurrence probabilities in a category are rejected, the inference result adjustment unit 145 determines that the category is unrecognizable. For example, in FIG. 20, the occurrence probabilities 412a, 413a, 414a, ... of all the labels in the category 411 are smaller than the inference threshold Tb 415, so they are rejected.

[0206] Here, when there are multiple occurrence probabilities to be adopted for each category, the inference result adjustment unit 145 adopts the highest occurrence probability and discards the other occurrence probabilities. In this way, the inference result adjustment unit 145 adopts the label corresponding to the highest occurrence probability.

[0207] Finally, the inference result adjustment unit 145 determines one label for each of all categories, or determines that the category is unrecognizable. For example, in FIG. 20, for the label 403 of the category 401, the occurrence probability 403a is adopted. Also, for the category 411, it is determined that it is unrecognizable (unknown label) 416. Also, for the label 422 of the category 421, the occurrence probability 422a is adopted.

[0208] In this manner, the inference result adjustment unit 145 generates an inference result R' for each of all categories, which includes either one determined label or information indicating that the category is unrecognizable.

[0209] 21 shows a final recognition result 440 (inference result R′) generated by the inference result adjustment unit 145. As shown in this figure, the final recognition result 440 includes multiple pieces of recognition information 441, and each piece of recognition information 441 is composed of a category 442 and a label 443.

[0210] The categories included in the plurality of pieces of recognition information 441 correspond to the categories shown in Fig. 6. The labels included in the plurality of pieces of recognition information 441 each match the labels in the categories shown in Fig. 6 or include information 444 indicating unknown (unrecognizable).

[0211] The inference result adjustment unit 145 outputs the inference result R′ thus generated to the inference result transfer unit 146 .

[0212] (c) Inference result transfer unit 146 The inference result transfer unit 146 receives the inference result R′ from the inference result adjustment unit 145. Upon receiving the inference result R′, the inference result transfer unit 146 outputs the received inference result R′ to the inference result output unit 126.

[0213] (4) Inference result output unit 126 The inference result output unit 126 receives the inference result R' from the inference result transfer unit 146. Upon receiving the inference result R', the inference result output unit 126 transmits the received inference result R' to an external information terminal via the network communication circuit 111.

[0214] (5) Control unit 140 The control unit 140 (control means) integrally controls the field data input unit 141, the feature extraction unit 142, and the inference unit 143. The control unit 140 also controls the field data input unit 141, the feature extraction unit 142, and the inference unit 143, respectively.

[0215] Furthermore, the control unit 140 controls the neural network so as to calculate an inference result for each label for the image data read by the field data input unit 141.

[0216] 1.12 Operation of the Learning Recognizer 10 (1) Operation in the training recognition processing unit 124 The operation of the training recognition processing unit 124 will be described with reference to the flowchart shown in FIG.

[0217] Although a flowchart usually describes a procedure in chronological order, due to the nature of neural networks, the procedure in the flowchart shown below may not necessarily follow a chronological order. In particular, the repeated processing for each training label in the following steps S104 to S113 functionally indicates that processing for each training label is performed, and the processing does not necessarily proceed in chronological order, so care should be taken. The same applies to the flowchart shown in FIG. 23.

[0218] The control unit 140 repeats steps S102 to S113 for each image data held in the training data DB 151 (steps S101 to S114).

[0219] Next, each of steps S102 to S113 will be described.

[0220] The training data input unit 131 reads out one image data from the training data DB 151 (step S102).

[0221] The feature extraction unit 132 extracts features from the image data (step S103).

[0222] The feature learning unit 133 repeats steps S105 to S111 for each training label included in the training data corresponding to the image data in the training data table 230 (steps S104 to S113).

[0223] Next, each of steps S105 to S111 will be described.

[0224] The label inference unit 134 infers a label (step S105).

[0225] Next, the label inference unit 134 calculates the inference error d (step S106).

[0226] Next, the parameter acquiring unit 123 calculates a label weighting value LW for each training label (step S107).

[0227] Next, the weighting unit 135 multiplies the inference error d by the label weighting LW to calculate a corrected inference error d'.

[0228] Corrected inference error d' = inference error d × label weight LW (step S108) The feature learning unit 133 refers to the missing information table 290 (step S109) and determines whether the label is missing information (step S110). If it is determined to be missing information ("YES" in step S110), the feature learning unit 133 transfers control to step S113.

[0229] On the other hand, if it is determined that the information is not missing ("NO" in step S110), the feature learning unit 133 compares the corrected inference error d' with a predetermined convergence threshold and determines whether the corrected inference error d' is greater than the predetermined convergence threshold (step S111).

[0230] If it is determined that the corrected inference error d' is greater than the predetermined convergence threshold (step S111: ">"), the label inference unit 134 corrects the neuron weight and the neuron threshold (step S112). Next, the feature learning unit 133 transfers control to step S105 and repeats the process.

[0231] On the other hand, if it is determined that the corrected inference error d' is not greater than the predetermined convergence threshold ("≦" in step S111), feature learning unit 133 transfers control to step S113.

[0232] In this way, the inference error and the corrected inference error are calculated for each label, and the neuron weights and neuron thresholds are corrected and the inference is repeated for all labels so that the corrected inference error is smaller than a predetermined convergence threshold.

[0233] (2) Operation in the actual recognition processing unit 125 The operation of the actual recognition processing unit 125 will be described with reference to the flowchart shown in FIG.

[0234] The field data input unit 141 reads out the image data 289 stored in the field data DB 154. The feature extraction unit 142 extracts a feature amount from the read out image data (step S131).

[0235] The control unit 140 repeats steps S133 to S140 for each of all the categories shown in FIG. 6 (steps S132 to S141).

[0236] Next, each of steps S133 to S140 will be described.

[0237] The label inference unit 144 infers a label within the category (step S133).

[0238] The inference unit 143 repeats steps S135 to S138 for each label (steps S134 to S139).

[0239] Next, each of steps S135 to S138 will be described.

[0240] The parameter acquiring unit 123 calculates the inference threshold T (step S135).

[0241] The inference result adjustment unit 145 compares the occurrence probability in the inference result R with the inference threshold T for each label of each category (step S136).

[0242] If the inference result is equal to the inference threshold T or is greater than the inference threshold T (step S136: "≧"), the inference result adjustment unit 145 adopts the occurrence probability of the inference result as the label of the category (step S137).

[0243] On the other hand, if the inference result is smaller than the inference threshold T ("<" in step S136), the inference result adjustment unit 145 rejects the occurrence probability of the inference result (step S138).

[0244] When the repetition for each label is completed (step S139), the inference result adjustment unit 145 determines one label for one category, or determines that the category is unrecognizable (step S140).

[0245] When the repetition for each category is completed (step S141), the inference result output unit 126 outputs the inference result R' (step S142).

[0246] 1.13 The pattern of categories to be established for each object of reasoning In the above description, a "vehicle" object in an image is used as the subject of inference in the learning recognition device 10.

[0247] The object of inference in the learning recognition device 10 is not limited to a "vehicle" in an image. As shown in Fig. 24, it may be, for example, "food," "industrial products," "plants," "marine products," "animals," "people," etc.

[0248] Furthermore, labels for "food" can be classified into categories such as "type," "color," "size," "quality level," and "freshness." Labels for "industrial products" can be classified into categories such as "type," "color," "shape," "presence of blemishes," and "stage of production process." Labels for "plants," "seafood products," "animals," "people," and the like can be classified into the categories shown in FIG. 24.

[0249] In this way, learning and recognition in the learning recognition device 10 can be performed on these objects as well.

[0250] 1.14 Summary The training recognition processing unit 124 can perform learning so as to maintain a certain level of recognition performance even when there is a bias in the learning characteristics for each label.

[0251] Furthermore, the on-the-spot recognition processing unit 125 can maintain a certain level of recognition performance even when there is a bias in the learning characteristics for each label.

[0252] 2. Other Modifications Although the present invention has been described based on the above embodiment, the present invention is not limited to the above embodiment. The following may be adopted.

[0253] (1) According to the above embodiment, the learning recognition device 10 is composed of a single device equipped with the training recognition processing unit 124 and the field recognition processing unit 125.

[0254] However, the present invention is not limited to this embodiment.

[0255] One aspect of the present disclosure may be a learning and recognition system that is configured not with a single learning and recognition device 10 but with a learning device having a training recognition processing unit 124 and a recognition device having an on-the-job recognition processing unit 125. The learning device and the recognition device are connected via a network.

[0256] Specifically, the learning device includes a CPU 101, a ROM 102, a RAM 103, a storage unit 104, and a network communication circuit 111 connected to a bus B1 shown in Fig. 1, and a GPU 105, a ROM 106, and a RAM 107 connected to a bus B2. The buses B1 and B2 are connected to each other.

[0257] The recognition device is also composed of a CPU 101, a ROM 102, a RAM 103, a storage unit 104, and a network communication circuit 111 connected to a bus B1 shown in Fig. 1, and a GPU 108, a ROM 109, and a RAM 110 connected to a bus B3. The bus B1 and the bus B3 are connected to each other.

[0258] The learning device transmits the contents of the learning result DB to the recognition device via the network. The recognition device receives the contents of the learning result DB and trains the on-site recognition processing unit based on the received contents.

[0259] (2) The subject of learning and recognition in the learning and recognition device 10 of the above embodiment is image data.

[0260] However, the present invention is not limited to this embodiment.

[0261] (a) Learning The target of learning and recognition in the recognition device 10 may be voice data. In this case, examples of categories may be "music," "human voice," "natural sounds," and so on.

[0262] Examples of labels for the category "music" are "classical music," "ethnic music," "pop," "Latin music," etc. Examples of labels for the category "human voices" are "news audio," "lecture audio," "conversation audio," etc. Examples of labels for the category "sounds of nature" are "birds singing," "wind sounds," "flowing river sounds," etc.

[0263] The voice data may also be nursing records, nursing records, etc. The voice data may consist of, for example, the patient's own voice talking about his / her symptoms. A specific example of the voice data is, "My runny nose and cough won't stop since this morning, and my appetite is okay, but I haven't had a bowel movement. I haven't taken my temperature, but I don't think I have a fever." Categories "symptoms," "appetite," "excretion," etc. are inferred from the voice data. Example labels for the category "symptoms" are "fever," "cough," "sore throat," etc. Example labels for the category "appetite" are "yes," "moderate," "no," etc. Example labels for the category "excretion" are "yes," "moderate," "no," etc.

[0264] (b) The subject of learning and recognition in the learning recognition device 10 may be character data in natural language processing. In this case, examples of categories may be "conversation", "literature", "newspaper", "paper", etc. Examples of labels for the category "conversation" are "Japanese", "English", "Italian", etc. Examples of labels for the category "literature" are "contemporary Japanese literature", "modern Japanese literature", "medieval Japanese literature", "ancient Japanese literature", etc. Examples of labels for the category "newspaper" are "political news", "economic news", "science news", etc.

[0265] The character data may be character data such as nursing records or nursing records. For example, the character data is a written record of a patient's symptoms recorded by a nurse. A specific example of character data is "Runny nose and cough won't stop since this morning. Appetite is okay. No bowel movement. Temperature not taken, but no fever." Categories such as "symptoms," "appetite," and "excretion" are inferred from the character data. Examples of labels for each category are as above.

[0266] (3) In the embodiment, the parameter acquisition unit 123 uses frequency information, difficulty information, and proficiency information, that is, calculates a label weighting value for each label in each category based on the learning characteristics of the teacher label.

[0267] However, this is not limited to this. A person may determine the label weighting value for each label in each category using the frequency information, the difficulty information, and the proficiency information. For example, an information processing device (specifically, a personal computer, etc.) accepts an input of a label weighting value for each label in each category from its user. The information processing device transmits the inputted label weighting value to the learning recognition device 10. The parameter acquisition unit 123 of the learning recognition device 10 receives the label weighting value, and outputs the received label weighting value to the weighting value assignment unit 135 of the feature learning unit 133.

[0268] (4) In the embodiment, the parameter acquisition unit 123 uses frequency information, difficulty information, and proficiency information, that is, calculates an inference threshold for each label in each category based on the learning characteristics of the teacher label.

[0269] However, this is not limited to this. A person may determine an inference threshold for each label in each category using frequency information, difficulty information, and proficiency information. For example, an information processing device (specifically, a personal computer, etc.) accepts an input of an inference threshold for each label in each category from its user. The information processing device transmits the inference threshold that has been input to the learning recognition device 10. The parameter acquisition unit 123 (threshold acquisition means) of the learning recognition device 10 receives (acquires) the inference threshold, and outputs the received inference threshold to the inference unit 143 of the field recognition processing unit 125.

[0270] In addition, the inference threshold may not only be calculated based on the learning characteristics, but may also be set manually based on prior information as described above, or the parameter acquisition unit 123 (threshold acquisition means) may dynamically generate (calculate) and set the inference threshold based on some fluctuating external factor, as shown below.

[0271] After the operation of the learning recognition device 10, that is, the operation of the on-site recognition processing unit 125, has started, there are the following patterns for generating an inference threshold value.

[0272] Here, for example, it is assumed that multiple cameras are installed in a public facility such as a baseball field. The multiple cameras each capture the ground, the spectator seats, the entrance to the baseball field, the surroundings of the baseball field, etc., to obtain image data representing the images (still images or video). The large number of image data thus obtained is stored in the field data DB 154 of the learning recognition device 10. The field recognition processing unit 125 of the learning recognition device 10 performs recognition processing on the large number of image data stored in the field data DB 154. It is to be noted that the public facility is not limited to a baseball field. It may also be applied to stadiums where other sports are played, theaters where plays are performed, movie theaters where movies are screened, etc.

[0273] Here, the on-site recognition processing unit 125 uses, for example, a large number of image data stored in the on-site data DB 154 to recognize categories such as “crowded number of people,” “age distribution,” and “relationships” as a crowd based on images of spectators captured in the image data, as shown in FIG. 24, and also recognizes labels for each category.

[0274] In this case, the parameter acquiring unit 123 may dynamically generate the inference threshold value, for example, as follows.

[0275] (a) The parameter acquisition unit 123 generates an inference threshold using tuning information transmitted from a specific camera among multiple cameras installed in the same baseball stadium. Specifically, the age of each spectator is recognized using image data obtained from a camera (specific camera) installed near the entrance that accurately captures the faces of spectators entering the baseball stadium, and the frequency is tallied for each age group (i.e., teens, twenties, thirties, etc.). The parameter acquisition unit 123 may generate an inference threshold for each age group (i.e., for each label) according to the frequency thus tallied.

[0276] (b) For image data obtained from a camera installed in a location that is subject to disturbances such as afternoon sun, raindrops, etc. among multiple cameras installed in the same baseball field, the parameter acquisition unit 123 may generate an inference threshold using tuning information based on changes in brightness depending on the weather, season, time, etc. For example, the parameter acquisition unit 123 may generate an inference threshold for color judgment using brightness information according to the weather, season, time (label), etc.

[0277] (c) For example, the parameter acquisition unit 123 may generate an inference threshold by using tuning information such as the number of visitors to a baseball stadium, the degree of congestion in the seats at the baseball stadium, etc. As an example, the parameter acquisition unit 123 may determine that when the seats at a baseball stadium are crowded, a large number of occlusions occur (labels), and that in an image, the upper halves of the spectators seated in the lower tier of seats and the backs of the seats hide the lower halves of the spectators seated in the upper tier, making it difficult to recognize images of the lower halves of the spectators in the upper tier, and generate an inference threshold.

[0278] (d) For example, the training recognition processing unit 124 of the learning recognition device 10 may perform on-site learning using image data obtained from a camera installed in a baseball field, and the parameter acquisition unit 123 may generate an inference threshold using tuning information fed back from the on-site learning obtained. In other words, the inference threshold may be generated from recognition performance that depends on the site.

[0279] Specifically, the training recognition processing unit 124 recognizes and learns the age and gender of each spectator using image data obtained from a camera (a specific camera) installed near the entrance of the stadium that accurately captures the faces of spectators entering the stadium. The parameter acquisition unit 123 may generate an inference threshold for each label of the category "spectator relationship" (whose labels are "family," "couple," "friend," and "business relationship") according to the learning results. Next, the on-site recognition processing unit 125 recognizes the "spectator relationship" according to the generated inference threshold using image data obtained from a camera capturing the spectator seats.

[0280] (5) In an embodiment, the inference error and the corrected inference error are calculated for each label, and the neuron weights and neuron thresholds are corrected and the inference is repeated until the corrected inference error becomes zero for all labels.

[0281] When the corrected inference error converges, the inference repetition may be stopped when a predetermined number of iterations is reached, or when the corrected inference error reaches a predetermined convergence threshold.

[0282] (6) The above-described embodiments and modifications may be combined with each other.

[0283] (7) One aspect of the present disclosure is a learning device for training a neural network, comprising: an acquisition means for acquiring teacher data and a plurality of corresponding teacher labels; a control means for controlling the neural network to calculate an inference error for each teacher label of the teacher data; and a correction means for correcting the inference error calculated for each teacher label by a weighting value set based on the learning characteristics of the teacher label, to calculate a corrected inference error, wherein the control means repeatedly applies the control to the neural network so as to reduce the corrected inference error for each teacher label.

[0284] Here, the information processing apparatus may further comprise a calculation means for calculating the weight value for each of the truth labels based on a learning characteristic of the truth label.

[0285] Here, in the learning device, learning is performed on a plurality of teacher data, and a plurality of teacher labels are associated with each of the plurality of teacher data, and the acquisition means may further acquire frequency information indicating a frequency of appearance of each teacher label, and the calculation means may calculate the weighted value based on the acquired frequency information.

[0286] Here, the difficulty of recognition varies for each of the plurality of teacher labels, and the acquisition means may further acquire difficulty information indicating the difficulty of each teacher label, and the calculation means may calculate the weighted value based on the acquired difficulty information.

[0287] Here, the proficiency level in the neural network differs for each of the multiple teacher labels, and the acquisition means further acquires proficiency information indicating the proficiency level for each teacher label, and the calculation means calculates the weighted value based on the acquired proficiency information.

[0288] Here, a plurality of categories may be set for the teacher data, and some of the plurality of teacher labels may belong to one of the plurality of categories.

[0289] Here, the acquisition means may further acquire a missing label that cannot be recognized from the teacher data in the neural network, and the control means may further determine whether any of the plurality of teacher labels matches the missing label, and if it is determined that there is a match, suppress the repeated action of the missing label on the neural network.

[0290] Another aspect of the present disclosure is a learning method used in a learning device that causes a neural network to learn, the method including: an acquisition step of acquiring teacher data and a plurality of corresponding teacher labels; a control step of controlling the neural network to calculate an inference error for each teacher label of the teacher data; and a correction step of correcting the inference error calculated for each teacher label by a weighting value set based on the learning characteristics of the teacher label, for each teacher label, to calculate a corrected inference error, the control step repeatedly acting on the neural network so as to reduce the corrected inference error for each teacher label.

[0291] Another aspect of the present disclosure is a learning program used in a learning device that trains a neural network and stored in a computer-readable storage medium, the learning device being a computer, which executes an acquisition step of acquiring teacher data and a plurality of corresponding teacher labels, a control step of controlling the neural network to calculate an inference error for each teacher label for the teacher data, and a correction step of correcting the inference error calculated for each teacher label by a weighting value set based on the learning characteristics of the teacher label, for each teacher label, to calculate a corrected inference error, and the control step repeatedly acts on the neural network so that the corrected inference error for each teacher label becomes smaller.

[0292] Another aspect of the present disclosure is a recognition device that acquires a plurality of labels from data to be recognized by performing recognition using a neural network, the recognition device comprising: an acquisition means for acquiring the data; a control means for controlling the neural network to calculate an inference result for the data for each of the labels; and a determination means for determining whether to output the inference result based on an inference threshold value for each of the labels.

[0293] Here, the method may further include a threshold acquisition means for acquiring an inference threshold for each label, and the judgment means may reject the inference result if the inference result calculated for each label is smaller than the inference threshold, and may output the inference result if the inference result calculated for each label is equal to or larger than the inference threshold.

[0294] Here, the threshold obtaining means may calculate the inference threshold based on learning characteristics of the label.

[0295] Here, in the neural network, learning is performed on a plurality of teacher data, and a plurality of teacher labels are associated with each of the plurality of teacher data, and the acquisition means may further acquire frequency information indicating a frequency of each of the teacher labels, and the threshold acquisition means may calculate the inference threshold based on the acquired frequency information.

[0296] Here, a level of difficulty of recognition may differ for each of the plurality of labels, and the acquisition means may further acquire difficulty information indicating the difficulty of each label, and the threshold acquisition means may calculate the inference threshold based on the acquired difficulty information.

[0297] Here, the proficiency level in the neural network differs for each of the multiple labels, and the acquisition means further acquires proficiency information indicating the proficiency level for each label, and the threshold acquisition means may calculate the inference threshold based on the acquired proficiency information.

[0298] Here, the threshold value obtaining means may dynamically calculate the inference threshold value.

[0299] Here, the threshold value obtaining means may calculate the inference threshold value based on a fluctuating external factor.

[0300] Here, the threshold acquisition means may acquire the inference threshold input by a user.

[0301] Here, a plurality of categories may be set for the data, and some of the plurality of labels may belong to one of the plurality of categories.

[0302] Another aspect of the present disclosure is a recognition method used in a recognition device that acquires multiple labels from data to be recognized by performing recognition using a neural network, the method including an acquisition step of acquiring the data, a control step of controlling the neural network to calculate an inference result for the data for each of the labels, and a determination step of determining whether to output the inference result based on an inference threshold value for each of the labels.

[0303] Another aspect of the present disclosure is a recognition program stored in a computer-readable storage medium for use in a recognition device that acquires multiple labels from data to be recognized by performing recognition using a neural network, the program causing the recognition device, which is a computer, to execute an acquisition step of acquiring the data, a control step of controlling the neural network to calculate an inference result for the data for each label, and a determination step of determining whether to output the inference result based on an inference threshold for each label.

[0304] Another aspect of the present disclosure is a learning and recognition device including the learning device and the recognition device. [Industrial Applicability]

[0305] The learning device according to the present disclosure has the excellent effect of being able to learn so as to maintain a certain level of recognition performance even when there is a bias in the learning characteristics of each label, and is useful as a technology for making a neural network learn, particularly as a learning technology when multiple teacher labels correspond to one teacher data. Also, the recognition device according to the present disclosure has the excellent effect of being able to maintain a certain level of recognition performance even when there is a learning bias in the characteristics of each label, and is useful as a technology for performing recognition using a neural network, particularly as a recognition technology when multiple teacher labels correspond to one teacher data. [Explanation of symbols]

[0306] 10 Learning recognition device 50 Neural Networks 50a Input layer 50b Feature Extraction Layer 50c recognition layer 101 CPU 102 ROM 103 RAM 104 Storage section 105 GPU 106 ROM 107 RAM 108 GPU 109 ROM 110 RAM 111 Network communication circuits 121 Main control unit 122 General Control Unit 123 Parameter Acquisition Unit 124 Training Recognition Processing Unit 125 Field Recognition Processing Unit 126 Inference result output section 130 Control section 131 Training Data Input Section 132 Feature Extraction Unit 133 Feature Learning Unit 134 Label Inference Part 135 Weighting Unit 136 Inference Error Transfer Unit 140 Control section 141 Field Data Entry Section 142 Feature Extraction Unit 143 Reasoning part 144 Label Inference Part 145 Inference result adjustment section 146 Inference result transfer unit

Claims

[Claim 1] A learning device for training a neural network, comprising: An acquisition means for acquiring teacher data and a plurality of corresponding teacher labels; A control means for controlling the neural network so as to calculate an inference error for each of the teacher labels with respect to the teacher data; a correction means for correcting the inference error calculated for each teacher label by a weighting value set based on the learning characteristics of the teacher label, to calculate a corrected inference error, the control means repeatedly operates the neural network so that the corrected inference error for each teacher label becomes smaller; The acquiring means further acquires missing labels that cannot be recognized from the training data in the neural network, The control means further determines whether any of the plurality of teacher labels matches the missing label, and when it is determined that the missing label matches, suppresses the repeated action of the missing label on the neural network. Learning device.

Images