Information processing device, imaging device, information processing method, and program

Abstract

To accurately estimate a category of an object in an image.SOLUTION: An information processing device comprises: extracting means for applying a model to an image and performing extraction processing for extracting a feature amount; estimating means for performing estimation processing for estimating a likelihood representing a probability classified into each category about an object in the image on the basis of the feature amount; first calculating means for calculating a likelihood error of each category on the basis of correct answer information of the image and the likelihood of each category estimated by the estimation processing; second calculating means for calculating the whole likelihood error representing an error of the likelihood of the whole category on the basis of the likelihood error of each category; and learning means for learning the model on the basis of the whole likelihood error. When calculating the whole likelihood error, the second calculating means assigns weight based on a relation between the category to be targeted and other categories per category to be targeted with respect to the likelihood error of each category.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to machine learning. [Background technology]

[0002] Digital cameras are equipped with various functions for acquiring images suitable for subjects that are automatically detected by the camera or selected by the user. For example, there is an autofocus (AF) function that automatically adjusts the focus, an autoexposure (AE) function that automatically adjusts the brightness, and an autowhite balance (AWB) function that automatically adjusts the color to an appropriate level. In addition, cameras with a tracking function that continues to track a selected subject in subsequent frames to keep the focus on the subject or to keep adjusting the brightness and color are also known. These functions can perform highly accurate processing by estimating the area in which the subject exists and the category of the subject in the input image. Patent Document 1 describes a method for classifying objects in an input image into appropriate classes using a convolutional network. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-64604 A

[0004] [Non-Patent Document 1] Multi-Task Learning as Multi-Objective Optimization,In Neural Information Processing Systems (NeurIPS) 2018 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to learn category classification, which classifies object categories using a multi-layered neural network called a deep net (or deep neural net, deep learning), it is necessary to prepare learning images of categories and GT information and perform learning. Note that here, GT (Ground Truth) information is frame information that indicates the position and size within the learning image. When actually captured images are used as learning images, there are cases where a specific category is frequently included in the GT of a certain category. For example, people are often included in the GT of a motorcycle. Such relationships between specific categories are not taken into consideration when learning. Therefore, when learning for multi-category classification is performed using a dataset that includes data in which multiple categories overlap in the GT, there is a risk that a category other than the category originally intended by the user will be learned as a category feature. As a result, there is a problem that the accuracy of the final model does not improve.

[0006] In view of the above problems, an object of the present invention is to improve the accuracy of estimating the category of an object in an image. [Means for solving the problem]

[0007] The present invention comprises an extraction means for performing an extraction process that applies a model to an image to extract features; an estimation means for performing an estimation process that estimates a likelihood representing the likelihood that an object in the image will be classified into each category based on the features; a first calculation means for calculating a likelihood error of each category based on correct answer information of the image and the likelihood of each category estimated by the estimation process; a second calculation means for calculating an overall likelihood error that represents the error of the likelihood of the entire category based on the likelihood error of each category; and a learning means for learning the model based on the overall likelihood error, wherein the second calculation means, when calculating the overall likelihood error, weights the likelihood error of each category for each target category based on the relationship between the target category and other categories. Effect of the Invention

[0008] According to the present invention, the accuracy of the estimation of the category of an object in an image can be improved. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a camera system. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of a learning device. [Diagram 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a camera. [Figure 4] 4 is a flowchart showing a process executed by the camera. [Diagram 5] FIG. 1 is a diagram showing a network structure of a neural network. [Figure 6] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 7] 13 is a flowchart showing a process executed by the learning device. [Figure 8] FIG. 13 is a diagram showing an example of a learning image. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. [Figure 10] FIG. 2 is a diagram illustrating an example of a functional configuration of a learning device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. Note that the same reference numerals in the drawings have the same operation, and therefore the repeated description will be omitted.

[0011] <Embodiment 1> In this embodiment, a method of learning for estimating the area of ​​an object in an image and the category of the object in order to perform AF (autofocus) processing on an object detected from an image will be described. FIG. 1 shows an example of a camera system according to this embodiment. The camera system includes a camera 100 and a learning device 150. The camera 100 is configured with an image analysis device 110. The camera 100 has an AF function. In addition to or instead of the AF function, the camera 100 may have an AE (auto exposure) function, an AWB (auto white balance) function, or the like. In the following, a case will be described in which a tracking process is performed by detecting an area of ​​a person or a specific part of a person as a frame from a frame of a moving image or a still image captured continuously, and performing AF processing on the detection frame. The category of the object to be detected is not particularly limited, and may be a person, an animal such as a horse, a vehicle such as a train, or the like. In addition, the object to be detected also includes a specific part of the object, such as a person's face or an animal's face. The image analysis device 110 acquires image data to be processed and outputs the processing result. The learning device 150 learns parameters of a model used for processing by the image analysis device 110. The learning device 150 is an example of an information processing device.

[0012] 2 shows an example of the hardware configuration of a learning device according to this embodiment. The learning device 150 includes a control device 101, a storage device 102, an input device 103, an output device 104, and a communication I / F 105. These devices are connected to each other via a bus 106. The control device 101 is configured with a CPU (Central Processing Unit). Note that the control device 101 may use a GPU (Graphics Processing Unit) instead of the CPU or together with the CPU. The control device 101 executes programs stored in the storage device 102. The control device 101 of the learning device 150 is responsible for the overall control of the learning device 150.

[0013] The memory device 102 is a storage device such as a magnetic memory device or a semiconductor memory, and stores programs executable by the control device 101, learned parameters acquired by learning, and the like. The memory device 102 of the learning device 150 also stores learning data. The input device 103 is a keyboard, a mouse, a touch panel device, and the like, and inputs various user instructions. The output device 104 is a liquid crystal display, an external monitor, and the like, and outputs processing results, and the like. The communication I / F (interface) 105 communicates with an external device and transmits and receives data. The communication I / F 105 may be connected to an external device via a wired or wireless LAN, or may be connected to an external device via a USB cable, and the like. The control device 101 of the learning device 150 transmits learned parameters to the image analysis device 110 via the communication I / F 105.

[0014] The image analysis device 110 can also be realized with a hardware configuration similar to that of Fig. 2. Moreover, the camera 100 is configured to have the image analysis device 110 and an imaging section (not shown). The storage device 102 of the image analysis device 110 stores the captured image captured by the imaging section. The input device 103 of the image analysis device 110 includes buttons, levers, etc. provided on the camera 100. The output device 104 of the image analysis device 110 displays the captured image captured by the imaging section.

[0015] FIG. 3(a) shows an example of the functional configuration of the camera. The camera 100 includes an image analysis device 110, an AF processing unit 360, and a camera control unit 361. The AF processing unit 360 performs AF processing to focus on the attention area output from the image analysis device 110. The camera control unit 361 controls the operation of an imaging unit (not shown). The image analysis device 110 includes an image acquisition unit 310, a feature extraction unit 320, a detection area estimation unit 330, a category likelihood estimation unit 340, and an attention area selection unit 350. In this embodiment, the control device 101 of the image analysis device 110 executes a program stored in the storage device 102 of the image analysis device 110, thereby realizing each function of the camera 100.

[0016] The image acquisition unit 310 sequentially acquires successive frames (images) in time series from a captured moving image. For example, the image acquisition unit 310 acquires full HD (1920×1280 pixels) image data in real time (60 frames per second). Note that the image acquisition unit 310 may acquire a single image (still image). The feature extraction unit 320 extracts features from the image acquired by the image acquisition unit 310 using a known machine learning method such as a neural network. For a model for extracting the features, learned parameters acquired by learning using the learning device 150 are used. In this embodiment, the feature extraction unit 320 outputs three feature maps: a center map, a size map, and a category likelihood map.

[0017] The detection area estimation unit 330 estimates the detection area of ​​the object in the image based on the center map and size map output by the feature extraction unit 320. In this embodiment, the center coordinate, width, and height of the frame are estimated as the detection area. Specifically, the detection area estimation unit 330 acquires the peak position of the feature amount in the center map, and sets this position as the center position of the object. In addition, the value of the size map at the same position as this center position is acquired, and this is acquired as the size of the object. The detection area estimation unit 330 outputs the frame with the center position and size acquired by the above processing as the result of the estimation of the detection area. FIG. 3(b) shows an example of the frame estimated by the detection area estimation unit 330. As shown in FIG. 3(b), four frames 371 to 374 representing the position and size of the object are estimated from the image 370 as candidates for the attention area described later.

[0018] The category likelihood estimation unit 340 estimates the likelihood of each preset candidate category for the detection area estimated by the detection area estimation unit 330 from the category likelihood map output by the feature extraction unit 320, and identifies the category of the detection area based on the estimated value. A specific method for identifying the category is to adopt the category with the highest likelihood. The category likelihood estimation unit 340 outputs the category identified by the above process as the category estimation result.

[0019] The attention area selection unit 350 selects an attention area from the detection area based on the detection area estimated by the detection area estimation unit 330 and the category specified by the category likelihood estimation unit 340. A specific method for selecting an attention area is to select a frame with the highest score in the center as the attention area from among the frames estimated by the detection area estimation unit 330. Another method is to select a frame with the highest score in the center as the attention area from among the frames whose category specified by the category likelihood estimation unit 340 matches a preset category. FIG. 3(c) shows that frame 373 has been selected as the attention area from the four frames 371 to 374 shown in FIG. 3(b). The AF processing unit 360 performs AF processing on the image captured by the imaging unit, based on information about the region of interest (frame to be noted) selected by the region of interest selection unit 350. The camera control unit 361 controls the operation of the imaging unit of the camera 100 .

[0020] Hereinafter, a series of steps in which the image analyzing device 110 according to this embodiment selects a frame of interest from a moving image captured and executes AF processing will be described. FIG. 4 is a flowchart showing the processing executed by the image analyzing device 110. In the following description, each process (step) will be written with an S at the beginning, and the description of the process (step) will be omitted. Each process of this flowchart is realized by the control device 101 of the image analyzing device 110 executing a program stored in the storage device 102 of the image analyzing device 110. Note that the image analyzing device 110 does not need to execute all the processes described in this flowchart, and may execute only some of the processes.

[0021] In S401, the image acquisition unit 310 acquires an image from a moving image captured by the imaging unit 310. The image acquired in S401 is, for example, bitmap data expressed in RGB 8 bits. In S402 , the feature extraction unit 320 extracts a feature amount from the image acquired by the image acquisition unit 310 .

[0022] The process executed in S402 will be specifically described below. In this embodiment, the feature extraction unit 320 applies a neural network to extract features. FIG. 5 shows an example of the network structure of a neural network. Here, a neural network that performs a recognition task by repeating a convolution process and a pooling process is used. For example, the one described in Non-Patent Document 1 can be used. Note that the neural network is an example of a machine learning model, and is not limited to a method of extracting features using a neural network, and known machine learning methods such as SIFT and HOG may be used.

[0023] The feature extraction unit 320 performs multiple convolution processes (511, 513, 515) and multiple pooling processes (512, 514) to extract features from the input image 530 based on parameters representing weighting coefficients in the multiple convolution processes that are prepared in advance. These parameters are acquired by learning, which will be described later.

[0024] In the convolution process, for example, a 3×3 filter is set for multiple channels for the input image 530 and feature maps (552, 554), a convolution operation is performed centered on the pixel of interest, and multiple feature maps corresponding to the multiple channels are output. In the example of Fig. 5, the convolution process 1 (511) has a filter size of 3×3 and 4 channels, the convolution process 2 (513) has a filter size of 3×3 and 12 channels, and the convolution process 3 (515) has a filter size of 3×3 and 24 channels.

[0025] In the pooling process, the feature map output from the convolution process is reduced. When pooling is performed in a range of 2×2, the feature map is reduced to 1 / 4. In the pooling process, methods such as maximum pooling and average pooling can be used. In the example of FIG. 5, both pooling process 1 (512) and pooling process 2 (514) perform pooling in a range of 2×2. In the example of FIG. 5, since there are two pooling processes in a range of 2×2, the resolution of the input image 530 is reduced to 1 / 16 at the stage of the output layer 520. The network structure of the neural network is not limited to that shown in FIG. 5. It may be made more multi-layered than that shown in FIG. 5, and the number of channels may be changed. In this embodiment, a center map, a size map, and a category likelihood map are output from the output layer 520.

[0026] In S403, based on the feature amount extracted by feature extraction section 320, detection area estimation section 330 estimates the center coordinates, width, and height of the object frame as the detection area of ​​the object in the image. In S404, the category likelihood estimation unit 340 estimates the likelihood of each category based on the feature amount extracted by the feature extraction unit 320 for the frame estimated by the detection area estimation unit 330. The likelihood here indicates the likelihood of being classified into each category.

[0027] In S405, the attention area selection unit 350 selects a detection frame to be an attention area based on the frame (detection frame) estimated by the detection area estimation unit 330 and the estimated value of the likelihood of each category estimated by the category likelihood estimation unit 340 for that detection frame. In S406, the camera control unit 361 determines whether or not to execute AF processing. For example, the camera control unit 361 determines whether or not to execute AF processing based on whether or not tracking processing is being performed. If the camera control unit 361 determines to execute AF processing, the process proceeds to S407, and if it determines not to execute AF processing, the process proceeds to S408.

[0028] In S407 , the AF processing unit 360 performs AF processing to bring the detection frame selected by the attention area selection unit 350 into focus. In S408, the camera control unit 361 determines whether or not to continue the operation of the imaging unit. If the camera control unit 361 determines that the operation of the imaging unit is to be continued, the process returns to S401, and if the camera control unit 361 determines that the operation of the imaging unit is to be ended, the process of this flowchart ends. In this way, the camera 100 sequentially acquires successive images of the moving image captured by the imaging unit, and continues to perform AF processing on the area of ​​the object of interest, so that in S407, a tracking process is performed to track the object of interest.

[0029] Next, a method for training the model used in the feature extraction unit 320 will be described. 6 shows an example of the functional configuration of a learning device according to this embodiment. In this embodiment, each function of the learning device 150 is realized by the control device 101 of the learning device 150 executing a program stored in the storage device 102 of the learning device 150. The learning device 150 includes a learning data storage unit 601, a learning data acquisition unit 602, an image acquisition unit 603, a feature extraction unit 604, a detection region estimation unit 605, and a category likelihood estimation unit 606.

[0030] 6 are the same as the feature extraction unit 320 and the detection region estimation unit 330 in Fig. 3, and therefore description thereof will be omitted. The feature extraction unit 604 in Fig. 6 also extracts features using the neural network shown in Fig. 5.

[0031] The learning data storage unit 601 stores learning data for the learning device 150 to perform learning in the storage device 102 of the learning device 150. The learning data is a set of a learning image and correct answer information. The correct answer information includes area information and category information. The area information is information that represents the area of ​​an object to be analyzed in the learning image. Specifically, the area information is a correct answer value of a frame (center coordinates, width, height). The category information is a correct answer value of the likelihood of each category assigned to each area information. Specifically, the category information is data having a value of 1 for the correct answer category and a value of 0 for other categories. The learning data storage unit 601 stores a group of learning data prepared in advance.

[0032] The learning data acquiring unit 602 acquires the learning data stored in the learning data storage unit 601. Note that the learning data acquiring unit 602 may acquire the learning data from an external storage device connected via the communication I / F 105, instead of acquiring the learning data from the storage device 102 of the learning device 150.

[0033] The image acquisition unit 603 acquires the training images from the training data acquisition unit 602 . The category likelihood estimation unit 606 estimates the likelihood of each category from the feature amount extracted by the feature extraction unit 604 for the detection region estimated by the detection region estimation unit 605 .

[0034] The learning device 150 also includes a region error calculation unit 607 , a category likelihood error calculation unit 608 , a parameter learning unit 609 , a category correlation dictionary holding unit 610 , and a category correlation dictionary correction unit 611 . The region error calculation unit 607 calculates a region error from the estimated value of the detection region obtained by the detection region estimation unit 605 and the region information (correct answer information) of the learning data.

[0035] The category likelihood error calculation unit 608 calculates the category likelihood error from the estimated value of the likelihood of each category obtained by the category likelihood estimation unit 606 and the category information (correct answer information) of the training data. The category likelihood error is the sum of the squared error between the estimated value and the correct answer value in the training data for all training data in the same category. The category likelihood error of category i is defined as loss i Let us assume that.

[0036] The parameter learning unit 609 updates the parameters used in the feature extraction unit 320, the detection region estimation unit 330, and the category likelihood estimation unit 340 so as to reduce the region error and the total loss of learning described below. The category correlation dictionary storage unit 610 stores a category correlation dictionary used when calculating the total loss of learning. The category correlation dictionary is updated by a category correlation dictionary update process described later. The category correlation dictionary correction unit 611 adjusts the category correlation dictionary held in the category correlation dictionary holding unit 610 based on the error for each category.

[0037] 7(a) is a flowchart showing processing executed by the learning device 150. Each step of this flowchart is realized by the control device 101 of the learning device 150 executing a program stored in the storage device 102 of the learning device 150. In S701, the learning data acquisition unit 602 acquires learning data stored in the learning data storage unit 601. The image acquisition unit 603 acquires learning images from the learning data acquired by the learning data acquisition unit 602. As described above, the learning data is composed of learning images and correct answer information.

[0038] In S702, the feature extraction unit 604 extracts feature amounts from the learning images acquired by the image acquisition unit 603. The extraction process here is similar to that in S402. In S703, the detection area estimation unit 605 estimates the center coordinates, width, and height of the object frame as the detection area of ​​the object in the learning image based on the feature amount extracted by the feature extraction unit 604. The estimation process here is the same as in S403.

[0039] In S704, the category likelihood estimation unit 606 estimates the likelihood of each category for the frame estimated by the detection area estimation unit 605, based on the feature amount extracted by the feature extraction unit 604. The estimation process here is similar to that in S404. In S705, the area error calculation unit 607 calculates an error from the estimated value of the detection area obtained by the detection area estimation unit 605 and the area information of the learning data. The error is calculated by adding up the deviation amounts for the center coordinates, position, and height between the frame (detection frame) estimated by the detection area estimation unit 605 and the correct frame having center coordinates closest to that frame. In S706, the category likelihood error calculation unit 608 calculates an error from the estimated value of the likelihood of each category obtained by the category likelihood estimation unit 606, the category information (correct answer information) of the training data, and the category correlation dictionary. The category likelihood error calculation unit 608 first calculates the squared error between the correct answer value of the training data and the estimated value of the likelihood of each category obtained from the training images of the training data, and calculates the category likelihood error of each category by summing up the squared errors within the same category of all the training data. The process of calculating the category likelihood error of each category is an example of the first calculation process.

[0040] Here, the category correlation dictionary is a dictionary having one value for each combination of categories, and this value represents a coefficient by which the category likelihood error of each category is multiplied when calculating the total loss. The category likelihood error calculation unit 608 multiplies the category likelihood error of the candidate category i for the input training image (the correct category is assumed to be A) by the value of a coefficient stored in the category correlation dictionary according to the combination of the candidate category i and the correct category A. Then, the sum according to the total number of candidate categories is calculated as the total loss of the training. The total loss is a value obtained by calculating the loss of the entire candidate categories by weighting the category likelihood error of the candidate category i according to the relationship between the candidate category i and the correct category A, and is an example of an overall likelihood error. When the correct category of the training data group is A, the total loss is calculated by the following formula (1).

[0041]

number

[0042] Here, each symbol represents the following. loss i : Category likelihood error for category i loss_weight[A][i]: weighting value for the combination of category A and category i

[0043] As described above, the total loss in this embodiment weights the likelihood error of each category according to the relationship between the category and the correct category. As a result, for example, the larger the weighting coefficient, the greater the impact of the category combination on the total loss and the greater the contribution rate to learning. In other words, by changing the value of the category correlation dictionary according to the specific relationship between categories, it is possible to adjust the impact on the entire learning.

[0044] In S707, the parameter learning unit 609 updates the parameters so that the domain error calculated in S705 and the total loss calculated in S706 become smaller. The parameters to be updated are the weight coefficients of the convolution layer of the neural network. The parameter update is performed by calculating a loss value by taking the sum of the domain error calculated in S705 and the total loss calculated in S706, and using the backpropagation method. The parameter update process is an example of a process for learning a model.

[0045] In S708, the learning device 150 updates the category correlation dictionary. The process of updating the category correlation dictionary will be described in detail later. In S709, the parameter learning unit 609 judges whether or not to end the learning according to a preset end condition. If the parameter learning unit 609 judges to end the learning, the process of this flowchart ends, and if it judges to continue the learning, the process returns to S701. As a method for judging whether to end the learning, there is a method of preparing learning data for accuracy verification separately from the learning data for parameter update, and executing the processes of S701 to S708 on the learning data for accuracy verification as well, and judging whether the loss value calculated in S707 has become equal to or less than a predetermined value. There is also a method of judging whether the number of repetitions of the processes of S701 to S708 has become equal to or more than a predetermined value.

[0046] The following describes the process of updating the category correlation dictionary executed in S708. Fig. 7(b) is a flowchart showing the details of the process of updating the category correlation dictionary. In S711, the category correlation dictionary correction unit 611 adds up the squared errors between the estimated values ​​and the correct values ​​in the learning data for each combination of the correct category and the candidate category. The errors for each combination of the correct category and the candidate category are tallied for each combination of the correct category a and the candidate category i in a format such as sum_loss_weight[a][i], and are stored in the storage device 102. The count value of the number of additions in S711 is also stored in the storage device 102.

[0047] 7A, the addition in S711 is continued until the count value of the number of additions reaches a specified number, and in S712, the category correlation dictionary correction unit 611 determines whether the number of additions reaches the specified number. If the number of additions reaches the specified number, the category correlation dictionary correction unit 611 advances the process to S713. In S713, the category correlation dictionary correction unit 611 calculates the average value for each of the sum_loss_weight[a][i] obtained as the result of the aggregation in S711, and sets this as the average likelihood error.

[0048] In S714, the category correlation dictionary correction unit 611 calculates a category correlation error. The category correlation error is a value used to compare the characteristics of correlation between two categories, and for example, the absolute value of the difference between the average likelihood error when the correct category is A and the candidate category is B, and the average likelihood error when the correct category is B and the candidate category is A is used. This value is large when B is likely to be erroneously detected when the correct category is A, but A is unlikely to be erroneously detected when the correct category is B.

[0049] In S715, the category correlation dictionary correction unit 611 calculates a correction value for the category correlation dictionary based on the category correlation error in S714. In S716, the category correlation dictionary correction unit 611 uses the correction value calculated in S715 to correct the category correlation dictionary held in the category correlation dictionary holding unit 610. After that, this flowchart ends, and the process proceeds to S709 in FIG.

[0050] Usually, it is considered that the false positive rate of category B being erroneously detected when the correct category is A and the false positive rate of category A being erroneously detected when the correct category is B are about the same. Therefore, if the category is normally trained, it is considered that the average likelihood error is low to the same extent between all categories. On the other hand, when the category correlation error is large, it is considered that there is a difference between the false positive rate of category B when the correct category is A and the false positive rate of category A when the correct category is B. It is considered that this is not caused by the characteristics between the categories, but by external factors such as the training data. Therefore, for a combination of categories with a larger category correlation error than others, the category correlation dictionary correction unit 611 reduces the value of the category correlation dictionary so as to reduce the influence on the category likelihood error. In this way, it is possible to suppress the influence between specific categories caused by the characteristics of the training data on the entire training of the multi-category classification model.

[0051] A method for adjusting the category correlation dictionary will be specifically described. FIG. 8 shows an example of a learning image. A frame 801 of this learning image 800 is labeled with a motorcycle as the correct category, but the frame 801 contains a human. In this way, there are cases where objects of a specific category frequently appear in a frame of a certain category. When a model for category classification is trained using such training data, the motorcycle category also learns human characteristics, so even if the correct answer is motorcycle, the estimated value of the likelihood of the human category becomes high. On the other hand, there are almost no cases where a motorcycle appears in a frame whose correct answer category is human. Therefore, since the human category does not learn the characteristics of motorcycles, the estimated value of the likelihood of the motorcycle category does not become high when the correct answer is human. In other words, the category correlation error in the combination of motorcycles and humans becomes large. For a combination of specific categories such as motorcycles and humans identified by using the category correlation error, the value of the category correlation dictionary is reduced. This reduces the influence of the relationship between specific categories such as motorcycles and humans on learning.

[0052] According to the present embodiment, by using the category correlation dictionary, it is possible to finely adjust the contribution rate to learning based on the relationship between categories. By using a multi-category classification model trained using the category likelihood error adjusted in this way, it is possible to accurately classify the category of objects in an image during inference.

[0053] <Modification 1 of the First Embodiment> In the first embodiment, a method has been described in which a category correlation error between categories is large, a specific category between categories that may hinder normal learning is specified, and the value of the category correlation dictionary between the categories is adjusted. The method of adjusting the value of the category correlation dictionary using the category correlation error is not limited to the method of the first embodiment. For example, when the category correlation error between categories A and B is small and the average likelihood error between the categories is large, these categories are considered to have similar characteristics and are difficult to distinguish. Therefore, the category correlation dictionary correction unit 611 increases the value of the category correlation dictionary for such a combination of categories. This increases the influence on the learning between the categories in the entire learning, and the entire learning progresses smoothly. By adjusting the category correlation dictionary in this way, it becomes possible to flexibly adjust the learning efficiency between specific categories.

[0054] <Embodiment 2> In the first embodiment, a method for updating the category correlation dictionary in parallel with learning a model has been described. In the second embodiment, a method for analyzing a group of learning data used for learning at the start of model learning and updating the category correlation dictionary to values ​​suitable for learning based on the characteristics of the group of learning data will be described. Note that a description of the points common to the first embodiment will be omitted, and the description will focus on the differences from the first embodiment.

[0055] 9 shows an example of the functional configuration of a learning device according to this embodiment. In FIG. 9, the function of a category correlation dictionary correcting unit 811 is different from that of the category correlation dictionary correcting unit 611 in FIG. The category correlation dictionary correction unit 811 analyzes the learning data acquired by the learning data acquisition unit 602, and corrects the values ​​of the category correlation dictionary held in the category correlation dictionary holding unit 810 based on the results of the analysis. As a method for correcting the values ​​of the category correlation dictionary based on the results of the analysis of the learning data, for example, there is a method using correct answer information (area information, category information) of the learning data. Specifically, the category correlation dictionary correction unit 811 counts up information on the frames of other categories included in the frames of the learning images of the learning data for each correct answer category, and corrects the values ​​of the category correlation dictionary based on the results of the counting. At this time, for frames of other categories with a high overlap rate, there is a possibility that the characteristics of the other categories are erroneously learned as the characteristics of the correct answer category. Therefore, the values ​​of the category correlation dictionary corresponding to the "correct answer category that is overlapped a lot with objects of another category" and the "other category that overlaps with the above-mentioned 'correct answer category that is overlapped a lot with another category'" are corrected to be smaller. This makes it possible to reduce the influence of such inter-category relationships on the entire learning.

[0056] 7(a) begins, the category correlation dictionary correction unit 811 analyzes a group of learning data used for learning, and updates the values ​​of the category correlation dictionary held in the category correlation dictionary holding unit 610 based on the results of the analysis. In S706, the category likelihood error calculation unit 608 uses the values ​​of the category correlation dictionary updated based on the results of the analysis of the learning data. Also, the process of updating the category correlation dictionary in S708 is skipped.

[0057] According to the present embodiment as described above, even when learning is performed using a huge amount of learning data that the user cannot grasp the whole picture, it is possible to grasp the relationship between categories resulting from the characteristics of the learning data. Then, by using a model trained using a category likelihood error whose contribution rate to learning is adjusted according to the grasped relationship between categories, it is possible to accurately classify the category of objects in an image during inference.

[0058] <Embodiment 3> In the first and second embodiments, a method for automatically correcting a category correlation dictionary has been described. In the third embodiment, a method for a user to manually edit a category correlation dictionary will be described. Note that a description of the points common to the first embodiment will be omitted, and the differences from the first embodiment will be mainly described.

[0059] 10 shows an example of the functional configuration of a learning device according to this embodiment. Fig. 10 differs from Fig. 6 in that a category correlation dictionary editing unit 911 is provided instead of the category correlation dictionary correcting unit 611. The category correlation dictionary editing unit 911 edits the category correlation dictionary based on information input by any method such as a GUI (Graphical User Interface), a CUI (Character User Interface), or a text editor. The category correlation dictionary editing unit 911 displays a GUI or CUI for editing the category correlation dictionary on the output device 104 of the learning device 150, and receives operations from the user from the input device 103. For example, the category correlation dictionary editing unit 911 updates the values ​​of the category correlation dictionary held in the category correlation dictionary holding unit 610 based on values ​​input by an input operation or a selection operation for each combination of a correct category and a candidate category. The category correlation dictionary editing unit 911 may also update the values ​​of the category correlation dictionary based on values ​​input using external software.

[0060] 7(a) begins, the category correlation dictionary editing unit 911 updates the values ​​of the category correlation dictionary held in the category correlation dictionary holding unit 610 based on information input by a user operation. In S706, the category likelihood error calculation unit 608 uses the values ​​of the category correlation dictionary updated based on the information input by a user operation. Also, the process of updating the category correlation dictionary in S708 is skipped.

[0061] According to the present embodiment as described above, learning can be carried out in a direction that is in line with the user's desire, such as a learning direction based on a criterion that is in line with a use case other than learning efficiency.

[0062] <Other embodiments> Although the present invention has been described above with reference to the embodiments, the above embodiments are merely illustrative of the specific examples of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these embodiments. In other words, the present invention can be embodied in various forms without departing from its technical concept or main features.

[0063] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0064] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) An extraction means for performing an extraction process for extracting feature quantities by applying a model to an image; an estimation means for performing an estimation process of estimating a likelihood representing a probability that an object in the image is classified into each category based on the feature amount; a first calculation means for calculating a likelihood error for each category based on the correct answer information of the image and the likelihood of each category estimated by the estimation process; A second calculation means for calculating an overall likelihood error representing an error of the likelihood of the entire category based on the likelihood error of each category; A learning means for learning the model based on the overall likelihood error; having The information processing device is characterized in that, when calculating the overall likelihood error, the second calculation means weights the likelihood error of each category for each target category based on the relationship between the target category and other categories. (Configuration 2) the first calculation means calculates a likelihood error for each category based on the likelihood of each category obtained by performing the extraction process and the estimation process on a plurality of the images having the same correct category; The information processing device according to configuration 1, wherein the second calculation means weights the likelihood error of each category for each target category based on a relationship between the target category and the correct category. (Configuration 3) an adjustment means for adjusting the weighting based on a result of analyzing a learning data group including the image and correct answer information of the image, which is a subject of learning of the model; 3. The information processing device according to configuration 1 or 2, further comprising: (Configuration 4) an adjustment means for adjusting the weighting based on a result of tallying up the likelihood estimated by the estimation process for each combination of a correct category of the image and a category estimated by the estimation process; 4. The information processing device according to any one of configurations 1 to 3, further comprising: (Configuration 5) The information processing device according to configuration 4, wherein the adjustment means calculates a first error based on the likelihood of a second category estimated by the estimation process for an image whose correct category is a first category, and a second error based on the first category estimated by the estimation process for an image whose correct category is the second category, and adjusts the weighting based on a result of aggregating a difference between the first error and the second error. (Configuration 6) 6. The information processing apparatus according to configuration 5, wherein the adjustment means reduces the weighting for the combination of the first category and the second category when the difference is large. (Configuration 7) The information processing device according to configuration 5, wherein the adjustment means increases the weighting for a combination of the first category and the second category when the difference is small and the first error and the second error are both large. (Configuration 8) A storage means for storing the weighting for each combination of categories; and 8. The information processing device according to any one of configurations 1 to 7, wherein the second calculation means obtains the weighting according to a combination of a target category and another category from the storage means. (Configuration 9) An adjustment means for adjusting the weighting according to the combination of categories in response to an operation by a user. 9. The information processing device according to configuration 8, further comprising: (Configuration 10) An adjustment means for adjusting the weighting according to the combination of categories according to information input from an external software. 9. The information processing device according to configuration 8, further comprising: (Configuration 11) 11. The information processing device according to any one of configurations 1 to 10, wherein the second calculation means calculates the overall likelihood error by using the following formula (1).

number

[0065] 100: camera, 110: image analysis device, 150: learning device

Claims

1. An extraction means for performing an extraction process for extracting features by applying a model to an image; an estimation means for performing an estimation process of estimating a likelihood representing a probability that an object in the image is classified into each category based on the feature amount; a first calculation means for calculating a likelihood error for each category based on correct answer information of the image and the likelihood of each category estimated by the estimation process; a second calculation means for calculating an overall likelihood error representing an error in the likelihood of the entire category based on the likelihood error of each category; A learning means for learning the model based on the overall likelihood error; having The information processing device is characterized in that, when calculating the overall likelihood error, the second calculation means weights the likelihood error of each category for each target category based on the relationship between the target category and other categories.

2. the first calculation means calculates a likelihood error for each category based on the likelihood of each category obtained by performing the extraction process and the estimation process on a plurality of the images having the same correct category; 2 . The information processing apparatus according to claim 1 , wherein the second calculation means weights the likelihood error of each category for each target category based on a relationship between the target category and the correct category.

3. an adjustment means for adjusting the weighting based on a result of analyzing a learning data group including the image and correct answer information of the image, which is a subject of learning of the model; 2. The information processing apparatus according to claim 1, further comprising:

4. an adjustment means for adjusting the weighting based on a result of tallying up the likelihood estimated by the estimation process for each combination of a correct category of the image and a category estimated by the estimation process; 2. The information processing apparatus according to claim 1, further comprising:

5. The information processing device according to claim 4, characterized in that the adjustment means calculates a first error based on the likelihood of a second category estimated by the estimation process for an image whose correct category is a first category, and a second error based on the first category estimated by the estimation process for an image whose correct category is the second category, and adjusts the weighting based on a result of tallying up a difference between the first error and the second error.

6. 6. The information processing apparatus according to claim 5, wherein said adjustment means reduces the weighting for the combination of said first category and said second category when said difference is large.

7. 6. The information processing device according to claim 5, wherein the adjustment means increases the weighting for a combination of the first category and the second category when the difference is small and the first error and the second error are both large.

8. A storage means for storing the weighting for each combination of categories; Further comprising: 2 . The information processing apparatus according to claim 1 , wherein the second calculation means acquires the weighting according to a combination of a target category and another category from the storage means.

9. An adjustment means for adjusting the weighting according to the combination of categories in response to an operation by a user.

9. The information processing apparatus according to claim 8, further comprising:

10. An adjustment means for adjusting the weighting according to the combination of categories according to information input from an external software.

9. The information processing apparatus according to claim 8, further comprising:

11. 2 . The information processing apparatus according to claim 1 , wherein the second calculation means calculates the overall likelihood error by using the following formula (1): [0010] Here, each symbol represents the following. total_loss[A]: Overall likelihood error for images with the correct category A Loss i : Category likelihood error for category i loss_weight[A][i]: weighting value according to the combination of category A and category i

12. The information processing apparatus according to claim 1 , wherein the estimation means detects regions of a plurality of objects in the image based on the feature amount, and performs the estimation process for each of the detected regions.

13. 2. The information processing apparatus according to claim 1, wherein the model is a neural network.

14. 1. An imaging device, comprising: a selection means for applying the model learned by the information processing device according to claim 1 to a captured image to perform the extraction process for extracting feature amounts, and for selecting a region to be processed from among the plurality of regions by performing the estimation process for each of a plurality of object regions in the captured image based on the feature amounts; An imaging device comprising:

15. An extraction step of performing an extraction process of extracting features by applying a model to an image; an estimation step of performing an estimation process of estimating a likelihood representing a probability that an object in the image is classified into each category based on the feature amount; a first calculation step of calculating a likelihood error for each category based on correct answer information of the image and the likelihood of each category estimated by the estimation process; a second calculation step of calculating an overall likelihood error representing an error in the likelihood of the entire category based on the likelihood error of each category; a learning step of learning the model based on the overall likelihood error; Including, An information processing method characterized in that, in the second calculation step, when calculating the overall likelihood error, the likelihood error of each category is weighted for each target category based on the relationship between the target category and other categories.

16. The computer of the information processing device, An extraction means for performing an extraction process for extracting features by applying a model to an image; an estimation means for performing an estimation process of estimating a likelihood representing a probability that an object in the image is classified into each category based on the feature amount; a first calculation means for calculating a likelihood error for each category based on correct answer information of the image and the likelihood of each category estimated by the estimation process; a second calculation means for calculating an overall likelihood error representing an error in the likelihood of the entire category based on the likelihood error of each category; a learning means for learning the model based on the overall likelihood error; A program for causing the device to function as a The second calculation means is a program characterized in that, when calculating the overall likelihood error, the second calculation means weights the likelihood error of each category for each target category based on a relationship between the target category and other categories.

Images