Information processing device, machine learning model, information processing method, and program

Abstract

To reduce computation cost of a machine learning model designed to perform a recognition task using an image and related information as an input.SOLUTION: An information processing method is provided, comprising: inputting pixel information to a first part of a machine learning model designed to perform recognition processing on a recognition target in a captured image on the basis of the pixel information of the capture image and information on the captured image in addition to the pixel information; and perform recognition processing by inputting corrected information obtained by correcting an output of the first part of the machine learning model using the information on the capture image to a second part of the machine learning model succeeding the first part.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to an information processing device, a machine learning model, an information processing method, and a program. [Background technology]

[0002] Numerous CNNs have been proposed to perform image recognition tasks such as image classification, object detection, or semantic segmentation. Non-Patent Documents 1 and 2 disclose CNNs that perform semantic segmentation. These CNNs take an image as input, extract features using a convolutional layer and a pre-pooling layer, perform bilinear interpolation and upsampling with an inverse convolutional layer, and then output a map of region categories with a resolution equivalent to the input image.

[0003] Furthermore, CNNs that perform recognition processing using information other than images in addition to images have also been proposed. Non-Patent Document 3 discloses a CNN that performs semantic domain segmentation using a depth map in addition to an RGB image as input. Non-Patent Document 4 discloses a CNN that performs action recognition using optical flow images from multiple frames in addition to an RGB image. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Jonathan Long, Evan Shelhamer, Trevor Darrell, “Fully Convolutional Networks for Semantic Segmentation”, CVPR2015, [online], November 14, 2014, [Retrieved August 11, 2021], Internet [Non-Patent Document 2] Olaf Ronneberger, Philipp Fischer, Thomas Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation", MICCAI 2015, [online], May 18, 2015, [Retrieved August 11, 2021], Internet [Non-Patent Document 3] Caner Hazirbasy, Lingni May, Csaba Domokos, and Daniel Cremers, “FuseNet: Incorporating Depth into Semantic Segmentation via Fusion-based CNN Architecture”, ACCV2016, [online], March 10, 2017, [Retrieved August 11, 2021], Internet [Non-Patent Document 4] Karen Simonyan, Andrew Zisserman, “Two-Stream Convolutional Networks for Action Recognition in Videos”, NIPS2014, [online], June 9, 2014, [Retrieved August 11, 2021], Internet [Non-Patent Document 5] Fisher Yu, Dequan Wang, Evan Shelhamer, Trevor Darrell, “Deep Layer Aggregation”, CVPR2018, [online], July 7, 2018, [Retrieved August 11, 2021], Internet [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] However, in the CNNs described in Non-Patent Documents 3 and 4, since they take maps of different modalities as input in addition to RGB images, the computational cost is often higher due to the network structure compared to when only RGB images are input. In the method described in Non-Patent Document 3, the input image is encoded using two branches, one for the RGB image and the other for the depth map, so the computational cost is higher due to the CNN branch that processes the depth map. Also, in the method described in Non-Patent Document 4, two streams, spatial and temporal, are processed by separate CNNs, and the recognition results from each are finally integrated. In this case, one frame of optical flow image input to the temporal stream is decomposed into two axes, the X-axis and Y-axis, to obtain a two-channel image.

[0006] The present invention aims to reduce the computational cost of a machine learning model that performs a recognition task using an image and related information as input. [Means for solving the problem]

[0007] To achieve the objectives of the present invention, for example, an information processing apparatus according to one embodiment has the following configuration. That is, An information processing device for training a machine learning model that performs recognition processing of a recognition target in an captured image, based on pixel information of the captured image and information relating to the captured image in addition to the pixel information, comprising: acquisition means for acquiring second correct answer data that indicates the correct output of the machine learning model for the captured image; creation means for creating first correct answer data that indicates the correct correction information obtained by correcting the output of a first part of the machine learning model that takes the pixel information as input with information relating to the captured image; and learning means for training the machine learning model based on the error between the correction information and the first correct answer data, and the error between the output when the correction information is input to a second part of the machine learning model that follows the first part and the second correct answer data. It is characterized by the following: [Effects of the Invention]

[0008] For machine learning models that perform recognition tasks using images and related information as input, the computational cost can be reduced. [Brief explanation of the drawing]

[0009] [Figure 1] A diagram illustrating an example of an input image, GT, and image recognition processing according to Embodiment 1. [Figure 2] A diagram illustrating an example of the learning mechanism of a CNN according to Embodiment 1. [Figure 3] A diagram showing an example of the functional configuration of the recognition device according to Embodiment 1, and a diagram showing an example of the functional configuration of the learning device. [Figure 4] Flowchart (a) showing an example of the processing by the recognition device according to Embodiment 1, and flowcharts (b), (c) showing examples of the processing by the learning processing. [Figure 5] Diagram showing an example of the functional configuration of the learning device according to Embodiment 2. [Figure 6] Diagram (a) for explaining an example of the learning mechanism of the CNN according to Embodiment 1, and diagrams (b), (c) showing examples of a network that repeatedly aggregates high-dimensional features into low-dimensional features. [Figure 7] Diagram showing an example of the functional configuration of the learning device according to Embodiment 3. [Figure 8] Diagram for explaining an example of the recognition processing in the moving image according to Embodiment 3. [Figure 9] Diagram showing an example of the functional configuration of the recognition device according to Embodiment 3. [Figure 10] Diagram showing an example of the recognition processing including the assignment processing according to Embodiment 3. [Figure 11] Diagram showing the hardware configuration of the computer according to Embodiment 4.

Mode for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0011] [Embodiment 1] In one embodiment, the recognition device 1000 and the learning device 2000, as information processing devices, recognize objects to be recognized in input data using a machine learning model. In this embodiment, image recognition processing is performed by semantic domain segmentation using a convolutional neural network (CNN), with captured images and information related to those captured images as input data. Here, the learning device 2000 trains the machine learning model, and the recognition device 1000 performs recognition processing using the training results. The recognition device and the learning device may be implemented in the same device or as separate devices.

[0012] Figure 1 is a schematic diagram illustrating the image recognition processing performed by the recognition device 1000. The input image 101 shown in Figure 1(a) is an example of image data input to the recognition device 1000 according to this embodiment. Here, the input image 101 is assumed to be an RGB image, but the format, such as the color space, is not particularly limited as long as image recognition processing can be performed, such as in CMYK format.

[0013] Furthermore, in the recognition processing performed by the recognition device 1000 and learning device 2000 according to this embodiment, the subject in the captured image is classified into one of the following categories: Plant, Sky, or Other. Here, the input image 101 has a flower (classified as Plant) in the foreground center, and the sky (classified as Sky) and ground (classified as Other) in the background. These are just examples, and the recognition device 1000 and learning device 2000 may classify the subjects into different categories, and different subjects may be used in the input image 101 and the correct answer (GT) 102 described later.

[0014] GT102 shown in Figure 1(b) is an example of a ground truth (GT) corresponding to the input image 101. As described above, in this embodiment, flowers are associated with the Plant category, the sky with the Sky category, and the ground with the Other category. Also, as shown in Figure 1(b), in GT102, a label corresponding to the category is assigned to the region where the target object of each category exists. The label is information indicating the category assigned to each region, and in each figure, the labels assigned as a result of classification (or assigned to the ground truth data) are shown by color coding (a mesh pattern). In this embodiment, as semantic region segmentation, an image recognition task is performed in which the region in the input image is divided into sub-regions according to specific categories, as in GT102.

[0015] Figure 1(c) shows an example of the input / output of the CNN 103 provided in the recognition device 1000 according to this embodiment. The computation mechanism of the CNN 103 according to this embodiment will be described below.

[0016] CNN103 has a hierarchical structure in which multiple modules, each consisting of layers that perform convolution, activation, pooling, and normalization, are linked together. It takes an input image 101 as input and outputs an inference result 110, which is the result of categorizing the image. As shown in Non-Patent Document 1 or 2, CNN103 can output the inference result 110 by upsampling the intermediate features of higher-order layers to match the output size, thereby aligning the sizes of the intermediate features from lower-order to higher-order layers, and by utilizing 1x1 convolution.

[0017] Here, CNN103 is described in two parts: CNN104, which performs the preceding processing, and CNN108, which performs the subsequent processing. CNN103 also includes an input terminal 105 that accepts side information. In this embodiment, side information refers to information about the image that affects its pixel values, and is input to the intermediate layer of the machine learning model (CNN103) in addition to the input image.

[0018] By performing an image recognition task using side information in addition to the image as input to a machine learning model, it becomes possible to obtain output based on information different from the appearance of the image. Side information may be, for example, imaging parameters of the imaging device that captures the input image, or values ​​calculated from the input image. Examples of side information include white balance (WB) coefficient, motion vector, automatic exposure evaluation value Brightness value (Bv), subject distance from the imaging device, aperture value, or focal length. The following describes an example using Bv as side information, but it is not limited to this, and any side information that affects the pixel value of the image may be used. Side information may be a scalar value, a one-dimensional vector, a two-dimensional vector, or any format that can be processed. In this embodiment, CNN103 is trained so that the correction information obtained by correcting the output of the intermediate layer of CNN103 with side information is output as a side map, which is a map of the side information. A detailed explanation of the side map and the side map GT, which is the GT of the side map, will be described later.

[0019] In this embodiment, the output of CNN104, i.e., the output of the intermediate layer of CNN103, is corrected using side information. The intermediate layer 106 is an example of the output of the intermediate layer corrected in this way. The recognition device 1000, as an information processing device according to this embodiment, adds an activation layer to any channel of the intermediate layer 106 and obtains a GT for the output of that activation layer. Next, the recognition device 1000 calculates the loss between the output of the activation layer and the GT, and can train the CNN so that the output of the intermediate layer 106 corresponds to the GT. Here, channel 107 is one of the channels of the output of the intermediate layer 106 and is the channel for estimating the side map. The intermediate layer 106 has multiple channels with the same resolution as the input after upsampling, but this resolution may differ from that of the input image.

[0020] The output of each channel, including channel 107, is input to CNN108. The output layer 109 outputs the inference result 110 using a 1x1 convolution and an activation layer. Here, the inference result 110 is assumed to have three normalized channels with the same height and width as the input image 101, corresponding to the likelihoods of the Plant, Sky, and Other categories, respectively. That is, in these three channels, the sum of the likelihoods of the Plant, Sky, and Other categories at the same position is 1.0, and each value is assumed to be a real number in the range [0,1]. The final activation layer of the output layer 109 may use a softmax function. Furthermore, any activation layer commonly used in CNN network configurations can be used for the activation layer of CNN103, such as ReLU (Rectified linear unit, ramp function) or Leaky ReLU.

[0021] Figure 2 is a schematic diagram illustrating the learning mechanism in the learning device as an information processing device in this embodiment. Input image 201 is the same image as input image 101 and is input to CNN 203. CNN 203 is a CNN with the same configuration as CNN 103, and comprises CNN 204 for processing the preceding stage, an input terminal 205 for receiving side information input, an intermediate layer 206, CNN 208 for processing the subsequent stage, and an output layer 209.

[0022] Output 202 is an example of the output result of CNN203, and is the result of categorical classification for the input image 201, similar to the inference result 110 in Figure 1. GT211 is the ground truth data corresponding to the input image, similar to GT102 in Figure 1. Output 210 is an example of the output of the hidden layer 206 via a predetermined activation layer, relating to the response of one channel of the hidden layer 206. Output 210 is the output of a channel that has been pre-trained to estimate a sidemap, and GT212 is the GT of the sidemap corresponding to output 210. The learning device 2000 calculates the loss 213 for output 202 and output 210 with respect to the ground truth data (GT211 and GT212, respectively). Here, the loss 213 is calculated using cross-entropy.

[0023] During the single update process in training, backpropagation is performed based on the loss calculated by the loss function, and the updated weights and biases of each layer are calculated and updated. In this example, the GT212 is obtained for the response of one channel of the 206 hidden layers, and the loss is calculated, thereby training for that one channel of the hidden layer. This training process is not limited to one channel; corresponding GTs may be prepared and training performed for multiple channels of the 106 hidden layers.

[0024] Figure 3(a) is a block diagram showing an example of the functional configuration of a recognition device as an information processing device according to this embodiment. The recognition device 3000 performs runtime processing of the CNN 103 described above and has an image acquisition unit 3001, a side acquisition unit 3002, an estimation unit 3003, and a dictionary storage unit 3004 for this purpose. Figure 3(b) is a block diagram showing an example of the functional configuration of a learning device as an information processing device according to this embodiment. The learning device 3100 performs processing in the learning mechanism shown in Figure 2. The learning device 3100 includes a learning storage unit 3101, a data acquisition unit 3102, a GT creation unit 3103, an estimation unit 3104, a loss calculation unit 3105, an update unit 3106, and a dictionary storage unit 3107 as storage units for each data. The functions of each block will be explained in the flowchart of Figure 4.

[0025] Figure 4 is a flowchart illustrating an example of the processing performed by the recognition device 3000 and learning device 3100 according to this embodiment. Figure 4(a) shows an example of the processing performed by the recognition device 3000 during the runtime of the CNN 103 described above. In S4001, the dictionary storage unit 3004 sets the dictionary to be used by the estimation unit 3003. Here, the dictionary is described below as representing parameters such as weights and biases used in each layer of the CNN. That is, in S4001, the weights and biases of each layer of the convolutional neural network used by the estimation unit 3003 are loaded.

[0026] In S4002, the image acquisition unit 3001 acquires an image for recognition processing (i.e., the input image 1001). The image acquisition unit 3001 resizes the input image 1001 to match the input size of the CNN 103 and further preprocesses each pixel as necessary. For example, as preprocessing for each pixel, the image acquisition unit 3001 may subtract the average RGB value of a previously acquired image set from the RGB channel of each pixel of the input image, or it may perform any other processing depending on the environment. In the following explanation, the image data transformed by such preprocessing will also be referred to as the input image.

[0027] In S4003, the side information acquisition unit 3002 acquires side information to be input to the intermediate layer of the CNN. The side information in this embodiment is Bv, as described above, and here it is assumed to be a scalar value. Bv is information usable within the camera, calculated based on brightness information detected by the photometering sensor in the camera. Hereafter, the output corrected using the side information will be collectively referred to as the Bv map.

[0028] In S4004, the estimation unit 3003 recognizes the object to be recognized in the input data using a machine learning model having a hierarchical structure consisting of multiple layers. In this embodiment, the estimation unit 3003 recognizes the category of each pixel in the input image. That is, the processing in S4004 is a feedforward processing by CNN103, first the preceding feedforward processing by CNN104 is performed, then side information is input to the intermediate layer, and the output of the intermediate layer 106 is obtained. In this embodiment, as described above, the sidemap is estimated in one channel of the intermediate layer output.

[0029] Here, the side information is input as the bias of the convolutional layer, but this is just one example, and the side information may be used in any way as long as the final output can be obtained using the side information input to the hidden layer. For example, the estimation unit 3003 may calculate the Bv map by multiplying the elements at the corresponding positions when the side information is the same size as the output of the hidden layer. The estimation unit 3003 may also perform preprocessing on the side information before performing the convolution calculation to calculate the Bv map. Here, the estimation unit 3003 can perform a 1x1 convolution on the side information as preprocessing, and then normalize it. Here, the weights and biases used in the 1x1 convolution, and the parameters used in normalization are assumed to be learned and recorded during training.

[0030] Furthermore, in cases where the number of features obtained in the preceding forward propagation process is almost zero (such as an image that is entirely gray), the final output may depend heavily on the side information. To address such cases, by setting the channel to which side information is added as a bias to only a portion of the whole (one channel in this example) and providing a channel to which no side information is added at all, the dependence on side information in special cases can be reduced.

[0031] After obtaining an output including channel 107 of the Bv map, the estimation unit 3003 takes the Bv map as input to the CNN 108 and performs a forward propagation process up to the output layer 109 to obtain the inference result 110. In this CNN 108 process, feature extraction for region category determination is performed based on both the features extracted from the image pixel information and the Bv map, and the region category determination is performed in the output layer 109 using the extracted features.

[0032] A Bv map corrected using Bv is a map that reflects the absolute value of the absolute light intensity in each region of the image. Therefore, by performing inference processing using Bv, it is possible to perform recognition processing of the object to be recognized using both the appearance information of the RGB image and the light intensity of each region. With such processing, for example, when classifying the sky region of a cloudy point outdoors (Sky region, white, high Bv) and a white wall surface indoors (Other region, white, low Bv), the accuracy of classification can be improved by referring to side information.

[0033] The above describes the runtime processing. Next, we will explain the training process with reference to the flowchart in Figure 4(b).

[0034] In S4101, the learning memory unit 3101 sets the parameters (weights and biases) for each layer of the CNN. If pre-trained parameters exist for each layer of the CNN, the learning memory unit 3101 may set the parameters for each layer to the pre-trained parameters instead of initializing them. In addition, the learning memory unit 3101 sets hyperparameters related to learning. The parameters set here are common CNN parameters such as mini-batch size, learning rate, or parameters for the stochastic gradient descent solver, and a detailed explanation of the setting process is omitted.

[0035] In S4102, the data acquisition unit 3102 acquires training data. Here, the data acquisition unit 3102 can acquire training data from the learning memory unit 3101, which functions as a storage device. To this end, the learning memory unit 3101 can store training images and side information in association with their corresponding GTs. The data acquisition unit 3102 may also perform preprocessing on each image, such as augmentation processing like random cropping or color conversion, or normalization.

[0036] In S4103, the GT creation unit 3103 creates a side map GT based on the side information acquired in S4102. Hereinafter, an example of the process of creating a side map GT using the side information Bv and the RAW image will be described.

[0037] The GT creation unit 3103 obtains Bv(i) for each pixel by correcting the pixel value of the RAW image with Bv based on the following formula (1). L (i) =0.25·r (i) +0.5g (i) +0.25·b (i) Bv (i) =Bv+log2(L (i) / opt) Formula (1)

[0038] Here, i is the index of the pixel, and r (i) ,g (i) , and b (i) are the pixel values of the R, G, and B channels corresponding to the i-th pixel of the RGB3-channel image obtained by demosaicking the RAW image, respectively. Also, opt is a constant obtained from the reference values of the aperture value, exposure time, and sensitivity of the image sensor, and Bv<000001##>is the Bv of the i-th pixel. The weights of r (i) ,g (i) , and b (i) are an example, and different values may be used.

[0039] The range of Bv can be set arbitrarily. Generally, Bv has a range of about -10 to +15, and considering that it has a value of about -5 in a dark indoor environment and about +10 in a bright outdoor environment, the GT creation unit 3103 may clip the effective Bv range according to the recognition target. For example, the GT creation unit 3103 may set the range of Bv to [0,10] for the purpose of improving the classification accuracy between the Sky region (sky, clouds) and the Other region (white wall, etc.) outdoors during the day. Furthermore, the GT creation unit 3103 creates a map with an appropriate range according to the application, such as [0,1] or [0,4], as the side map to be learned in the intermediate layer.

[0040] The projection of Bv values ​​onto map values ​​when creating a Bv map is not particularly limited, and any effective transformation can be selected. The GT creation unit 3103 may select an effective transformation method from, for example, linear transformations or nonlinear transformations (polynomial functions, sigmoid functions, logarithmic functions), or it may combine these transformation methods, and it may perform these transformations once or multiple times.

[0041] By creating a sidemap GT in this way, it becomes possible to improve the accuracy of classification when the side information of a region sample in a certain category is concentrated in a specific range. In this embodiment, the range of Bv is [0,10] and the range of map values ​​is [0,1], and projection onto the map is performed by a linear transformation. In this case, the sidemap GT is 0 when the value of Bv is 0 or less, and takes the value of 1 when the value of Bv is 10.

[0042] In S4104, the estimation unit 3104 performs category recognition of images within a minibatch using the forward propagation process of CNN203. This process is performed in the same way as in S4004, so a redundant explanation is omitted.

[0043] In S4105, the loss calculation unit 3105 calculates the loss based on a predetermined loss function from the forward propagation output, which is the target of training for CNN203, and its corresponding GT. The loss calculation unit 3105 uses the output 210 of one channel of the hidden layer 206 (hereinafter referred to as "response" as appropriate) and the final network output 202 as the forward propagation output. The GT corresponding to output 210 is the sidemap GT212, and the GT corresponding to output 202 is the GT102 for each category. Output 202 is the output of three channels corresponding to Plant, Sky, and Other, and the GT for each corresponding category is also data with three channels. The number of channels in the sidemap GT212 is one channel, the same as the Bv map and output 210. In this embodiment, the loss calculation unit 3105 calculates the cross-entropy loss for each specific domain GT and each category GT from these output-GT pairs, and adds the two calculated cross-entropy losses together with appropriate weights. By increasing the weighting of the side map GT, the influence of side information on perception can be greatly increased, and this weighting will be set arbitrarily by the user.

[0044] In S4106, the update unit 3106 updates the parameters of the CNN. In this embodiment, the update unit 3106 calculates the update amount for the weights and biases of each layer of the CNN by backpropagation using the overall loss calculated in S4105, and updates them accordingly. The updated weight and bias values ​​are stored in the dictionary storage unit 3107.

[0045] S4102 to S4106 is a loop process (L4001) that is repeated until the loss calculated in S4105 converges sufficiently. Here, a threshold used to determine whether the loss has converged sufficiently is set in advance as desired, and it is determined whether the loss is below this threshold or not. If it is determined that the loss has converged sufficiently, the loop process ends; otherwise, the process returns to step S4102.

[0046] This process allows the CNN to be trained to estimate a Bv map on a specific output channel of the hidden layer by inputting an RGB image as its input and side information (Bv) into the hidden layer. This enables inference using side information to be performed within the CNN at a lower computational cost than when both the RGB image and the Bv map are input to the CNN's input layer.

[0047] In this embodiment, image recognition processing is described as being performed by semantic region segmentation, but the type of image recognition processing is not limited to this. For example, as a recognition task similar to semantic region segmentation, image recognition processing may be performed to estimate the ratio of region labels within the corresponding block of the input image for each pixel of the output map. In this case, the output map has a lower resolution than the input image, one pixel of the output map corresponds to a block consisting of multiple pixels of the input image, and the ratio of region labels can be the ratio of region label pixels within that block. For example, when a VGA image (640×480) is input and an 80×60 map is output, one pixel of the output map corresponds to a block consisting of 8×8 pixels of the input image, and the ratio of region labels is the ratio of region label pixels within that 8×8 block. For example, if 32 pixels in the block of the input image corresponding to a certain output pixel belong to the Sky category, the Sky ratio of that output pixel will be 0.5.

[0048] For example, the learning device 3100 according to this embodiment can perform image recognition accuracy evaluation by setting appropriate evaluation metrics for known image classification techniques or object detection techniques instead of semantic segmentation or similar tasks, and can similarly perform learning using side information. When using object detection techniques, post-processing such as coordinate regression by a fully connected layer or non-maximum suppression is performed after the output of the map of the final inference result 110. Even in this case, the process of learning to estimate side maps in predetermined channels of the intermediate layer can be performed similarly. Therefore, even when using different recognition tasks, recognition accuracy can be improved with less computational cost by inputting side information into the intermediate layer and performing inference based on the side information at the output of the intermediate layer of the CNN.

[0049] [Embodiment 2] The recognition and learning apparatus according to Embodiment 1 achieves an effect similar to that of inputting an RGB-Bv image to the input layer, but with low computational cost, by using Bv as side information and training one channel of the intermediate layer of the CNN to estimate the Bv map. The GT used for learning Bv map estimation could be created using a pre-set creation method that takes into account the characteristics of the recognition target. Here, it is conceivable that the optimal selection of parameters used to create the side map GT will change depending on the characteristics and state of the recognition target. In view of this, the information processing apparatus according to this embodiment prepares verification data and searches for parameters to be used to create the side map GT so as to optimize the estimation accuracy with respect to the verification data (for example, by grid search). The network configuration used for the recognition and learning processes of the CNN according to this embodiment is the same as that of Embodiment 1, so redundant explanations will be omitted.

[0050] Figure 5 is a block diagram showing an example of the functional configuration of the learning device 5000 according to this embodiment. The learning device 5000 has the same configuration as the learning device 3100 of Embodiment 1, except that it additionally has a verification storage unit 5001 and a selection unit 5002.

[0051] Figure 4(c) is a flowchart showing an example of a parameter selection process performed in addition to the process shown in Figure 4(b) in the learning process according to this embodiment. In the process shown in Figure 4(c), a grid search loop is performed to select the parameters to be used when creating the side map GT.

[0052] In S4201, the selection unit 5002 selects one parameter related to the creation of the side map GT as a parameter to be used. Here, the selection unit 5002 can select a parameter to be used from parameters of a type / range defined in the search space explored by grid search. In this embodiment, the parameter is selected using the lower or upper limit of Bv, the lower or upper limit of the map, the projection function (linear or sigmoid function), positive or negative (positive map or negative map), or learning on / off for each output channel of the hidden layer as the search space. Here, learning on / off for each output channel of the hidden layer is a setting that switches whether or not to learn the side map for each output channel of the hidden layer that is learned to output a side map. This learning on / off may be a discrete switching setting or a continuous setting. A continuous setting may be, for example, setting the reflection rate of the side map with a real value in the range [0,1] for each output channel, and setting it so that the closer to 1, the higher the learning rate of the side map.

[0053] The selection unit 5002 does not need to search the entire search space described above; it may select only some parameters, or it may set a different search range. For example, the selection unit 5002 may fix the output channel of the intermediate layer that outputs the side map to one channel, fix the projection function to be linear, and further fix the map range to [0,4], and then perform selection on other parameters. In that case, the search space is narrowed down to three dimensions (lower bound of Bv, upper bound of Bv, positive / negative), which makes it possible to speed up the selection process. In the processing of S4201, the selection unit 5002 selects the parameters corresponding to the grid of the search space as the parameters to be used when creating the GT.

[0054] In S4202, the learning device 5000 performs CNN training using the parameters selected in S4201. The training process performed in S4202 is the same as the flowchart in Figure 4(b), except that the parameters selected in S4201 are used.

[0055] In S4203, the selection unit 5002 uses the validation data to evaluate the recognition accuracy of the recognition target by the CNN trained in S4202. For example, the selection unit 5002 can calculate the error in the output using the input image and its GT included in the validation data, and evaluate the recognition accuracy using the sum of the errors calculated from each validation data as an indicator. To this end, the validation storage unit 5001 can store multiple sets of images to be input to the CNN and their output GTs as validation data.

[0056] In S4204, the selection unit 5002 determines whether the selection of all parameters to be used has been completed. Here, the selection unit 5002 can determine whether the selection is complete based on whether processing has been completed for all grids in the search space. If the selection is complete, the process is terminated; otherwise, the process returns to S4201.

[0057] If processing is completed in S4204, the selection unit 5002 compares the recognition accuracy evaluated in S4203 for each usage parameter, identifies the one with the highest recognition accuracy, and selects it as the final usage parameter. By using the parameter identified here at runtime, it becomes possible to perform recognition processing using the optimal parameter.

[0058] In this embodiment, an example of optimization using grid search has been described, but the method is not limited to this method as long as it allows for optimization of the parameters used, and any known method may be used. For example, the selection unit 5002 can use a different method that performs optimization using a search space, such as a genetic algorithm or a simplex method, instead of grid search.

[0059] [Embodiment 3] In Embodiment 1, the side information was described assuming that it is basically a scalar value, but as mentioned above, it is not limited to scalar values. In this embodiment, the processing performed when the side information is not a scalar value will be described in detail.

[0060] The side information may be, for example, a one-dimensional vector or a two-dimensional vector. If the side information is a two-dimensional vector map, it may have a lower resolution than the input image. Also, side maps GT may be prepared from multiple side information sources, and all corresponding side maps may be estimated simultaneously in the intermediate layer. The depth map as side information does not need to have a lower resolution than the original image; for example, it may be distance information (scalar value) to the focused subject measured using a distance measuring sensor such as that of a single-lens reflex camera.

[0061] In this embodiment, we will describe an example in which the subject distance is used along with Bv as side information. Here, a depth map with a lower resolution than the input image is set as information indicating the subject distance, and the recognition device estimates a depth map with the same resolution as the input image as a side map, which is used for region category discrimination.

[0062] Figure 6(a) is a schematic diagram of the network used to illustrate the recognition process performed by the recognition device according to this embodiment. Here, the basic recognition process can be performed in the same way as shown in Figure 1(c), so redundant explanations are omitted.

[0063] CNN603 in Figure 6 consists of CNN604, input terminal 605, hidden layer 606, CNN609, and output layer 610. In this example, the same processing as in Figure 1(c) is performed, except that a depth map (subject distance) is input to input terminal 605 in addition to the Bv map, and a depth map is estimated in addition to the Bv map in channel 608 of the output of hidden layer 606.

[0064] Figure 7 shows an example of the CNN network configuration during training according to this embodiment. In Figure 7, in addition to the network configuration in Figure 2, a depth map is additionally input as side information to the input terminal 705 (corresponding to the input terminal 205), and the depth map is estimated as a side map 708 along with the Bv map at the output 707 of the hidden layer 706. Furthermore, the errors between the outputs 711 and 712 from the activation layer of the side map 708 and their GT714 and 715 are calculated, and the final training process is performed using the error between the output of the final activation layer 710 and GT713. This is the configuration in Figure 2 with the addition of the output 712 and depth map GT715 corresponding to the depth map.

[0065] The recognition process performed by the recognition device 3000 according to this embodiment is basically the same as that shown in Figure 4(a) of Embodiment 1. The differences from the process in Embodiment 1 will be explained below with reference to Figure 4(a). Processes S4001 to S4002 are performed in the same way as in Embodiment 1.

[0066] In S4003, the side acquisition unit 3002 acquires side information. In this embodiment, the side acquisition unit 3002 acquires multiple pieces of side information (here, Bv and subject distance). Here, Bv is acquired as a scalar value, and the depth map indicating the subject distance is acquired as a two-dimensional vector.

[0067] Here, we will explain how the side acquisition unit 3002 acquires a depth map. The side acquisition unit 3002 may, for example, acquire the subject distance using contrast AF (autofocus) and use that as the depth map. When using inexpensive digital still cameras that do not have a distance measuring sensor, such as compact cameras, automatic focusing may be performed by measuring the contrast value that changes in conjunction with the position of the focus lens and searching for the peak of the contrast value. Here, we will call this type of automatic focusing contrast AF. In contrast AF, the contrast value is measured for each block on the image, and the focus lens is moved in the direction where the contrast value is large to search for the peak (also called the hill-climbing method). When the peak of the contrast value is found, the search is terminated there.

[0068] Alternatively, for example, the side acquisition unit 3002 may acquire the subject distance using image plane phase-detection AF and use it as a depth map. Image plane phase-detection AF is an autofocus system that automatically focuses using the amount of focus shift detected by sparsely arranged phase-detection elements on the image sensor. Since this amount of focus shift can be converted into distance, a sparse depth map can be acquired. Image plane phase-detection AF is used, for example, in interchangeable-lens cameras such as SLR cameras or mirrorless cameras. These are just examples, and other known methods may be used to acquire the depth map.

[0069] In S4004, the estimation unit 3003 recognizes the object to be recognized in the input data using a machine learning model having a hierarchical structure consisting of multiple layers. In the processing of S4004 according to this embodiment, as described above, in addition to Bv, a depth map is input to the intermediate layer as side information, and side maps based on each are estimated.

[0070] Figure 6(b) shows an example of a network structure that iteratively aggregates high-dimensional features into low-dimensional features. The configuration of the CNN604, input terminal 605, hidden layer 606, and channel 608 constituting the CNN603 according to this embodiment may be, for example, the configuration shown in Figure 6(b). This configuration is used, for example, in Non-Patent Document 5, and makes it possible to obtain feature maps with higher resolution.

[0071] In Figure 6(b), Down sample is a process that reduces the resolution by pooling, etc. Up sample is a process that increases the resolution by bilinear interpolation, etc., and Keep resolution is a process that does not change the resolution. Sum represents the element-wise sum of the feature map. Here, 621 represents the input of side information, which is a scalar value or a one-dimensional vector. When the side information is a scalar value or a one-dimensional vector, it is processed using weights and biases as in Embodiment 1 and input into the feature map of the hidden layer. Here, when the side information is a one-dimensional vector, the weights are a matrix (input dimension × feature dimension), and the bias is a vector of the feature dimension. These weights and biases are also learned during CNN training, just like other CNN parameters.

[0072] Furthermore, 622 represents the input of side information, which is a low-resolution 2D map. Here, side information, which is a 2D vector, is input to a feature map with a resolution downsampled to 1 / 16. Figure 6(c) is a diagram illustrating an example of inputting this 2D vector side information. 623 is the feature map, which is assumed to have a resolution of 1 / 16 of the original resolution of the image. 624 is the 2D vector side information, and 625 represents the operation when combining 623 and 624. As an operation for this combination, for example, the element at the corresponding position of the side information is added to or multiplied for a specific channel of the feature map. Alternatively, as an operation for combination, the side information may be concatenated in the channel direction of the feature map. Similar to the side information in Embodiment 1, this 2D vector side information may also undergo preprocessing such as processing using weights or biases, or normalization. 266 is the feature map after the above combination processing.

[0073] Through this process, a sidemap is estimated using the output of a specific channel in the intermediate layer 607 of CNN603.

[0074] In S4004, the estimation unit 3003 performs the final task of determining the region category based on the image's pixel information and image features derived from the Bv map and depth map. In an image where a white wall (Other) is located close to the camera and a white cloudy sky (Sky) is in the background, the depth map shows that the wall is nearby and the cloudy sky is at infinity. By considering such cases and using the depth map for training, it is possible to improve the classification accuracy of recognition objects that have similar features based on pixel information but different subject distances. Furthermore, by using the Bv map in addition to the depth map for training, it is possible to further improve classification accuracy by using the light intensity of each region as a judgment criterion.

[0075] The above describes the runtime processing, and next we will explain the training process. The training process is basically the same as the process shown in Figure 4(b) of Embodiment 1, so we will omit any redundant explanations.

[0076] In steps S4102 to S4103 of this embodiment, a side map GT is created in the same manner as in Embodiment 1. In this example, a side map GT is created for both Bv and the subject distance. As the depth map GT, a depth map with a resolution (higher than the side information) that is somewhat close to the resolution of the input image may be prepared. This high-resolution depth map can be acquired by any method, such as by the stereo method or by using a TOF sensor.

[0077] By using such sidemap GT for training, it becomes possible to train a CNN that performs the final recognition task using a feature map obtained by the CNN and a 2D depth map (lower resolution than the original image) as input to the input image.

[0078] As mentioned above, side information is not limited to Bv or subject distance. For example, the aperture value or focal length (one-dimensional vector), or both, of the lens may be used as side information to estimate a defocus map (map of the amount of blur) as a side map. The GT of the defocus map can be acquired, for example, using a camera with image plane phase-detection AF where phase-detection elements are densely arranged. By training the model to estimate the defocus map in the intermediate layer, it becomes possible to improve recognition accuracy by also considering the amount of blur for each region. Therefore, it is expected to be effective in cases where the pixel characteristics are similar but the amount of blur differs, such as classifying blurred green plant leaves (Plant, high blur amount) and flat green artificial objects (Other, low blur amount) in macro imaging.

[0079] Alternatively, for example, the white balance processing coefficient (WB coefficient) may be used as side information to estimate the RGB values ​​before white balance processing as a side map. This can be achieved by having the intermediate layer 606 learn to recalculate the RGB values ​​before white balance processing for each region based on the pixel features extracted by CNN604 and the WB coefficient. With such a configuration, recognition processing can be performed based on both the pixel values ​​of the input image whose influence of illumination color has been reduced by white balance processing, and the RGB values ​​before white balance processing, i.e., the pixel values ​​whose influence of illumination color is strong. Therefore, even in images that have been converted to abnormal colors due to incorrect white balance processing that removes the color of the light source, the possibility of failure in region category determination can be reduced.

[0080] [Embodiment 4] In Embodiments 1 to 3, the image input to the CNN was described as a single still image. In this embodiment, we will describe the case where tracking of a recognition target is performed in a moving image composed of multiple images that are sequential in time.

[0081] The recognition and learning devices according to this embodiment can perform the same processing as in Embodiment 1 on each of the multiple images input to the CNN, for example, as shown in Figures 4(a) to 4(b). Here, as the side information according to this embodiment, motion vectors created by motion compensation in video compression can be used. The following explanation will use motion vectors as the side information and optical flow as the side map.

[0082] Figure 8 is a diagram illustrating the recognition process performed by the recognition device according to this embodiment. In the example in Figure 8, a frame (image) at time t is input to the CNN802 from a video, and a heatmap showing the probability of existence for each position of the tracked object at that time, and the bounding box size of the tracked object are output. At the same time, a heatmap showing the probability of existence for each position of the tracked object at time t+1, which follows time t, and the bounding box size of the tracked object are also output.

[0083] In Figure 8, the input image 801 is a frame at time t included in the video. CNN802 consists of CNN803, an input terminal for motion vectors, a hidden layer 806, CNN809, and an output layer 810. The basic network configuration, excluding parameters, is the same as that in Figure 1(c) or Figure 6(a). In this embodiment, convolutional layers with recursive connections may be included in CNN803, hidden layer 806, and CNN809. In that case, past time-series information is featured and reflected in the tracking and estimation process, which is expected to improve the accuracy of optical flow estimation.

[0084] In the example in Figure 8, side information 804 is a motion vector. Here, the block size for motion estimation can be set to any size (e.g., 16x16 or 8x8), but the setting will vary depending on the compression method or compression ratio of the video. When input to the input terminal, side information 804 is appropriately resized so that a motion vector with uniform resolution is input to CNN802. In this embodiment, the motion vector for one frame at time t is set to be estimated using the image at time t and the image at time t-1 which is temporally continuous with time t. However, if a corresponding motion vector can be set at each time, there is no need to limit the process to this, and for example, the motion vector estimated from the image at time t and the image at time t+1 may be used as the motion vector at time t.

[0085] 807 is the output channel of the intermediate layer 806, and 808 is the sidemap. In the example in Figure 8, the sidemap 808 is assumed to be the optical flow and has a higher resolution than the motion vector. In this embodiment, the recognition device 9000 is trained to estimate the optical flow at time t and time t+1 as the GT using the motion vectors from images at time t and time t-1. With such a configuration, it is possible to provide a recognition device that is trained to predict future motion using side information.

[0086] CNN809 takes the output of each channel of the hidden layer 806 as input and outputs information for estimating and predicting the heatmap and bounding box size described above. The output layer 810 consists of a 1x1 convolutional layer and an activation layer with the required number of output channels, and outputs outputs 811 and 812.

[0087] Outputs 811 and 812 correspond to time t and time t+1, respectively. Outputs 811 and 812 each include a heatmap and a map showing the estimated size of the bounding box in both the X-axis and Y-axis directions for each time point. In other words, in this example, these outputs are output as three-channel maps for each time point.

[0088] Here, post-processing such as NMS is performed on the heatmap to detect peaks, and the position of these peaks is set as the center position of the bounding box. Next, the size of the bounding box (in this case, width and height) is obtained by reading values ​​near the peak position from the bounding box size map. Through this process, the coordinates (X,Y) of the bounding box indicating the tracking target, as well as its width and height, are determined. In the tracking process according to this embodiment, an ID is assigned to each tracking target, and this process will be described later as runtime processing, referring to the flowchart in Figure 10.

[0089] Figure 9 is a block diagram showing an example of the functional configuration of the recognition device 9000 according to this embodiment. The recognition device 9000 has the same configuration as the recognition device 3000 in Figure 3, except that it additionally has an allocation unit 9001 and a result storage unit 9002, so redundant explanations will be omitted. The processing performed by these functional units will be explained with reference to the flowchart in Figure 10.

[0090] Figure 10 is a flowchart showing an example of the processing performed by the recognition device 9000 according to this embodiment at runtime. In S10001, the dictionary storage unit 3004 sets the dictionary to be used by the estimation unit 3003, in the same manner as in S4001 of Embodiment 1. In S10002, the image acquisition unit 3001 acquires an image for recognition processing, in the same manner as in S4002. Here, an image at a certain time t (1 ≤ t ≤ T) is acquired.

[0091] In S10003, the side acquisition unit 3002 acquires motion vectors, which are side information. These motion vectors are assumed to be of lower resolution than the images acquired in S10002, as described above, and to be resized to an appropriate size for input to the intermediate layers of the CNN. Furthermore, for the image at time t, the motion vectors are calculated from the frame images at times t-1 and t.

[0092] In S10004, the estimation unit 3003 estimates and recognizes the recognition target in the input data, similar to the processing in S4004. Here, the estimation unit 3003 estimates the optical flow calculated from the images at time t and t+1 as a side map in the output of the intermediate layer, and outputs the heat map and bounding size box at time t and time t+1 as maps. The estimation unit 3003 also determines the parameters of the bounding box (center coordinates (X,Y), width, and height) as explained in Figure 8 for each tracking target, and stores these results in the result storage unit 9002.

[0093] In S10005, the assignment unit 9001 assigns a person ID to each tracking target. To do this, the assignment unit 9001 first reads the bounding box estimation result from the result storage unit 9002 at the previous time and creates an affinity matrix between it and the bounding box estimation result at the current time. The similarity of the estimation results evaluated here may be calculated using Intersection over Union (IoU), the Euclidean distance of the bounding box parameters, or any other evaluation method. IoU is an evaluation index that represents the overlap between bounding boxes; the closer to 1, the higher the similarity, and the closer to 0, the lower the similarity, and this is called the score matrix. The Euclidean distance is a value that is small when the similarity is high and large when the similarity is low, and this is called the cost matrix.

[0094] If the number of detected objects at time t is m and the number of detected objects at time t-1 is n, a simple similarity matrix would be an n × m matrix. However, here we will perform calculations using a square matrix that matches the larger of the two values ​​of n and m. In this square matrix, elements that do not originally have a value will be assigned 0 when using the score matrix, or a sufficiently large value when using the cost matrix.

[0095] ID assignment is performed using an appropriate assignment problem algorithm. Here, the assignment unit 9001 may use the Hungarian algorithm to assign IDs. In this case, the assignment unit 9001 seeks the assignment that maximizes the score when using a score matrix, and seeks the assignment that minimizes the cost when using a cost matrix.

[0096] S10002~S10005 is a loop process (L10001) that is repeated until the process is completed for all times t=1...T. If the process is completed for all times, the process ends; otherwise, the process returns to S10002. This process makes it possible to track a person in the video from time 1 to time T.

[0097] In this embodiment, the GT used for training the CNN includes, as described above, a heatmap and bounding box size in addition to the optical flow. The creation of the optical flow GT can be done using any known method for estimating optical flow from video, or a method that generates a highly computationally intensive and dense optical flow, such as Dual TV-L1, may be used.

[0098] In this embodiment, the heatmap's GT is a map created using a two-variable Gaussian function where the center of the human body is the peak and the value at the peak position is 1.0. The bounding box size GT (2 channels) is a map in which the values ​​near this peak position indicate the height or width of the bounding box, and the other values ​​are 0. Here, the center of the bounding box is assumed to coincide with the peak position of the heatmap.

[0099] The peak position of the heatmap can be any position convenient for GT annotation and does not have to be centered on the human body. For example, the peak position could be at the waist or at the center of the head. If the peak position of the heatmap is not centered on the human body, an additional GT for the center position of the bounding box may be prepared and trained. That is, maps for two channels of the bounding box center offset (X-axis and Y-axis) may be added as GT and sidemaps, and the CNN may be trained to output a total of five channels of maps at each time step. The bounding box center offset should be trained to output the offset from that position to the center of the bounding box. In this case, the GT for the bounding box center offset will be a two-channel map where the area around the peak position of the heatmap is a vector from a specific position on the human body to the center of the bounding box, and all other values ​​are zero.

[0100] With this configuration, motion vectors are used as side information to estimate the optical flow at the next time step, and the bounding boxes of the target to be tracked at the current and next time steps can be estimated. Furthermore, by assigning an ID to the bounding box, the target tracking process can be performed. In addition, by estimating a dense optical flow using a CNN based on a sparse optical flow, it is possible to reduce the computational cost compared to existing processes that calculate dense optical flows.

[0101] In this embodiment, the optical flow GT was created using frames at time t and time t+1, and the CNN was trained to estimate heatmaps at time t and time t+1 as its output. This configuration makes it possible to reduce runtime processing latency and improve real-time performance, but different processing may be performed if real-time performance is not required. For example, the optical flow GT could be created using frames at time t-1 and time t, and the CNN could be trained to estimate heatmaps at time t-1 and time t as its output. In this case, a minimum latency of one frame would occur.

[0102] [Embodiment 4] In the embodiments described above, each processing unit shown in Figure 3, for example, may be implemented by dedicated hardware. Alternatively, some or all of the processing units of the recognition device (e.g., 3000) and the learning device (e.g., 3100) may be implemented by a computer. In this embodiment, at least a portion of the processing according to each embodiment described above is performed by a computer.

[0103] Figure 11 shows the basic configuration of a computer. In Figure 11, the processor 1101 is, for example, a CPU, which controls the operation of the entire computer. The memory 1102 is, for example, RAM, which temporarily stores programs and data. The computer-readable storage medium 1103 is, for example, a hard disk or CD-ROM, which stores programs and data long-term. In this embodiment, the programs that realize the functions of each part, stored in the storage medium 1103, are read into the memory 1102. Then, the processor 1101 operates according to the programs in the memory 1102, thereby realizing the functions of each part.

[0104] In Figure 11, the input interface 1104 is an interface for acquiring information from an external device. The output interface 1105 is an interface for outputting information to an external device. The bus 1106 connects the above-mentioned parts and enables data exchange.

[0105] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0106] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0107] 3000: Recognition device, 3001: Image acquisition unit, 3002: Side acquisition unit, 3003: Estimation unit, 3004: Dictionary storage unit, 3100: Learning device, 3101: Learning and storage unit, 3102: Data acquisition unit, 3103: GT creation unit, 3104: Estimation unit, 3105: Loss calculation unit, 3106: Update unit, 3107: Dictionary storage unit,

Claims

1. An information processing device that trains a machine learning model to perform recognition processing of a target to be recognized in an captured image, based on the pixel information of the captured image and information relating to the captured image in addition to the pixel information, Acquisition means for acquiring second ground truth data that shows the correct output of the machine learning model for the captured image, A creation means for creating first correct answer data that shows the correct corrected information obtained by correcting the output of the first part of the machine learning model that takes the pixel information as input with information about the captured image, A learning means for training the machine learning model based on the error between the correction information and the first correct data, and the error between the output when the correction information is input to the second part of the machine learning model following the first part and the second correct data, An information processing device characterized by comprising:

2. The system further includes an evaluation means for evaluating the accuracy of the recognition process when using the set of information relating to the captured image and the first correct data. The information processing apparatus according to claim 1, characterized in that the learning means trains the machine learning model using the set of the plurality of sets that yields the highest accuracy evaluation.

3. The information processing apparatus according to claim 1 or 2, characterized in that the first correct data is the RGB values ​​before white balance processing is applied to the captured image, a defocus map based on aperture value or focal length, a map showing the absolute value of light intensity due to automatic exposure, a depth map based on subject distance, or an optical flow based on motion vector.

4. The information processing apparatus according to claim 3, characterized in that the motion vector is calculated from captured images at a first time and a second time following the first time, and the optical flow is calculated from captured images at the second time and a third time following the second time.

5. An information processing method for training a machine learning model that performs recognition processing of a target to be recognized in an captured image, based on the pixel information of the captured image and information relating to the captured image in addition to the pixel information, A step of obtaining second ground truth data that shows the correct output of the machine learning model for the captured image, A step of creating first correct data that shows the correct correction information obtained by correcting the output of the first part of the machine learning model that takes the pixel information as input with information about the captured image, A step of training the machine learning model based on the error between the correction information and the first correct data, and the error between the output when the correction information is input to the second part of the machine learning model following the first part and the second correct data, An information processing method characterized by comprising:

6. A program for causing a computer to function as an information processing device according to any one of claims 1 to 4.

Images