Information processing device, information processing method, and program

Abstract

To improve the accuracy of multi-task processing on an image.SOLUTION: An information processing apparatus performs transformation processing on time-series images. The information processing apparatus performs a plurality of tasks different in the type of information to be output by using the images after the transformation processing. The information processing apparatus derives, based on a first image of the time-series images, a parameter of the transformation processing on a second image subsequent to the first image of the time-series images. The information processing apparatus performs the transformation processing on the second image by using the derived parameter of the transformation processing on the second image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program, and more particularly to improving the accuracy of a task using an image. [Background technology]

[0002] Various processing techniques using images are known. For example, a computer may perform recognition processing on an image. Here, the purpose of image processing is called a task, and the purpose of recognition processing is called a recognition task. For example, recognition tasks include a face detection task that detects human face areas from an image, and an object category recognition task that determines the category (human, animal, vehicle, etc.) of an object (subject) in an image. Recognition tasks also include a tracking task that searches for and tracks a specific subject, and a scene type recognition task that determines the type of scene (urban, mountainous, coastal, etc.). For example, Non-Patent Document 1 discloses a real-time object detection technology.

[0003] Neural networks (hereafter referred to as "NN") are known as a technology for performing the above-mentioned recognition tasks. NN is an abbreviation for Neural Networks. A deep (large number of layers) multi-layer NN is called a deep NN (DNN). DNN is an abbreviation for Deep Neural Networks. In particular, a deep convolutional NN is called a deep convolutional NN (DCNN). DCNN is an abbreviation for Deep Convolutional Neural Networks. DCNN has been attracting attention in recent years due to its high performance (recognition accuracy, recognition performance).

[0004] DCNN has a network structure in which each layer performs convolution processing on the output from the previous layer and outputs the results to the next layer. The final layer is the output layer, which represents the recognition results. Each layer is equipped with multiple filters (kernels) for convolution operations. Layers close to the output layer can have a fully connected structure like a conventional NN instead of convolutional connections. In the training phase of DCNN, the values ​​of the filters for convolution operations and the connection weights of the fully connected layer (collectively referred to as "learning parameters") are obtained through training using supervised data. Methods such as backpropagation (BP) can be used for training. BP stands for backpropagation. In the recognition phase of DCNN, data is input to a trained DCNN, and each layer sequentially processes the data using the learning parameters obtained through training, and the recognition results are output from the output layer.

[0005] There is a technique called multi-task learning / processing, in which multiple tasks are learned and executed using a single learning means (learner). For example, Non-Patent Document 2 describes a method for learning multiple recognition tasks using a single multi-layer NN with multiple output units for multiple recognition tasks. In Non-Patent Document 2, the same hidden layer is used for each recognition task, and the hidden layer is trained using data from all recognition tasks. Non-Patent Document 2 assumes that the multiple recognition tasks handled in multi-task learning have a certain degree of similarity in the content of the recognition processing. In other words, by training similar recognition tasks using a single learner, common knowledge used in each recognition task can be shared between the recognition tasks, which may improve recognition performance. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] T. -W. Chin et al. "AdaScale: Towards Real-time video object detection using adaptive scaling", Proceedings of the 2nd SysML Conference, 2019. [Non-patent document 2] R. Caruana. "Multitask learning" Machine Learning, 28, 41-75, 1997. [Non-patent document 3] B. Li et al. "High Performance Visual Tracking with Siamese Region Proposal Network", The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8971-8980. [Non-patent document 4] X. Zhou et al. "Objects as Points", arXiv technical report (arXiv 1904.07850). [Non-Patent Document 5] S. Madgwick et al. "Estimation of IMU and MARG orientation using a gradient descent algorithm", 2011 IEEE international conference on rehabilitation robotics, 1-7. [Non-patent document 6] A. Vaswani et al. "Attention Is All You Need", Advances in Neural Information Processing Systems 30(NIPS),2017,pp.5998-6008. Summary of the Invention [Problem to be solved by the invention]

[0007] In multitasking image processing, it is necessary to improve the accuracy of each task.

[0008] An object of the present invention is to improve the accuracy of multitasking processing on images. [Means for solving the problem]

[0009] An information processing device according to an embodiment of the present invention has the following configuration: a conversion means for performing a conversion process on each of the time-series images; a processing means for performing a plurality of tasks using the converted image to output different types of information; a derivation means for deriving, based on a first image of the time-series images, parameters for the conversion process for a second image of the time-series images that comes after the first image; The conversion means performs a conversion process on the second image using the parameters of the conversion process on the second image derived by the derivation means. stomach, the processing means performs the plurality of tasks using a multi-layer neural network; The multilayer neural network includes a first layer that extracts intermediate features from the converted image, a second layer that outputs information indicating a result of a first task of the plurality of tasks from the intermediate features, and a third layer that outputs information indicating a result of a second task of the plurality of tasks from the intermediate features. . [Effects of the Invention]

[0010] The accuracy of multitasking processing on images can be improved. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an information processing apparatus according to an embodiment. [Figure 2] 10 is a flowchart of image conversion and multitasking processing according to an embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the configuration of a learning device according to an embodiment. [Figure 4] 1 is a flowchart of a method for generating training data in one embodiment. [Figure 5]1 is a flowchart of a learning method according to one embodiment. [Figure 6] FIG. 1 is a diagram showing an example of the configuration of an information processing apparatus according to an embodiment. [Figure 7] 1 is a flowchart of a learning method according to one embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the hardware configuration of an information processing device and a learning device. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] An information processing device according to an embodiment of the present invention performs multiple tasks using time-series images. The types of tasks are not particularly limited, and examples include a task for detecting a specific object, a task for segmenting an area, and a task for detecting an anomaly. The multiple tasks may include a first recognition task for recognizing a first object and a second recognition task for recognizing a second object different from the first object. For ease of understanding, the following describes a camera system, which is an information processing device that performs a first recognition task for tracking an object and a second recognition task for detecting a specific feature included in the object. The object here is a specific person or animal photographed by an electronic camera. The specific feature of the object can be detected as the face of the object. Such a camera system can perform facial recognition of the object or display the area of ​​the detected specific feature on a display or viewfinder attached to the electronic camera.

[0014] Generally, the accuracy of a task depends on the state of the input image. Therefore, the accuracy of the task can be improved by performing preprocessing on the image depending on the task. Examples of preprocessing include cropping, which cuts out a specific portion of the image, and color correction on the image. For example, when performing tracking, the image of the next frame can be cropped based on the position or size of the subject, the subject's movement, or the surrounding conditions. In particular, if the subject is moving, the crop area of ​​the image can be set wide so as not to lose track of the subject. As another method, if there is a brightness bias in the image, such as when the entire image is too dark or too bright, a method such as histogram equalization can be used to eliminate the brightness bias before performing a recognition task on the image.

[0015] On the other hand, in multitasking, where multiple recognition tasks are performed from a single image, image preprocessing to improve the accuracy of a single recognition task may degrade the accuracy of other recognition tasks. For example, to accurately detect a specific part of a subject, an image cropped near the specific part can be used. For example, a smaller crop area may improve the accuracy of detecting the specific part. Thus, when multiple tasks that output different types of information, such as a tracking task and a detection task, are performed simultaneously, a larger crop area may improve the accuracy of the tracking task, and a smaller crop area may improve the accuracy of the detection task. Therefore, when performing a recognition task to track a subject and a recognition task to detect specific parts on a single image, the crop area can be determined to ensure the accuracy of both recognition tasks, which output different types of information.

[0016] As described above, the information processing device uses time-series images. Hereinafter, the acquisition time of a certain frame image will be designated as t=T, and this frame image will be referred to as frame image T. The acquisition time of the frame image T-1 immediately before frame image T will be designated as t=T-1, and the acquisition time of the next frame image T+1 will be designated as t=T+1. The acquisition time of the first frame image is t=1. In the following embodiments, the conversion process used for preprocessing on images is a cropping process, and the center coordinates of the cropping area are designated as (X, Y), and the size of the cropping area is designated as (W, H). The parameters (X, Y, W, H) are estimated as the parameters of the conversion process.

[0017] 1 is a diagram illustrating an example of the configuration of an information processing device 1 according to this embodiment. The information processing device 1 includes a conversion unit 101, a multitasking unit 103, and an estimation unit 109.

[0018] The conversion unit 101 performs a conversion process on each of the images in the time series. For example, the conversion unit 101 converts a frame image t and outputs a converted image 102. In this embodiment, the conversion unit 101 performs image conversion using parameters estimated by the estimation unit 109. For example, when the estimation unit 109 estimates parameters (X, Y, W, H), the conversion unit 101 crops the image according to the position and size indicated by these parameters. Furthermore, the conversion unit 101 can enlarge or reduce the cropped image to a predetermined image size.

[0019] 2 shows an example of the processing procedure for image conversion by the conversion unit 101 in this embodiment. The processing content of the conversion unit 101 will be described below with reference to FIG.

[0020] First, in S201, the conversion unit 101 acquires frame image t. Next, in S202, the conversion unit 101 acquires parameters (X, Y, W, H) indicating the cropping area estimated by the estimation unit 109 using frame image t-1. Note that for frame image 1, the position of the subject that was initially detected can be set as the center of the cropping area. Also, the size of the cropping area can be set as a size obtained by multiplying the size of the detected subject by predetermined parameters.

[0021] In steps S203 to S208, it is determined whether the parameters obtained in step S202 are outliers. That is, the conversion unit 101 determines whether the parameters derived by the estimation unit 109 satisfy a criterion, and if the criterion is not satisfied, corrects the parameters derived by the estimation unit 109 and performs conversion processing using the corrected parameters. Specifically, it is determined whether the size and movement amount of the cropping area according to the parameters obtained in step S202 exceed preset size and movement amount thresholds, and if the thresholds are exceeded, the parameters (X, Y, W, H) can be corrected. However, the correction method is not limited to the one described below, and only some of the following correction methods may be used, or another correction method may be used.

[0022] In S203, the conversion unit 101 determines whether the parameter derived by the estimation unit 109 is below a lower limit value, and if the parameter is below the lower limit value, the conversion unit 101 corrects the parameter to the lower limit value in S204. For example, if the conversion unit 101 determines in S203 that the size of the cropping area obtained in S202 is larger than a preset upper limit value, the conversion unit 101 corrects the size of the cropping area to the upper limit value in S204.

[0023] In S205, the conversion unit 101 determines whether the parameter derived by the estimation unit 109 exceeds an upper limit value, and if the parameter exceeds the upper limit value, corrects the parameter to the upper limit value. For example, if the conversion unit 101 determines in S203 that the size of the cropping area obtained in S202 is smaller than a preset lower limit value, the conversion unit 101 corrects the size of the cropping area to the lower limit value in S206.

[0024] Furthermore, if the conversion unit 101 determines in S207 that the amount of movement of the cropping region obtained in S202 exceeds a preset upper limit, then in S208 the conversion unit 101 corrects the position of the cropping region so that the amount of movement matches the upper limit. For example, the conversion unit 101 can determine whether the amount of movement between the center coordinates of the cropping region used to crop frame image t-1 and the center coordinates of the cropping region according to the parameters obtained in S202 exceeds a threshold. If it does, the center coordinates of the crop in S209 can be set to a point that is a threshold distance away from the center coordinates of the cropping region used to crop frame image t-1 toward the center coordinates of the cropping region according to the parameters obtained in S202.

[0025] In S209, the conversion unit 101 converts the image using the parameters obtained in S202 to S208. The conversion unit 101 performs crop processing on the frame image t in accordance with the parameters (X, Y, W, H). The conversion unit 101 can also enlarge or reduce the cropped image to a predetermined image size. In S210, the conversion unit 101 outputs the converted image (converted image 102). Thereafter, in S211, the multitasking unit 103 performs multitasking processing on the converted image 102 and outputs the results for each task. Furthermore, in S212, the estimation unit 109 derives parameters for the conversion processing on the frame image t+1, as will be described later.

[0026] The multitasking unit 103 performs multiple tasks using the image after the transformation process. In this embodiment, the multitasking unit 103 performs multiple recognition tasks using the transformed image 102 output by the transformation unit 101. The multitasking unit 103 in this embodiment can perform multiple tasks using a multilayer neural network. In this embodiment, the multilayer NN is a convolutional neural network (CNN). The multilayer NN has a first layer that extracts intermediate features from the image after the transformation process. The multilayer NN also has a second layer that outputs information indicating the result of a first task of the multiple tasks from the intermediate features, and a third layer that outputs information indicating the result of a second task of the multiple tasks from the intermediate features. In the example of FIG. 1, the multitasking unit 103 is configured as a multilayer NN and has a first layer, an intermediate layer 104, a second layer, a tracking unit 105, and a third layer, a detection unit 106.

[0027] The intermediate layer 104 extracts intermediate features from the image after the transformation process. The intermediate layer 104 is configured, for example, as a multi-layer CNN. This CNN may not have a layer called a fully connected layer, but may have only a type of layer called a convolutional layer. The intermediate layer 104 can generate an intermediate feature map from the transformed image 102. The intermediate feature map output by the intermediate layer 104 may be a feature map output from the final layer of the intermediate layer, or may be a combination of feature maps output from multiple layers that make up the intermediate layer 104. The intermediate feature map output by the intermediate layer 104 may also be converted into a one-dimensional array by flattening (smoothing) such a feature map.

[0028] The tracking unit 105 tracks the object using the intermediate features generated by the hidden layer 104. The specific configuration of the tracking unit 105 is not particularly limited. For example, as in Non-Patent Document 3, the tracking unit 105 may generate a likelihood map 107 representing the likelihood of the object using a trained multi-layer CNN model based on correlation matching between intermediate features of the object in an initial frame and intermediate features of the object in a current frame. Such likelihood map 107 is information indicating the result of a tracking task, and the object can be identified using this likelihood map 107. The likelihood map 107 may be configured to distinguish between the likelihood of the object to be tracked (represented by a circle in the figure for ease of understanding) and the likelihood of a similar object that is not the object to be tracked (represented by an x ​​in the figure for ease of understanding). Note that the configuration of the tracking unit 105 is not necessarily limited to a NN. For example, the tracking unit 105 may track the object using known techniques such as SVM or AdaBoost.

[0029] The detection unit 106 detects a specific part of the subject by using the intermediate features generated by the intermediate layer 104. The specific configuration of the detection unit 106 is not particularly limited. For example, as in Non-Patent Document 4, the detection unit 106 may output a likelihood map 108 using a multi-layer CNN trained so that the final layer outputs the likelihood of an area where a specific part exists. Such a likelihood map 108 is information indicating the result of the detection task, and the area of ​​the specific part can be detected by using this likelihood map 108. Note that the configuration of the detection unit 106 is not necessarily limited to a NN. For example, the detection unit 106 may detect a specific part by using known techniques such as SVM or AdaBoost.

[0030] The estimation unit 109 derives parameters for a conversion process for a second image, which is a subsequent image in the time series based on a first image. In this embodiment, the estimation unit 109 estimates a crop region (X, Y, W, H) for a frame image t+1 at the next time based on a frame image t. In this embodiment, the estimation unit 109 is configured as a multi-layer NN.

[0031] The estimation unit 109 can derive parameters for the transformation process on the second image using at least one of the first image, intermediate features from the first image, and information indicating the result of the task on the first image. Here, the intermediate features from the first image can be extracted from the first image after the transformation process by the multitasking unit 103 as described above. Furthermore, the information indicating the result of the task on the first image can be information indicating the result of at least one task out of the multiple tasks on the first image obtained by the multitasking unit 103 as described above.

[0032] In this embodiment, the estimation unit 109 uses intermediate features extracted by the multitasking unit 103 from the first image after the transformation process and information indicating the results of each of the multiple tasks performed on the first image by the multitasking unit 103. In this example, the estimation unit 109 derives parameters for the transformation process on the second image using information indicating the tracking result of the object from the first image and information indicating the detection result of a specific part from the first image, in addition to the intermediate feature map extracted from the first image. The information indicating the tracking result of the object from the first image is likelihood map 107 output by the tracking unit 105, and the information indicating the detection result of a specific part from the first image is likelihood map 108 output by the detection unit 106. When these maps are input to the multi-layer NN of the estimation unit 109, a crop region (X, Y, W, H) for frame image t+1 is output. Thereafter, the transformation unit 101 can perform the transformation process on the second image using the parameters for the transformation process on the second image derived by the estimation unit 109. In this embodiment, the estimation unit 109 derives parameters for the conversion process using a multi-layer NN. However, the configuration of the estimation unit 109 is not necessarily limited to an NN. For example, the estimation unit 109 may estimate parameters using known techniques such as SVM or AdaBoost.

[0033] (Learning method) A method for training the estimation unit 109 so that the parameters of the crop region of frame image t+1 can be estimated will be described below. Fig. 3 illustrates the configuration of a training device that generates a training model used by the estimation unit 109. The training device 3, which is an information processing device, can train a model that derives parameters for the conversion process, which is used to perform a conversion process on time-series images and to perform multiple tasks using the images after the conversion process.

[0034] The learning device 3 includes a parameter generation unit 301, a conversion unit 302, a multitasking unit 303, an evaluation unit 304, a parameter determination unit 305, an estimation unit 306, and a learning unit 307. In this embodiment, video data of one or more scenes is used. For each frame of the video data, teacher data exists that indicates correct answers to the subject tracking results and specific part detection results. The multitasking unit 303 has the same configuration as the multitasking unit 103, and uses trained inference models for tracking and specific part detection (i.e., the intermediate layer 104, the tracking unit 105, and the detection unit 106).

[0035] Teacher data is used to generate a learning model that estimates parameters of the conversion process (crop area in this embodiment). The learning device 3 can generate such teacher data according to the flowchart of FIG. 4. First, the configuration of the learning device 3 related to the generation of teacher data used for learning will be described below. The parameter generation unit 301 generates parameters for multiple conversion processes, as described below. The conversion unit 302 performs conversion processes on time-series images using each of the parameters for the multiple conversion processes, and generates multiple converted images from each image. The multitasking unit 303 performs multiple tasks using each of the multiple converted images. The evaluation unit 304 evaluates the processing accuracy of each of the multiple tasks for each of the multiple converted images. The parameter determination unit 305 selects one of the parameters for the multiple conversion processes as teacher data indicating the parameters of the conversion process for the image, based on the processing accuracy of each of the multiple tasks.

[0036] The procedure for generating training data used for learning will be described below. According to the method of Fig. 4, training data for training the estimation unit 109 is created using one scene in a video. If video data of multiple scenes exists, the process of Fig. 4 can be repeated for each scene.

[0037] In S401, the time t is set to 1. The loop of S402 to S414 is repeated for all frames of one scene.

[0038] In S403, the parameter generation unit 301 acquires a frame image t of one scene. In S404, the parameter generation unit 301 generates multiple sets of parameters (X, Y, W, H) for the acquired frame image t. The method of generating the parameters is not particularly limited. For example, the parameter generation unit 301 may randomly generate parameters according to upper and lower limits. Alternatively, the parameter generation unit 301 may divide the image into a grid, combine each divided region to generate multiple crop regions, and generate parameters indicating each crop region. In this way, the parameter generation unit 301 can define multiple crop regions for one frame image t.

[0039] In S405, the conversion unit 302 performs image conversion on the frame image t using the parameters generated in S404. In this way, a converted image is generated in S406. If n sets of parameters are generated, such as (X1, Y1, W1, H1), ..., (Xn, Yn, Wn, Hn), n converted images are generated.

[0040] In S407, the multitasking unit 303 performs multitasking processing, i.e., a tracking task and a detection task, for each of the n converted images generated in S406. In S408 to S409, the evaluation unit 304 evaluates the processing accuracy of each task for each of the n converted images. In this example, the evaluation unit 304 uses training data to evaluate the likelihood map output for the tracking task in S407. Here, the loss function used when training the tracking model can be used to calculate an evaluation value for the tracking task. Also, in S409, the evaluation unit 304 evaluates the likelihood map output for the detection task in S407. Here, the loss function used when training the detection model can be used to calculate an evaluation value for the detection task.

[0041] In S410, the evaluation unit 304 evaluates the processing accuracy of the multitasking process based on the processing accuracy of each task. The evaluation value of the multitasking process is calculated based on the evaluation value for the tracking task obtained in S408 and the evaluation value for the detection task obtained in S409. Here, the evaluation unit 304 can calculate the evaluation value of the multitasking process taking into account the correlation between each task. In this embodiment, a subject is tracked and a specific part of the tracked subject is detected. Therefore, to improve the accuracy of the multitasking process, it is desirable that the accuracy of both tasks reach a certain level or higher.

[0042] For example, it may be determined whether both the maximum likelihood value near the correct subject position in the likelihood map for the tracking task and the maximum likelihood value near the correct specific part in the likelihood map for the detection task are equal to or greater than a specified threshold. Only when both of these values ​​are equal to or greater than a specified threshold, can the loss function be evaluated, and the parameters used for the image conversion be selected as candidates for selection in S411. With this configuration, it is possible to select, as training data, parameters of the conversion process used to generate a converted image whose evaluation values ​​for the processing accuracy of each of the multiple tasks exceed a threshold.

[0043] Furthermore, taking into consideration the priority of the tasks, the loss function may be evaluated if both of these do not simultaneously exceed a specified threshold. For example, in this embodiment, if the tracking task fails, detection of a specific part of the tracking target also fails. For this reason, evaluation based on the likelihood map of the tracking task may be prioritized over evaluation based on the likelihood map of the tracking task. For example, if the maximum likelihood value near the correct subject position in the likelihood map of the tracking task is equal to or greater than a specified threshold, the loss function may be evaluated, and the parameters used for image conversion may be selected as candidates for selection in S411.

[0044] If a converted image with a certain level of accuracy for each task is not obtained, frame image t and the parameters used to convert frame image t can be excluded from the training data. For example, regardless of the parameters used, if the maximum likelihood value near the correct subject position in the likelihood map for the tracking task is not equal to or greater than a specified threshold, it can be determined that the subject cannot be tracked from frame image t. In this case, frame image t can be excluded from the data used to train the parameter estimation model. Then, training data can be generated using frame images at times when it is determined that the subject can be tracked.

[0045] Alternatively, the weighted sum of the loss functions of the recognition tasks may be used as the evaluation value for multitasking, with the weights being assigned taking into account the priority of the recognition tasks. According to this method, when selecting training data based on the evaluation values ​​for the processing accuracy of each of the tasks, the evaluation values ​​for the processing accuracy of each of the tasks can be weighted according to the priority of the task.

[0046] Alternatively, training data may be selected based on the product of evaluation values ​​for the processing accuracy of each of multiple tasks. For example, the product of the loss function value for the tracking task and the loss function for the detection task, or the square root of this product, may be used as the evaluation value for multitask processing. The product term of the loss functions has an effect similar to a regularization term. In other words, when training is performed based on such evaluation values, the accuracy of both recognition tasks can be balanced, and the parameter estimation model can be prevented from being adapted to only one of the recognition tasks.

[0047] As another alternative, the evaluation value of the multitasking process may be given in the form of a quadratic form with the loss functions of the tracking task and the detection task as variables. For example, if the loss function of the i-th recognition task is L i Then, we can define the loss function vector L with this as the i-th element. Then, we can use any symmetric matrix A to define the loss function L for multitasking. total can be expressed as follows: L total =L T AL

[0048] In this way, by preventing the evaluation of multitasking from becoming high (the evaluation value becoming low) when the loss function value of one recognition task is low and the loss function value of the other recognition task is high, it is possible to expect an improvement in the recognition accuracy of the entire multitasking process.

[0049] In S411, the parameter determination unit 305 selects one of the parameters of the multiple conversion processes as training data based on the processing accuracy of the multitasking determined in S410. Here, the parameter determination unit 305 determines the parameter used for the multitasking process that obtained the highest evaluation. Specifically, the parameter determination unit 305 selects the multitasking process that obtained the smallest evaluation value in S410 from among the multitasking processes for the n converted images. The parameter determination unit 305 then determines the parameter of the conversion process used in this multitasking process as the optimal parameter and selects this parameter. In S412, the parameter determination unit 305 outputs the conversion parameter selected in S411. The output conversion parameter can be said to be a conversion parameter that is suitable for both the tracking task and the detection task for the frame image t. This conversion parameter is used as training data for training the estimation unit 109. In S413, 1 is added to the time t, and the process then returns to S403.

[0050] By performing the above procedure for each frame image at each time, and for each scene as necessary, multiple sets of training data can be created.

[0051] Next, the learning method of the estimation unit 306 will be described with reference to Fig. 5. Fig. 5 shows the procedure for learning the estimation unit 306 based on one scene in a video. This video is the same as the video used to create the training data described with reference to Fig. 4. When there is video data of multiple scenes, the process of Fig. 5 can be repeated for each scene.

[0052] In S501, time t is set to 1. The loop of S502 to S514 is repeated for all frames of one scene. In S503, the learning unit 307 acquires frame image t. In S504, the learning unit 307 acquires optimal conversion processing parameters for frame image t created according to the procedure in FIG.

[0053] In S505, the conversion unit 302 performs image conversion on the frame image t using the parameters acquired in S504, and outputs the converted image obtained in S506. In S507, the converted image obtained in S506 is input to the multitasking unit 303, and the trained multitasking unit 303 performs multitasking processing. In S508, the multitasking unit 303 outputs an intermediate feature map, and also outputs likelihood maps for the tracking task and the detection task.

[0054] In S509, the intermediate feature map output from the multitasking unit 303 in S508, the likelihood map for the tracking task, and the likelihood map for the detection task are input to the estimation unit 306. Then, the estimation unit 306 estimates optimal parameters for the conversion process. Here, the parameters for the conversion process for frame image t+1 are estimated.

[0055] In S510, the learning unit 307 acquires optimal conversion processing parameters (teaching data) for the frame image t+1 created according to the procedure of FIG. 4. In S511, the learning unit 307 evaluates the error between the parameters estimated in S509 and the teaching data acquired in S510. For example, the learning unit 307 can calculate an error value between the estimation result in S509 and the teaching data. Here, the loss function used to calculate the error value can be a function suitable for regression problems, such as least squares error, mean absolute error, or mean square logarithmic error. In S512, the learning unit 307 trains the estimation unit 306 based on the error evaluated in S511. For example, the learning unit 307 can update the weights of the estimation unit 306 (e.g., the weights of each layer of the NN) using backpropagation based on the error value. In S513, 1 is added to time t, and the process then returns to S503.

[0056] By performing the above procedure for each frame image at each time, and for each scene as necessary, it is possible to estimate parameters for the conversion process suited to each recognition task in a multitask.

[0057] In the above embodiment, image processing parameters were estimated based on the intermediate feature map obtained by multitasking in the multitasking unit 103, the results of the tracking task (likelihood map), and the results of the detection task (likelihood map). However, image processing parameters may also be estimated using additional information. Such information may include sensor information. Examples of sensors include acceleration sensors and geomagnetic sensors, and time-series data from these sensors may be used as sensor information. By utilizing information obtained from a sensor attached to the camera, it is possible to consider not only the movement of the subject in the image but also the attitude or movement of the camera itself at the time of image capture. Therefore, it is expected that image processing parameters (e.g., crop area) can be more appropriately estimated even when the camera itself is moving.

[0058] (Use of sensor information) Such a modified example will be described with reference to Fig. 6. The information processing device 1 according to this modified example includes a sensor processing unit 601. Below, the sensor processing unit 601 and the estimation unit 109 will be described, and the other configurations are the same as those of the above embodiment, so description thereof will be omitted.

[0059] The sensor processing unit 601 performs filtering on sensor data from sensors attached to the camera. Time-series data from acceleration sensors or geomagnetic sensors often contains noise. Therefore, the sensor processing unit 601 filters out the noise in the sensor information to more accurately estimate the camera's orientation. By providing the estimation unit 109 with appropriate orientation information obtained in this way, it is expected that the estimation accuracy of image processing parameters will improve. Examples of filters used by the sensor processing unit 601 to estimate the camera orientation include a Kalman filter or a Madgwick filter described in Non-Patent Document 5.

[0060] The estimation unit 109 derives parameters for the conversion process based further on sensor information obtained by a sensor associated with the camera that captured the time-series images. For example, the estimation unit 109 estimates image processing parameters using the intermediate feature map, likelihood map 107, and likelihood map 108 output by the multitasking unit 103, as well as the output from the sensor processing unit 601. The image conversion process and multitasking process in this modified example can be performed in the same manner as in the flowchart of FIG. 2, except that the parameters for the conversion process in S212 are estimated in this manner.

[0061] Such learning by the estimation unit 109 can be performed by the learning device 3 as follows, according to the flowchart of Fig. 5. That is, in S509, the estimation unit 306 estimates image processing parameters using the intermediate feature map output by the multitasking unit 303, the likelihood map for the tracking task, and the likelihood map for the detection task, as well as the sensor data after filtering. The filtering of the sensor data can be performed in the same way as in the sensor processing unit 601. The other processes are the same as those in the flowchart of Fig. 5, and therefore a description thereof will be omitted.

[0062] To improve estimation accuracy, the acquisition time of the sensor information data can be synchronized with the acquisition time of the frame images of the video. If there is a discrepancy between these times, the sensor processing unit 601 can interpolate the time-series data of the sensor information to obtain the sensor information (e.g., camera posture information) at the acquisition time of the frame images. The sensor information synchronized with the frame images in this way can be used in S212 and S509.

[0063] As shown in Fig. 5, sensor information may be used for multitasking. With such a configuration, multitasking can be performed taking into account sensor information (for example, camera attitude information), which is expected to improve processing accuracy. For example, in the example of Fig. 5, output from the sensor processing unit 601 is also input to the tracking unit 105 and the detection unit 106.

[0064] (Example of multitasking) So far, we have described a case where a tracking task and a detection task are performed simultaneously on an image. Below, we will explain another example of multitask processing (multiple tasks that output different types of information). That is, we will explain a case where a subject of a specific category, such as a human or an animal, is detected from a single image, and a specific scene area, such as a mountain or a plant in the background, is extracted, with reference to FIGS. 1 to 5. Descriptions of configurations similar to those in the above-described embodiment will be omitted.

[0065] In such cases, if the image brightness is low, i.e., the image is dark, it is expected that the accuracy of detecting people will improve by using an image that has been converted so that the average brightness value is larger, i.e., brightened. On the other hand, if the image becomes brighter, the dark color characteristics specific to mountains or plants will be lost, which may actually reduce the accuracy of extracting the mountain or plant regions.

[0066] In this example, the estimation unit 109 estimates a luminance correction value Y that ensures recognition accuracy for both a task of detecting a subject of a specific category (category detection task) and a task of extracting an area of ​​a specific scene (scene area extraction task). The conversion unit 101 performs conversion processing on an image using the parameters estimated by the estimation unit 109, but the parameters in this modified example are the luminance correction value Y, and the conversion processing is a luminance correction processing in accordance with the correction value Y.

[0067] The multitasking unit 103 has a category detection task processing unit and a scene area extraction task processing unit instead of the tracking unit 105 and the detection unit 106. In this modification, the category detection task processing unit detects people using the intermediate features generated by the hidden layer 104 and outputs a likelihood map indicating the likelihood of an area where a person exists. Such a category detection task processing unit can be implemented in the same way as the detection unit 106. In addition, in this modification, the scene area extraction task processing unit extracts plant areas using the intermediate features generated by the hidden layer 104. Such a scene area extraction task processing unit can be implemented by a multi-layer CNN trained so that the final layer outputs a likelihood map indicating the likelihood of an area where a plant exists. Such a CNN can be implemented by training using a likelihood map of a specific scene (in this case, plants) as training data.

[0068] The image conversion process and multitasking process in this modified example will be described with reference to Fig. 2. The process in this modified example is performed in the same manner as in the above embodiment, except that the conversion process in S209 is a luminance value correction process, and the parameter acquired in S202 is a luminance value correction value Y. In addition, in S202 to S208, if the parameter acquired in S202 exceeds a set range, the conversion unit 101 corrects it so that it does not exceed the set range. With this method, if the estimated parameter is an outlier, the effect on the processing accuracy by the multitasking unit 303 is reduced.

[0069] In this modified example, the estimation unit 109 estimates a correction value Y for the luminance value of frame image t+1 based on frame image t. As described with reference to Figures 3 and 5, this estimation unit 109 can learn using video data (teacher data) of one or more scenes to which correct answers have been assigned for the category detection task and the scene region extraction task. In this case, the learned model for the category detection task and the model for the scene region extraction task are also used.

[0070] A method for generating training data used to train the estimation unit 109 so as to estimate a correction value for a luminance value as in this modification will be described with reference to Fig. 4. A description of the processing that has already been described will be omitted.

[0071] In this modification, the parameter generation unit 301 generates parameters (here, correction values ​​Y for multiple luminance values) for the conversion process on the image acquired in S403 in S404. The parameter generation unit 301 may generate the correction value Y randomly in accordance with an upper limit and a lower limit. Alternatively, the parameter generation unit 301 may generate multiple correction values ​​Y determined by dividing the range between the upper limit and the lower limit at regular intervals. Alternatively, the parameter generation unit 301 may acquire training data for the correction value Y for the luminance value determined for frame image t-1, and set multiple nearby values ​​as the correction values ​​Y. In S405 to S406, n converted images are generated using the n correction values ​​Y, respectively.

[0072] In steps S407 to S409, the multitask processing and the evaluation of the category detection task and the scene region extraction task are performed. For example, the error between the likelihood map obtained by each task and the training data is evaluated using the loss function used in learning each task, thereby calculating an evaluation value.

[0073] In S410, the evaluation unit 304 calculates an evaluation value for the multitasking process based on the evaluation value for the category detection task obtained in S408 and the evaluation value for the scene region extraction task obtained in S409. Even in this modification, the evaluation unit 304 can calculate the evaluation value for the multitasking process by taking into account the correlation between the tasks. That is, even in this modification, in order to improve the accuracy of the multitasking process, it is desirable that the accuracy of both tasks reach a certain level or higher.

[0074] For example, the loss function may be evaluated only when the accuracy of all tasks is above a certain level, and the parameters used in the image conversion may be selected as candidates for selection in S411. Specifically, the loss function may be evaluated only when the area ratio between the estimated region given by the likelihood map obtained in the scene region extraction task and the region of the correct data is above a certain level. Alternatively, the loss function may be evaluated only when the ratio of the region not estimated within the correct region (undetected region) or the region estimated within the incorrect region (overdetected region) to the correct region is equal to or less than a predetermined value. Furthermore, taking into account the priority of the tasks, the loss function may be evaluated only when both of these do not simultaneously exceed a specified threshold. Furthermore, regardless of the parameters used, if the accuracy of both tasks is not above a specified threshold, the frame image t may be excluded from the data used to train the parameter estimation model.

[0075] As already explained, the product or square root of the product of the loss functions of the tasks may be used as the evaluation value for multitasking. The weight of the product can be set so that the overall evaluation value does not become small when the loss function value of one recognition task is low and the loss function value of the other recognition task is high.

[0076] By using this method, in S411, conversion parameters suitable for each of the multiple tasks at time t are determined, and these conversion parameters are used as training data for training the estimation unit 109.

[0077] (Learning multitasking) In each of the above embodiments, the estimating unit 306 is trained using the output of the multitasking unit 303. On the other hand, the multitasking unit 303 may be trained using the output of the estimating unit 306. That is, the training of the multitasking unit 303 and the training of the estimating unit 306 can be alternately repeated.

[0078] For example, the learning unit 307 can train the estimation unit 306 according to the flowchart in Fig. 4. After this process is completed, the learning unit 307 can train the multitasking unit 303. In this way, learning can be performed alternately, such that the estimation unit 306 is trained when the learning of the multitasking unit 303 is completed, and the multitasking unit 303 is trained when the learning of the estimation unit 306 is completed. By repeating such learning a specified number of times, the multitasking unit 303 and the estimation unit 306 can be trained.

[0079] The learning unit 307 can train the multitasking unit 303 according to the flowchart of Fig. 7. In this example, the learning unit 307 performs learning for each task using an image transformed with parameters estimated by the estimation unit 306. In this way, the multitasking unit 303 (i.e., the multitasking unit 103) is trained so that it can perform estimation processing suitable for an image transformed according to parameters estimated by the estimation unit 306 (i.e., the estimation unit 109).

[0080] Specifically, the estimation unit 306 derives parameters for a conversion process for a fourth image, which is a subsequent image in the second time series, using the trained model based on a third image in the second time series. The time series images used for the training may be the same as or different from the time series images used for the training of the estimation unit 306. The conversion unit 302 then performs a conversion process on the fourth image using the parameters for the conversion process derived by the estimation unit 306. Furthermore, the multitasking unit 303 performs multiple tasks using the converted fourth image. Furthermore, the training unit 307 trains the multitasking unit 303 based on the error between the results of the multiple tasks performed by the multitasking unit 303 and training data indicating the results of the multiple tasks performed on the fourth image.

[0081] According to the method of Fig. 7, one scene in a video is used for learning by the multitasking unit 303. For each frame of the video data, there is training data indicating the correct answer for each task (in this example, the subject tracking task and the specific part detection task). When video data of multiple scenes exists, the process of Fig. 7 can be repeated for each scene.

[0082] In S1001, time t is set to 1. The loop of S1002 to S1013 is repeated for all frames of one scene. In S1003, the conversion unit 302 acquires frame image t of one scene. In S1004, the conversion unit 302 acquires the parameters of the conversion process estimated in S1011 for frame image t-1. In this example, the conversion unit 302 acquires parameters indicating the cropping area estimated by the estimation unit 306. Note that, as in S202, for frame image 1, the cropping area can be determined based on the position and size of the subject that was initially detected.

[0083] In S1005, the conversion unit 302 performs image conversion using the parameters acquired in S1004. In S1006, the conversion unit 302 outputs the converted image obtained. In S1007, the converted image obtained in S1006 is input to the multitasking unit 303. In S1008, the multitasking unit 303 outputs the results of multiple tasks. In this example, the multitasking unit 303 outputs a likelihood map obtained in the tracking task and a likelihood map obtained in the detection task.

[0084] In S1009, the learning unit 307 evaluates the error with respect to the training data for each recognition task. In this example, the learning unit 307 evaluates the error of the likelihood map output in S1008. For example, when evaluating the error in the likelihood of the output position, the loss function can be given using cross entropy, where the position of the subject is a positive example and the position other than the subject is a negative example. Furthermore, when evaluating the error in the output size, the loss function can be given using smooth L1 loss or IOU loss.

[0085] In S1010, the learning unit 307 performs learning of the multitasking unit 303 based on the error evaluated in S1009. For example, the learning unit 307 can update the weights of each layer of the CNN that constitutes the multitasking unit 303 using the backpropagation algorithm based on the error value calculated in S1009.

[0086] In S1011, the estimation unit 306 acquires the intermediate feature map obtained in S1008 by the multitasking unit 303 and the output for each task (in this example, the likelihood map obtained in the tracking task and the likelihood map obtained in the detection task). Then, based on this data, the estimation unit 306 estimates parameters for the transformation process for frame image t+1. The estimated parameters are used in S1004 for the transformation process for the next frame image t+1. In S1012, 1 is added to time t, and then the process returns to S1003.

[0087] By performing the above procedure for the frame image at each time, and for each scene as necessary, the multitasking unit 303 can be trained to perform multitasking processing that is compatible with the estimated parameters of the conversion processing.

[0088] According to the above-described embodiment, when multitasking is performed on an image, the accuracy of the multitasking can be improved overall by performing a conversion process as a preprocessing. In particular, as described above, when using a single network including an intermediate layer and layers for each task branching from the intermediate layer, multitasking can be performed with high accuracy even if the input image conditions for obtaining high processing accuracy for each task are different.

[0089] When estimating the parameters of the conversion process, the parameters of the conversion process for the second image may be derived based on an image prior to the first image in the time series of images. For example, the estimation unit 109 may use information on a frame image older than the current frame image t. As a specific example, the estimation unit 109 may use a past frame image, or an intermediate feature map, a likelihood map for a tracking task, or a likelihood map for a detection task output by the multitasking unit 103 using the past frame image. Furthermore, the estimation unit 109 may use sensor information at the time of capturing the past frame image. The estimation unit 109 can estimate the conversion process parameters for frame image t+1 using this information in addition to the information on frame image t. Similarly, the multitasking unit 103 may perform multitasking using information on past frame images.

[0090] When using information on past frame images in this way, the multitasking unit 103 or the estimation unit 109 may perform processing using a recurrent neural network (RNN). Alternatively, the multitasking unit 103 or the estimation unit 109 may use an NN model suitable for handling time series, such as the Transformer described in Non-Patent Document 6.

[0091] While the above describes multitasking, which involves simultaneously performing two tasks, multitasking, which involves simultaneously performing three or more tasks, can also be achieved with a similar configuration.

[0092] In the above-described embodiment, each processing unit of the information processing device and learning device shown in, for example, Figures 1 and 3 is implemented by a computer. However, some or all of the functions of the information processing device and learning device may be implemented by dedicated hardware. Furthermore, the information processing device and learning device according to one embodiment of the present invention may be configured by a plurality of information processing devices connected via a network, for example.

[0093] FIG. 8 is a diagram showing the basic configuration of a computer. In FIG. 8, a processor 810 is, for example, a CPU, and controls the operation of the entire computer. A memory 820 is, for example, a RAM, and temporarily stores programs, data, etc. A computer-readable storage medium 830 is, for example, a hard disk or a CD-ROM, and stores programs, data, etc. long-term. In this embodiment, a program that realizes the function of each unit, which is stored in the storage medium 830, is read into the memory 820. Then, the processor 810 operates in accordance with the program on the memory 820, thereby realizing the function of each unit.

[0094] 8, input interface 840 is an interface for acquiring information from an external device. Output interface 850 is an interface for outputting information to an external device. Bus 860 connects the above-mentioned components and enables data exchange.

[0095] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0096] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0097] 101: Conversion unit, 103: Multitasking unit, 104: Hidden layer, 105: Tracking unit, 106: Detection unit, 109: Estimation unit

Claims

1. a conversion means for performing a conversion process on each of the time-series images; a processing means for performing a plurality of tasks using the converted image to output different types of information; a derivation means for deriving, based on a first image of the time-series images, parameters for the conversion process for a second image of the time-series images that comes after the first image; the conversion means performs a conversion process on the second image using the parameters of the conversion process on the second image derived by the derivation means, the processing means performs the plurality of tasks using a multi-layer neural network; an information processing device, wherein the multilayer neural network comprises: a first layer that extracts intermediate features from the image after the conversion processing; a second layer that outputs information indicating a result of a first task of the plurality of tasks from the intermediate features; and a third layer that outputs information indicating a result of a second task of the plurality of tasks from the intermediate features.

2. 2. The information processing device according to claim 1, wherein the multilayer neural network is a convolutional neural network or a recurrent neural network.

3. a conversion means for performing a conversion process on each of the time-series images; a processing means for performing a plurality of tasks using the converted image to output different types of information; a derivation means for deriving, based on a first image of the time-series images, parameters for the conversion process for a second image of the time-series images that comes after the first image; the conversion means performs a conversion process on the second image using the parameters of the conversion process on the second image derived by the derivation means, The information processing device, wherein the derivation means derives the parameters for the conversion process using a multi-layer neural network.

4. The derivation means the first image, intermediate features extracted by the processing means from the first image after the transformation process; and information indicating the result of at least one of the tasks on the first image obtained by the processing means; 4. The information processing apparatus according to claim 1, wherein the parameters of the conversion process for the second image are derived using at least one of the following:

5. 4. The information processing device according to claim 1, wherein the derivation means derives parameters for the conversion process for the second image using intermediate features extracted by the processing means from the first image after the conversion process and information indicating the results of each of the plurality of tasks for the first image obtained by the processing means.

6. The information processing device according to claim 1 , wherein the derivation means derives the parameters for the conversion process for the second image based further on an image that precedes the first image among the images in the time series.

7. 7. The information processing device according to claim 1, wherein the deriving means derives the parameters of the conversion process further based on sensor information obtained by a sensor associated with a camera that captured the time-series images.

8. 8. The information processing device according to claim 1, wherein the conversion means determines whether the parameters derived by the derivation means satisfy a criterion, and if the criterion is not satisfied, corrects the parameters derived by the derivation means, and performs the conversion process using the corrected parameters.

9. The conversion means determining whether the parameter derived by the derivation means is below a lower limit value, and if the parameter is below the lower limit value, correcting the parameter to the lower limit value; or determining whether the parameter derived by the derivation means exceeds an upper limit value, and if the parameter exceeds the upper limit value, correcting the parameter to the upper limit value; 9. The information processing device according to claim 8, wherein:

10. 10. The information processing apparatus according to claim 1, wherein the plurality of tasks include a first recognition task for recognizing a first object and a second recognition task for recognizing a second object different from the first object.

11. the conversion process is a crop process, the first recognition task is a task of tracking an object, the second recognition task is a task for detecting a specific part included in the subject, The derivation means derives parameters for the conversion process for the second image using intermediate features extracted from the first image after the conversion process by the processing means, information indicating a tracking result of the subject from the first image, and information indicating a detection result of the specific portion from the first image. The information processing device according to claim 10 .

12. An information processing method performed by an information processing device, performing a transformation process on each of the images in the time series; performing a plurality of tasks using the converted image, the tasks outputting different types of information; deriving, based on a first image in the time series of images, parameters for the transformation process for a second image in the time series of images that follows the first image; In the conversion process, the conversion process is performed on the second image using the derived parameters of the conversion process on the second image, performing the plurality of tasks using a multilayer neural network; an information processing method, wherein the multilayer neural network comprises: a first layer that extracts intermediate features from the image after the conversion process; a second layer that outputs information indicating a result of a first task of the plurality of tasks from the intermediate features; and a third layer that outputs information indicating a result of a second task of the plurality of tasks from the intermediate features.

13. An information processing method performed by an information processing device, performing a transformation process on each of the images in the time series; performing a plurality of tasks using the converted image, the tasks outputting different types of information; deriving, based on a first image in the time series of images, parameters for the transformation process for a second image in the time series of images that follows the first image; In the conversion process, the conversion process is performed on the second image using the derived parameters of the conversion process on the second image, An information processing method, wherein the step of deriving comprises deriving parameters for the conversion process using a multilayer neural network.

14. A program for causing a computer to function as the information processing device according to any one of claims 1 to 11.

Images