Information processing apparatus, information processing method, and program

Abstract

To enable learning of a neural network resistant to variation of a target subject even when there is variation in the length of moving image data used for learning.SOLUTION: The information processing apparatus includes an image acquisition unit configured to acquire a plurality of time-series images obtained by imaging a target object, and a learning unit configured to infer, based on a reference image and a search image acquired from the plurality of time-series images, the target object of the search image with respect to the target object of the reference image, and update a parameter of a neural network based on an inference result and correct answer data. Then, the learning means selects the reference image and the search image from the plurality of images based on at least one of the time at which the plurality of time-series images are captured, the predetermined time interval, and the dissimilarity between the plurality of time-series images.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing technique for training a neural network. [Background technology]

[0002] Non-Patent Document 1 discloses an object tracking technology using a multilayer neural network, in which a reference image containing the object to be tracked is input, the object is searched for in an image provided as a search image, and its position and size are inferred. To enable such object tracking, it is necessary to prepare reference images, search images, and ground truth data indicating the position and size of the object to be tracked corresponding to those images, and to train the parameters of the multilayer neural network. To adequately train the parameters of the multilayer neural network, a large amount of data must be secured, and public datasets such as those disclosed in Non-Patent Document 2 are generally used. In Non-Patent Document 1, to train the parameters of the multilayer neural network, reference images and search images are selected from the video disclosed in Non-Patent Document 2 so that the frame interval is 100 or less. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] SiamFC++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines Yinda Xu et al., AAAI 2020 [Non-patent document 2] LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking Heng Fan et al., CVPR 2019 Summary of the Invention [Problem to be solved by the invention]

[0004] Object tracking functionality is often installed in devices such as cameras. Devices such as cameras are subject to limitations on the circuits, computing power, processing time, etc. that can be installed, making it difficult to use a multilayer neural network of the scale disclosed in Non-Patent Document 1. Therefore, it is necessary to use a model with a significantly reduced number of parameters. However, if the number of parameters of a multilayer neural network is significantly reduced, it becomes difficult to train a model that can accommodate all of the wide variety of objects to be tracked, as described in Non-Patent Document 2. In this case, it is conceivable to independently prepare a training dataset (a set of reference images and search images) specific to the functions of a device equipped with a multilayer neural network, for example, from video data of the objects to be tracked.

[0005] However, depending on the subject's movement, it is often impossible to capture a video that captures the subject on screen for a sufficiently long period of time, which can lead to variations in the length of the video captured. In particular, data from short videos tends to have less variation in the subject's appearance, and such short video data generates a dataset with little variation in the subject's appearance between the reference image and the search image. Furthermore, a multilayer neural network for object tracking trained using a dataset with little variation in the subject's appearance tends to be weak against variations in the subject's appearance due to changes in posture, etc.

[0006] Therefore, an object of the present invention is to enable learning of a neural network that is robust to variations in the target subject, even when there is variation in the length of the video data used for learning. [Means for solving the problem]

[0007] The information processing device of the present invention comprises an image acquisition means for acquiring a plurality of images in a time series of images of a target subject, and a learning means for inferring the target subject of the search image relative to the target subject of the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data, wherein the learning means selects the reference image and the search image from the plurality of images based on at least one of the times at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series. [Effects of the Invention]

[0008] According to the present invention, even when there is variation in the length of video data used for learning, it is possible to learn a neural network that is robust to variations in the target subject. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Figure 2] 10 is a flowchart of information processing according to the embodiment. [Figure 3] 10 is a flowchart of a process for selecting a reference image and a search image. [Figure 4] 10A and 10B are diagrams used to explain a reference feature acquisition process and a search feature acquisition process. [Figure 5] 10A and 10B are diagrams illustrating an example of a mesh grid and a positional deviation amount map. [Figure 6] FIG. 1 is a diagram used to explain similarity, dissimilarity, and sampling probability. [Figure 7] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the features described in the embodiments are necessarily essential to the solution of the present invention; these features may be combined in any desired manner. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the device to which the present invention is applied and various conditions (such as usage conditions and usage environment). Furthermore, the embodiments may be configured by appropriately combining parts of the embodiments described below. In the following embodiments, the same or similar configurations and processing steps are designated by the same reference numerals, and redundant explanations will be omitted.

[0011] First Embodiment In this embodiment, as an example, an example of applying object tracking using a multilayer neural network to the autofocus function of a camera is described. Subjects targeted by the camera's autofocus function are expected to be subjects that move rapidly, such as athletes playing sports, birds or animals moving around, or cars or motorcycles in motion. Furthermore, subjects that move rapidly are subjects whose appearances tend to fluctuate significantly due to changes in posture, etc. In this embodiment, an example is described that enables learning of a multilayer neural network that can perform sufficiently strong and efficient object tracking for subjects whose appearances fluctuate.

[0012] 1 is a diagram showing an example of a schematic functional configuration of an information processing apparatus according to the first embodiment. First, an overview of the information processing apparatus according to this embodiment will be described. The imaging device 110 is a digital camera or a surveillance camera that includes an imaging optical system, an imaging element, imaging and signal processing circuitry, etc., and outputs video data of an image of a subject or the like to the information processing device 100. The image acquisition unit 101 of the information processing device 100 acquires moving image data from the imaging device 110. In the present embodiment, the image acquisition unit 101 selects data of at least one moving image from among a plurality of moving images captured by the imaging device 110. Details of the moving image selection process performed by the image acquisition unit 101 will be described later.

[0013] The reference feature acquisition unit 102 selects an image to be used as a reference image from the video selected by the image acquisition unit 101, and extracts image features from the reference image. The reference image is an image containing the object to be tracked. As will be described in detail later, the reference feature acquisition unit 102 of this embodiment extracts image features using a multi-layer neural network. Hereinafter, image features extracted from the reference image will be referred to as reference features. Details of the acquisition of the reference image and the reference feature extraction process by the reference feature acquisition unit 102 will be described later. The reference features extracted by the reference feature acquisition unit 102 are sent to the tracking unit 104.

[0014] The search feature acquisition unit 103 selects an image to be used as a search image from the video selected by the image acquisition unit 101, and extracts image features from the search image. The search image is an image used to search for a target subject to be tracked. As will be described in detail later, the search feature acquisition unit 103 of this embodiment extracts image features using a multi-layer neural network. Hereinafter, the image features extracted from the search image will be referred to as search features. Details of the process of acquiring the search image and extracting the search features by the search feature acquisition unit 103 will be described later. The search features extracted by the search feature acquisition unit 103 are sent to the tracking unit 104.

[0015] The tracking unit 104 receives the reference features and the search features, and infers the position and size of the tracking target in the search image that corresponds to the subject to be tracked (hereinafter referred to as the tracking target) in the reference image. As will be described in detail later, the tracking unit 104 of this embodiment infers the position and size of the tracking target using a multi-layer neural network.

[0016] The update unit 105 receives the inference result from the tracking unit 104 and calculates the error between the inference result and previously input correct data. The update unit 105 then updates the parameters of the multilayer neural network according to the error, thereby performing learning to optimize the parameters. As will be described in detail later, the update unit 105 of this embodiment updates the adjustable parameters of the multilayer neural networks of the reference feature acquisition unit 102, the search feature acquisition unit 103, and the tracking unit 104 so as to optimize them. The update unit 105 may update the parameters of all the multilayer neural networks of the reference feature acquisition unit 102, the search feature acquisition unit 103, and the tracking unit 104, or may update the parameters of any one or two of the multilayer neural networks. The parameters of the multilayer neural network can be updated using, for example, a stochastic gradient descent method.

[0017] The result output unit 106 outputs the result of the tracking unit 104 tracking the tracking target using the multilayer neural network after learning to optimize the parameters as described above, i.e., the inference result of the position and size of the tracking target. That is, the result output unit 106 outputs the result of the tracking unit 104 tracking the tracking target using the feature amounts acquired by the reference feature acquisition unit 102 and the search feature acquisition unit 103 after learning to optimize the parameters. In this embodiment, the inference result output from the result output unit 106 is used, for example, to operate the autofocus function of the camera. That is, because the inference result is the tracking result of the tracking target, the camera performs autofocus on the subject of the tracking result.

[0018] Note that, although the present embodiment has exemplified an example in which the tracking results using a trained multilayer neural network are output from the result output unit 106, the tracking results can also be used to confirm the effectiveness of the update performed by the update unit 105. In other words, by outputting the tracking results using a multilayer neural network that is currently being trained from the result output unit 106 and checking the autofocus operation of the camera based on the tracking results, the user can confirm whether the learning is being performed appropriately.

[0019] Fig. 2 is a flowchart showing the flow of information processing in the information processing device 100 of this embodiment. An overview of the information processing according to this embodiment will be described with reference to the flowchart of Fig. 2, and then the details of the processing performed in each step of Fig. 2 will be described. In the information processing device 100, as step S200 for advance preparation before the processing of step S201 and subsequent steps is started, acquisition of learning data, predetermined conversion processing of the learning data, assignment of correct answer data, etc. In the case of this embodiment, it is assumed that a plurality of moving images (continuous images) composed of a plurality of frames (frames) consecutive in time series captured by the imaging device 110 are used as the learning data.

[0020] Next, in step S201, the image acquisition unit 101 selects a moving image to be used for learning from the multiple moving images prepared in advance in step S200. Next, in step S202, the reference feature acquisition unit 102 and the search feature acquisition unit 103 select a pair of a reference image and a search image, which is an image different from the reference image, from the video selected by the image acquisition unit 101 based on a pre-set time interval.

[0021] Next, in step S203, the reference feature acquisition unit 102 extracts reference features from the reference image and sends the reference features to the tracking unit 104. Also, in step S204, the search feature acquisition unit 103 extracts search features from the search image and sends the search features to the tracking unit 104.

[0022] Next, in step S205, the tracking unit 104 compares the reference features with the search features. Then, in the next step S206 , the tracking unit 104 infers the position and size of the tracking target relative to the search image based on the comparison result between the reference feature and the search feature, and sends the inference result to the update unit 105 .

[0023] Next, in step S207, the update unit 105 compares the inference result sent from the tracking unit 104 with correct answer data prepared in advance, and calculates the error therebetween. Furthermore, in step S208, the update unit 105 updates the parameters of the multilayer neural networks used in the reference feature acquisition unit 102, the search feature acquisition unit 103, and the tracking unit 104 based on the calculated error so as to optimize them.

[0024] Thereafter, in step S209, the information processing device 100 determines whether to end the process. For example, the information processing device 100 ends the process if the number of parameter updates reaches a specified number, and returns the process to step S201 if the specified number has not been reached. In this embodiment, it is assumed that the specified number is set to 10,000 times.

[0025] The processing of each step in the flowchart of FIG. 2 will be described in detail below. <Preparation process> In step S200, the information processing device 100 collects a sufficient number of pairs of training data and correct answer data, and performs a predetermined conversion process on the training data. The training data is assumed to be, for example, an RGB video image captured by an imaging device 110 such as a digital camera, with a width of 4000 pixels, a height of 3000 pixels, and 30 frames (30 frames). The predetermined conversion process on the training data is a process of converting the video into a sequence of images. The information processing device 100 then performs a process of assigning correct answer data to all images converted into example images. The correct answer data is data indicating the position, width, and height (i.e., size) of the tracking target in each image, and these data values ​​are assumed to be, for example, values ​​input by the user, or values ​​calculated from the area of ​​the tracking target detected from the image.

[0026] Furthermore, as will be described in detail later, the information processing device 100 of this embodiment selects pairs of reference images and search images from a video based on a preset time interval, and therefore refers to the capture time of each frame of the video when selecting the pairs of reference images and search images. Therefore, it is desirable that all learning data videos be captured with the same number of frames. However, learning data videos with the same number of frames are not always obtained, and videos with different numbers of frames, i.e., videos with varying lengths, are often obtained. In this embodiment, videos captured by the imaging device 110 are assigned timestamps indicating the capture time (capture date and time) of each image for each frame, and these timestamps are referenced when selecting reference images and search images in step S202, which will be described later. Note that in this embodiment, the method for storing learning data is not particularly limited. For example, the learning data may be stored in an external storage device such as a hard disk, or in cloud storage via a network.

[0027] <Video selection process> Next, the moving image selection process performed by the image acquisition unit 101 in step S201 of FIG. 2 will be described. In step S201, the image acquisition unit 101 selects a moving image to be used for learning from among a plurality of moving images collected as learning data.

[0028] For example, the image acquisition unit 101 randomly extracts one video without replacement from among multiple videos collected as training data. That is, when randomly selecting one video from multiple videos, the image acquisition unit 101 uses a method in which a video selected once is not subsequently selected. The image acquisition unit 101 also repeats the video selection in step S201 a specified number of times. If the number of videos prepared as training data is less than the specified number of times, video selection becomes impossible before the specified number of times is reached. Therefore, if the number of videos is less than the specified number of times, the image acquisition unit 101 again sets all videos prepared as training data as candidates for selection and performs non-replacement extraction until the specified number of times is reached. That is, if the number of videos prepared as training data is less than the specified number of times, the image acquisition unit 101 allows the same video to be selected again in the non-replacement extraction. The image acquisition unit 101 selects one video at a time from among multiple videos collected as training data, but multiple videos may also be selected at a time. When multiple videos are selected at once in this way, in step S208 (described later), the update unit 105 updates the parameters taking into account all of the selected videos. This type of parameter update is generally called batch learning.

[0029] <Reference image and search image selection process> Next, a description will be given of the reference image and search image selection process performed by the reference feature acquisition unit 102 and the search feature acquisition unit 103 in step S202 of Fig. 2. As described above, in step S202, the reference feature acquisition unit 102 and the search feature acquisition unit 103 select a pair of a reference image and a search image from among the images of each frame constituting the moving image selected by the image acquisition unit 101, based on a pre-set time interval.

[0030] FIG. 3(a) is a detailed flowchart of the reference image and search image selection process in step S202 of FIG. In the information processing device 100, a predetermined time interval is set as step S300 for advance preparation before the processing from step S301 onward is started. Hereinafter, the time interval set in step S300 will be referred to as the set time interval. In this embodiment, an example is given in which the set time interval in step S300 is set to 1 second. Note that if the frame rate of the video is 30 fps (frames per second), the set time interval of 1 second corresponds to a time interval of 30 frames. In this case, the pair of reference image and search image selected in the flowchart of FIG. 3 will be equivalent to a pair of images of frames spaced apart by at least 30 frames, which corresponds to the set time interval.

[0031] Next, in step S301, the information processing device 100 performs conditional branching processing according to the duration of the video selected by the image acquisition unit 101 in step S201. For example, if the duration of the video is less than the set time interval, the processing of the information processing device 100 proceeds to step S304 and subsequent steps, whereas if the duration is equal to or greater than the set time interval, the processing proceeds to step S302 and subsequent steps.

[0032] If the process proceeds to step S304 because the duration of the video is less than the set time interval, the reference feature acquisition unit 102 selects the image of the first frame of the video as the reference image. Then, in the next step S305, the search feature acquisition unit 103 selects the image of the last frame of the video as the search image.

[0033] For example, if the process proceeds to step S302 because the duration of the video is equal to or longer than the set time interval, the reference feature acquisition unit 102 selects an image of one of the frames in the first half of the video as a reference image. Furthermore, in the next step S303, the search feature acquisition unit 103 selects an image of one of the frames in the second half of the video as a search image. For example, if the video selected in step S201 is a video consisting of 50 frames, in step S302 the reference feature acquisition unit 102 selects one of the images of the first to twentieth frames of the video as a reference image. Furthermore, in the next step S303, the search feature acquisition unit 103 selects one of the images of the 31st to 50th frames of the video as a search image.

[0034] Next, details of the reference feature acquisition process performed by the reference feature acquisition unit 102 in step S203 of Fig. 2 and the search feature acquisition process performed by the search feature acquisition unit 103 in step S204 of Fig. 2 will be described with reference to Fig. 4. Fig. 4(a) is a conceptual diagram of the reference feature acquisition process in the reference feature acquisition unit 102, and Fig. 4(b) is a conceptual diagram of the search feature acquisition process in the search feature acquisition unit 103.

[0035] <Reference feature acquisition process> First, the reference feature acquisition process performed by the reference feature acquisition unit 102 will be described with reference to FIG. The reference image 400 shown in FIG. 4(a) is assumed to be an RGB image having a width of 4000 pixels and a height of 3000 pixels as described above. The reference feature acquisition unit 102 first cuts out a square area including the tracking target as a crop rectangle 401. The width of the ground truth data corresponding to the reference image 400 is defined as gt w and the height is gt h Then, the reference feature acquisition unit 102 calculates the length E (number of pixels) of one side of the square crop rectangle 401 using equation (1).

[0036]

number

[0037] In addition, A in the formula (1) is a parameter that represents the area ratio. According to the formula (1), the area E of the square crop rectangle 401 is 2 is the area of ​​the tracking target region gt w ×gt h In this embodiment, for example, the area ratio A is set to 5. Also, the x coordinate of the center of the correct answer data corresponding to the reference image 400 is set to gt x and the y coordinate is gt y The reference feature acquisition unit 102 calculates the (gt x ,gt y ) and cuts out a square rectangle 401 with a side length of E from the reference image 400. In the cropping process, for example, pixels in the crop rectangle that extend beyond the reference image 400 are filled with black (R, G, B=0, 0, 0).

[0038] Next, the reference feature acquisition unit 102 performs a scaling process on the image of the crop rectangle 401 to convert it into an image 402 with a specific resolution. Here, the resolution conversion process by scaling is a process for matching the input resolution to the subsequent process. That is, the reference feature acquisition unit 102 includes a reference feature extractor 403 shown in FIG. 4(a), and performs a resolution conversion process by scaling to match the input resolution of the reference feature extractor 403.

[0039] For example, if the length of one side of a square image input to the reference feature extractor 403 is F, the reference feature acquisition unit 102 needs to enlarge or reduce the image of the crop rectangle 401 described above by a magnification r shown in equation (2).

[0040] r=F / E formula (2)

[0041] In this embodiment, the square image input to the reference feature extractor 403 is assumed to have a width of 128 pixels and a height of 128 pixels. The reference feature extractor 403 outputs reference features 404 extracted from the image 402 after the aforementioned cropping and scaling processes as intermediate outputs. For example, the reference feature extractor 403 uses GoogLeNet, a type of convolutional neural network, and acquires, as the reference feature 404, the output of an intermediate layer whose resolution is 1 / 16 of the input resolution in the internal processing of GoogLeNet. If the length F of one side of the square image input to the reference feature extractor 403 is 128 pixels, the reference feature 404 will have a width of 8 pixels, a height of 8 pixels, and an output of 832 channels. In this embodiment, for example, if a convolutional neural network has a 3×3 convolutional layer, the 3×3 kernel parameters are adjustable parameters in the reference feature extractor 403.

[0042] <Search feature acquisition process> Next, the search feature acquisition process performed by the search feature acquisition unit 103 will be described with reference to FIG. The search feature acquisition unit 103 basically performs a process of acquiring search features from the search image 410 using the same procedure as the reference feature acquisition process described above. Only the processes that are different from the reference feature acquisition process will be explained below. In the reference feature acquisition process described above, the area ratio A used in the crop process was 5, but in the search feature acquisition process, the area ratio A is set to 20. In addition, the reference feature acquisition unit 102 acquires the width gt of the correct answer data. w and height gt h The x-coordinate of the center of gt x and y coordinate gt y A square crop rectangle 401 with a side length of E and centered at is cut out from the reference image 400. In response to this, the search feature acquisition unit 103 extracts the x coordinate d x and y-coordinate d y A square crop rectangle 411 with a side length of E is cut out with the center at .

[0043] (d x ,dy )=(gt x +e x ,gt y +e y ) Formula (3)

[0044] In addition, e in equation (3) x and e y are parameters that represent perturbations of the x- and y-coordinates of the center. In the example of FIG. 4(b), the arrows in the crop rectangle 411 represent perturbations of the x- and y-coordinates of the center. The search feature acquisition unit 103 selects the strength of the perturbation to be applied to the center coordinates from, for example, ±E / 2, depending on the time interval between the reference image and the search image. Note that while an example of applying a perturbation to the search image has been given here, a perturbation of a strength depending on the time interval between the reference image and the search image may be applied to the reference image, or perturbations may be applied to both the search image and the reference image.

[0045] Next, the search feature acquisition unit 103 performs a scaling process on the image of the crop rectangle 411 to convert it into an image 412 with a specific resolution. In the search feature acquisition unit 103, the resolution conversion process by scaling is also performed to match the input resolution to the subsequent process. That is, the search feature acquisition unit 103 is provided with a search feature extractor 413 shown in FIG. 4(b), and performs a resolution conversion process by scaling to match the input resolution of the search feature extractor 413.

[0046] In this embodiment, the square image input to the search feature extractor 413 is assumed to have a width of 256 pixels and a height of 256 pixels. The search feature extractor 413 then outputs search features 414 extracted from the image 412 after the aforementioned cropping and scaling processes as intermediate outputs. Similar to the reference feature extractor 403, the search feature extractor 413 uses, for example, GoogLeNet, a type of convolutional neural network, to obtain, as the search features 414, the output of an intermediate layer whose resolution is 1 / 16 of the input resolution. If the square image 412 input to the search feature extractor 413 has a width of 256 pixels and a height of 256 pixels, the search features 414 will be 16 pixels wide and 16 pixels high, resulting in an output of 832 channels. In this embodiment, for example, if a convolutional neural network has a 3×3 convolutional layer, the 3×3 kernel parameters are adjustable parameters in the search feature extractor 413.

[0047] <Comparison process between reference feature and search feature> Next, the comparison process between the reference feature and the search feature performed by the tracking unit 104 in step S205 of FIG. 2 will be described. First, the tracker 104 performs depthwise separable convolution on the reference features acquired by the reference feature acquisition unit 102 and the search features acquired by the search feature acquisition unit 103. Depthwise separable convolution is a convolution method that simplifies calculations by separating the 3D convolution process using a group of kernels in a typical 2D convolution layer into a two-stage process consisting of depthwise convolution (2D) and pointwise convolution (1D), which are equivalent to the 3D convolution process. Note that depthwise separable convolution is also sometimes called channelwise separable convolution. The tracker 104 also performs zero padding so that the sizes of the output features and the search features match. For example, the tracker 104 performs convolution on the obtained features with a kernel size of 3×3 pixels, 832 input channels, and 5 output channels. At this time, the tracker 104 performs zero padding to match the size, as in the depth-unit separable convolution, and as a result, a feature with a width of 16 pixels, a height of 16 pixels, and five channels is obtained. In this embodiment, for example, if a convolutional neural network has a 3 × 3 convolution layer, the 3 × 3 kernel parameters are adjustable parameters in the tracker 104.

[0048] <Inference of the position and size of the tracked target> Next, the process of inferring the position and size of the tracking target performed by the tracking unit 104 in step S206 of FIG. 2 will be described. First, the tracking unit 104 interprets the five channels of the features obtained in the comparison process in step S205 as a likelihood map M, an x-direction positional deviation map X, a y-direction positional deviation map Y, a width map W, and a height map H, respectively, and infers the position and size of the tracked target. Here, the x-coordinate at which the value of the likelihood map M is maximum is defined as p x , y coordinate of p y In this case, if there are multiple coordinates where the value of the likelihood map M is the maximum, the tracking unit 104 selects the coordinate that is closest to the center of the likelihood map. Also, if the distances to the center of the likelihood map are also equal, the tracking unit 104 prioritizes the coordinate that is first reached when raster scanning the likelihood map in the horizontal direction. Then, the tracking unit 104 calculates the estimated value t of the x-coordinate of the position of the tracking target.x and the inferred value of the y-coordinate t y is calculated using equation (4).

[0049] (t x ,t y )=(p x ×16+X(p x ,p y ),p y ×16+Y(p x ,p y )) Formula (4)

[0050] In addition, X(p x ,p y ) is the x-coordinate p in the x-direction position deviation amount map X x and the y-coordinate p y Y(px,py) is the x-coordinate p in the y-direction positional deviation map Y. x and the y-coordinate p y The estimated width of the tracking target is t w =W(p x ,p y ), the estimated height is t h =H(p x ,p y )

[0051] <Calculation of the error between the inference result and the correct data> Next, the error calculation process between the inference result and the correct answer data, which is performed by the update unit 105 in step S207 of FIG. 2, will be described. 2, the update unit 105 calculates the error between the inference result and the supervised data. In this embodiment, the update unit 105 calculates the error L between the inference result and the supervised data using equation (5).

[0052] L=I1L M +I2L d +I3L s Formula (5)

[0053] In addition, L in formula (5) M represents the error on the likelihood map, and L d is the error related to the positional deviation, and L srepresents the error related to size. The details of these will be described later. Also, I1, I2, and I3 are hyperparameters, and I1 is L M is a real number greater than or equal to 0 that represents the importance of d A real number greater than or equal to 0 that indicates the importance of s In this embodiment, the update unit 105 changes the values ​​of the hyperparameters I1, I2, and I3 according to the time interval between the reference image and the search image, thereby changing the importance of the hyperparameters I1, I2, and I3 in optimizing the parameters of the multilayer neural network.

[0054] Below, the error L M This article explains: Here, an example will be described in which an image is cut out with the area ratio A set to 20 in the search feature acquisition process in step S204. From the above-mentioned formula (3), the center coordinates of the tracking target are calculated by the perturbation e x ,e y The update unit 105 creates a map of the same size as the search image extracted in step S204, and initializes all elements to 0. Furthermore, the ... x and y-coordinate e y The update unit 105 draws a two-dimensional normal distribution centered on a coordinate system spaced apart by a distance of 1000. The update unit 105 also calculates the width and height of the correct data corresponding to the search image by dividing the diagonal components of the covariance matrix Σ of the normal distribution by the width gt w and height gt h In this embodiment, the update unit 105 makes the adjustment as shown in equation (6).

[0055]

number

[0056] Furthermore, the update unit 105 normalizes all elements of the map on which the normal distribution is drawn to a minimum value of 0 and a maximum value of 1 by dividing the elements by the maximum value of the map. The update unit 105 also reduces this to the same size as the search feature (16×16 pixels) and creates a correct likelihood map GTM Here, the error L M is the likelihood map M and the correct likelihood map GT M Using this, it can be calculated as in equation (7).

[0057]

number

[0058] In equation (7), i represents the index in the x direction of the likelihood map, and j represents the index in the y direction of the likelihood map. Next, the error L d This will be explained using FIG. First, the update unit 105 creates a two-dimensional mesh grid of the same size as the x-direction positional deviation amount map X or the y-direction positional deviation amount map Y. The mesh grid 500 in FIG. x The mesh grid 501 in FIG. 5(b) is a y-direction mesh grid G y In this embodiment, the size of each mesh grid is 16x16 pixels, and each element takes a value from 0 to 15. Also, the ratio between the input resolution and the output resolution of the search feature extractor 413 is set to z. In this embodiment, the ratio z between the input resolution and the output resolution is 16 (=256 / 16). Furthermore, the update unit 105 performs the calculation of equation (8) on each element of the mesh grid to generate a correct x-direction positional deviation amount map GT X and the correct y-direction position deviation map GT Y and calculate.

[0059]

number

[0060] In equation (8), i represents the index in the x direction of the mesh grid, and j represents the index in the y direction of the mesh grid. x ) is the x-direction mesh grid Gx is the largest element of the y-direction mesh grid G y The largest element max(G y ), which is 15. r is the scaling factor of the cropped image, as shown in equation (2). The mesh grid 502 in FIG. 5(a) is x =-102, e y Correct x-direction position deviation map GT when =83 X The mesh grid 503 in FIG. 5(a) is an example of x =-102, e y Correct y-direction position deviation map GT when =83 Y This is an example. The update unit 105 also updates the correct x-direction positional deviation map GT X and the inference map X and the correct y-direction position deviation map GT Y and the inference map Y, the error L regarding the positional deviation amount is calculated. d is calculated as in equation (9).

[0061]

number

[0062] In equation (9), i represents the index in the x direction of the likelihood map, and j represents the index in the y direction of the likelihood map. Next, the size error L s This article explains: First, the correct width map GT w Or Correct Height Map GT h The update unit 105 calculates the elements as in equation (10) and creates a correct width map GT w and ground truth height map GT h Get.

[0063]

number

[0064] i in equation (10) is the correct width map GTw Or Correct Height Map GT h j represents the index in the x direction of the correct width map GT w Or Correct Height Map GT h r is the scaling factor of the cropped image as shown in the above formula (2). The update unit 105 updates the correct width map GT w and the inference map W and the correct height map GT h and the inference map H, the error L s is calculated as in equation (11).

[0065]

number

[0066] <Parameter update process based on errors> Next, the parameter update process performed by the update unit 105 in step S208 of FIG. 2 will be described. In step S208 of Fig. 2, the update unit 105 updates the parameters of the reference feature acquisition unit 102, the search feature acquisition unit 103, and the tracking unit 104 of Fig. 1 based on the error L calculated by equation (5). The update unit 105 updates the parameters using, for example, a stochastic gradient descent method so as to minimize the error. Thereafter, in step S209, the information processing device 100 ends the process after, for example, a predetermined number of updates have been performed.

[0067] <Advantages of the First Embodiment> The effects of the information processing device of this embodiment will be described below using an example in which a gymnast performing floor exercises is photographed and tracked with a digital camera and continuously focused using an autofocus function. To achieve a tracking function that continuously tracks gymnasts performing floor exercises, it is necessary to collect a large number of videos of people performing floor exercises and prepare learning data to which correct answer data is added. Existing learning techniques randomly select adjacent frames or sets of images within a predetermined frame interval from a given video, and use these as reference and search images for learning. However, with existing learning techniques, frames showing a gymnast taking a run-up during a floor exercise are often selected as reference and search images. Meanwhile, a tracking function for a gymnast performing floor exercises is desired that can track not only a gymnast taking a run-up, but also a gymnast undergoing posture changes due to jumps, twists, and the like. However, if, for example, images of frames showing a gymnast taking a run-up are selected as reference and search images, learning for posture changes such as jumps and twists is inefficient, making it difficult to ensure accurate tracking of a gymnast during posture changes.

[0068] In contrast, the information processing device of this embodiment performs learning using reference images and search images that are likely to be at least a set time interval away from the video. With such a set of images, it becomes easier to select reference images and search images with pose variations between images compared to the conventional learning technology described above. In other words, the information processing device of this embodiment makes it possible to efficiently learn subjects with varying poses, enabling learning that can achieve object tracking that is resistant to changes in the tracking target even from video data with varying lengths. Therefore, the information processing device of this embodiment can track athletes and other objects whose poses change with greater accuracy.

[0069] <Second embodiment> Next, as a second embodiment, an example will be described in which a reference image and a search image to be used for learning are selected by focusing on the similarity between the reference image and the search image. Note that the configuration of the information processing device 100 of the second embodiment is the same as in Fig. 1, and the flow of information processing is also generally the same as the flowchart in Fig. 2, so illustration and description thereof will be omitted. In the second embodiment, the selection process of the reference image and the search image in step S202 in FIG. 2 is different from that in the first embodiment.

[0070] FIG. 3B is a detailed flowchart of the reference image and search image selection process performed in step S202 of the second embodiment. In step S310, the reference feature acquisition unit 102 or the search feature acquisition unit 103 performs preprocessing on the image, similar to the cropping and scaling processes described in the first embodiment, on the image of each frame of the moving image selected by the image acquisition unit 101. The preprocessing in step S310 may be performed by either the reference feature acquisition unit 102 or the search feature acquisition unit 103, but may also be performed by both.

[0071] Next, in step S311, the reference feature acquisition unit 102 or the search feature acquisition unit 103 calculates the similarity between the preprocessed images in step S310 for all frames in the video. The similarity calculation process in step S311 may be performed by either the reference feature acquisition unit 102 or the search feature acquisition unit 103, or may be performed by both.

[0072] Next, in step S312, the reference feature acquisition unit 102 or the search feature acquisition unit 103 calculates a sampling probability based on the similarity calculated in step S311. The calculation process of the sampling probability in step S312 may be performed by either the reference feature acquisition unit 102 or the search feature acquisition unit 103, or may be performed by both. Note that, although this embodiment gives an example in which the sampling probability is calculated based on the similarity, the sampling probability may also be determined based on the time interval described in the first embodiment.

[0073] Then, the reference feature acquisition unit 102 selects a reference image based on the sampling probability from the moving image selected by the image acquisition unit 101. Also, the search feature acquisition unit 103 selects a search image based on the sampling probability from the moving image selected in step S201.

[0074] The following describes in detail the image preprocessing, similarity calculation processing, and sampling probability calculation processing in the second embodiment. In this embodiment, the image preprocessing, similarity calculation processing, and sampling probability calculation processing are performed by either or both of the reference feature acquisition unit 102 and the search feature acquisition unit 103, but here we will not distinguish between the two and will describe them as processing performed in the feature acquisition unit.

[0075] <Image pre-processing> First, the image pre-processing in step S310 in FIG. 3(b) will be described. As a preprocessing step for step S310, the feature acquisition unit performs cropping and scaling on all images converted into the image sequence in step S200 of FIG. 2, i.e., the RGB images with a width of 4000 pixels and a height of 3000 pixels as described above.

[0076] Here, we focus on one image in the image sequence and calculate the width of the correct answer data corresponding to that image as gt w and the height gt h In the second embodiment, the feature acquisition unit calculates the length E of one side of the square crop rectangle using equation (1), as in the first embodiment. In the case of image preprocessing in the second embodiment, the above-mentioned area ratio A is set to 3. Also, the x-coordinate of the center of the correct answer data corresponding to the image of interest is set to gt x and the y coordinate is gt y The feature acquisition unit performs cropping in the preprocessing of the image by determining the x-coordinate of the center of the correct data gt x and y coordinate gt yA rectangle with a side length of E and centered at is cut out from the image. Note that even during the pre-processing of the second embodiment, pixels that extend beyond the image in the crop rectangle are filled with black (R, G, B = 0, 0, 0). The feature acquisition unit performs a scaling process on the cropped rectangular image cut out by the cropping process using the scaling factor r shown in the above-mentioned equation (2). Note that the length F of one side of the image after the scaling process is set to 32 pixels. In the second embodiment, the feature acquisition unit applies the above pre-processing to all images included in the image sequence.

[0077] <Similarity calculation process> In step S311, the feature acquisition unit calculates the similarity between the preprocessed images corresponding to all frames in the video. First, the feature acquisition unit inputs all preprocessed images into an image feature extractor to acquire feature vectors. In this embodiment, the feature vector is a 1024-dimensional vector obtained by grayscaling the preprocessed image and raster scanning it horizontally. Here, if the two feature vectors are u and v, respectively, the similarity sim can be calculated using the absolute value of the cosine similarity as shown in Equation (12).

[0078]

number

[0079] In equation (12), the numerator indicates the absolute value of the inner product of the feature vectors, and the denominator indicates the product of the norms of each feature vector. The value range of similarity sim is 0≦sim≦1, where 0 is the value when the two features are orthogonal, and 1 is the value when they match regardless of sign. The feature acquisition unit calculates similarity sim for all pairs of feature vectors.

[0080] FIG. 6(a) is a table showing an example of similarity sim when an image sequence consists of, for example, five images, frame 1 to frame 5. Because feature vectors corresponding to the same image are identical, the diagonal elements of FIG. 6(a), which indicate the similarity between the same images, are 1. Furthermore, since the result of Equation (12) remains unchanged even if feature vector u and feature vector v are swapped, FIG. 6(a) is a symmetric matrix. The feature vector extraction method and similarity calculation method described in this embodiment are merely examples. Color histograms, HOG features, SIFT features, etc. may also be used to extract feature vectors, and meta-information such as the image capture time interval and location data of the shooting locations may also be used. Furthermore, similarity calculation methods such as Mahalanobis distance and Bhattacharyya distance may also be used. Similarity may also be obtained using a multilayer neural network trained to combine feature vector extraction and similarity calculation.

[0081] <Calculation of sampling probability> In step S313, the feature acquisition unit calculates the sampling probability based on the similarity sim calculated in step S311. In this embodiment, the feature acquisition unit first calculates the dissimilarity nsim from the similarity sim using equation (13).

[0082] nsim=1-sim Equation (13)

[0083] FIG. 6(b) is a diagram showing the dissimilarity corresponding to FIG. 6(a) in the form of a table. Then, assuming that the sampling probability of the i-th frame as the reference feature and the j-th frame as the search feature is p(i,j), the feature acquisition unit calculates the sampling probability p(i,j) using the temperature softmax function according to equation (14).

[0084]

number

[0085] In addition, nsim in equation (14) i,jis the dissimilarity between the i-th frame and the j-th frame. k and m represent all pairs of frames, and nsim k,m is the dissimilarity between the kth frame and the mth frame. T is a hyperparameter such that 0≦T. For example, when T=0, the sampling probability for any pair of images other than those with the maximum dissimilarity is 0. As the value of T increases, the probability of selecting a pair of images with low dissimilarity increases, and when T=∞, the sampling probability becomes uniform.

[0086] Figure 6(c) is a table showing the sampling probability corresponding to Figure 6(b), where T=0.1. In the second embodiment, the reference feature acquisition unit 102 selects the i-th frame sampled based on the sampling probability described above as the reference image, and the search feature acquisition unit 103 selects the j-th frame as the search image. In this case, the selected reference image and search image are RGB images with a width of 4000 pixels and a height of 3000 pixels before the image preprocessing in step S301 of FIG. 3 .

[0087] <Effects of the second embodiment> The effect of the information processing device of the second embodiment will be described using an example in which a flapping bird is photographed and tracked with a digital camera and continuously focused using an autofocus function. To achieve a tracking function that continuously tracks a flapping bird, it is necessary to collect a large number of videos of the flapping bird, assign ground truth data, and prepare training data. In the case of existing learning techniques, a set of images is often selected from a video of a bird flying linearly, and learning is performed using these as reference and search images. However, the posture change of a bird flying linearly is periodic. Therefore, if, for example, reference and search images with the same posture are selected, efficient learning that corresponds to the posture change due to the bird's flapping wings cannot be performed, and as a result, it is likely to be difficult to ensure tracking accuracy.

[0088] In contrast, in the information processing device of the second embodiment, a reference image and a search image are selected based on dissimilarity, so that even if the posture change is periodic, for example, a reference image and a search image of different postures are selected, and learning that corresponds to the posture change can be performed efficiently. That is, the information processing device of the second embodiment enables efficient learning of a subject whose posture changes, in other words, learning that can achieve object tracking that is resistant to posture changes of the tracking target. As a result, the information processing device of the second embodiment makes it possible to track, with high accuracy, a bird whose posture changes due to flapping of its wings, for example.

[0089] <Third embodiment> Next, as a third embodiment, an example will be described in which a search image is obtained by applying a perturbation based on the similarity in the search feature acquisition process, that is, the higher the similarity (or, the lower the dissimilarity), the greater the perturbation intensity. The perturbation based on the similarity may be applied not only to the search image but also to the reference image, or may be applied to both the reference image and the search image. The configuration and processing of the information processing device 100 according to the third embodiment are the same as those shown in FIGS. 1 and 2, and therefore illustrations and descriptions thereof will be omitted.

[0090] In the case of the first embodiment described above, in step S204 of FIG. 2, the search feature acquisition unit 103 calculates a perturbation e for the x-coordinate and y-coordinate of the center according to the time interval between the reference image and the search image. x ,e y In contrast, in the third embodiment, a perturbation e according to the time interval between the reference image and the search image is given. x ,e y After applying correction according to the similarity to x ,ce y Equation (15) gives the corrected perturbation ce according to the similarity. x ,ce y is the formula.

[0091] (ce x ,se y )=(sim×e x , sim×ey ) Formula (15)

[0092] In equation (15), the corrected perturbation ce x , ce y The higher the similarity, the stronger the perturbation, and the lower the similarity, the weaker the perturbation. Note that in this embodiment, only perturbation is given to the central coordinates, but other perturbations such as affine transformation and color transformation may also be added in combination.

[0093] In the following, an image transformation using a combination of affine transformations will be described as an example of image transformation that applies perturbation according to similarity. In the first embodiment, when a search image is cut out, a perturbation e x ,e y In the third embodiment, however, instead of applying perturbations when cutting out an image, image transformation is performed using a synthesis of affine transformations according to the degree of similarity. In this embodiment, the length E of one side of a clock rectangle cut out from a search image without applying perturbations is assumed to be 256 pixels. In the third embodiment, the search feature acquisition unit 103 applies a process of synthesizing affine transformations shown in equations (16) to (20) to the image of the cropped rectangle cut out by cropping processing, with the center of the image as a reference, based on the degree of similarity.

[0094]

number

[0095] In equation (16), FLX represents a mirror transformation in the x direction. In equation (17), SHI represents a parallel translation transformation of the image. x is the amount of translation in the direction, and shi y indicates the amount of translation in the y direction. SCA in equation (18) indicates the scaling process of the image, and sca x is the scaling factor in the x direction, and sca y indicates the scaling ratio in the y direction. SHE in equation (19) indicates the shearing process of the image, and deg x is the shear angle in the x direction, deg yrepresents the shear angle in the y direction. In equation (20), ROT represents the image rotation process, and deg represents the rotation angle. In the third embodiment, each value is a random number determined from the range shown in equation (21).

[0096]

number

[0097] Moreover, the composite matrix C of the affine transformation in this embodiment is defined by equation (22).

[0098]

number

[0099] In equation (22), · (dot operator) represents the dot product of matrices. The function sel() generates a uniform random number rand such that 0≦rand≦1. If the condition rand≦sim is satisfied, it takes the value of a given transformation matrix (such as ROT), and if the condition is not satisfied, it takes the identity matrix of 3 rows and 3 columns. The corrected perturbation ce in the third embodiment x ,ce y is the element c of the composite matrix C of the affine transformation 11 From c 23 can be calculated using equation (23).

[0100]

number

[0101] <Fourth embodiment> Next, as a fourth embodiment, an example will be described in which the update unit 105 takes dissimilarity into consideration when calculating the error between the inference result and the ground truth data in step S207 of Fig. 2. Note that the configuration and processing of the information processing device 100 according to the fourth embodiment are the same as those shown in Figs. 1 and 2, and therefore illustration and description thereof will be omitted. In the first embodiment, the error L is calculated using equation (5), but in the case of the fourth embodiment, the update unit 105 calculates the corrected error cL as shown in equation (24) based on the dissimilarity nsim.

[0102] cL=nsim×L Equation (24)

[0103] From equation (24), the higher the dissimilarity nsim, the larger the corrected error cL. Therefore, when updating parameters using the corrected error cL as in step S208 of FIG. 2, the larger the error, the more intensively the parameters are updated to reduce the error. In other words, according to equation (24), images with high dissimilarity are given more importance in parameter optimization, and are more likely to be given more priority in parameter updating than images with low dissimilarity.

[0104] Furthermore, in the fourth embodiment, the number of times that parameter updates have been performed may be added as the importance given to parameter optimization. In the fourth embodiment, the predetermined number determined in step S209 of FIG. 2 is set to, for example, 10,000 times. When the current number of parameter updates is set to iter, the update unit 105 calculates the corrected error cL as shown in equation (25).

[0105]

number

[0106] According to equation (25), when the number of parameter updates is small, updates are performed without considering dissimilarity, and after updating the parameters half the number of times (500 times) of the preset number of times, the update unit 105 calculates the corrected error cL according to the dissimilarity. Note that the preset number of times of 1000 in this embodiment and the number of times 500 at which dissimilarity consideration begins are merely examples, and the present invention is not limited to these.

[0107] <Hardware configuration of information processing device> FIG. 7 is a diagram showing an example of a hardware configuration applicable to the information processing apparatus of each of the above-described embodiments. The CPU 700 is a central processing unit that performs calculations and logical decisions for various processes. The ROM (Read-Only-Memory) 701 stores a control program. The RAM (Random Access Memory) 702 is used as a temporary storage area such as the main memory and work area of ​​the CPU 700.

[0108] The mass storage device 703 is a storage device for storing various data such as the information processing program according to this embodiment, image data, and correct answer data. The CPU 700 executes the information processing program according to this embodiment that is read from the mass storage device 703 and loaded into the RAM 702, thereby realizing the information processing described in each of the above-mentioned embodiments. An external storage device may be used that performs a similar function to the mass storage device 703. The external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include a flexible disk (FD), CD-ROM, DVD, USB memory, MO, and flash memory. The external storage device may also be a server device connected via a network.

[0109] The input unit 704 is composed of a keyboard, a touch panel, various buttons, etc., and accepts input from the user. The display unit 705 is composed of a liquid crystal display, etc., and can display various data, information processing results, etc. to the user. In addition, this device can communicate with other devices, such as the imaging device 110, via the communication unit 706. This device may receive user instructions from other devices, or may output processing results to other devices, via the communication unit 706. The information processing device of this embodiment can be realized by a general-purpose personal computer, tablet terminal, smartphone, etc., having the above-mentioned configuration.

[0110] In the first embodiment described above, a pair of reference image and search image is selected based on a set time interval, and in the second embodiment, a pair of reference image and search image is selected based on dissimilarity, but a pair of reference image and search image may be selected using both the set time interval and dissimilarity. For example, even if two images are spaced apart by an interval equal to or greater than the set time interval, there may be little or no change in the posture of the tracking subject, in which case it may be possible to select two images with high dissimilarity.

[0111] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0112] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) an image acquisition means for acquiring a plurality of time-series images of a target subject; a learning means for inferring a target subject in the search image relative to a target subject in the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data; and The information processing device is characterized in that the learning means selects the reference image and the search image from the plurality of images based on at least one of the times at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series. (Configuration 2) The learning means a reference feature acquisition means for acquiring image features from a reference image acquired from the plurality of images in the time series using a neural network; a search feature acquisition means for acquiring image features from a search image acquired from the plurality of images in the time series using a neural network; an inference means for inferring the target subject of the search image relative to the target subject of the reference image using a neural network based on image features of the reference image and image features of the search image; an update means for updating, based on the inference result by the inference means and correct answer data, parameters of at least one of the neural network used by the reference feature acquisition means to acquire image features of the reference image, the neural network used by the search feature acquisition means to acquire image features of the search image, and the neural network used by the inference means to make the inference; 2. The information processing device according to configuration 1, comprising: (Configuration 3) 3. The information processing device according to configuration 2, wherein the inference means tracks the target subject by inferring the target subject in the search image from the target subject in the reference image. (Configuration 4) 4. The information processing device according to any one of configurations 1 to 3, wherein the learning means updates the parameters only when a time interval between the reference image and the search image is equal to or greater than the predetermined time interval. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the learning means selects the reference image and the search image according to a sampling probability based on a time interval between the reference image and the search image. (Configuration 6) The information processing device described in any one of configurations 1 to 6, characterized in that the learning means selects the reference image and the search image according to a sampling probability based on the dissimilarity between the reference image and the search image. (Configuration 7) The information processing device according to any one of configurations 1 to 6, wherein the learning means applies a perturbation to at least one of the reference image and the search image, the perturbation having an intensity that varies depending on the time interval between the reference image and the search image. (Configuration 8) The information processing device according to any one of configurations 1 to 7, wherein the learning means applies a perturbation to at least one of the reference image and the search image, the perturbation having a strength that varies depending on the dissimilarity between the reference image and the search image. (Configuration 9) 9. The information processing apparatus according to any one of configurations 1 to 8, wherein the learning means changes the importance of updating the parameters according to the time interval between the reference image and the search image. (Configuration 10) 10. The information processing device according to any one of configurations 1 to 9, wherein the learning means changes the importance of updating the parameters depending on the dissimilarity between the reference image and the search image. (Configuration 11) The information processing device according to configuration 10, wherein the learning means uses, as the importance, a corrected error obtained by correcting the error between the target subject inferred from the search image and the correct data based on the dissimilarity. (Configuration 12) 12. The information processing apparatus according to configuration 11, wherein the learning means calculates the corrected error in accordance with the number of times the parameter is updated. (Method 1) an image acquisition step of acquiring a plurality of time-series images of a target subject; a learning process of inferring a target subject in the search image relative to a target subject in the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data; and An information processing method characterized in that, in the learning process, the reference image and the search image are selected from the plurality of images based on at least one of the times at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series. (Program 1) 13. A program that causes a computer to function as the information processing device according to any one of configurations 1 to 12. [Explanation of symbols]

[0113] 100: Information processing device, 101: Image acquisition unit, 102: Reference feature acquisition unit, 103: Search feature acquisition unit, 104: Tracking unit, 105: Update unit, 106: Result output unit, 110: Imaging device

Claims

1. an image acquisition means for acquiring a plurality of time-series images of a target subject; a learning means for inferring a target subject in the search image relative to a target subject in the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data; and The information processing device is characterized in that the learning means selects the reference image and the search image from the plurality of images based on at least one of the times at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series.

2. The learning means a reference feature acquisition means for acquiring image features from a reference image acquired from the plurality of images in the time series using a neural network; a search feature acquisition means for acquiring image features from a search image acquired from the plurality of images in the time series using a neural network; an inference means for inferring the target subject of the search image relative to the target subject of the reference image using a neural network based on image features of the reference image and image features of the search image; an update means for updating, based on the inference result by the inference means and correct answer data, parameters of at least one of the neural network used by the reference feature acquisition means to acquire image features of the reference image, the neural network used by the search feature acquisition means to acquire image features of the search image, and the neural network used by the inference means to make the inference; 2. The information processing apparatus according to claim 1, further comprising:

3. 3. The information processing apparatus according to claim 2, wherein the inference means tracks the target subject by inferring the target subject in the search image from the target subject in the reference image.

4. 2. The information processing apparatus according to claim 1, wherein said learning means updates said parameters only when the time interval between said reference image and said search image is equal to or greater than said predetermined time interval.

5. 2. The information processing apparatus according to claim 1, wherein said learning means selects said reference image and said search image in accordance with a sampling probability based on a time interval between said reference image and said search image.

6. 2. The information processing apparatus according to claim 1, wherein the learning means selects the reference image and the search image in accordance with a sampling probability based on a dissimilarity between the reference image and the search image.

7. 2. The information processing apparatus according to claim 1, wherein the learning means applies a perturbation to at least one of the reference image and the search image, the perturbation having an intensity that varies depending on the time interval between the reference image and the search image.

8. 2. The information processing apparatus according to claim 1, wherein the learning means applies a perturbation to at least one of the reference image and the search image, the perturbation having an intensity that varies depending on the dissimilarity between the reference image and the search image.

9. 2. The information processing apparatus according to claim 1, wherein said learning means changes the importance of updating said parameters in accordance with the time interval between said reference image and said search image.

10. 2. The information processing apparatus according to claim 1, wherein said learning means changes the importance of updating said parameters according to the degree of dissimilarity between said reference image and said search image.

11. 11. The information processing apparatus according to claim 10, wherein the learning means uses, as the importance, a corrected error obtained by correcting an error between the target subject inferred from the search image and ground truth data based on the dissimilarity.

12. 12. The information processing apparatus according to claim 11, wherein the learning means calculates the corrected error in accordance with the number of times the parameter is updated.

13. an image acquisition step of acquiring a plurality of time-series images of a target subject; a learning process of inferring a target subject in the search image relative to a target subject in the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data; and An information processing method characterized in that, in the learning process, the reference image and the search image are selected from the plurality of images based on at least one of the times at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series.

14. Computer, an image acquisition means for acquiring a plurality of time-series images of a target subject; a learning means for inferring a target subject in the search image relative to a target subject in the reference image based on a reference image and a search image acquired from the plurality of images in the time series, and updating parameters of a neural network based on the inference result and ground truth data; and The learning means is a program that functions as an information processing device that selects the reference image and the search image from the plurality of images based on at least one of the time at which the plurality of images in the time series were captured and a predetermined time interval, and the dissimilarity between the plurality of images in the time series.

Images