Information processing device, inference method, inference program, and method for generating feature generation model

Abstract

In order to enable context-based inference while suppressing calculation cost, this information processing device (2) is provided with: a feature value generation unit (21) that uses time information indicating timing at which a frame image was captured and position information indicating the detection position of an object in the frame image to generate a feature value according to the context; and an inference unit (22) that makes an inference on the basis of the generated feature value.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and the like that makes an inference about an object using a moving image of the object.

Background Art

[0002] In recent years, by using technologies such as deep learning, it has become possible to perform object detection, object identification, etc. (hereinafter collectively referred to as inferences) on images with very high accuracy. Research on inferences for moving images is also underway.

[0003] As one of the measures to improve the accuracy of inferences for moving images, it is known to perform inferences in consideration of the context of the moving image. For example, Non-Patent Document 1 below discloses a technique for extracting context feature amounts from a moving image using a learned convolutional neural network and identifying the actions of a person shown in the moving image using the context feature amounts.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the above-described conventional technologies have a problem that the computational cost when inputting a moving image into a convolutional neural network to extract context feature amounts is very large. One aspect of the present invention aims to realize an information processing apparatus and the like that can perform inferences considering the context while suppressing the computational cost. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present invention comprises a feature generation means that generates feature quantities of the object captured in a frame image according to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and inference means that performs predetermined inferences regarding the object based on the feature quantities.

[0007] An inference method relating to one aspect of the present invention includes, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, a processor generating context-dependent feature quantities of the object captured in the frame image using position information indicating the detection position of the object in the frame image and time information indicating the timing when the frame image was captured, and performing predetermined inferences about the object based on the feature quantities.

[0008] An inference program according to one aspect of the present invention causes a computer to function as a feature generation means that generates feature quantities of the object depicted in a frame image according to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and an inference means that performs predetermined inferences about the object based on the feature quantities.

[0009] A method for generating a feature generation model according to one aspect of the present invention includes, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, a feature generation model for generating context-specific features, which is then used to input time information indicating when the frame image was captured and position information indicating the detected position of the object in the frame image, and to perform a predetermined inference about the object based on the calculated features, and to update the feature generation model so that the result of the inference approaches predetermined ground truth data. [Effects of the Invention]

[0010] According to one aspect of the present invention, it becomes possible to perform context-aware inference while reducing computational costs. [Brief explanation of the drawing]

[0011] [Figure 1] This is a block diagram showing the configuration of an information processing device according to an exemplary embodiment 1 of the present invention. [Figure 2] This is a flowchart showing the flow of a method for generating and inferring a feature generation model according to exemplary embodiment 1 of the present invention. [Figure 3] This is a diagram illustrating the method for foreign object detection inspection. [Figure 4] This figure illustrates an overview of the inference method according to exemplary embodiment 2 of the present invention. [Figure 5] This is a block diagram showing an example configuration of an information processing device according to Exemplary Embodiment 2 of the present invention. [Figure 6] This figure shows an example where a difference occurred between the context during training and during inference. [Figure 7] This is a flowchart showing the processing flow performed by the information processing device according to Exemplary Embodiment 2 of the present invention during learning. [Figure 8] This is a flowchart showing the processing flow performed by the information processing device according to Exemplary Embodiment 2 of the present invention during inference. [Figure 9]FIG. 0 is a diagram showing an example of a computer that executes program instructions of software that realizes each function of each device according to each exemplary embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

[0012] 〔Exemplary Embodiment 1〕 The first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later. First, the information processing apparatuses 1 and 2 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configurations of the information processing apparatuses 1 and 2.

[0013] (Configuration of Information Processing Apparatus 1) As shown in the figure, the information processing apparatus 1 includes an inference unit 11 and a learning unit 12. The inference unit 11, for each of a plurality of frame images extracted from a moving image obtained by photographing an object moving along a predetermined context, inputs time information indicating the timing at which the frame image was taken and position information indicating the detection position of the object in the frame image into a feature quantity generation model for generating a feature quantity according to the context, and performs a predetermined inference regarding the object based on the calculated feature quantity

[0014] The learning unit 12 updates the feature quantity generation model so that the result of the inference by the inference unit 11 approaches predetermined correct answer data.

[0015] As described above, the information processing apparatus 1 according to the present exemplary embodiment performs a predetermined inference regarding the object based on feature amounts calculated by inputting, for each of a plurality of frame images extracted from a moving image capturing an object moving along a predetermined context, time information indicating the timing at which the frame image was captured and position information indicating the detection position of the object in the frame image into a feature amount generation model for generating feature amounts corresponding to the context. The information processing apparatus 1 includes an inference unit 11 and a learning unit 12. The learning unit 12 updates the feature amount generation model so that the result of the inference by the inference unit 11 approaches predetermined correct data.

[0016] According to the above configuration, it is possible to generate a feature amount generation model capable of generating feature amounts corresponding to the context from the time information and the position information. As a result, an effect is obtained in that it is possible to perform inference taking the context into account while suppressing the calculation cost as compared with the prior art that generates context feature amounts from the entire moving image.

[0017] (Configuration of Information Processing Apparatus 2) The information processing apparatus 2 includes a feature amount generation unit 21 and an inference unit 22. The feature amount generation unit 21 generates a feature amount corresponding to the context of the object captured in the frame image by using time information indicating the timing at which the frame image was captured and position information indicating the detection position of the object in the frame image for each of a plurality of frame images extracted from a moving image capturing an object moving along a predetermined context.

[0018] The inference unit 22 performs a predetermined inference regarding the object based on the feature amount generated by the feature amount generation unit 21.

[0019] As described above, the information processing device 2 according to this exemplary embodiment includes a feature generation unit 21 that generates feature quantities corresponding to the context of an object captured in a frame image, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and an inference unit 22 that performs predetermined inferences about the object based on the feature quantities generated by the feature generation unit 21.

[0020] Objects moving within a predetermined context are affected differently at different times based on that context. Therefore, according to the above configuration, feature quantities corresponding to the object's context can be generated, and inference can be performed based on those feature quantities.

[0021] Furthermore, location and time information have significantly smaller data sizes compared to video. Therefore, according to the information processing device 2 of this exemplary embodiment, it is possible to perform context-aware inference while reducing computational costs compared to conventional techniques that generate context features from the entire video.

[0022] (Learning Program) The functions of the information processing device 1 described above can also be implemented by a program. In this exemplary embodiment, the learning program causes the computer to function as an inference unit 11 that performs predetermined inferences about the object based on features calculated by inputting time information indicating when each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and position information indicating the detection position of the object in the frame image, into a feature generation model for generating context-appropriate features, and a learning unit 12 that updates the feature generation model so that the result of the inference by the inference unit 11 approaches predetermined ground truth data. According to this learning program, it is possible to generate a feature generation model that can generate context-appropriate features from time information and position information, thereby achieving the effect of performing context-aware inference while reducing computational costs.

[0023] (Inference program) Similarly, the functions of the information processing device 2 described above can also be implemented by a program. In this exemplary embodiment, the inference program causes the computer to function as a feature generation unit 21 that generates feature quantities corresponding to the context of the object in a frame image, using time information indicating when the frame image was taken and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and an inference unit 22 that performs predetermined inferences about the object based on the feature quantities generated by the feature generation unit 21. This inference program has the effect of being able to perform context-aware inference while keeping computation costs down.

[0024] (Flowchart generation and inference methods for feature generation models) The flow of the method for generating and inferring the feature generation model according to this exemplary embodiment will be explained with reference to Figure 2. Figure 2 is a flowchart showing the flow of the method for generating and inferring the feature generation model. Note that the entity executing each step in the determination method shown in Figure 2 may be a processor provided in the information processing device 1 or 2, or a processor provided in another device, or the entity executing each step may be a processor provided in a different device.

[0025] The flowchart shown on the left side of Figure 2 illustrates the method for generating a feature generation model according to this exemplary embodiment. In S11, at least one processor inputs time information indicating when the frame image was captured and position information indicating the detected position of the object in the frame image to a feature generation model for generating features according to the context, and performs a predetermined inference about the object based on the features calculated by this model.

[0026] In S12, at least one processor updates the feature generation model so that the inference results from S11 approach predetermined ground truth data.

[0027] As described above, the method for generating a feature generation model according to this exemplary embodiment includes: at least one processor inputting time information indicating when a frame image was taken and position information indicating the detection position of the object in the frame image to a feature generation model for generating context-appropriate features, and performing a predetermined inference about the object based on the calculated features (S11); and updating the feature generation model so that the result of the inference in S11 approaches predetermined ground truth data (S12). Thus, a feature generation model capable of generating context-appropriate features from time information and position information can be generated, thereby achieving the effect of performing context-aware inference while reducing computational cost.

[0028] On the other hand, the flowchart shown on the right side of Figure 2 illustrates the inference method according to this exemplary embodiment. In S21, at least one processor generates context-dependent feature quantities for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image.

[0029] In S22, at least one processor performs predetermined inference about the object based on the features generated in S21.

[0030] As described above, the inference method according to this exemplary embodiment includes, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, a processor generating feature quantities of the object in the frame image that correspond to the context, using position information indicating the detection position of the object in the frame image and time information indicating the timing when the frame image was captured (S21), and performing predetermined inference about the object based on the feature quantities generated in S21 (S22). Thus, the effect of performing context-aware inference while keeping computation costs down is obtained.

[0031] [Exemplary Embodiment 2] A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Below, an example will be described in which the inference method according to this exemplary embodiment (hereinafter referred to as "this inference method") is used for an inspection to confirm whether or not foreign matter is present in a liquid (for example, a pharmaceutical or beverage) sealed in a transparent container (hereinafter referred to as "foreign matter confirmation inspection").

[0032] (Method for detecting foreign objects) Prior to explaining this reasoning method, the method of foreign object detection inspection will be explained based on Figure 3. Figure 3 is a diagram illustrating the method of foreign object detection inspection. In foreign object detection inspection, a container containing a predetermined liquid, which is the object to be inspected, is fixed inside a device (not shown in Figure 3), and the device is used to shake the container. The control sequence for shaking the container is predetermined. For example, the control sequence in the example in Figure 3 is to rotate the container in a vertical plane for a predetermined time, then to keep it still for a predetermined time, and then to rotate the container in a horizontal plane for a predetermined time. This control sequence may be repeated multiple times.

[0033] In this foreign object detection inspection, the container is shaken using the control described above, and a video of the liquid inside the container is captured. Next, frame images are extracted from the captured video, and object detection is performed on each frame image. For each object detected by this object detection, it is determined whether it is a bubble or a foreign object. If no foreign objects are found, the inspected item is judged as good; if even one foreign object is found, it is judged as defective.

[0034] In the foreign object detection inspection described above, the container is oscillated by controlling a predetermined pattern using the device, so the object (bubble or foreign object) inside the container moves according to a predetermined context based on this pattern. For example, for a while after the container starts rotating, the flow of liquid inside the container accelerates, so the object is also accelerated by this flow. Also, the direction of movement of the object is in line with the direction of rotation of the container. After the rotation of the container ends, the liquid flow velocity gradually slows down and stabilizes in a steady state. During this time, the speed and direction of movement of the object are also in line with the liquid flow velocity and direction of flow. The same applies to subsequent control.

[0035] In this inference method, for each of the multiple frame images extracted from a video of an object moving in the context described above, features corresponding to the context of the object in that frame image are generated. Then, based on the generated features, it is determined whether the object is a bubble or a foreign object. In other words, the inference in this method is to determine whether the object is a bubble or a foreign object.

[0036] As will be explained in more detail later, this inference method allows for context-aware inference while reducing computational costs compared to conventional techniques that generate context features from the entire video image.

[0037] (Overview of the inference method) Next, we will explain the outline of this inference method based on Figure 4. Figure 4 is a diagram illustrating the outline of this inference method. Prior to executing this inference method, frame images are first extracted from the video footage of the target object. In the example in Figure 4, from FR1 to FR n Up to n frame images have been extracted. The illustrated frame image FR1 shows a container filled with liquid, along with objects OB1 and OB2. The other frame images are similar.

[0038] In this inference method, objects are first detected from each frame image. As mentioned above, the objects are bubbles and foreign objects. Since both are small in size and have similar appearances, it is difficult to accurately distinguish whether the detected object is a bubble or a foreign object based on only one frame image.

[0039] Next, in this inference method, frame images FR1~FR n Based on the detection results of objects OB1 and OB2, trajectory data showing the movement path of the objects is generated. Figure 4 schematically shows the trajectory data A1 for object OB1 and the trajectory data A2 for object OB2.

[0040] The trajectory data A1 includes time information indicating the timing when the frame image in which the object OB1 was detected was taken, and position information indicating the detection position of the object OB1 in that frame image. For example, from frame image FR1 to FR 10 Assume that object OB1 is detected in each of these locations. In this case, trajectory data A1 will contain frame images FR1~FR 10 Time information indicating the timing of each shot, and frame images FR1~FR 10 This includes location information indicating the detection location of object OB1.

[0041] Furthermore, trajectory data A1 may include information that indicates the characteristics of the detected object OB1. For example, trajectory data A1 may include an image patch which is an image extracted from the region in the frame image in which object OB1 is visible, or feature quantities extracted from the image patch, information indicating the size of the detected object, and information indicating the movement speed of the detected object.

[0042] Furthermore, in this inference method, normalized positional information may be used to avoid being affected by differences in container size or liquid volume. In this case, for example, normalized positional information can be generated by applying at least one of translation, rotation, and scaling transformations to the positional information indicating the position of the object on the frame image.

[0043] Furthermore, time information may also be normalized. In this case, for example, the time information of frame image FR1 is set to 0, and the frame image FR taken at the time when the series of control sequences was completed is also normalized. n The time information is set to 1, and these values ​​are used as a reference for frame images FR2~FR n-1 You just need to determine the time information.

[0044] The same applies to trajectory data A2, which includes time information indicating when the frame image in which object OB2 was detected was taken, and position information indicating the detection position of object OB2 in that frame image. In addition, trajectory data A2 may also include information indicating the characteristics of the detected object OB2.

[0045] Furthermore, if the object is, for example, an air bubble, it may appear and disappear during the oscillation, so the range of time information in trajectory data A1 and A2 may differ. Also, although only trajectory data A1 and A2 for two objects OB1 and OB2 are shown here, a larger number of objects may be detected in actual foreign object detection inspections.

[0046] Next, in this inference method, for each of the trajectory data A1 and A2, the time information and position information contained therein are input into a feature generation model to generate context-appropriate features B1 and B2. These features are generated for each time point. For example, the time information and position information at time t1 generate features for that time t1. Alternatively, for example, a pre-generated judgment function may be used to determine whether a frame image is inside or outside the liquid region, and the value indicating this determination result may be included in features B1 and B2. This makes it possible to obtain reasonable inference results by eliminating the influence of objects detected outside the liquid region.

[0047] According to the feature generation model, context-specific features can be generated using time and location information, which are significantly smaller in data size compared to video. Then, in this inference method, the trajectory data and features generated as described above are integrated to generate integrated data, and this integrated data is input into the inference model to output the inference result. This completes the inference method. Specifically, the inference result indicates whether objects OB1 and OB2 are bubbles or foreign objects, respectively.

[0048] In the foreign object detection inspection described above, both objects OB1 and OB2 are small in size and have similar appearances. Therefore, accurately determining whether the detected object is a bubble or a foreign object is difficult. However, this inference method uses integrated data that reflects features generated using a feature generation model to perform context-aware estimation, enabling highly accurate estimation of this difficult task.

[0049] (Configuration of Information Processing Device 3) The configuration of the information processing device 3 will be explained with reference to Figure 5. Figure 5 is a block diagram showing an example configuration of the information processing device 3. The information processing device 3 includes a control unit 30 that controls all parts of the information processing device 3, and a storage unit 31 that stores various data used by the information processing device 3. The information processing device 3 also includes a communication unit 32 for the information processing device 3 to communicate with other devices, an input unit 33 that receives input of various data to the information processing device 3, and an output unit 34 for the information processing device 3 to output various data.

[0050] The control unit 30 includes an object detection unit 301, a trajectory data generation unit 302, a feature generation unit 303, an integration unit 304, an inference unit 305, a learning unit 306, a difference identification unit 307, an adjustment unit 308, and a similarity calculation unit 309. The storage unit 31 stores trajectory data 311, a feature generation model 312, an inference model 313, and training data 314. The learning unit 306, the similarity calculation unit 309, and the training data 314 will be explained in the "About Learning" section below, and the difference identification unit 307 and the adjustment unit 308 will be explained in the "About Absorbing Contextual Differences" section below.

[0051] The object detection unit 301 detects a predetermined object from each of multiple frame images extracted from the video. If the target video was shot at a high frame rate, the object detection unit 301 can detect the object from each frame image with relatively light image processing by utilizing the continuity of position. The method of object detection is not particularly limited. For example, the object detection unit 301 may detect the object using a detection model that has been trained to detect the object using images of the object as training data. The algorithm of the detection model is not particularly limited. For example, the object detection unit 301 may use a detection model such as a convolutional neural network, a recurrent neural network, or a transformer, or it may use a detection model that combines several of these.

[0052] The trajectory data generation unit 302 generates trajectory data showing the trajectory of an object based on the object detection result from multiple frame images by the object detection unit 301. As described above, the trajectory data includes time information indicating the timing when the frame image in which the object was detected was taken, and position information indicating the detection position of the object in the frame image. It may also include image patches or other information indicating the characteristics of the detected object. Furthermore, the time information may include information indicating the time difference between the frame images. The generated trajectory data is stored in the storage unit 31 as trajectory data 311.

[0053] The feature generation unit 303 generates feature quantities corresponding to the context of the object captured in a frame image, using time information indicating when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of the multiple frame images extracted from a video image of an object moving along a predetermined context. Specifically, the feature generation unit 303 generates feature quantities by inputting the time information and position information shown in the trajectory data 311 into the feature generation model 312.

[0054] The feature generation model 312 is a pre-trained model that has been trained to generate context-aware features. More specifically, the feature generation model 312 is generated by learning the relationship between time information indicating when an object moving along a given context was photographed, location information indicating the detection position of the object in the image taken at that time, and the features of the object at that time. The feature generation model 312 may also be a function with the above-mentioned features as the target variable and time information and location information as the explanatory variables.

[0055] Furthermore, the context that influences the movement of the objects used for learning may be the same as, or similar to, the context that influences the movement of the objects being inferred. Also, the algorithm of the feature generation model 312 is not particularly limited. For example, the feature generation model 312 may be a convolutional neural network, a recurrent neural network, a transformer, or a combination of these models.

[0056] The integration unit 304 integrates the trajectory data 311 generated by the trajectory data generation unit 302 and the feature quantities generated by the feature quantity generation unit 303 to generate integrated data. The method of integration is not particularly limited. For example, the integration unit 304 may combine the feature quantities as an additional dimension with the components of the trajectory data 311 at each time point (specifically, position information, image patches, etc., associated with one time point). Alternatively, the integration unit 304 can generate integrated data that reflects the feature quantities by adding the feature quantities to the components at each time point, or by multiplying the components at each time point by the feature quantities. Furthermore, the integration unit 304 may reflect the feature quantities in the components at each time point using an attention mechanism.

[0057] The inference unit 305 performs predetermined inferences about the object based on the features generated by the feature generation unit 303. Specifically, the inference unit 305 inputs integrated data that reflects the features generated by the feature generation unit 303 into the inference model 313 to obtain an inference result, that is, a determination result of whether the object is a bubble or a foreign object.

[0058] The inference model 313 is a model generated by learning whether an object in an image is a bubble or a foreign object using integrated data generated from images showing bubbles or foreign objects. The algorithm of the inference model 313 is not particularly limited. For example, the inference model 313 may be a convolutional neural network, a recurrent neural network, or a transformer, or a model that combines several of these.

[0059] As described above, the information processing device 3 includes a feature generation unit 303 that generates feature quantities corresponding to the context of the object captured in a frame image, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of the multiple frame images extracted from a video image of an object moving in a predetermined context, and an inference unit 305 that performs predetermined inferences about the object based on the feature quantities generated by the feature generation unit 303. This provides the effect of being able to perform context-aware inference while keeping computation costs down, similar to the information processing device 2 in the exemplary embodiment 1. Note that a time-series of still images obtained by continuously capturing still images is also included in the category of "multiple frame images extracted from a video image".

[0060] Furthermore, as described above, the feature generation unit 303 may generate features using a feature generation model 312 that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the context in which the object is moving was photographed, position information indicating the detection position of the object in the image taken at that timing, and features corresponding to the context of the object at that timing. This provides the effect of generating reasonable features based on the learning results, in addition to the effects achieved by the information processing device 2 according to the exemplary embodiment 1.

[0061] Furthermore, as described above, the information processing device 3 includes a trajectory data generation unit 302 that generates trajectory data 311 showing the trajectory of an object based on the detection results of the object from multiple frame images, and an integration unit 304 that integrates the trajectory data 311 and the feature quantities generated by the feature quantity generation unit 303 to generate integrated data. The feature quantity generation unit 303 generates feature quantities using position information and time information extracted from the trajectory data 311, and the inference unit 305 performs inference using the integrated data. As a result, in addition to the effects achieved by the information processing device 2 according to the exemplary embodiment 1, the effect of performing context-aware inference within a framework that uses trajectory data 311 generated based on frame images is obtained.

[0062] (Regarding learning) This section describes the learning process performed by the learning unit 306. It also describes the training data 314 and the similarity calculation unit 309. The learning unit 306 updates the feature generation model 312 and the inference model 313 through learning using the training data 314.

[0063] The training data 314 is data in which information indicating whether an object is a bubble or a foreign object is associated with the trajectory data of a given object as ground truth data. The training data 314 may also include multiple frame images that formed the basis of the trajectory data. As described above, the feature generation unit 303 generates features from the time information and position information contained in the trajectory data, and the integration unit 304 integrates the generated features with the trajectory data to generate integrated data. Then, the inference unit 305 performs inference using the integrated data, thereby obtaining the inference result based on the trajectory data contained in the training data 314.

[0064] The learning unit 306 updates the inference model 313 and the feature generation model 312 so that the inference result based on the trajectory data contained in the training data 314 approaches the predetermined ground truth data shown in the training data 314. For example, the learning unit 306 may update the inference model 313 and the feature generation model 312 respectively using gradient descent to minimize the loss function, which is the sum of the errors between the inference result and the ground truth data.

[0065] As mentioned above, during inference, feature generation is not performed using video or frame images directly, but during training, frame images may be used. For example, since similar frame images are likely to have similar contexts, the similarity between frame images may be used to update the feature generation model 312.

[0066] The similarity calculation unit 309 calculates the similarity between frame images and is configured to be used when this similarity is to be used to update the feature generation model 312. When the similarity calculation unit 309 calculates the similarity, the learning unit 306 updates the feature generation model 312 so that the similarity between multiple frame images is reflected in the similarity between features generated by the feature generation model 312 for those frame images. For example, by adding a normalization term to the loss function described above, the feature generation model 312 can be updated so that the similarity between frame images becomes closer to the similarity of the features.

[0067] (Regarding the absorption of contextual differences) As mentioned above, the context that influences the movement of the object used for learning may be the same as or similar to the context that influences the movement of the object being inferred. Furthermore, these contexts may be identical or similar in at least part, but may not be identical or similar in whole.

[0068] The difference identification unit 307 and the adjustment unit 308 are used when there is a difference between the context of the movement of the object used for learning and the context of the movement of the object that is the target of inference.

[0069] The adjustment unit 308 adjusts at least one of the time information and position information used to generate features so as to absorb the difference between the context of the movement of the object used for learning and the context of the movement of the object that is the target of inference.

[0070] The difference identification unit 307 identifies the difference between the context of the object's movement and the context of the object's movement based on the difference between the environment surrounding the object used for learning and the environment surrounding the object. Alternatively, the difference identification unit 307 may identify the difference between the context of the object's movement and the context of the object's movement based on the difference between the object used for learning and the object being inferred.

[0071] The difference identification unit 307 and the adjustment unit 308 will be explained in more detail below with reference to Figure 6. Figure 6 shows an example where a difference occurs between contexts. In Example EX1 shown in Figure 6, it is assumed that a foreign object detection inspection is performed by a control sequence of rotation, rest, rotation, similar to Figure 3, and the controls included in each control sequence during learning and inference, and their execution order, are the same. However, the rest period during inference is longer than during learning. More specifically, in Example EX1, in both learning and inference, rotation starts at time t1, ends at time t2, and the container becomes resting, and the movement of the liquid in the container becomes steady at time t3. After this, during learning, the second rotation starts at time t4 and ends at time t5, whereas during inference, the second rotation starts at time t4', which is Δt earlier than time t4. Consequently, the time when the second rotation ends is also Δt earlier than time t5, at time t5'. Thus, in Example EX1, there is a Δt difference in the context from time t4' onwards between learning and inference.

[0072] In this case, the adjustment unit 308 performs an adjustment by adding Δt to the time indicated in each time information corresponding to the period from time t4' to time t5' among the time information used for feature generation. This makes it possible to absorb the difference in context between training and inference. Conversely to example EX1, if the quiescence period is made longer by Δt during inference than during training, the adjustment unit 308 should perform an adjustment by subtracting Δt from the time indicated in each time information after the end of the quiescence period.

[0073] Thus, if the control sequences that affect the context differ between the learning phase and the inference phase, the adjustment unit 308 can absorb the differences in context by adjusting the time information accordingly.

[0074] Furthermore, if the direction in which the container is moved is reversed between training and inference, the direction in which the liquid flows will also be reversed, and in this case, a difference in context will occur between training and inference. In such cases where a difference in the positional context occurs between training and inference, the adjustment unit 308 can adjust the positional information used for feature generation to absorb that difference.

[0075] For example, suppose that by reversing the direction in which the container is moved, the movement pattern of the object becomes a left-right inversion of the object's movement pattern during learning. In this case, the adjustment unit 308 may absorb the difference between contexts by applying a left-right inversion process to the object's position information. For example, if coordinate values ​​are used as position information, the adjustment unit 308 can apply a left-right inversion transformation to each coordinate value during the period in which the movement pattern is left-right inverted, based on a predetermined axis. Also, for example, if the object's movement pattern and the object's movement pattern during learning are rotationally symmetric, the adjustment unit 308 may absorb the difference between contexts by performing a rotational transformation on the object's position information. Note that the adjustment unit for adjusting time information and the adjustment unit for adjusting position information may be separate blocks.

[0076] Even when the control sequence that affects the context is the same during learning and inference, the time information corresponding to each frame image used during inference may deviate from the time information corresponding to each frame image used for learning. For example, in Example EX1 of FIG. 6, the time of the frame image at the start of the first rotation among the frame images used for learning is t1. At this time, if the time of the frame image at the start of the first rotation among the frame images used for inference is t1' (t1' < t1), a deviation of (t1 - t1') occurs between the contexts. In such a case, the adjustment unit 308 may perform an adjustment of adding the value of (t1 - t1') to the time indicated by each time information used for inference. Also, if the time of the frame image at the start of the first rotation among the frame images used for inference is t1'' (t1'' > t1), the adjustment unit 308 may perform an adjustment of subtracting the value of (t1'' - t1) from the time indicated by each time information used for inference. By such adjustment, the relationship between the time indicated by the time information and the control timing can be made consistent with that during learning, and thereby, the feature amount generation model 312 can be made to output appropriate feature amounts.

[0077] Also, the elements that cause differences in the context are not limited to the control sequence. For example, when there is a difference between the object used for learning and the object of inference, or when there is a difference between the environment around the object used for learning and the environment around the object of inference, differences may occur between the contexts.

[0078] The difference specifying unit 307 specifies the difference between the context in the movement of the object and the context in the movement of the object of inference based on at least any one of the differences as described above, that is, the difference between the object used for learning and the object of inference, and the difference between the environment around the object used for learning and the environment around the object of inference. Therefore, according to the information processing apparatus 3 according to this exemplary embodiment, in addition to the effects exhibited by the information processing apparatus 2 according to Exemplary Embodiment 1, an effect of automatically specifying the difference in the context can be obtained. Also, since the information processing apparatus 3 includes the adjustment unit 308, the adjustment unit 308 can be made to perform an adjustment to absorb the difference specified by the difference specifying unit 307.

[0079] For example, Example EX2 shown in Figure 6 illustrates a case where the viscosity of the liquid sealed in the container differs between the learning phase and the inference phase in a foreign object detection inspection performed in a sequence of rotation, rest, rotation. In other words, in Example EX2, the environment surrounding the object used for learning is different from the environment surrounding the object in question. Specifically, the liquid in the container used for inference has a higher viscosity than the liquid used for learning, and therefore, the time it takes for the container to stabilize after being brought to rest is shorter during inference than during learning. In other words, the time t3' when the container stabilizes during inference is earlier than the time t3 when it stabilizes during learning (t3 > t3').

[0080] In this case, the difference identification unit 307 identifies the time t3', which is the difference between the contexts during learning and during inference, based on the viscosity of the liquid in the container used for inference. If the relationship between viscosity and the time required to stabilize is identified and modeled in advance, the difference identification unit 307 can identify time t3' using this model and the viscosity of the liquid in the container used for inference.

[0081] The adjustment unit 308 then absorbs the above-mentioned difference by adjusting the time information based on the identification result of the difference identification unit 307. Specifically, the adjustment unit 308 can perform an adjustment by adding the value of (t3-t3') to the time indicated in each time information from time t3' to t4.

[0082] The adjustment unit 308 may also adjust all time points in the steady state to the same value. In this case, the adjustment unit 308 may replace the time points shown in each time information from time t3' to t4 with, for example, time t3. Thus, the adjustment unit 308 may set the time information for the period during inference when the context is constant to a constant value. Furthermore, this constant value can be selected from the time values ​​during the period when the object was moving according to the same or similar context as the above context during learning (in the example above, the period from time t3 to t4).

[0083] As described above, the information processing device 3 includes an adjustment unit 308 that adjusts at least one of the time information and position information used for feature generation in order to absorb the difference between the context of the movement of the object used for learning and the context of the movement of the object that is the target of inference. Therefore, according to the information processing device 1 of this exemplary embodiment, even if there is a difference between the context of the movement of the object and the context of the movement of the object used for learning, it is possible to generate valid features using the same feature generation model 312.

[0084] (Process flow during learning) Figure 7 is a flowchart showing the processing flow performed by the information processing device 3 during learning. Prior to learning, the training data 314 and the feature generation model 312 are stored in the memory unit 31. The feature generation model 312 stored in the memory unit 31 may have its parameters in their initial state, or it may have undergone some degree of learning.

[0085] In S31, the learning unit 306 retrieves the training data 314 stored in the storage unit 31. As described above, the training data 314 is data in which information indicating whether an object is a bubble or a foreign object is associated with the trajectory data of a certain object as correct data. Here, it is assumed that the training data 314 also includes multiple frame images that formed the basis of the trajectory data.

[0086] In S32, the feature generation unit 303 generates features corresponding to the context of the object used for training, using the training data 314 acquired in S31. More specifically, the feature generation unit 303 generates features by inputting the time information and position information shown in the trajectory data included in the training data 314 acquired in S31 into the feature generation model 312.

[0087] In S33, the integration unit 304 integrates the trajectory data included in the training data 314 acquired in S31 with the feature quantities generated in S32 to generate integrated data. Then, in S34, the inference unit 305 performs predetermined inference using the integrated data generated in S33. Specifically, the inference unit 305 inputs the integrated data generated in S33 into the inference model 313 to obtain the inference result, i.e., the determination result of whether the object is a bubble or a foreign object.

[0088] In S35, the similarity calculation unit 309 calculates the similarity between frame images included in the training data 314 acquired in S31. These frame images may be cropped around locations corresponding to the position information shown in the trajectory data. The similarity calculation unit 309 may calculate the similarity for all combinations of multiple frame images (corresponding to one trajectory data) included in the training data 314, or it may calculate the similarity for some combinations. Furthermore, the processing in S35 may be performed before S36, for example, before S32, or in parallel with the processing in S32 to S34.

[0089] In S36, the learning unit 306 updates the feature generation model 312 so that the inference result in S34 approaches the predetermined ground truth data shown in the training data 314. In this update, the learning unit 306 updates the feature generation model 312 so that the similarity between frame images calculated in S35 is reflected in the similarity between features generated by the feature generation model 312 for those frame images.

[0090] In S37, the learning unit 306 determines whether or not to terminate the learning process. The conditions for terminating the learning process can be predetermined; for example, the learning process may be terminated when the number of updates of the feature generation model 312 reaches a predetermined number. If the learning unit 306 determines NO in S37, it returns to the process in S31 and acquires new training data 314. On the other hand, if the learning unit 306 determines YES in S37, it stores the updated feature generation model 312 in the storage unit 31 and completes the process shown in Figure 7.

[0091] In S31, instead of the training data 314, a video image showing a predetermined object (specifically, at least one of bubbles and foreign objects) or a frame image extracted from such video image may be acquired. In this case, the object detection unit 301 detects the object from the acquired frame image, and the trajectory data generation unit 302 generates trajectory data 311 of the detected object. By labeling this trajectory data 311 with correct data, the training data 314 is generated. The processing after the generation of the training data 314 is the same as the processing from S32 onwards described above.

[0092] As described above, the method for generating the feature generation model 312 according to this exemplary embodiment includes: for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, inputting time information indicating the timing when the frame image was taken and position information indicating the detection position of the object in the frame image into the feature generation model 312 for generating features according to the context, and performing a predetermined inference about the object based on the calculated features (S34); and updating the feature generation model 312 so that the result of the inference approaches predetermined ground truth data (S36).

[0093] The above configuration allows for the creation of a feature generation model 312 capable of generating context-dependent features. This enables the creation of a feature generation model capable of generating context-dependent features from time and location information, resulting in the ability to perform context-aware inference while reducing computational costs.

[0094] Furthermore, as described above, the method for generating the feature generation model 312 according to this exemplary embodiment includes calculating the similarity between multiple frame images (S35), and in updating the feature generation model 312, the feature generation model 312 is updated so that the similarity between multiple frame images is reflected in the similarity between the features generated by the feature generation model 312 for those frame images. Since similar frame images are considered to have similar contexts, the above configuration makes it possible to generate a feature generation model 312 that can generate more valid features that take into account the similarity between frame images.

[0095] (Process flow during inference) Figure 8 is a flowchart showing the flow of processing (inference method) performed by the information processing device 3 during inference. Figure 8 also shows the processing after multiple frame images extracted from the video image to be inferred are input to the information processing device 3. This video image contains an object to be determined as either a bubble or a foreign object. The information processing device 3 may also perform the process of extracting frame images from the video image.

[0096] In S41, the object detection unit 301 detects an object from each of the frame images described above. Subsequently, in S42, the trajectory data generation unit 302 generates trajectory data 311 showing the trajectory of the object based on the object detection result in S41. The following describes the process when one object and one trajectory data 311 showing the trajectory of that object are generated. If multiple trajectory data 311 are generated, the processes described in S43 to S47 below are performed for each trajectory data 311.

[0097] In S43, the difference identification unit 307 identifies the difference between the context of the object's movement and the context of the object's movement, based on at least one of the difference between the object used for learning and the object being inferred, and the difference between the environment surrounding the object used for learning and the environment surrounding the object being inferred. For example, if the viscosity of the liquid sealed in the container differs between the learning phase and the inference phase, the difference identification unit 307 may calculate the time at which the liquid in the container becomes steady based on the difference in the viscosity of the liquid, and then calculate the difference between that time and the time at which the liquid in the container became steady during the learning phase.

[0098] In S44, the adjustment unit 308 adjusts at least one of the time information and location information used to generate features so as to absorb the differences between contexts identified in S43. For example, if the difference in the time of steady-state development is calculated in S43 as described above, the adjustment unit 308 adjusts the time information to absorb that time difference. If there are no differences between contexts, the processing in S43 and S44 is omitted. Also, if at least one of the time information and location information was normalized during training, the adjustment unit 308 similarly normalizes the time information and location information used to generate features.

[0099] In S45, the feature generation unit 303 generates context-appropriate features. Specifically, for each of the multiple frame images corresponding to a single trajectory data 311, the feature generation unit 303 extracts the position information and time information of the object depicted in that frame image from the trajectory data 311. Then, the feature generation unit 303 inputs the extracted time information and position information into the feature generation model 312 to generate features. As a result, for each frame image, features appropriate to the context of the object depicted in that frame image are generated.

[0100] In S46, the integration unit 304 integrates the trajectory data 311 generated in S42 with the feature quantities generated in S45 to generate integrated data. Then, in S47, the inference unit 305 performs a predetermined inference about the object based on the feature quantities generated in S45. Specifically, the inference unit 305 obtains the inference result by inputting the integrated data, which reflects the feature quantities generated in S45, into the inference model 313, and the process in Figure 8 is completed. The inference unit 305 may output the inference result to the output unit 34, etc., or store it in the storage unit 31, etc.

[0101] As described above, the inference method according to this exemplary embodiment includes generating context-appropriate feature quantities for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, using position information indicating the detection position of the object in the frame image and time information indicating the timing when the frame image was captured (S45), and performing predetermined inference about the object based on the generated feature quantities (S47). This provides the effect of performing context-aware inference while keeping computation costs down.

[0102] [Variation] The entities executing each process described in the above exemplary embodiments are arbitrary and not limited to the examples given. In other words, an information processing system with the same functions as information processing devices 1 to 3 can be constructed using multiple devices that can communicate with each other. For example, the processes in the flowchart of Figure 7 and the processes in the flowchart of Figure 8 may be executed by different information processing devices (or processors). Alternatively, each process in the flowchart of Figure 7 or Figure 8 can be divided and executed by multiple information processing devices (or processors).

[0103] Furthermore, the content of the predetermined inference performed by the inference units 11, 22, and 305 is not particularly limited and can be anything other than relating to the object. For example, in addition to classification or identification as described in Exemplary Embodiment 2, it may also be prediction, transformation, etc.

[0104] Furthermore, the factors that create the context are arbitrary. For example, information processing device 2 or 3 can perform inference that takes into account the context of an object moving in accordance with a context that arises from various devices whose operation changes at predetermined intervals, or from natural phenomena that change at predetermined intervals. In addition, information processing device 1 or 3 can generate a feature generation model that enables the above-mentioned inference that takes the context into account.

[0105] For example, the movement of objects (vehicles, people, etc.) around a traffic signal is affected by the periodic light control of the traffic signal. In other words, the aforementioned objects move according to the context caused by the light control of the traffic signal.

[0106] Therefore, the information processing device 1 or 3 can generate a feature generation model that corresponds to the above context by repeatedly performing a predetermined inference about the moving object based on features calculated by inputting time information and position information into a feature generation model for each of the multiple frame images extracted from a video image of a moving object moving in accordance with the above context, and updating the feature generation model so that the inference result approaches predetermined ground truth data. Then, the information processing device 2 or 3 can obtain a highly valid inference result that takes the above context into account by performing a predetermined inference about the moving object based on the features generated using the feature generation model thus generated. The content of the inference is not particularly limited and may be, for example, the prediction of the position of the moving object after a predetermined time, the classification of the behavior of the moving object, or the detection of abnormal behavior of the moving object. In these inferences, it is preferable to also consider interactions between vehicles and pedestrians, between vehicles, etc.

[0107] [Examples of implementation using software] Some or all of the functions of the information processing devices 1 to 3 may be implemented by hardware such as integrated circuits (IC chips), or by software.

[0108] In the latter case, information processing devices 1 to 3 are implemented by a computer that executes instructions for programs (inference programs / learning programs), which are software that realize each function. An example of such a computer (hereinafter referred to as computer C) is shown in Figure 9. Computer C comprises at least one processor C1 and at least one memory C2. Memory C2 stores a program P that causes computer C to operate as one of the information processing devices 1 to 3. In computer C, processor C1 reads program P from memory C2 and executes it, thereby realizing one of the functions of information processing devices 1 to 3.

[0109] Processor C1 can include, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), DSP (Digital Signal Processor), MPU (Micro Processing Unit), FPU (Floating Point Number Processing Unit), PPU (Physics Processing Unit), microcontroller, or a combination thereof. Memory C2 can include, for example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination thereof.

[0110] Computer C may also be equipped with RAM (Random Access Memory) for loading program P at runtime and for temporarily storing various data. Furthermore, computer C may be equipped with communication interfaces for sending and receiving data with other devices. Additionally, computer C may be equipped with input / output interfaces for connecting input / output devices such as keyboards, mice, displays, and printers.

[0111] Furthermore, program P can be recorded on a non-temporary, tangible recording medium M that is readable by computer C. Such a recording medium M could be, for example, tape, disk, card, semiconductor memory, or programmable logic circuitry. Computer C can acquire program P via such a recording medium M. Program P can also be transmitted via a transmission medium. Such a transmission medium could be, for example, a communication network or broadcast waves. Computer C can also acquire program P via such a transmission medium.

[0112] [Additional Note 1] The present invention is not limited to the embodiments described above, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the embodiments described above are also included in the technical scope of the present invention.

[0113] [Additional Note 2] Some or all of the embodiments described above may also be described as follows. However, the present invention is not limited to the embodiments described below.

[0114] (Note 1) An information processing device comprising: a feature generation means that generates feature quantities of the object captured in a frame image according to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving in a predetermined context; and an inference means that performs predetermined inferences regarding the object based on the feature quantities.

[0115] (Note 2) The information processing device described in Appendix 1, wherein the feature generation means generates the feature using a feature generation model that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the context was photographed, position information indicating the detection position of the object in the image taken at that timing, and the feature of the object according to the context at that timing.

[0116] (Note 3) An information processing apparatus according to Appendix 1 or 2, comprising: trajectory data generation means for generating trajectory data showing the trajectory of an object based on the detection results of the object from a plurality of frame images; and integration means for integrating the trajectory data and features generated by the feature generation means to generate integrated data, wherein the feature generation means generates the features using the position information and time information extracted from the trajectory data, and the inference means performs the inference using the integrated data.

[0117] (Note 4) The information processing apparatus according to Appendix 2, comprising adjustment means for adjusting at least one of the time information and position information used to generate the feature quantity so as to absorb the difference between the context in the movement of the object and the context in the movement of the target object.

[0118] (Note 5) The information processing apparatus according to Appendix 4, comprising difference identification means for identifying the difference between the context in the movement of the object and the context in the movement of the object, based on at least one of the difference between the object and the target object, and the difference between the environment surrounding the object and the environment surrounding the target object.

[0119] (Note 6) An inference method comprising: at least one processor generating context-dependent feature quantities of the object captured in a frame image for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, using position information indicating the detection position of the object in the frame image and time information indicating the timing when the frame image was captured; and performing predetermined inference about the object based on the feature quantities.

[0120] (Note 7) An inference program that causes a computer to function as a feature generation means, which generates feature quantities of the object depicted in a frame image according to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving in a predetermined context, and an inference means, which performs predetermined inferences about the object based on the feature quantities.

[0121] (Note 8) A method for generating a feature generation model, comprising: at least one processor performing a predetermined inference about an object based on features calculated by inputting time information indicating when the frame image was taken and position information indicating the detected position of the object in the frame image to a feature generation model for generating context-specific features for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context; and updating the feature generation model so that the result of the inference approaches predetermined ground truth data.

[0122] (Note 9) A method for generating a feature generation model according to Appendix 8, comprising at least one processor calculating the similarity between the plurality of frame images, and updating the feature generation model such that the similarity between the plurality of frame images is reflected in the similarity between the features generated by the feature generation model for the frame images.

[0123] [Additional Note 3] Some or all of the embodiments described above can also be expressed as follows: An information processing device comprising at least one processor, the processor performing the following processes for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context: using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, to generate context-appropriate feature quantities of the object captured in the frame image; and performing predetermined inferences about the object based on the feature quantities.

[0124] Furthermore, this information processing device may also be equipped with memory, and this memory may store an inference program that causes the processor to perform the process of generating the feature quantities and the process of performing the predetermined inference. In addition, this inference program may be recorded on a computer-readable, non-temporary, tangible recording medium. [Explanation of symbols]

[0125] 1. Information Processing Device 11 Reasoning part 12. Learning Department 2. Information Processing Device 21 Feature Generation Unit 22 Reasoning part 3. Information Processing Device 311 Track Data 312 Feature Generation Models 302 Track Data Generation Unit 303 Feature Generation Unit 304 Integration Department 305 Reasoning Department 306 Learning Department 307 Difference identification part 308 Adjustment section

Claims

1. A feature generation means generates feature quantities of the object captured in a frame image that correspond to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving in a predetermined context. The system comprises inference means that perform predetermined inferences regarding the object based on the aforementioned feature quantities, The feature generation means generates the features using a feature generation model that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the context was photographed, position information indicating the detection position of the object in the image taken at that timing, and the features of the object according to the context at that timing. Information processing device.

2. Trajectory data generation means generates trajectory data indicating the trajectory of the object's movement based on the detection results of the object from the plurality of frame images, The system includes an integration means that integrates the trajectory data and the features generated by the feature generation means to generate integrated data, The feature generation means generates the feature using the position information and time information extracted from the trajectory data, The information processing apparatus according to claim 1, wherein the inference means performs the inference using the integrated data.

3. The information processing apparatus according to claim 1, further comprising adjustment means for adjusting at least one of the time information and position information used to generate the feature quantity so as to absorb the difference between the context in the movement of the object and the context in the movement of the target object.

4. The information processing apparatus according to claim 3, comprising difference identification means for identifying the difference between the context in the movement of the object and the context in the movement of the object, based on at least one of the difference between the object and the target object, and the difference between the environment surrounding the object and the environment surrounding the target object.

5. At least one processor, For each of the multiple frame images extracted from a video of an object moving in a predetermined context, the system generates feature quantities of the object in the frame image that correspond to the context, using positional information indicating the detection position of the object in the frame image and time information indicating the timing when the frame image was captured. This includes performing a predetermined inference about the object based on the aforementioned feature quantities, The features are generated using a feature generation model that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the aforementioned context was photographed, position information indicating the detected position of the object in the image taken at that timing, and the features of the object according to the aforementioned context at that timing. Reasoning method.

6. Computer A feature generation means that generates feature quantities of the object captured in a frame image according to the context, using time information indicating the timing when the frame image was captured and position information indicating the detection position of the object in the frame image, for each of a plurality of frame images extracted from a video image of an object moving along a predetermined context, and It functions as an inference means that performs predetermined inferences about the object based on the aforementioned feature quantities, The features are generated using a feature generation model that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the aforementioned context was photographed, position information indicating the detected position of the object in the image taken at that timing, and the features of the object according to the aforementioned context at that timing. An inference program.

7. At least one processor, For each of the multiple frame images extracted from a video image of an object moving along a predetermined context, a predetermined inference about the object is performed based on the calculated features obtained by inputting time information indicating when the frame image was captured and position information indicating the detected position of the object in the frame image into a feature generation model for generating features according to the context. This includes updating the feature generation model so that the result of the inference approaches predetermined ground truth data, The features are generated using a feature generation model that has learned the relationship between time information indicating the timing at which an object moving along the same or similar context as the aforementioned context was photographed, position information indicating the detected position of the object in the image taken at that timing, and the features of the object according to the aforementioned context at that timing. Methods for generating feature generation models.

8. The process includes at least one processor calculating the similarity between the plurality of frame images, A method for generating a feature generation model according to claim 7, wherein in updating the feature generation model, the feature generation model is updated so that the similarity between the plurality of frame images is reflected in the similarity between the features generated by the feature generation model for the frame images.

Images