Information processing device, information processing method, and recording medium

Abstract

This information processing device includes: an acquisition means for acquiring an image from a video; a detection means for detecting the position of a tracking target included in the image; a conversion means for converting position information related to the position of the tracking target into a feature amount indicating a feature of the position information; an update means for updating the feature amount by using a cross attention mechanism capable of cross-checking the tracking target between the plurality of images; and a restoration means for restoring the updated feature amount to position information. According to such an information processing device, it is possible to accurately track a tracking target included in a video.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] One such device is known to track an object included in a video. For example, Patent Literature 1 discloses generating a first feature vector indicating the position information of an object in a first image and a second feature vector indicating the position information of the object in a second image, and generating correspondence information indicating the correspondence between the objects using the first feature vector and the second feature vector, thereby tracking the object.

[0003] International Publication No. 2021 / 130951

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques disclosed in prior art documents.

[0005] One aspect of the information processing device disclosed herein includes an acquisition means for acquiring an image from a video, a detection means for detecting the position of a tracked object included in the image, a conversion means for converting position information regarding the position of the tracked object into a feature quantity indicating the characteristics of the position information, an update means for updating the feature quantity using a cross-attention mechanism capable of matching the tracked object between a plurality of the images, and a restoration means for restoring the updated feature quantity to the position information.

[0006] One aspect of the information processing method disclosed herein involves using at least one computer to acquire images from a video, detect the position of a tracked object contained in the images, convert position information regarding the position of the tracked object into features indicating characteristics of the position information, update the features using a cross-attention mechanism that can match the tracked object between multiple images, and restore the updated features to the position information.

[0007] One aspect of a recording medium of this disclosure is a recording medium having recorded thereon a computer program that causes at least one computer to execute an information processing method, which includes acquiring an image from a video, detecting a position of a tracked object included in the image, converting position information regarding the position of the tracked object into a feature that indicates characteristics of the position information, updating the feature using a cross-attention mechanism that can match the tracked object between a plurality of the images, and restoring the updated feature to the position information.

[0008] 1 is a block diagram showing a hardware configuration of a first information processing device. FIG. 2 is a conceptual diagram showing an example of a tracking method using query propagation. FIG. 3 is a block diagram showing a functional configuration of the first information processing device. FIG. 4 is a flowchart showing the flow of tracking processing by the first information processing device. FIG. 5 is a block diagram showing a configuration of a cross-attention mechanism in the first information processing device. FIG. 6 is a plan view showing an example of an affinity matrix calculated by the cross-attention mechanism. FIG. 7 is a block diagram showing a functional configuration of a second information processing device. FIG. 8 is a conceptual diagram showing a method of generating training data according to a comparative example. FIG. 9 is a conceptual diagram showing a method of generating training data according to the second information processing device. FIG. 10 is a conceptual diagram showing query propagation in a learning operation by the second information processing device. FIG. 11 is a flowchart showing the flow of learning operation by the second information processing device.

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings.

[0010] First Embodiment A first information processing apparatus will be described with reference to FIGS. 1 to 6. FIG.

[0011] (Hardware Configuration) First, the hardware configuration of the first information processing apparatus will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the first information processing apparatus.

[0012] 1, the first information processing device 1 includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, a storage device 14, an input device 15, and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are connected to each other via a data bus 17. Note that the data bus 17 may be an interface other than a data bus (for example, a LAN, a USB, etc.).

[0013] The processor 11 loads a computer program. For example, the processor 11 is configured to load a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the processor 11 may load a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may acquire (i.e., load) the computer program from a device (not shown) located outside the first information processing device 1 via a network interface. The processor 11 executes the loaded computer program to perform various processes. In particular, in this embodiment, when the processor 11 executes the loaded computer program, functional blocks related to the tracking process performed by the first information processing device 1 are realized within the processor 11. In other words, the processor 11 may function as a controller that executes each control in the first information processing device 1.

[0014] The processor 11 may be configured as, for example, a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), or a quantum processor. The processor 11 may be configured as one of these, or may be configured to use multiple processors in parallel.

[0015] The RAM 12 temporarily stores computer programs executed by the processor 11. The RAM 12 temporarily stores data that the processor 11 temporarily uses while it is executing the computer programs. The RAM 12 may be, for example, a dynamic random access memory (D-RAM) or a static random access memory (SRAM). Alternatively, other types of volatile memory may be used instead of the RAM 12.

[0016] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store fixed data. The ROM 13 may be, for example, a programmable read-only memory (PROM) or an erasable read-only memory (EPROM). Alternatively, other types of non-volatile memory may be used instead of the ROM 13.

[0017] The storage device 14 stores data that is to be saved long-term by the first information processing device 1. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may store computer programs executed by the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives input instructions from a user of the information processing device 1. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel. The input device 15 may also be, for example, a device that includes a microphone and is capable of voice input. The input device 15 may also be configured as various terminals, such as a smartphone, a tablet, or a laptop computer.

[0019] The output device 16 is a device that outputs information related to the information processing device 1 to the outside. For example, the output device 16 may be a display device (e.g., a display or digital signage) that can display information related to the information processing device 1. The output device 16 may also be a speaker or the like that can output information related to the first information processing device 1 as audio. The output device 16 may be configured as various terminals, such as a smartphone, a tablet, or a laptop computer.

[0020] 1. For example, the first information processing device 1 may include a processor 11, a RAM 12, and a ROM 13. In this case, the storage device 14, the input device 15, and the output device 16 may each be configured as an external device connected to the first information processing device 1. Furthermore, some of the calculation functions of the first information processing device 1 may be realized by an external server, a cloud, or the like.

[0021] (Tracking Method) Next, a tracing method executed by the first information processing device 1 will be described with reference to Fig. 2. Fig. 2 is a conceptual diagram showing an example of a tracing method using query propagation.

[0022] In FIG. 2 , the first information processing device 1 is configured to be able to execute a tracking process for tracking a tracking target included in a video. The tracking target may be, for example, a person or an animal, or may be an object such as luggage or a vehicle. In the tracking process executed by the first information processing device, a feature based on position information of the tracking target detected from the video is acquired as a "detection query." Then, this detection query is used to update a "tracking query," which is a query for tracking the target. The first information processing device 1 tracks the tracking target by propagating this tracking query in chronological order. Note that these queries are set for each tracking target. Therefore, when a video includes multiple tracking targets, the queries corresponding to each of the multiple tracking targets are updated.

[0023] 2, objects A and B are detected from a frame captured at time T1. Therefore, the detection query at time T1 includes detection queries for objects A and B. Then, the tracking query is updated using the detection query at time T1. As a result, the tracking query at time T2 includes tracking queries for objects A and B.

[0024] Subsequently, a new object C is detected in the frame captured at time T2. Therefore, the detection query at time T2 includes the detection query for object C. The tracking query is then updated using the detection query at time T2. As a result, the tracking query at time T3 includes the tracking queries for objects A and B that were included in the tracking query at time T2 (in other words, propagated from the previous time) and the tracking query for the newly detected object C.

[0025] Note that, from the frame captured at time T3, object A, object B, and object C are each detected, but from the frame captured at time T4, only object A and object C are detected, and object B is not detected. Therefore, the tracking query for object B disappears from the tracking query at time T4, and tracking queries for objects A and C are included.

[0026] (Functional Configuration) Next, the functional configuration for executing the above-mentioned tracking process will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of the first information processing apparatus.

[0027] 3, the first information processing device 10 includes, as components for realizing its functions, an image acquisition unit 110, a target position detection unit 120, a feature conversion unit 130, a feature update unit 140, a location information restoration unit 150, and a storage unit 155. Note that each of the image acquisition unit 110, the target position detection unit 120, the feature conversion unit 130, the feature update unit 140, and the location information restoration unit 150 may be a processing block realized by the above-mentioned processor 11 (see FIG. 1). The storage unit 155 may be realized by the above-mentioned storage device 14 (see FIG. 1), etc.

[0028] The image acquisition unit 110 is configured to be able to acquire videos and images. More specifically, the image acquisition unit 110 may acquire images of each frame constituting a video sequentially in time series. Alternatively, the image acquisition unit 110 may acquire images arranged in time series. For example, images may be acquired from each frame constituting a video at predetermined intervals. The image acquisition unit 110 may be configured to acquire images in real time while capturing a video with a camera. The images acquired by the image acquisition unit 110 are configured to be output to the target position detection unit 120.

[0029] The target position detection unit 120 is configured to be able to detect the position of the tracking target from the image acquired by the image acquisition unit 110. The target position detection unit 120 may be configured to, for example, detect a person included in the image and output position information indicating the position of the detected person. Note that, if the image includes multiple tracking targets, the target position detection unit 120 may detect position information for each of the multiple tracking targets. The target position detection unit 120 may be configured as a trained detection model configured by, for example, a neural network. The position information of the tracking target detected by the target position detection unit 120 is configured to be output to the feature conversion unit 130.

[0030] The feature conversion unit 130 is configured to convert the position information of the tracking target detected by the target position detection unit 120 into a feature. The feature here may be a feature vector indicating the position information of the tracking target. The feature conversion unit 130 may be configured as an encoder including a plurality of feature extraction blocks. The feature conversion unit 130 may be constructed, for example, with approximately three fully connected layers in a neural network. The feature converted by the feature conversion unit 130 may be used, for example, as a detection query (see FIG. 2 ) used in the tracking process. The feature converted by the feature conversion unit 130 is configured to be output to the feature update unit 140.

[0031] The feature update unit 140 is configured to be able to update the features converted by the feature conversion unit 130 using a cross-attention mechanism 200. The cross-attention mechanism 200 has a function of comparing position information of a tracked target between multiple images. Specifically, the cross-attention mechanism 200 has a function of associating targets included in each of consecutive images acquired in time series to determine whether they are the same person. The specific configuration of the cross-attention mechanism 200 will be described in detail later. The features updated by the feature update unit 140 (hereinafter referred to as "updated features") may be used, for example, as a tracking query (see FIG. 2) used in the tracking process. In this case, the updated features may be temporarily stored in a storage unit 155 that stores tracking queries. The updated features are also configured to be output to the position information restoration unit 150.

[0032] The location information restoration unit 150 is configured to be able to restore the updated feature amounts updated by the feature amount updating unit 140 into location information. The location information restoration unit 150 may be configured as a decoder including a plurality of feature extraction blocks. The location information restoration unit 150 may be constructed, for example, with approximately three fully connected layers in a neural network. The location information restoration unit 150 may have a function of outputting the restored location information.

[0033] The storage unit 155 is configured to store feature amounts and location information for each tracking target. For example, the storage unit 155 may be configured to store a detection query and a tracking query for each tracking target. For example, the storage unit 155 may be provided with multiple memory areas, and may store tracking queries for tracking one person in one memory area so as to accumulate them in chronological order. Furthermore, the storage unit 155 may be configured to store feature amounts and location information for each tracking target in association with a corresponding ID.

[0034] (Flow of Tracking Process) Next, the flow of tracking process executed by the first information processing device 1 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing the flow of tracking process by the first information processing device.

[0035] 5, when the tracking process by the first information processing device 1 is started, the image acquisition unit 110 first acquires an image from a moving image (step S101). Then, the target position detection unit 120 detects the position of the tracking target included in the image from the image acquired by the image acquisition unit 110 (step S102).

[0036] Next, the feature conversion unit 130 converts the position information of the tracked target detected by the target position detection unit 120 into a feature (step S103). Then, the feature update unit 140 updates the feature converted by the feature conversion unit 130 using the intersection attention mechanism 200 (step S104).

[0037] Thereafter, the location information restoring unit 150 restores the updated feature amount updated by the feature amount updating unit 140 to the location information (step S105). Thereafter, the first information processing device 1 determines whether or not to end the tracking process (step S106).

[0038] If the tracking process is not to be ended (step S106: NO), the process may be executed again from step S101. That is, the image of the next frame may be acquired, and the above-described process may be executed repeatedly. On the other hand, if the tracking process is not to be ended (step S106: YES), the series of processes ends.

[0039] (Intersection Attention Mechanism) Next, the configuration and operation of the intersection attention mechanism 200 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the intersection attention mechanism in the first information processing apparatus.

[0040] As shown in FIG. 5 , the cross-attention mechanism 200 includes three feature embedding units 210, 220, and 230 corresponding to the query, key, and value, respectively, a matrix multiplication unit 240, a normalization unit 250, a matrix multiplication unit 260, a residual processing unit 270, and a memory update unit 280.

[0041] The feature embedding processing unit 210 is configured to be able to extract a query from the feature amount at time t (i.e., the feature amount corresponding to the frame captured at time t) input from the feature amount conversion unit 130. The feature embedding processing unit 220 is configured to be able to extract a key from the feature amount at time t-τ calculated in the past tracking process (i.e., the feature amount corresponding to the frame captured at time t-τ before time t). The feature embedding processing unit 230 is configured to be able to extract a value from the feature amount at time t-τ calculated in the past tracking process. The query and the key are configured to be output to the matrix multiplication calculation unit 240. On the other hand, the value is configured to be output to the matrix multiplication calculation unit 260.

[0042] The matrix multiplication calculation unit 240 is configured to be able to calculate a weight (Attention Weight) indicating the correlation between the query and the key by calculating the matrix product of the query and the key. That is, the matrix calculation unit 240 is configured to be able to calculate a weight indicating the correlation between a feature amount corresponding to a frame captured at time t and a feature amount corresponding to a frame captured at time t-τ. The matrix multiplication calculation unit 240 may calculate (use) an affinity matrix, for example, in which the vertical axis represents the feature amount corresponding to the frame captured at time t and the horizontal axis represents the feature amount corresponding to the frame captured at time t-τ, as the weight (Attention Weight) of the cross-attention mechanism 200.

[0043] The normalization unit 250 is configured to be able to perform normalization processing on the weights calculated by the matrix multiplication unit 240. For example, the normalization unit 250 may perform processing to normalize the similarity matrix calculated by the matrix multiplication unit 240 using a cross-softmax function. The weights normalized by the normalization unit 250 are configured to be output to the matrix multiplication unit 260.

[0044] The matrix multiplication unit 260 is configured to perform a process of reflecting the weight in the value by calculating the matrix product of the output from the normalization unit 250 and the value. Note that the matrix product in this embodiment may typically be a tensor product (in other words, a direct product). For example, the matrix product may be a Kronecker product. The calculation result of the matrix multiplication unit 260 is configured to be output to the residual processing unit 270.

[0045] The residual processing unit 270 is configured to perform residual processing on the calculation result of the matrix multiplication unit 260. This residual processing may be a process of adding the calculation result of the matrix multiplication unit 260 and the feature amount input to the cross-attention mechanism 200 (specifically, the feature amount at time t). This is to prevent the feature amount from being generated as the calculation result of the cross-attention mechanism 200 even if a correlation is not calculated. For example, if 0 is calculated as the correlation (weight), the value is multiplied by 0, and the feature value in the calculation result of the matrix calculation unit 260 becomes 0 (disappears). To prevent this, the residual processing unit 270 performs the above-described residual processing. The calculation result of the residual processing unit 270 is output from the cross-attention mechanism 200 as the feature amount at time t.

[0046] The memory update unit 280 updates the stored feature amounts corresponding to the tracking target. The memory update unit 280 may update only the feature amounts stored in the memory means corresponding to the updated feature amounts output by the matrix multiplication calculation unit 260, or may update the feature amounts output by the calculation results of the residual processing unit 270 by overwriting them in the memory means. For example, the tracking target may be identified by weights calculated from the query and key by the matrix multiplication calculation unit 240, and it may be determined which tracking target to update from among the multiple tracking targets stored in the memory unit 155. Furthermore, the updated feature amounts calculated from the normalized weights and values ​​by the matrix multiplication calculation unit 260 may be determined as the updated amounts of the feature amounts of the tracking target stored in the memory unit 155.

[0047] Note that the first information processing device 1 essentially focuses on the similarity between the tracking process and the operation performed by the cross-attention mechanism 200, and can be said to perform an operation of updating features using information generated when matching an object. For example, the tracking process includes a process of detecting a tracked object, a process of matching the tracked object, and a process of updating the detection result of the tracked object. On the other hand, the cross-attention mechanism 200 includes a process of extracting features related to the tracked object, a process of calculating weights, and a process of updating the features related to the tracked object. The first information processing device 1 essentially uses the process of calculating weights in the cross-attention mechanism 200 as a process of matching a tracked object in the tracking process. In other words, the first information processing device 1 essentially uses the process of matching a tracked object in the tracking process as a process of calculating weights in the cross-attention mechanism 200. Therefore, it can also be said that the first information processing device 1 realizes the operation of detecting an object, the operation of matching the object, and the operation of updating the detection result using the cross-attention mechanism 200.

[0048] More specifically, as described above, the intersection attention mechanism 200 uses the feature quantity corresponding to the frame captured at time t as a query, and acquires and uses the feature quantities contained in frames captured up to time t-τ before time t as a key and value from the storage unit 160, which stores the feature quantities. The memory update unit 280 may update the feature quantity output by the calculation result of the residual processing unit 270 by overwriting it in the storage unit. Alternatively, the tracking target may be identified from the weight calculated by the matrix multiplication calculation unit 240, and the feature quantity corresponding to the ID of the identified tracking target stored in the storage unit may be updated using the updated feature quantity output by the matrix multiplication calculation unit 260. In this way, it is possible to accurately track a tracking target included in a video using a relatively simple algorithm.

[0049] (Similarity Matrix) Next, the affinity matrix calculated by the above-described intersection attention mechanism 200 will be specifically described with reference to Fig. 6. Fig. 6 is a plan view showing an example of the affinity matrix calculated by the intersection attention mechanism.

[0050] As shown in FIG. 6, the similarity matrix AM used as weights by the cross-attention mechanism 200 is ｔ－τ and the tracked object O at time t ｔ For example, the similarity matrix AM is information indicating the correspondence between the multiple tracking targets O ｔ－τ The first tracked object O ｔ－τ However, there are multiple tracking targets O ｔ The first tracked object O ｔ (i.e., they are the same person), and (2) there are multiple tracking targets O ｔ－τ The second tracking target O ｔ－τ However, there are multiple tracking targets O ｔ The second tracking target O ｔ , (N) multiple tracking targets O ｔ－τ The Nth tracking target O ｔ－τ However, there are multiple tracking targets O ｔ The Nth tracking target O ｔ The similarity matrix AM is information indicating that the tracking target O ｔ－τ and tracking target O ｔ Since it is information indicating the correspondence relationship between the

[0051] Specifically, the vertical axis of the affinity matrix AM is the feature vector CV ｔ－τ and its horizontal axis corresponds to the vector component of the feature vector CV ｔ Therefore, the vertical axis of the affinity matrix AM corresponds to the vector components of the feature vector CV. ｔ－τ , which corresponds to the size (i.e., the number of pixels) of the image captured at time t-τ. Similarly, the size of the horizontal axis of the similarity matrix AM corresponds to the size of the feature vector CV ｔ The size of the similarity matrix AM corresponds to the size of the image captured at time t (i.e., the number of pixels). In other words, the vertical axis of the similarity matrix AM is the size of the tracked object O appearing in the image at time t-τ. ｔ－τ The detection result of the tracking target O ｔ－τ The horizontal axis corresponds to the detected position of the tracking target O ｔThe detection result of the tracking target O ｔ In this case, the vertical axis of the matrix can be regarded as a matrix corresponding to the position of a tracked object O ｔ－τ The vector components corresponding to the same tracked object O on the horizontal axis ｔ The elements of the affinity matrix AM react (typically have a non-zero value) at the positions where the vector components corresponding to the tracked object O on the vertical axis intersect. ｔ－τ The detection result of the tracked object O on the horizontal axis ｔ The elements of the affinity matrix AM react at the positions where the detection results of the feature vector CV intersect. ｔ－τ Tracking target O included in ｔ－τ The vector components corresponding to the feature vector CV ｔ The same tracking target O included in ｔ The value of the element at the position where the vector component corresponding to the vector intersects with the vector component corresponding to the vector component is the value obtained by multiplying both vector components (i.e., a value other than 0), while the values ​​of the other elements are 0.

[0052] For example, in the example shown in FIG. 5, the feature vector CV ｔ－τ , and the feature vector CV ｔ The elements of the similarity matrix AM react at positions where the vector components corresponding to the same tracking target O#k included in the image captured at time t-τ intersect with the vector components corresponding to the same tracking target O#k included in the image captured at time t. In other words, the elements of the similarity matrix AM react at positions where the detection result of the tracking target O#k reflected in the image captured at time t intersect with the detection result of the tracking target O#k reflected in the image captured at time t.

[0053] Conversely, the feature vector CV ｔ－τ Tracking target O included in ｔ－τ The vector components corresponding to the feature vector CV ｔ The same tracking target O included in ｔ If the elements of the similarity matrix AM do not react (typically become 0) at the position where the vector components corresponding to the similarity matrix AM intersect, the tracking target O reflected in the image taken at time t-τ isｔ－τ is estimated not to be reflected in the image captured at time t (for example, it has gone outside the angle of view of the camera).

[0054] In this way, the similarity matrix AM is ｔ－τ and tracking target O ｔ In other words, the similarity matrix AM can be used as information indicating the correspondence between the tracking target O and the image captured at time t-τ. ｔ－τ and the object O reflected in the image taken at time t. ｔ Therefore, the similarity matrix AM can be used as information indicating the result of matching with the tracking target O that was captured in the image taken at time t-τ. ｔ－τ The similarity matrix AM can be used as information for tracking the position of a target object in an image captured at time t. By using the similarity matrix AM in this way, it is possible to accurately track a target object included in a video.

[0055] (Technical Effects) Next, technical effects obtained by the first information processing device 1 will be described.

[0056] As described with reference to FIGS. 1 to 6 , in the first information processing device 1, feature quantities related to the position information of an object in an image acquired from a video are updated using a cross-attention mechanism 200 having a function of matching the object. In the first information processing device 1, the operation of this cross-attention mechanism 200 is used to perform tracking processing on the tracking object. In this way, tracking processing can be performed more appropriately compared to when the cross-attention mechanism 200 described in this embodiment is not used. For example, when using a self-attention mechanism for tracking processing, learning is required so that the weights in the self-attention mechanism react strongly between the same tracking objects. However, to realize such a configuration, a large number of self-attention mechanisms are required, which creates a technical problem in that the algorithm used for tracking processing becomes complicated. However, by using the cross-attention mechanism 200 described in this embodiment, the tracking processing algorithm can be constructed with a simple structure, making it possible to achieve high-precision tracking processing while reducing computational costs.

[0057] Second Embodiment A second information processing device 1 will be described with reference to Figures 7 to 11. The second information processing device 1 differs in some configurations and operations from the first information processing device 1 described above, but other parts may be similar to the first information processing device 1. Therefore, the following will describe in detail the parts that differ from the first embodiment, and will omit explanations of other overlapping parts as appropriate.

[0058] (Functional Configuration) First, the functional configuration of the second information processing device 1 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing the functional configuration of the second information processing device.

[0059] 7, the second information processing device 1 includes, as components for realizing its functions, an image acquisition unit 110, a target position detection unit 120, a feature conversion unit 130, a feature update unit 140, a position information restoration unit 150, and a learning unit 160. That is, the second information processing device 1 further includes the learning unit 160 in addition to the configuration already described in the first embodiment (see FIG. 3). Note that the learning unit 160 may be a processing block realized by the above-described processor 11 (see FIG. 1).

[0060] The learning unit 160 is configured to be able to perform learning regarding the tracking process of the tracking target executed by the second information processing device 1. More specifically, the learning unit 160 may perform machine learning on the tracking model 50 that performs the tracking process (i.e., a model having the functions of the target position detection unit 120, the feature conversion unit 130, the feature update unit 140, and the target information restoration unit 140) so that tracking can be performed with higher accuracy. The learning unit 160 may perform learning regarding the operation of the cross-attention mechanism 200 used by the feature update unit 140. For example, the learning unit 160 may perform learning so that the operation of matching the tracking target performed by the cross-attention mechanism 200 can be performed more accurately. Specifically, the learning unit 160 may train the similarity matrix AM used by the cross-attention mechanism 200 so that it reacts more strongly to the same tracking target. Specific learning techniques used by the learning unit 160 will be described in detail below.

[0061] (Learning Method) Next, the learning method executed by the learning unit will be specifically described with reference to Fig. 8 to Fig. 10. Fig. 8 is a conceptual diagram showing a learning data generation method according to a comparative example. Fig. 9 is a conceptual diagram showing a learning data generation method according to a second information processing device. Fig. 10 is a conceptual diagram showing query propagation in a learning operation by the second information processing device.

[0062] 8, a learning method according to a comparative example will be described first. In the learning method according to the comparative example, multiple videos are used as learning data. Specifically, the multiple videos are each converted into mini-batches and used as learning data. When attempting to apply such a learning method to learning a tracking process using query propagation, for example, due to limitations on memory capacity, it is not possible to use many frames for learning, and as a result, it is not possible to obtain an appropriate learning effect.

[0063] 9 , the learning unit 160 in the second information processing device 1 performs mini-batch conversion on one video and uses the converted data as training data. For example, the learning unit 160 may use all video frames included in the mini-batch for training query propagation of the tracking model 50. The training data may include each frame included in one video and ground truth data indicating the correspondence between the tracked targets captured in each frame (i.e., which people are the same person).

[0064] 10 , the learning unit 160 uses, as learning data, multiple frames (frame t1, frame t2, frame t3, ...) arranged in chronological order from one video. In this way, it is possible to significantly increase the number of frames that can be used in learning query propagation in the tracking model 50. It also becomes possible to learn query propagation by the tracking model 50 in chronological order.

[0065] (Learning Operation Flow) Next, the flow of the learning operation (i.e., the operation when the learning unit 160 learns the tracking model 50) executed by the second information processing device 1 will be described with reference to Fig. 11. Fig. 11 is a flowchart showing the flow of the learning operation by the second information processing device.

[0066] 11, when the second information processing device 10 starts a learning operation, the learning unit 160 first batch-converts one video to generate learning data (step S201). Then, the learning unit 160 inputs the learning data 202 into the tracking model 50 (step S202).

[0067] Next, the learning unit 160 compares the output result of the tracking model 50 with the ground truth data to calculate a loss function (step S203). Then, the learning unit 160 calculates the gradient of the loss function (step S204).

[0068] Next, the learning unit 160 updates the parameters of the tracking model based on the calculated gradient so as to reduce the loss function (step S205). After that, the learning unit 160 determines whether learning has been performed using all frames of the learning data (step S206).

[0069] If all frames have not been used for learning (step S206: NO), the learning unit 160 starts the process again from step S202. That is, the learning unit 160 repeats the process from inputting images, which are learning data, into the tracking model 50 to updating the parameters. On the other hand, if all frames have been used for learning (step S206: YES), the learning unit 160 determines that learning is complete and saves the learned model (step S207).

[0070] (Technical Effects) Next, technical effects obtained by the second information processing device 1 will be described.

[0071] As described with reference to FIGS. 6 to 11 , in the second information processing device 1, multiple frames included in one video are converted into one mini-batch for learning. In this way, compared to, for example, a case where multiple videos are each converted into a mini-batch, it is possible to significantly increase the number of frames that can be used in learning query propagation of the tracking model 50. Note that the tracking process performed by the second information processing device 1 does not impose any particular limit on the number of frames. Therefore, by realizing learning using many frames (in other words, long-term time-series learning), it is possible to effectively improve tracking accuracy.

[0072] The information processing device 1 according to each of the above-described embodiments can be applied to various systems that perform tracking processing on an object. For example, the information processing device 1 can be applied to a gateless authentication system that tracks an object passing through a predetermined area and performs authentication processing using biometric information (e.g., face information, iris information, etc.) of the object being tracked.

[0073] The scope of each embodiment also includes a processing method in which a program that operates the configuration of each embodiment to realize the functions of the above-described embodiments is recorded on a recording medium, the program recorded on the recording medium is read as code, and the program is executed on a computer. In other words, a computer-readable recording medium is also included in the scope of each embodiment. Furthermore, each embodiment includes not only a recording medium on which the above-described program is recorded, but also the program itself.

[0074] Examples of recording media that can be used include floppy disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, magnetic tapes, non-volatile memory cards, and ROMs. Furthermore, the scope of each embodiment is not limited to programs that execute processes by themselves, but also includes programs that execute processes by operating on an OS in conjunction with other software or expansion board functions. Furthermore, the program itself may be stored on a server, and part or all of the program may be downloadable from the server to a user terminal. The program may be provided to the user in, for example, a SaaS (Software as a Service) format.

[0075] <Supplementary Notes> The above-described embodiment may be further described as in the following supplementary notes, but is not limited to the following.

[0076] (Supplementary Note 1) The information processing device described in Supplementary Note 1 is an information processing device including: an acquisition means for acquiring an image from a video; a detection means for detecting a position of a tracked object included in the image; a conversion means for converting position information regarding the position of the tracked object into a feature quantity indicating a characteristic of the position information; an update means for updating the feature quantity using a cross-attention mechanism capable of matching the tracked object between a plurality of the images; and a restoration means for restoring the updated feature quantity to the position information.

[0077] (Supplementary Note 2) The information processing device described in Supplementary Note 2 is the information processing device described in Supplementary Note 1, in which the cross-attention mechanism matches the tracked target using a weight calculated from a first feature that is the feature related to a first image and a second feature that is the feature related to a second image that was taken before the first image.

[0078] (Supplementary Note 3) The information processing device described in Supplementary Note 3 is the information processing device described in Supplementary Note 2, in which the cross-attention mechanism matches the tracked target by using, as the weight, a similarity matrix obtained by calculating a matrix product of the first feature and the second feature.

[0079] (Appendix 4) The information processing device described in Appendix 4 is the information processing device described in any one of Appendices 1 to 3, further comprising a storage means capable of storing the feature amounts, and a storage update means that updates the feature amounts stored in the storage means based on the feature amounts updated by the update means.

[0080] (Supplementary Note 5) The information processing device described in Supplementary Note 5 is the information processing device described in any one of Supplementary Notes 1 to 3, further including a learning unit that converts multiple frames included in one video into one mini-batch to use as training data, and performs learning related to matching of the tracked target.

[0081] (Supplementary Note 6) The information processing method described in Supplementary Note 6 is an information processing method that, by at least one computer, acquires an image from a video, detects a position of a tracked object included in the image, converts position information regarding the position of the tracked object into a feature amount indicating characteristics of the position information, updates the feature amount using a cross-attention mechanism that can match the tracked object between a plurality of the images, and restores the updated feature amount to the position information.

[0082] (Supplementary Note 7) The recording medium described in Supplementary Note 7 is a recording medium having recorded thereon a computer program for causing at least one computer to execute an information processing method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into feature quantities indicating characteristics of the position information, updating the feature quantities using a cross-attention mechanism capable of matching the tracked target between a plurality of the images, and restoring the updated feature quantities to the position information.

[0083] (Supplementary Note 8) The computer program described in Supplementary Note 8 is a computer program that causes at least one computer to execute an information processing method of acquiring images from a video, detecting a position of a tracked target included in the images, converting position information regarding the position of the tracked target into feature quantities indicating characteristics of the position information, updating the feature quantities using a cross-attention mechanism that can match the tracked target between a plurality of the images, and restoring the updated feature quantities to the position information.

[0084] (Supplementary Note 9) The tracking device described in Supplementary Note 9 is a tracking device including: an acquisition means for acquiring an image from a video; a detection means for detecting a position of a tracking target included in the image; a conversion means for converting position information on the position of the tracking target into a feature quantity indicating a characteristic of the position information; an update means for updating the feature quantity using a cross-attention mechanism capable of matching the tracking target between a plurality of the images; a restoration means for restoring the updated feature quantity to the position information; and a tracking means for tracking the tracking target based on the restored position information.

[0085] (Supplementary Note 10) The tracking method described in Supplementary Note 10 is a tracking method that, by at least one computer, acquires an image from a video, detects a position of a tracked object included in the image, converts position information regarding the position of the tracked object into a feature amount indicating characteristics of the position information, updates the feature amount using a cross-attention mechanism that can match the tracked object between a plurality of the images, restores the updated feature amount to the position information, and tracks the tracked object based on the restored position information.

[0086] (Supplementary Note 11) The recording medium described in Supplementary Note 11 is a recording medium having recorded thereon a computer program for causing at least one computer to execute a tracking method of acquiring images from a video, detecting a position of a tracked object included in the images, converting position information on the position of the tracked object into feature amounts indicating characteristics of the position information, updating the feature amounts using a cross-attention mechanism capable of matching the tracked object between a plurality of the images, restoring the updated feature amounts to the position information, and tracking the tracked object based on the restored position information.

[0087] (Supplementary Note 12) The computer program described in Supplementary Note 12 is a computer program that causes at least one computer to execute a tracking method of acquiring images from a video, detecting a position of a tracked object included in the images, converting position information on the position of the tracked object into feature amounts indicating characteristics of the position information, updating the feature amounts using a cross-attention mechanism that can match the tracked object between a plurality of the images, restoring the updated feature amounts to the position information, and tracking the tracked object based on the restored position information.

[0088] This disclosure may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0089] REFERENCE SIGNS LIST 10 Information processing device 11 Processor 12 RAM 13 ROM 14 Storage device 15 Input device 16 Output device 17 Data bus 50 Tracking model 110 Image acquisition unit 120 Target position detection unit 130 Feature conversion unit 140 Feature update unit 150 Position information restoration unit 155 Memory unit 160 Learning unit 200 Intersection attention mechanism 210 Query 220 Key 230 Value 240 Matrix multiplication unit 250 Normalization unit 260 Matrix multiplication unit 270 Residual processing unit 280 Memory update unit

Claims

1. An information processing device comprising: an acquisition means for acquiring images from a video; a detection means for detecting the position of a tracked object contained in said images; a conversion means for converting position information relating to the position of said tracked object into a feature quantity indicating the characteristics of said position information; an update means for updating said feature quantity using a cross-attention mechanism capable of matching said tracked object between a plurality of said images; and a restoration means for restoring said updated feature quantity to said position information.

2. The information processing device according to claim 1, wherein the cross-attention mechanism matches the tracked object using a weight calculated from a first feature that is the feature related to a first image and a second feature that is the feature related to a second image taken before the first image.

3. The information processing device according to claim 2, wherein the cross-attention mechanism matches the tracked target by using, as the weight, a similarity matrix obtained by calculating a matrix product of the first feature amount and the second feature amount.

4. An information processing device according to any one of claims 1 to 3, further comprising: a storage means capable of storing the feature; and a storage update means for updating the feature stored in the storage means based on the feature updated by the update means.

5. An information processing device according to any one of claims 1 to 3, further comprising a learning unit that converts multiple frames included in one video into one mini-batch as learning data and performs learning related to matching of the tracked object.

6. An information processing method, comprising: acquiring images from a video using at least one computer; detecting the position of a tracked object contained in the images; converting position information relating to the position of the tracked object into features indicating characteristics of the position information; updating the features using a cross-attention mechanism capable of matching the tracked object between multiple images; and restoring the updated features to the position information.

7. A recording medium having recorded thereon a computer program for causing at least one computer to execute an information processing method, which comprises acquiring images from a video, detecting the position of a tracked object contained in the images, converting position information relating to the position of the tracked object into features indicating characteristics of the position information, updating the features using a cross-attention mechanism capable of matching the tracked object between a plurality of the images, and restoring the updated features to the position information.

Images