Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises a determination means that determines whether a certainty factor for determining a correspondence between a second element and a first element is higher than a predetermined threshold value or not with the first element serving as a reference of the correspondence between the two elements included in time-series data, the first element being obtained at a first time, the second element being obtained at a second time preceded by the first time, and a selection means that selects the second element as another reference of the correspondence between the two elements if it is determined that the certainty factor is higher than the predetermined threshold value, and selects the first element as the reference of the correspondence between the two elements if it is determined that the certainty factor is lower than the predetermined threshold value.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] For example, a device has been proposed that tracks a specific object from images captured at multiple times, and that simultaneously tracks the target and an object similar to the target (see Patent Document 1).Other prior art documents related to this disclosure include Patent Documents 2 to 7.

[0003] International Publication No. 2022 / 019076 International Publication No. 2021 / 130951 International Publication No. 2020 / 194497 JP 2022-030852 A JP 2022-019339 A JP 2020-016901 A JP 2018-077807 A

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of an information processing device includes a determination means for determining whether a certainty level when determining correspondence between a first element included in time series data, the first element being acquired at a first time and a second element being acquired at a second time later than the first time, is higher than a predetermined threshold, using the first element as a criterion for correspondence between the two elements; and a selection means for selecting the second element as a new criterion for correspondence between the two elements if it is determined that the certainty level is higher than the predetermined threshold, and selecting the first element as a criterion for correspondence between the two elements if it is determined that the certainty level is lower than the predetermined threshold.

[0006] One aspect of the information processing method uses a first element included in time series data, acquired at a first time, and a second element acquired at a second time later than the first time, as a criterion for correspondence between two elements, and determines whether a certainty factor when determining correspondence between the second element and the first element is higher than a predetermined threshold; if it is determined that the certainty factor is higher than the predetermined threshold, selects the second element as a new criterion for correspondence between the two elements; and if it is determined that the certainty factor is lower than the predetermined threshold, selects the first element as the criterion for correspondence between the two elements.

[0007] One aspect of the storage medium has a computer program recorded on it to cause a computer to execute an information processing method in which a first element included in time series data, acquired at a first time, and a second element acquired at a second time later than the first time, are used as a criterion for correspondence between the two elements, and the certainty level when determining correspondence between the second element and the first element is higher than a predetermined threshold; if it is determined that the certainty level is higher than the predetermined threshold, the second element is selected as a new criterion for correspondence between the two elements; and if it is determined that the certainty level is lower than the predetermined threshold, the first element is selected as the criterion for correspondence between the two elements.

[0008] 1 is a block diagram showing an example of the configuration of an information processing device. FIG. 2 is a block diagram showing another example of the configuration of an information processing device. FIG. 3 is a diagram showing an example of frames included in video data. FIG. 4 is a block diagram showing the configuration of an object matching unit. FIG. 5 is a flowchart showing an object matching operation according to a second embodiment. FIG. 6 is a diagram showing an example of a similarity matrix. FIG. 7 is a block diagram showing the configuration of a refinement unit. FIG. 8 is a flowchart showing a refinement operation according to a second embodiment. FIG. 9 is a diagram showing an example of a change over time in the state of a tracked object. FIG. 10 is a block diagram showing another example of the configuration of an information processing device. FIG. 11 is a block diagram showing another example of the configuration of an information processing device. FIG. 12 is a diagram showing an example of a face authentication gate device. FIG. 13 is a diagram showing an example of an ID correspondence table.

[0009] An information processing device, an information processing method, and a recording medium according to an embodiment will be described.

[0010] First Embodiment A first embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 1. In the following, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1.

[0011] In FIG. 1 , the information processing device 1 includes a determination unit 11 and a selection unit 12. The determination unit 11 determines whether a certainty factor for determining a correspondence between a first element acquired at a first time and a second element acquired at a second time later than the first time, which are included in time-series data, is higher than a predetermined threshold value, using the first element as a criterion for correspondence between the two elements. The certainty factor may be calculated using a score for determining whether the second element corresponds to the first element. Time-series data refers to a data sequence acquired in chronological order and decomposable into multiple elements. Specific examples of time-series data include video data, multiple images captured periodically or irregularly of the same object or location, and sound data. When the time-series data is video data, the multiple elements included in the time-series data may be multiple frames constituting the video, or may be objects included in each frame.

[0012] Elements included in time-series data may change over time. For example, if an element is an object included in each of multiple frames constituting a video, at least one of the object's position and state may change over time. When associating elements that change over time, a first element that precedes the first element may be used as a reference, and it may be determined whether a second element that follows the first element in time corresponds to the first element. If it is determined that the second element corresponds to the first element, it may be determined that a third element that follows the second element in time corresponds to the second element using the second element as a new reference. On the other hand, if it is determined that the second element does not correspond to the first element, it is often determined that there is no element that corresponds to the first element, and the association of the first element is terminated. However, elements may change anomalously temporarily. Due to temporary anomalous changes, it may be determined that the second element does not correspond to the first element. If the association of the first element is terminated in this case, the element association may not be performed appropriately.

[0013] If the determination unit 11 determines that the certainty is higher than the predetermined threshold (specifically, if the score for determining whether the second element corresponds to the first element indicates that the second element corresponds to the first element and the certainty is higher than the predetermined threshold), the selection unit 12 selects the second element as a new criterion for the correspondence between the two elements. On the other hand, if the determination unit 11 determines that the certainty is lower than the predetermined threshold (specifically, if the score for determining whether the second element corresponds to the first element indicates that the second element corresponds to the first element, but the certainty is lower than the predetermined threshold), the selection unit 12 selects the first element as the criterion for the correspondence between the two elements (i.e., maintains the criterion for the correspondence between the two elements). In this case, the correspondence between the first element and a third element, which occurs chronologically after the second element, may be determined. This configuration can suppress the impact of temporary irregular changes in elements on the correspondence. Therefore, the information processing device 1 can appropriately associate elements. Note that if the certainty is equal to the predetermined threshold, either case may be included.

[0014] In the information processing device 1, the determination unit 11 may determine whether a certainty factor when determining a correspondence between a first element and a second element included in time series data, the first element being acquired at a first time and the second element being acquired at a second time after the first time, is higher than a predetermined threshold, using the first element as a criterion for correspondence between the two elements. The certainty factor may be calculated using a score for determining whether the second element corresponds to the first element. If the selection unit 12 determines that the certainty factor is higher than the predetermined threshold, the selection unit 12 may select the second element as a new criterion for correspondence between the two elements. If the selection unit 12 determines that the certainty factor is lower than the predetermined threshold, the selection unit 12 may select the first element as a criterion for correspondence between the two elements.

[0015] Such an information processing device 1 may be realized, for example, by a computer reading a computer program recorded on a recording medium. In this case, it can be said that the recording medium has recorded thereon a computer program for causing the computer to execute an information processing method that uses a first element included in time-series data acquired at a first time and a second element acquired at a second time later than the first time as a criterion for correspondence between the two elements, determines whether a certainty factor for determining correspondence between the second element and the first element is higher than a predetermined threshold, and selects the second element as a new criterion for correspondence between the two elements if it is determined that the certainty factor is higher than the predetermined threshold, and selects the first element as a criterion for correspondence between the two elements if it is determined that the certainty factor is lower than the predetermined threshold.

[0016] The information processing device 1 may be realized by a server device (for example, a cloud server) or a terminal device (for example, at least one of a smartphone, a tablet terminal, and a notebook personal computer).

[0017] Second Embodiment A second embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 2 to Fig. 9. In the following, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2.

[0018] (1) Configuration of Information Processing Device 2 As shown in Fig. 2, the information processing device 2 includes a calculation device 21, a storage device 22, and a communication device 23. The information processing device 2 may include an input device 24 and an output device 25. Note that the information processing device 2 does not necessarily include at least one of the input device 24 and the output device 25. In the information processing device 2, the calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0019] The arithmetic device 21 may include, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a TPU (Tensor Processing Unit), and a quantum processor.

[0020] The storage device 22 may include, for example, at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and an optical disk array. In other words, the storage device 22 may include a non-transitory recording medium. The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that is temporarily used by the arithmetic device 21 when the arithmetic device 21 is executing a computer program. The storage device 22 may include video data 221. The video data 221 corresponds to an example of the "time series data" in the first embodiment described above.

[0021] The communication device 23 may be capable of communicating with devices external to the information processing device 2 via a network (not shown). The communication device 23 may perform wired communication or wireless communication.

[0022] The input device 24 is a device capable of accepting information input to the information processing device 2 from the outside. The input device 24 may include an operation device (e.g., a keyboard, a mouse, a touch panel, etc.) that can be operated by an operator of the information processing device 2. The input device 24 may include a recording medium reading device that can read information recorded on a recording medium that is detachable from the information processing device 2, such as a USB (Universal Serial Bus) memory. Note that when information is input to the information processing device 2 via the communication device 23 (in other words, when the information processing device 2 obtains information via the communication device 23), the communication device 23 may function as an input device.

[0023] The output device 25 is a device capable of outputting information to the outside of the information processing device 2. The output device 25 may output visual information such as text or images, auditory information such as sound, or tactile information such as vibration, as the information. The output device 25 may include, for example, at least one of a display, a speaker, a printer, and a vibration motor. The output device 25 may be capable of outputting information to a recording medium detachable from the information processing device 2, such as a USB memory. Note that when the information processing device 2 outputs information via the communication device 23, the communication device 23 may function as the output device.

[0024] The arithmetic device 21 may have an object tracking unit 211, a calculation unit 215, a determination unit 216, and a selection unit 217 as logically realized functional blocks or as physically realized processing circuits. The object tracking unit 211 may have an object detection unit 212, an object matching unit 213, and a refinement unit 214. At least one of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. When at least some of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 are functional blocks, at least some of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 may be realized by the arithmetic device 21 executing a predetermined computer program.

[0025] The arithmetic device 21 may acquire (in other words, read) the predetermined computer program from the storage device 22. The arithmetic device 21 may read the predetermined computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 2. The arithmetic device 21 may acquire (in other words, download or read) the predetermined computer program from a device (not shown) external to the information processing device 2 via the communication device 23. Note that the recording medium for recording the predetermined computer program executed by the arithmetic device 21 may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program.

[0026] (2) Object Tracking Operation Performed by Object Tracking Unit 211 The object tracking operation performed by the object tracking unit 211 will be described. The object tracking operation may include an object detection operation, an object matching operation, and a refinement operation. The object detection operation, the object matching operation, and the refinement operation will be described in order below. As shown in FIG. 3, the video data 221 stored in the storage device 22 may include frames FR1, FR2, and FR3. Frame FR1 is a frame captured at time t-τ. Frame FR2 is a frame captured at time t. Frame FR3 is a frame captured at time t+τ. Note that "τ" is a time corresponding to the imaging cycle. Note that the object tracking unit 211 performs object tracking operation, and therefore may also be referred to as a tracking means.

[0027] (2-1) Object Detection Operation The object detection operation performed by the object detection unit 212 will now be described. The object detection unit 212 reads a frame (e.g., at least one of frames FR1, FR2, and FR3) included in the video data 221 and performs an object detection operation on the read frame. The object detection unit 212 may detect an object O included in a frame using an existing method for detecting an object O included in the frame (in other words, an object O captured in the frame). However, it is preferable that the object detection unit 212 perform the object detection operation using a method capable of acquiring information regarding the position of the object O within the frame (hereinafter referred to as "object position information PI") by detecting the object O included in the frame. The object position information PI acquired by the object detection unit 212 indicates the result of the object detection operation by the object detection unit 212, and therefore may also be referred to as object detection information. In the following description, it is assumed that the object detection unit 212 detects the object O using a method capable of acquiring the object position information PI.

[0028] The object detection unit 212 generates a heat map (so-called score map) indicating the central position (Key Point) KP (see FIG. 3 ) of the object O within the frame as the object position information PI. More specifically, the object detection unit 212 generates a heat map indicating the central position KP of the object O within the frame for each object O. Note that the heat map indicating the central position KP is a map related to position, and therefore may also be referred to as a position map.

[0029] The object detection unit 212 may generate, as the object position information PI, information indicating the size of a detection bounding box BB (see FIG. 3 ) of the object O as a score map. The information indicating the size of the detection bounding box BB of the object O may be considered to be information indicating the size of the object O. Note that the map information indicating the size of the detection bounding box BB is also a map related to the position, and may therefore be referred to as a position map.

[0030] The object detection unit 212 may generate, as the object position information PI, information indicating the correction amount (local offset) of the detection frame BB of the object O as a score map. Note that the map information indicating the correction amount of the detection frame BB is also a map related to position, and therefore may also be referred to as a position map.

[0031] In the frame FR1 captured at time t-τ, four objects O ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ In this case, the object detection unit 212 detects the object position information PI ｔ－τ As four objects O ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ At least one of information indicating the center position KP of each of #4, information indicating the size of the detection frame BB, and information indicating the correction amount of the detection frame BB may be generated.

[0032] In the frame FR2 captured at time t, four objects O ｔ #1, O ｔ #2, O ｔ #3 and O ｔ In this case, the object detection unit 212 detects the object position information PI ｔ As four objects O ｔ #1, O ｔ #2, O ｔ #3 and O ｔ At least one of information indicating the center position KP of each of #4, information indicating the size of the detection frame BB, and information indicating the correction amount of the detection frame BB may be generated.

[0033] The object detection unit 212 may perform the object detection operation using a computation model that outputs object position information PI when a frame is input. An example of such a computation model is a computation model that uses a neural network (for example, a CNN: Convolutional Neural Network). The parameters of the computation model may be optimized to output appropriate object position information PI. In this case, the parameters of the computation model are optimized based on the object position information PI (for example, object position information PI) acquired by the object detection unit 212.ｔ－τ and P.I. ｔ The object detection unit 212 may calculate the loss of the object position information PI based on the loss function.

[0034] (2-2) Object Matching Operation The object matching operation performed by the object matching unit 213 will be described with reference to Fig. 4 and Fig. 5. The object matching unit 213 reads out the object position information PI acquired by the object detection unit 212, and performs the object matching operation using the read out object position information PI. As shown in Fig. 4, the object matching unit 213 has a feature map conversion unit 2131, a feature vector conversion unit 2132, a feature conversion unit 2133, and a normalization unit 2134.

[0035] In the following, four objects O included in the frame FR1 are ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ #4 and the four objects O included in frame FR2 ｔ #1, O ｔ #2, O ｔ #3 and O ｔ The object matching operation for matching four objects O #4 included in the frame FR1 will be described below. ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ #4 is appropriately "Object O ｔ－τ Furthermore, the four objects O included in the frame FR2 are called ｔ #1, O ｔ #2, O ｔ #3 and O ｔ #4 is appropriately "Object O ｔ "

[0036] In the flowchart of FIG. 5, the feature map conversion unit 2131 converts the feature map of an object O included in a frame FR1. ｔ－τ (i.e., four objects O ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ #4) Object position information PI ｔ－τ (Step S101). The feature map conversion unit 2131 acquires the object position information PIｔ－τ From, feature map commercial ｔ－τ The feature map conversion unit 2131 generates the feature map of the object O included in the frame FR2 (step S102). ｔ (i.e., four objects O ｔ #1, O ｔ #2, O ｔ #3 and O ｔ #4) Object position information PI ｔ (Step S101). The feature map conversion unit 2131 acquires the object position information PI ｔ From, feature map commercial ｔ (Step S102). Note that the feature map CM (for example, the feature map CM ｔ－τ and commercials ｔ ) is the object position information PI (e.g., object position information PI ｔ－τ and P.I. ｔ ) for each channel.

[0037] The feature map conversion unit 2131 may generate the feature map CM using a computational model that outputs the feature map CM when the object position information PI is input. An example of such a computational model is a computational model using a neural network (e.g., a CNN). The parameters of the computational model may be optimized to output an appropriate feature map CM (e.g., a feature map CM suitable for generating an affinity matrix AM, which will be described later).

[0038] In the flowchart of FIG. 5, after the process of step S102, the feature vector conversion unit 2132 converts the feature map CM ｔ－τ From the feature vector CV ｔ－τ (Step S103). The feature vector conversion unit 2132 generates the feature map CM ｔ From the feature vector CV ｔ (Step S103). The object matching unit 213 may generate a feature vector CV directly from the object position information PI without generating a feature map CM. The feature vector conversion unit 2132 generates the feature vector CV, and therefore may be referred to as a first generation unit.

[0039] In the flowchart of FIG. 5, after the process of step S103, the feature conversion unit 2133 converts the feature vector CV ｔ－τ and the feature vector CV ｔ In the process of step S104, the feature conversion unit 2133 may generate an affinity matrix AM using the feature vector CV ｔ－τ and the feature vector CV ｔ The affinity matrix AM may be generated using a computational model that outputs the affinity matrix AM when input is a neural network (e.g., a CNN).

[0040] In the process of step S104, the normalization unit 2134 normalizes the affinity matrix AM. ｔ and the feature vector CV ｔ－τ The normalization unit 2134 may normalize the affinity matrix AM by normalizing the matrix product of σ and σ. The normalization unit 2134 may perform any normalization process on the affinity matrix AM, such as a normalization process using at least one of a sigmoid function and a softmax function.

[0041] A specific description will be given of a case where the normalization unit 2134 performs normalization processing on the affinity matrix AM using a softmax function. The normalization unit 2134 may perform normalization processing on row vector components using a softmax function so that the sum of row vector components consisting of multiple components in each row of the affinity matrix AM becomes 1. The normalization unit 2134 may perform normalization processing on column vector components using a softmax function so that the sum of column vector components consisting of multiple components in each column of the affinity matrix AM becomes 1. The normalization unit 2134 may use a matrix including components obtained by multiplying the normalized row vector components by the normalized column vector components as the normalized affinity matrix AM.

[0042] Feature Vector CV ｔ The vector components of (x １ , x ２ , …, x ｎ ) and the feature vector CVｔ－τ The vector component of (y １ , y ２ , ..., y ｎ ) In this case, the feature vector CV ｔ and the feature vector CV ｔ－τ The elements of the first row of the affinity matrix AM obtained by the calculation process of calculating the Hadamard product with (x １ *y １ , x １ *y ２ , …x １ *y ｎ The elements of the second row of the affinity matrix AM may be (x ２ *y １ , x ２ *y ２ , …x ２ *y ｎ The n-th row element of the affinity matrix AM may be (x ｎ *y １ , x ｎ *y ２ , …x ｎ *y ｎ ) where "*" indicates an element product by the Hadamard product.

[0043] Therefore, the elements of each row of the affinity matrix AM are the feature vector CV ｔ vector components and feature vector CV ｔ－τ Therefore, the vertical axis of the affinity matrix AM is the product of the feature vector CV ｔ In other words, the vertical axis of the affinity matrix AM corresponds to the vector components of the object O included in the frame FR2 at time t. ｔ The detection result (for example, object O ｔ The elements of each column of the similarity matrix AM correspond to the feature vector CV ｔ－τ vector components and feature vector CV ｔ Therefore, the horizontal axis of the affinity matrix AM is the product of the feature vector CV ｔ－τ In other words, the horizontal axis of the similarity matrix AM corresponds to the vector components of the object O included in the frame FR1 at time t-τ. ｔ－τ The detection result (for example, object Oｔ－τ It can be said that this corresponds to the position of

[0044] The feature conversion unit 2133 converts the feature vector CV ｔ－τ and the feature vector CV ｔ The features obtained by the element product of the feature vector CV and the convolutional neural network (CNN) may be generated as an affinity matrix AM. In this case, the elements of each row of the affinity matrix AM are the feature vector CV ｔ－τ vector components and feature vector CV ｔ Therefore, the vertical axis of the affinity matrix AM is the product of the feature vector CV ｔ－τ In other words, the vertical axis of the affinity matrix AM corresponds to the vector components of the object O included in the frame FR1 at time t-τ. ｔ－τ The detection result (for example, object O ｔ－τ The elements of each column of the similarity matrix AM correspond to the feature vector CV ｔ vector components and feature vector CV ｔ－τ Therefore, the horizontal axis of the affinity matrix AM is the product of the feature vector CV ｔ In other words, the horizontal axis of the affinity matrix AM corresponds to the vector components of the object O included in the frame FR2 at time t. ｔ The detection result (for example, object O ｔ It can be said that this corresponds to the position of

[0045] An object O on the vertical axis ｔ Vector components corresponding to the object O on the horizontal axis ｔ－τ The components of the affinity matrix AM react (for example, become non-zero) at the positions where the vector components corresponding to the object O on the vertical axis intersect. ｔ Detection results and object O on the horizontal axis ｔ－τ The elements of the affinity matrix AM react at the positions where the detection results of the feature vector CV intersect. ｔ An object O included in ｔ and the vector components corresponding to the feature vector CV ｔ－τ An object O included in ｔ－τThe matrix may be one in which the value of the component at the position where the vector component corresponding to the vector intersects with the vector component corresponding to the vector component is a value obtained by multiplying both vector components together (e.g., a value other than 0), while the values ​​of the other components are 0.

[0046] In the affinity matrix AM shown in FIG. 6, the feature vector CV ｔ Object O included in ｔ Vector components corresponding to #1 and feature vector CV ｔ－τ Object O included in ｔ－τ #1, Object O ｔ－τ #2, Object O ｔ－τ #3 and Object O ｔ－τ #4 The elements of the affinity matrix AM at the positions where the corresponding vector elements intersect are defined as a １１ , a １２ , a １３ and a １４ Let's say.

[0047] In the similarity matrix AM, the feature vector CV ｔ Object O included in ｔ Vector components corresponding to #2 and feature vector CV ｔ－τ Object O included in ｔ－τ #1, Object O ｔ－τ #2, Object O ｔ－τ #3 and Object O ｔ－τ #4 The elements of the affinity matrix AM at the positions where the corresponding vector elements intersect are defined as a ２１ , a ２２ , a ２３ and a ２４ Let's say.

[0048] In the similarity matrix AM, the feature vector CV ｔ Object O included in ｔ Vector components corresponding to #3 and feature vector CV ｔ－τ Object O included in ｔ－τ #1, Object O ｔ－τ #2, Object O ｔ－τ #3 and Object O ｔ－τ #4 The elements of the affinity matrix AM at the positions where the corresponding vector elements intersect are defined as a ３１ , a ３２ , a ３３ and a ３４ Let's say.

[0049] In the similarity matrix AM, the feature vector CV ｔ Object O included in ｔ Vector components corresponding to #4 and feature vector CV ｔ－τ Object O included in ｔ－τ #1, Object O ｔ－τ #2, Object O ｔ－τ #3 and Object O ｔ－τ #4 The elements of the affinity matrix AM at the positions where the corresponding vector elements intersect are defined as a ４１ , a ４２ , a ４３ and a ４４ Let's say.

[0050] In the similarity matrix AM, the feature vector CV ｔ An object O included in ｔ and the vector components corresponding to the feature vector CV ｔ－τ An object O included in ｔ－τ The components at the positions where the vector components corresponding to the object O intersect react (for example, become values ​​other than 0). ｔ and object O ｔ－τ In other words, the similarity matrix AM can be used as information indicating the correspondence between the object O included in the frame FR2 and the object O ｔ and object O included in frame FR1 ｔ－τ The similarity matrix AM can be used as information indicating the result of matching with the object O included in the frame FR1. ｔ－τ The similarity matrix AM can be used to track the position of the object O in the frame FR2. ｔ and object O ｔ－τ The feature conversion unit 2133 generates the affinity matrix AM, which may be referred to as correspondence information, and may therefore be referred to as second generation means.

[0051] (2-3) Refining Operation The refining operation performed by the refining unit 214 will be described with reference to Fig. 7 and Fig. 8. The refining operation is an operation for correcting the object position information PI acquired by the object detection unit 212. In Fig. 7, the refining unit 214 has a feature map conversion unit 2141, a feature vector conversion unit 2142, a matrix calculation unit 2143, and a residual processing unit 2144. Note that the refining unit 214 may also be referred to as a correction means, since it performs a refining operation to correct the object position information PI.

[0052] In the flowchart of FIG. 8, the feature map conversion unit 2141 converts the feature map of an object O included in a frame FR1. ｔ－τ (i.e., four objects O ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ #4) Object position information PI ｔ－τ (Step S201). The feature map conversion unit 2141 acquires the object information PI ｔ－τ From,Feature Map CM' ｔ－τ The feature map conversion unit 2141 generates the feature map of the object O included in the frame FR2 (step S202). ｔ (i.e., four objects O ｔ #1, O ｔ #2, O ｔ #3 and O ｔ #4) Object position information PI ｔ (Step S201). The feature map conversion unit 2141 acquires the object information PI ｔ From,Feature Map CM' ｔ may be generated (step S202).

[0053] The feature map conversion unit 2141 of the refinement unit 214 and the feature map conversion unit 2131 of the object matching unit 213 convert the object position information PI (for example, the object position information PI ｔ－τ and P.I. ｔ) in common in that they generate a feature map (e.g., feature map CM or CM') from the similarity matrix AM. However, the feature map conversion unit 2131 of the object matching unit 213 generates the feature map CM for the purpose of generating the similarity matrix AM (i.e., for the purpose of performing the object matching operation). In contrast, the feature map conversion unit 2141 of the refinement unit 214 generates the feature map CM' for the purpose of correcting the object position information PI using the similarity matrix AM (i.e., for the purpose of performing the refinement operation). Therefore, the feature map conversion unit 2131 of the object matching unit 213 can generate a feature map CM that is more suitable for generating the similarity matrix AM. The feature map conversion unit 2141 of the refinement unit 214 can generate a feature map CM' that is more suitable for correcting the object value information PI.

[0054] The feature map conversion unit 2141 converts the object position information PI (for example, the object position information PI ｔ－τ and P.I. ｔ ) is input, the feature map CM' (for example, the feature map CM' ｔ－τ and CM' ｔ An example of such a computational model is a computational model using a neural network (e.g., CNN). Note that the parameters of the computational model may be optimized to output an appropriate feature map CM' (particularly, a feature map CM' suitable for correcting the object position information PI).

[0055] In the flowchart of FIG. 8, after the process of step S202, the feature vector conversion unit 2142 converts the feature map CM′ ｔ－τ From the feature vector CV′ ｔ－τ (Step S203). The feature vector conversion unit 2142 generates the feature map CM′ ｔ From the feature vector CV′ ｔ may be generated (step S203).

[0056] 8, in parallel with or before or after the processing of steps S201 to S203, the matrix calculation unit 2143 may acquire the similarity matrix AM generated by the object matching unit 213 (specifically, the feature conversion unit 2133) (step S204). ｔ and the similarity matrix AM to generate a feature vector CV_res (step S205). ｔ The information obtained by the calculation process of calculating the matrix product of the similarity matrix AM and the feature vector CV_res (i.e., the matrix product) may be generated as the feature vector CV_res.

[0057] 8, after the processing of step S205, the feature vector conversion unit 2142 may generate a feature map CM_res from the feature vector CV_res (step S206). In the processing of step S206, the feature vector conversion unit 2142 may generate the feature map CM_res by converting the feature vector CV_res into the feature map CM_res.

[0058] In the flowchart of FIG. 8, after the process of step S206, the feature map conversion unit 2141 converts the object position information PI ｔ＿ｒｅｓ In the process of step S207, the feature map conversion unit 2141 converts the dimension of the feature map CM_res to generate the object position information PI ｔ＿ｒｅｓ may be generated.

[0059] For example, when the feature map CM_res is input, the feature map conversion unit 2141 converts the object position information PI ｔ＿ｒｅｓ Using a calculation model that outputs ｔ＿ｒｅｓ An example of such a computation model is a computation model using a neural network (e.g., CNN). Note that the parameters of the computation model are determined by appropriate object position information PI ｔ＿ｒｅｓ may be optimized to output

[0060] The feature map conversion unit 2141 converts (i) the object O in the frame FR2 from the feature map CM_res. ｔ (ii) map information indicating the center position KP of the object O in the frame FR2; ｔ (iii) map information indicating the size of the detection frame BB of the object O in the frame FR2; ｔ and map information indicating the correction amount of the detection frame BB. ｔ＿ｒｅｓ may be generated.

[0061] The process of step SS207 is essentially to calculate the object position information PI using an attention mechanism that uses the similarity matrix AM as a weight. ｔ＿ｒｅｓ In other words, the refinement unit 214 may constitute at least a part of an attention mechanism. ｔ＿ｒｅｓ is the refined object position information PI ｔ In this case, the process of step S207 is essentially the same as using an attention mechanism that uses the similarity matrix AM as a weight to calculate the object position information PI ｔ may be considered equivalent to the process of correcting (in other words, updating, adjusting or improving) .

[0062] Here, the object position information PI ｔ＿ｒｅｓ is the original object position information PI ｔ (i.e., object position information PI that has not been subjected to refinement operation) ｔ ) may have been lost. ｔ＿ｒｅｓ This is because the similarity matrix AM indicating the part to which attention should be paid (here, the detected position of the object O) in the attention mechanism is used as a weight. For this reason, there is a possibility that information parts of the object detection information that are different from information related to the detected position of the object O will be lost.

[0063] The refinement unit 214 refines the original object position information PI ｔ Specifically, the residual processing unit 2144 may perform processing to prevent the loss of information included in the object position information PI ｔ＿ｒｅｓ The original object position information PIｔ By adding to the object position information PI ｔ＿ｒｅｆ may be corrected (step S208).

[0064] In the process of step S208, the residual processing unit 2144 calculates the object position information PI ｔ＿ｒｅｓ Object O included in ｔ map information indicating the center position KP of the object, and original object position information PI ｔ Object O included in ｔ The residual processor 2144 may add the object position information PI ｔ＿ｒｅｓ Object O included in ｔ The map information indicating the size of the detection frame BB and the original object position information PI ｔ Object O included in ｔ The residual processing unit 2144 may add the object position information PI ｔ＿ｒｅｓ and the original object position information PI ｔ and map information indicating the correction amount of the detection frame BB included in the detection frame BB.

[0065] The process of step S208 is essentially the same as the process of step S208 in which the object position information PI is calculated using a residual attention mechanism including the residual processing unit 2144. ｔ＿ｒｅｆ In other words, the refiner 214 may constitute at least a part of a residual attention mechanism.

[0066] Object position information PI ｔ＿ｒｅｆ is the original object position information PI ｔ For example, the object O included in the frame FR2 ｔ and object O included in frame FR3. ｔ＋τ When an object matching operation is performed to match the object position information PI ｔ Instead, object position information PI ｔ＿ｒｅｆ That is, the feature map conversion unit 213 may acquire the object position information PI ｔ＿ｒｅｆ Feature Map CMｔ may be generated.

[0067] The refinement unit 214 refines the original object position information PI ｔ In this case, the refinement unit 214 does not need to include the residual processing unit 2144. ｔ＿ｒｅｓ and P.I. ｔ＿ｒｅｆ Based on a loss function relating to at least one of the above, the object position information PI ｔ＿ｒｅｓ and P.I. ｔ＿ｒｅｆ At least one of the losses may be calculated.

[0068] (3) Matching Operation The matching operation of the object O using the similarity matrix AM generated by the object matching unit 213 (specifically, the feature conversion unit 2133) will be described below. As an example, the matching operation of the object O included in the frame FR1 will be described below. ｔ－τ (i.e., four objects O ｔ－τ #1, O ｔ－τ #2, O ｔ－τ #3 and O ｔ－τ #4) and object O included in frame FR2 ｔ (i.e., four objects O ｔ #1, O ｔ #2, O ｔ #3 and O ｔ The operation of associating with #4) will be described.

[0069] In the affinity matrix AM shown in FIG. １１ , a １２ , a １３ and a １４ Among them, component a １１ The value of component a is the maximum. ２１ , a ２２ , a ２３ and a ２４ Among them, component a ２２ The value of component a is the maximum. ３１ , a ３２ , a ３３ and a ３４ Among them, component a ３３ The value of component a is the maximum. ４１ , a４２ , a ４３ and a ４４ Among them, component a ４４ The value of is assumed to be maximum.

[0070] The calculation unit 215 calculates the object O included in the frame FR2. ｔ is the object O included in the frame FR1. ｔ－τ As described above, the similarity matrix AM is calculated by calculating an index indicating the likelihood that the object O corresponds to the object O. ｔ and object O ｔ－τ Since the similarity matrix AM is information indicating the correspondence relationship between the object O and the object O, each element of the similarity matrix AM is ｔ and object O ｔ－τ Here, a class indicating "correspondence" is defined as class pos, and a class indicating "not correlating" is defined as class neg. The calculation unit 215 calculates the correspondence score between the object O included in the frame FR2 based on the similarity matrix AM. ｔ may be classified into class pos or class neg.

[0071] Component a of the affinity matrix AM １１ , a １２ , a １３ and a １４ Among them, component a １１ In this case, the value of object O included in frame FR2 is the largest. ｔ #1 is the object O included in the frame FR1. ｔ－τ In this case, the calculation unit 215 calculates that the object O included in the frame FR2 is highly likely to correspond to the object O ｔ #1 is an object O included in frame FR1 ｔ－τ #1 (in other words, the probability that the object O included in the frame FR2 ｔ The result of this calculation is "p(pos|O ｔ #1)”. For example, “p(pos|O ｔ #1) = a １１ The calculation unit 215 may calculate the object O included in the frame FR2. ｔ #1 is an object O included in frame FR1 ｔ－τ#1 (in other words, the probability that the object O included in frame FR2 ｔ The result of this calculation is "p(neg|O ｔ #1)”. For example, “p(neg|O ｔ #1) = 1-a １１ " can be.

[0072] The calculation unit 215 calculates the object O included in the frame FR2. ｔ #1 is the object O included in the frame FR1. ｔ－τ As an index showing the likelihood that the possibilty of ... ｔ #1) / p(neg|O ｔ #1)" may be calculated. ｔ #1)” is the object O included in the frame FR2. ｔ #1 is an object O included in frame FR1 ｔ－τ This may be referred to as first information indicating that the negation signal corresponds to the negation signal #1. ｔ #1)” is the object O included in the frame FR2. ｔ #1 is an object O included in frame FR1 ｔ－τ This may be referred to as second information indicating that it does not correspond to #1.

[0073] Incidentally, the calculation unit 215 calculates the object O included in the frame FR2. ｔ is an object O included in the frame FR1. ｔ－τ An index indicating the likelihood that the ｔ ) / p(neg | O ｔ )")") is an object O included in the frame FR2. ｔ and object O included in frame FR1 ｔ－τ In this case, the index may be calculated by taking into consideration the correlation with "p(pos|O ｔ , O ｔ－τ ) / p(neg | O ｔ , O ｔ－τ )" in this embodiment. ｔ and object O ｔ－τ (in other words, the correspondence between object O ｔ and object Oｔ－τ By using the similarity matrix AM, it is possible to use the similarity matrix AM, which is information indicating the relevance of the object O to the object O. ｔ and object O ｔ－τ Therefore, according to this embodiment, the calculation cost for the calculation unit 215 to calculate the above index can be reduced.

[0074] As mentioned above, component a ２１ , a ２２ , a ２３ and a ２４ Among them, component a ２２ In this case, the value of object O included in frame FR2 is the largest. ｔ #2 is the object O included in the frame FR1. ｔ－τ The calculation unit 215 calculates the object O included in the frame FR2. ｔ #2 is the object O included in the frame FR1. ｔ－τ As an index showing the likelihood that the possibilty of ... ｔ #2) / p(neg|O ｔ #2)" may be calculated.

[0075] As mentioned above, component a ３１ , a ３２ , a ３３ and a ３４ Among them, component a ３３ In this case, the value of object O included in frame FR2 is the largest. ｔ #3 is the object O included in the frame FR1. ｔ－τ The calculation unit 215 calculates that the object O included in the frame FR2 is highly likely to correspond to the object O ｔ #3 is the object O included in the frame FR1. ｔ－τ As an index showing the likelihood that the possibilty of the possibilty corresponds to #3, the likelihood ratio “p(pos|O ｔ #3) / p(neg|O ｔ #3)" may be calculated.

[0076] As mentioned above, component a ４１ , a ４２ , a ４３ and a ４４ Among them, component a ４４In this case, the value of object O included in frame FR2 is the largest. ｔ #4 is the object O included in the frame FR1. ｔ－τ The calculation unit 215 calculates that the object O included in the frame FR2 is highly likely to correspond to the object O ｔ #4 is the object O included in the frame FR1. ｔ－τ As an index showing the likelihood that #4 corresponds to #1, the likelihood ratio “p(pos|O ｔ #4) / p(neg|O ｔ #4)" may be calculated.

[0077] The calculation unit 215 calculates the object O included in the frame FR2. ｔ is the object O included in the frame FR1. ｔ－τ The log-likelihood ratio (e.g., Log{p(pos|O ｔ ) / p(neg | O ｔ )}) may be calculated. The above index (for example, likelihood ratio, log likelihood ratio) may be referred to as a confidence factor.

[0078] The determination unit 216 determines whether the object O included in the frame FR2 is a vertex or a pixel, based on the index (for example, likelihood ratio) calculated by the calculation unit 215. ｔ is the object O included in the frame FR1. ｔ－τ The determination unit 216 determines whether or not the object O included in the frame FR2 corresponds to the object O. ｔ For #1, the likelihood ratio "p(pos|O ｔ #1) / p(neg|O ｔ #1)” is greater than a threshold th1. ｔ #1) / p(neg|O ｔ If the threshold th1 is greater than the threshold th1, the determination unit 216 determines that the object O ｔ It may be determined that #1 is suitable as a reference source for matching in the next frame. ｔ #1) / p(neg|O ｔ If the threshold th1 is less than the threshold th1, the determination unit 216 determines that the object O ｔIt may be determined that #1 is inappropriate as a reference source for matching in the next frame. ｔ #1) / p(neg|O ｔ If the threshold value th1 is equal to the threshold value th1, the case can be treated as one of the cases.

[0079] When the index calculated by the calculation unit 215 is a log-likelihood ratio, the threshold value th1 may be "1". This is because when the likelihood ratio exceeds 1, p(pos|O ｔ ) > p(neg | O ｔ ), it is appropriate to classify it into the class pos, which indicates "associated."

[0080] The determination unit 216 determines whether the object O included in the frame FR2 ｔ For #2, the likelihood ratio “p(pos|O ｔ #2) / p(neg|O ｔ #2)” is greater than a threshold th1. ｔ #2) / p(neg|O ｔ If the threshold th1 is greater than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #2 is suitable as a reference source for matching in the next frame. ｔ #2) / p(neg|O ｔ If the threshold th1 is less than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #2 is inappropriate as a reference source for matching in the next frame. ｔ #2) / p(neg|O ｔ #2)" is equal to the threshold value th1, it can be treated as one of the cases.

[0081] The determination unit 216 determines whether the object O included in the frame FR2 ｔ For #3, the likelihood ratio “p(pos|O ｔ #3) / p(neg|O ｔ #3)” is greater than a threshold value th1. ｔ#3) / p(neg|O ｔ If the threshold th1 is greater than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #3 is suitable as a reference source for matching in the next frame. ｔ #3) / p(neg|O ｔ If the threshold th1 is less than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #3 is inappropriate as a reference source for matching in the next frame. ｔ #3) / p(neg|O ｔ #3)" is equal to the threshold value th1, it can be treated as one of the cases.

[0082] The determination unit 216 determines whether the object O included in the frame FR2 ｔ For #4, the likelihood ratio “p(pos|O ｔ #4) / p(neg|O ｔ #4)” may be determined to be greater than a threshold value th1. ｔ #4) / p(neg|O ｔ If the threshold th1 is greater than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #4 is suitable as a reference source for matching in the next frame matching. ｔ #4) / p(neg|O ｔ If the threshold th1 is less than the threshold th1, the determination unit 216 determines that the object O included in the frame FR2 ｔ It may be determined that #4 is inappropriate as a reference source for matching in the next frame. ｔ #4) / p(neg|O ｔ #4)" is equal to the threshold value th1, it can be treated as one of the cases.

[0083] The selection unit 217 selects the object O included in the frame FR2 based on the result of the determination of the certainty factor in the log likelihood ratio by the determination unit 216. ｔ and object O included in frame FR1 ｔ－τThe selection unit 217 associates the O ｔ The association may be performed by the determining unit 216 instead of the selecting unit 217.

[0084] For example, the determination unit 216 determines whether the object O included in the frame FR2 is ｔ #1 is the object O included in the frame FR1. ｔ－τ If it is determined that the confidence level for object O #1 is high (for example, the log-likelihood ratio is higher than a threshold), the selection unit 217 selects object O #1 included in frame FR2. ｔ Specifically, the selection unit 217 may select the object O included in the frame FR2 as a reference for matching in the next frame. ｔ #1, object O included in frame FR1 ｔ－τ After assigning the same tracking ID as the tracking ID assigned to #1, the information required by the object matching unit 213 of the next frame is stored as a feature vector CV ｔ－τ It may be used as.

[0085] In this case, the selection unit 217 selects the object O included in the frame FR2. ｔ #1, Object O ｔ As a result, the object tracking unit 211 uses frames FR2 and FR3 to locate the object O included in frame FR2. ｔ In this case, the object matching unit 213 may perform an object tracking operation for the object position information PI ｔ Instead, object position information PI ｔ＿ｒｅｓ or P.I. ｔ＿ｒｅｆ The object position information PI ｔ is the object O included in the frame FR2 detected by the object detection unit 212. ｔ The object O in the frame FR2 is obtained by detecting ｔ The object position information PI ｔ＿ｒｅｓ or P.I. ｔ＿ｒｅｆ is the refined object position information PI generated by the refinement unit 214. ｔ is.

[0086] On the other hand, the determination unit 216 determines that the object O included in the frame FR2 ｔ #1 is the object O included in the frame FR1. ｔ－τ If it is determined that the confidence level for object O #1 is low (for example, the log-likelihood ratio is lower than a threshold), the selection unit 217 selects object O #1 included in frame FR2. ｔ #1 is the object O included in the frame FR1. ｔ－τ In this case, the selection unit 217 does not need to associate the object O included in the frame FR2 with the object O ｔ #1 as a new object (i.e., object O included in frame FR1). ｔ－τ In this case, the selection unit 217 may determine that the object O included in the frame FR2 is a different object from the object O included in the frame FR2. ｔ #1 may be given a new tracking ID (in other words, an unused tracking ID).

[0087] In this case, the selection unit 217 selects the object O included in the frame FR1. ｔ－τ #1, Object O ｔ－τ The frame FR2 may be selected as a reference (e.g., a reference source) for tracking the position of the object O included in the frame FR1 in the frame FR3. ｔ－τ This is because the object corresponding to #1 is not included in the frame FR1. As a result, the object tracking unit 211 uses the frames FR1 and FR3 to track the object O included in the frame FR1. ｔ－τ An object tracking operation may be performed for #1.

[0088] For example, the determination unit 216 determines whether the object O included in the frame FR2 is ｔ #1 is the object O included in the frame FR1. ｔ－τ While the reliability is determined to be high for object #1, the determining unit 216 determines that object O included in frame FR2 is ｔ #2 is the object O included in the frame FR1. ｔ－τ If it is determined that the reliability of object O2 included in frame FR2 is low, the selection unit 217 ｔ #1, Object O ｔ 1 is selected as a reference (e.g., a reference source) for tracking the position of object O in frame FR3, and is included in frame FR1. ｔ－τ #2, Object Oｔ－τ #2 may be selected as a reference (e.g., a reference source) for tracking its position within frame FR3.

[0089] As a result, the object tracking unit 211 uses the frames FR2 and FR3 to locate the object O included in the frame FR2. ｔ The object tracking unit 211 may perform an object tracking operation for the frame FR1 using the frames FR1 and FR3. ｔ－τ An object tracking operation may be performed for #2.

[0090] The above-described operations of the information processing device 2 may be realized by the information processing device 2 reading a computer program recorded on a recording medium. In this case, it can be said that the recording medium has recorded thereon a computer program for causing the information processing device 2 to execute the above-described operations.

[0091] (Technical Effect) When tracking an object included in a plurality of images (e.g., video) captured by a camera as time-series data, the following technical problem may occur. For example, the camera may be temporarily unable to capture the tracked object because the tracked object is obscured by another object. In this case, tracking of an object included in one image may be terminated because the object is not included in another image captured after the first image. For example, the tracked object may undergo anomalous changes. Specifically, if the object is a person, the person may suddenly crouch or change direction. In this case, even if the same object is included in one image and another image captured after the first image, the object included in the first image may not be associated with the object included in the other image. In this case, the object included in the other image may be recognized as a new object.

[0092] As shown in FIG. 9 , the state of a person P as a tracked object changes. Specifically, at times t1 and t2, the person P is walking. At times t3 and t4, the person P jumps up. At times t5 and t6, the person P is walking again. In this case, when tracking of the person P is performed using an image including the person P captured at time t2 and an image including the person P captured at time t3, it may be determined that the person P included in the image captured at time t2 does not correspond to the person P included in the image captured at time t3. This is because there is a relatively large difference between the state (e.g., posture) of the person P at time t2 and the state of the person P at time t3. In this case, the person P at time t2 and the person P at time t3 may be treated as different people. In other words, tracking of the tracking ID assigned to the person P at time t2 may be terminated, and a new tracking ID may be assigned to the person P at time t3.

[0093] In addition, when tracking of person P is performed using an image including person P captured at time t4 and an image including person P captured at time t5, it may be determined that person P included in the image captured at time t4 does not correspond to person P included in the image captured at time t5. This is because there is a relatively large difference between the state (e.g., posture) of person P at time t4 and the state of person P at time t5. In this case, person P at time t4 and person P at time t5 may be treated as different people. In other words, tracking of the tracking ID assigned to person P at time t4 may be terminated, and a new tracking ID may be assigned to person P at time t5.

[0094] To address this technical issue, a method of tracking an object (in other words, matching objects) using, for example, three or more images can be considered. However, since three or more images must be processed in one object tracking operation, real-time processing is extremely difficult. Furthermore, when the time-series data is a video of 30 frames per second (FPS), from the viewpoint of computational cost, only object movements of about 0.1 seconds can be considered.

[0095] For example, the determination unit 216 determines whether the object O included in the frame FR2 is ｔ is the object O included in the frame FR1.ｔ－τ It may be determined whether the object O included in the frame FR2 corresponds to the object O ｔ is the object O included in the frame FR1. ｔ－τ If it is determined that the object O included in the frame FR2 corresponds to the object O ｔ may be selected as a reference (e.g., a reference source) for tracking the position of the object O in the frame FR3. As a result, the object tracking unit 211 uses the frames FR2 and FR3 to locate the object O included in the frame FR2. ｔ On the other hand, the object O included in the frame FR2 may be tracked. ｔ is the object O included in the frame FR1. ｔ－τ If it is determined that the object O does not correspond to the frame FR1, the selection unit 217 ｔ－τ may be selected as a reference (e.g., a reference source) for tracking the position of the object O in the frame FR3. As a result, the object tracking unit 211 uses the frames FR1 and FR3 to locate the object O included in the frame FR1. ｔ－τ An object tracking operation may be performed on the object.

[0096] 9 , the determination unit 216 may determine that the person P included in the image captured at time t2 does not correspond to the person P included in the image captured at time t3. In this case, the selection unit 217 may select the person P included in the image captured at time t2 as a reference (e.g., a reference source) for tracking the location of the person P in the image captured at time t4.

[0097] The object tracking unit 211 may perform an object tracking operation using an image captured at time t2 and an image captured at time t4. The determination unit 216 may determine that the person P included in the image captured at time t2 does not correspond to the person P included in the image captured at time t4. In this case, the selection unit 217 may select the person P included in the image captured at time t2 as a reference (e.g., a reference source) for tracking the location of the person P in the image captured at time t5.

[0098] The object tracking unit 211 may perform an object tracking operation using an image captured at time t2 and an image captured at time t5. The determination unit 216 may determine that the person P included in the image captured at time t2 corresponds to the person P included in the image captured at time t5. In this case, the selection unit 217 may assign the same tracking ID to the person P included in the image captured at time t5 as the tracking ID assigned to the person P included in the image captured at time t2.

[0099] According to the information processing device 2, even if the object to be tracked cannot be captured temporarily or if the object temporarily changes irregularly, the object can be appropriately tracked. In addition, the object tracking operation performed by the object tracking unit 211 is performed using two images, which reduces calculation costs and enables real-time processing.

[0100] The object to be tracked is not limited to a person (for example, person P). The object to be tracked may be a moving body such as a vehicle. The information processing device 2 may be realized by a server device (for example, a cloud server) or a terminal device (for example, at least one of a smartphone, a tablet terminal, and a notebook personal computer).

[0101] (Modification) When the object to be tracked is a person (e.g., person P), a facial recognition operation may be performed in addition to the object tracking operation. In FIG. 10 , the information processing device 2 a may include a facial recognition unit 218 for performing the facial recognition operation. The storage device 22 may include a facial feature database 222 (hereinafter referred to as "facial feature DB 222"). Note that existing technology (e.g., at least one of a two-dimensional (2D) authentication method and a three-dimensional (3D) authentication method) can be applied to the facial recognition operation.

[0102] The face authentication unit 218 detects the object position information PI (for example, the object position information PI ｔ－τ and P.I. ｔThe face of an object O (here, a person) included in a frame (for example, at least one of frames FR1 and FR2) may be detected based on the frame (for example, at least one of frames FR1 and FR2). Note that existing technology can be applied to the method of detecting a person's face from a frame (image), so detailed description thereof will be omitted.

[0103] If a face is detected, the face authentication unit 218 may generate a face image including a face region in the frame. The face authentication unit 218 may extract features from the generated face image. The face authentication unit 218 may calculate a matching score (or a similarity score) based on the extracted features and the features registered in the face feature DB 222. The face authentication unit 218 may compare the calculated matching score with a threshold th2. If the matching score is greater than the threshold th2, the face authentication unit 218 may determine that face authentication has been successful. In this case, the face authentication unit 218 may associate an object O (here, a person) included in the frame with an authentication ID registered in the face feature DB 222.

[0104] If the matching score is smaller than the threshold th2, the face authentication unit 218 may determine that face authentication has failed. Note that if the matching score and the threshold th2 are "equal," either case may be included. Note that if a face is not detected in a certain frame, the face authentication unit 218 does not need to perform face authentication for that frame.

[0105] Third Embodiment A third embodiment of an information processing device, an information processing method, and a recording medium will be described with reference to Fig. 11 and Fig. 12. In the following, the third embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 3.

[0106] 11 , the information processing device 3 includes a calculation device 31, a storage device 32, and a communication device 33. The information processing device 3 may include an input device 34 and an output device 35. Note that the information processing device 3 does not necessarily include at least one of the input device 34 and the output device 35. In the information processing device 3, the calculation device 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36. The storage device 32 may include a facial feature database 321 (hereinafter referred to as "facial feature DB 321") and an ID correspondence table 322.

[0107] The basic configurations of the arithmetic unit 31, the storage unit 32, the communication unit 33, the input unit 34, and the output unit 35 may be similar to those of the arithmetic unit 21, the storage unit 22, the communication unit 23, the input unit 24, and the output unit 25 in the second embodiment described above. Therefore, a description of the basic configurations of the arithmetic unit 31, the storage unit 32, the communication unit 33, the input unit 34, and the output unit 35 will be omitted.

[0108] The arithmetic device 31 may have the face tracking unit 311 and the face authentication unit 316 as logically realized functional blocks or as physically realized processing circuits. At least one of the face tracking unit 311 and the face authentication unit 316 may be realized in a form in which a logical functional block and a physical processing circuit (i.e., hardware) are mixed. When at least a part of the face tracking unit 311 and the face authentication unit 316 is a functional block, at least a part of the face tracking unit 311 and the face authentication unit 316 may be realized by the arithmetic device 31 executing a predetermined computer program.

[0109] The arithmetic device 31 may acquire (in other words, read) the predetermined computer program from the storage device 32. The arithmetic device 31 may read the predetermined computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (not shown) included in the information processing device 3. The arithmetic device 31 may acquire (in other words, download or read) the predetermined computer program from a device (not shown) external to the information processing device 3 via the communication device 33. Note that the recording medium on which the predetermined computer program executed by the arithmetic device 31 is recorded may be at least one of an optical disk, a magnetic medium, a magneto-optical disk, a semiconductor memory, and any other medium capable of storing a program.

[0110] The information processing device 3 is assumed to constitute a part of the facial recognition gate device 4 shown in Fig. 12. Note that the information processing device 3 may be a device different from the facial recognition gate device 4. In this case, the information processing device 3 may be configured to be able to communicate with the facial recognition gate device 4 via the communication device 33. In this case, the information processing device 3 may be realized by a server device (e.g., a cloud server) or a terminal device (e.g., at least one of a smartphone, a tablet terminal, and a notebook personal computer).

[0111] The facial recognition gate device 4 includes a camera CAM. The facial recognition unit 316 of the information processing device 3 may perform facial recognition using a facial image generated by the camera CAM capturing an image of the face of the person to be authenticated (e.g., a person attempting to pass through the facial recognition gate device 4). If facial recognition of the person to be authenticated is successful, the facial recognition gate device 4 allows the person to pass. If the facial recognition gate device 4 is a flap-type gate device, the facial recognition gate device 4 may open the flap. On the other hand, if facial authentication of the person to be authenticated is unsuccessful, the facial recognition gate device 4 does not allow the person to pass. In this case, the facial recognition gate device 4 may close the flap. Note that the facial recognition gate device 4 is not limited to a flap-type gate device, but may also be an arm-type gate device or a slide-type gate device.

[0112] The camera CAM captures multiple images of the face of the person to be authenticated approaching the facial recognition gate device 4. As a result, multiple temporally consecutive facial images may be generated. These multiple facial images correspond to another example of "time-series data" in the first embodiment described above. The facial authentication unit 316 may perform facial authentication using at least one of the multiple facial images. Therefore, if facial authentication is successful, the facial recognition gate device 4 can open the flap before the person to be authenticated reaches the facial recognition gate device 4. As a result, the person to be authenticated can pass through the facial recognition gate device 4 without stopping at the facial recognition gate device 4. In other words, the facial recognition gate device 4 is a so-called walk-through type facial recognition gate device.

[0113] 12, when the face authentication unit 316 is performing face authentication using a face image generated by the camera CAM capturing an image of the face of person P11 (i.e., the person to be authenticated), person P12 may cut in front of person P11. In this case, if the face authentication of person P11 is successful and the flap of the face authentication gate device 4 is in the open state, person P12 may pass through the face authentication gate device 4. Note that in FIG. 12, the dotted arrows indicate the traveling directions of people P11 and P12.

[0114] The face tracking unit 311 of the computing device 31 may perform face tracking using a plurality of face images generated by the camera CAM capturing an image of the person to be authenticated (for example, at least one of the persons P11 and P12) multiple times. For example, a face F included in a face image at time t-τ may be ｔ－τ is the face of person P11. ｔ－τ A unique tracking ID is assigned to the face of the person P11 as the person P11. The tracking ID assigned to the face of the person P11 is assumed to be "00001".

[0115] The tracking ID is registered in an ID correspondence table 322. As shown in Fig. 13, the ID correspondence table 322 indicates the correspondence between the tracking ID and the authentication ID. The ID correspondence table 322 may include a matching time, which is the time when the face authentication operation was performed.

[0116] The face authentication unit 316 may perform face authentication using a face image including a face to which a tracking ID has been assigned. The face authentication unit 316 may extract features from the face image including the face to which the tracking ID has been assigned. The face authentication unit 316 may calculate a matching score (or a similarity score) based on the extracted features and features registered in the face feature DB 321. The face authentication unit 316 may compare the calculated matching score with a threshold value th3.

[0117] If the matching score is greater than the threshold th3, the face authentication unit 316 may determine that face authentication has been successful. In this case, the face authentication unit 316 may associate the tracking ID (in other words, the face included in the face image) with the authentication ID registered in the face feature DB 321. The face authentication unit 316 may associate the tracking ID with the authentication ID by registering the authentication ID in the ID correspondence table 322.

[0118] If the matching score is smaller than the threshold th3, the face authentication unit 316 may determine that face authentication has failed. In this case, the face authentication unit 316 may register information indicating that there is no match (for example, "N / A (Not Applicable)") in the ID correspondence table 322. Note that if the matching score and the threshold th3 are "equal," the case may be treated as including either case.

[0119] Here, it is assumed that face authentication for person P11 is successful, and the tracking ID "00001" is associated with the authentication ID "00121".

[0120] The face tracking unit 311 has a face matching unit 312, a calculation unit 313, a determination unit 314, and a selection unit 315. The face matching unit 312 may extract features of a face image from time t-τ (here, a face image including the face of person P11) and may also extract features of a face image from time t. The face matching unit 312 may calculate a matching score based on the features of the face image from time t-τ and the features of the face image from time t. Note that the method of calculating the matching score can be the same as the method of calculating the matching score in the face authentication operation. Note that the operation of the face matching unit 312 may be performed by the face authentication unit 316. In this case, the face tracking unit 311 does not need to have the face matching unit 312.

[0121] The calculation unit 313 calculates the face F included in the face image at time t based on the matching score calculated by the face matching unit 312. ｔ is the face F included in the face image at time t-τ ｔ－τ The determining unit 314 may calculate an index indicating the likelihood that the signal corresponds to the signal. The index may be a likelihood ratio or a log-likelihood ratio. The determining unit 314 may compare the index calculated by the calculating unit 313 with a threshold value th4.

[0122] If it is determined that the calculated index is greater than the threshold value th4, the determination unit 314 determines whether the face F included in the face image at time t is a ｔ is the face F included in the face image at time t-τ ｔ－τ In this case, the selection unit 315 may determine that the face F included in the face image at time t corresponds to the face F (here, the face of the person P11). ｔ , the face F included in the face image at time t-τ ｔ－τ In this case, the selection unit 315 may select the face image at time t as a reference for tracking the face of the person P11.

[0123] If it is determined that the calculated index is smaller than the threshold value th4, the determination unit 314 determines whether the face F included in the face image at time t is ｔ is the face F included in the face image at time t-τ ｔ－τ In this case, the selection unit 314 may determine that the face F included in the face image at time t does not correspond to the face F (here, the face of the person P11). ｔ , the face F included in the face image at time t-τ ｔ－τ In this case, the selection unit 314 may select the face image at time t-τ as a reference for tracking the face of the person P11.

[0124] The facial recognition gate device 4 may determine whether to allow the person to be authenticated to pass based on the ID correspondence table 322 and the tracking ID assigned to the face contained in the facial image generated by the camera CAM by capturing an image of the person to be authenticated (e.g., at least one of persons P11 and P12).

[0125] For example, if the tracking ID assigned to the face included in the most recently generated facial image is "00001" (i.e., the person to be authenticated is person P11), the tracking ID is associated with the authentication ID "00121." In this case, the facial recognition gate device 4 may allow the person to be authenticated (i.e., person P11) to pass through. As a result, the facial recognition gate device 4 may open the flap.

[0126] For example, if the tracking ID assigned to the face included in the most recently generated facial image is "00002" (e.g., if the person being authenticated is person P12), the tracking ID is associated with "N / A." In this case, the facial recognition gate device 4 does not need to allow the person being authenticated (e.g., person P12) to pass through. As a result, the facial recognition gate device 4 may close the flap.

[0127] (Technical Effect) The facial recognition gate device 4 may determine whether to allow the person to be authenticated to pass based on the ID correspondence table 322 and the tracking ID assigned to the face included in the most recent facial image. For example, the tracking ID assigned to the face of person P11 is different from the tracking ID assigned to the face of person P12. Therefore, when person P12 cuts in front of person P11, even if facial recognition of person P11 is successful, the flap of the facial recognition gate device 4 will be closed if facial recognition of person P12 is not successful. As a result, it is possible to prevent person P12 from passing through the facial recognition gate device 4 before the facial recognition operation for person P12, who cuts in front of person P11, is completed.

[0128] For example, suppose that the face of person P11 is included in the face image at time t-τ. The face image at time t does not include the face of person P11 but includes the face of person P12. The face image at time t+τ does not include the face of person P12 but includes the face of person P11.

[0129] In this case, the determination unit 314 may determine that the face included in the face image at time t (i.e., the face of person P12) does not correspond to the face included in the face image at time t-τ (i.e., the face of person P11). In this case, the selection unit 314 may select the face image at time t-τ as a reference for tracking the face of person P11. As a result, a face tracking operation may be performed using the face image at time t-τ and the face image at time t+τ. In this case, the determination unit 314 may determine that the face included in the face image at time t+τ (i.e., the face of person P11) corresponds to the face included in the face image at time t-τ (i.e., the face of person P11). In this case, the selection unit 315 may assign the same tracking ID to the face included in the face image at time t+τ as the tracking ID assigned to the face included in the face image at time t-τ.

[0130] With this configuration, even if the camera CAM is temporarily unable to capture an image of the face of the person P11 (i.e., the person to be authenticated), the face of the person P11 can be appropriately tracked. For example, if facial authentication for the person P11 was successful before the camera CAM was unable to capture an image of the face of the person P11, when the camera CAM is able to capture an image of the face of the person P11 again, the person P11 may be permitted to pass through the facial authentication gate device 4 without performing the facial authentication operation for the person P again.

[0131] <Supplementary Notes> The following supplementary notes are further disclosed regarding the above-described embodiment.

[0132] (Supplementary Note 1) An information processing device comprising: a determination means for determining whether a certainty level when determining correspondence between a first element and a second element included in time series data, the first element being acquired at a first time and the second element being acquired at a second time after the first time, is higher than a predetermined threshold, using the first element as a criterion for correspondence between the two elements; and a selection means for selecting the second element as a new criterion for correspondence between the two elements if it is determined that the certainty level is higher than the predetermined threshold, and selecting the first element as a criterion for correspondence between the two elements if it is determined that the certainty level is lower than the predetermined threshold.

[0133] (Supplementary Note 2) The information processing device according to Supplementary Note 1, wherein the time series data is a video including a plurality of images, the first element is an object in a first image among the plurality of images that was captured at the first time, and the second element is an object in a second image among the plurality of images that was captured at the second time, the determination means determines whether the degree of certainty when determining a correspondence between the object in the second image and the object in the first image is higher than the predetermined threshold value, using the object in the first image as a reference, and the selection means selects the object in the second image as a new reference if it is determined that the degree of certainty is higher than the predetermined threshold value, and selects the object in the first image as a reference if it is determined that the degree of certainty is lower than the predetermined threshold.

[0134] (Supplementary Note 3) The information processing device according to Supplementary Note 2, further comprising a tracking means for tracking an object in the plurality of images, wherein the tracking means, when the object in the first image is selected as a reference by the selection means, tracks the object in the first image using the first image and a third image among the plurality of images that is captured at a third time after the second time, and when the object in the second image is selected as a new reference by the selection means, tracks the object in the second image using the second image and the third image.

[0135] (Supplementary Note 4) The information processing device according to Supplementary Note 2 or 3, comprises: a first generation means that generates, based on first position information about a position of an object in the first image and second position information about a position of the object in the second image, a first feature vector that indicates a feature amount of the first position information and a second feature vector that indicates a feature amount of the second position information; a second generation means that generates information obtained by arithmetic processing using the first feature vector and the second feature vector as correspondence information that indicates a correspondence relationship between the object in the first image and the object in the second image; and a calculation means that calculates, based on the correspondence information, the degree of certainty when determining correspondence between the object in the second image and the object in the first image.

[0136] (Supplementary Note 5) The information processing device according to Supplementary Note 4, wherein the correspondence information includes first information indicating that the object in the second image corresponds to the object in the first image and second information indicating that the object in the second image does not correspond to the object in the first image, and the calculation means calculates the certainty factor based on the first information and the second information.

[0137] (Supplementary Note 6) The information processing device according to Supplementary Note 6, wherein the calculation means calculates, as the certainty factor, a likelihood ratio which is a ratio between the probability that the object in the second image corresponds to the object in the first image based on the first information and the probability that the object in the second image does not correspond to the object in the first image based on the second information.

[0138] (Supplementary Note 7) The information processing device according to any one of Supplementary Notes 4 to 6, further comprising: a correction unit that corrects the second position information using the correspondence information.

[0139] (Supplementary Note 8) The information processing device according to Supplementary Note 7, wherein the correction means corrects the second position information using an attention mechanism that uses the correspondence information as a weight.

[0140] (Supplementary Note 9) The information processing device according to Supplementary Note 7 or 8, wherein when an object in the second image is selected as a new reference by the selection means, the first generation means generates a corrected second feature vector indicating a feature amount of the corrected second position information based on the second position information corrected by the correction means.

[0141] (Supplementary Note 10) An information processing method, comprising: determining whether a certainty factor when determining a correspondence between a first element included in time series data, the first element being acquired at a first time and a second element being acquired at a second time after the first time, is higher than a predetermined threshold value when determining a correspondence between the second element and the first element, when it is determined that the certainty factor is higher than the predetermined threshold value, selecting the second element as a new criterion for the correspondence between the two elements, and when it is determined that the certainty factor is lower than the predetermined threshold value, selecting the first element as the criterion for the correspondence between the two elements.

[0142] (Supplementary Note 11) A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method, the information processing method comprising: determining whether a certainty factor when determining a correspondence between a first element included in time series data, the first element being acquired at a first time and a second element being acquired at a second time after the first time, is higher than a predetermined threshold, using the first element as a criterion for correspondence between the two elements; selecting the second element as a new criterion for correspondence between the two elements if it is determined that the certainty factor is higher than the predetermined threshold; and selecting the first element as a criterion for correspondence between the two elements if it is determined that the certainty factor is lower than the predetermined threshold.

[0143] This disclosure is not limited to the above-described embodiments, but may be modified as appropriate within the scope that does not contradict the gist or idea of ​​the invention that can be read from the claims and the entire specification, and information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical scope of this disclosure.

[0144] 1, 2, 2a, 3 Information processing device 11, 216, 314 Determination unit 12, 217, 315 Selection unit 21, 31 Arithmetic unit 211 Object tracking unit 212 Object detection unit 213 Object matching unit 214 Refinement unit 215, 313 Calculation unit 218, 316 Face authentication unit 311 Face tracking unit 312 Face matching unit

Claims

1. Of the first element acquired at the first time and the second element acquired at the second time after the first time, which are included in the time-series data, when determining the correspondence between the second element and the first element with the first element as the reference for the correspondence between two elements, a determination means for determining whether the confidence level is higher than a predetermined threshold; A selection means for selecting the second element as a new reference for the correspondence between the two elements when it is determined that the confidence level is higher than the predetermined threshold, and selecting the first element as a reference for the correspondence between the two elements when it is determined that the confidence level is lower than the predetermined threshold; An information processing apparatus comprising the above.

2. The time-series data is a moving image including a plurality of images, The first element is an object in the first image captured at the first time among the plurality of images, The second element is an object in the second image captured at the second time among the plurality of images, The determination means determines whether the confidence level is higher than the predetermined threshold when determining the correspondence between the object in the second image and the object in the first image with the object in the first image as a reference, The selection means selects the object in the second image as a new reference when it is determined that the confidence level is higher than the predetermined threshold, and selects the object in the first image as a reference when it is determined that the confidence level is lower than the predetermined threshold The information processing apparatus according to Claim 1.

3. The information processing apparatus includes a tracking means for tracking the object in the plurality of images, The tracking means is When the object in the first image is selected as a reference by the selection means, the object in the first image is tracked using the first image and the third image captured at the third time after the second time among the plurality of images, When the object in the second image is selected as a new reference by the selection means, the object in the second image is tracked using the second image and the third image The information processing apparatus according to Claim 2.

4. The information processing apparatus is A first generation means for generating a first feature vector indicating a feature amount of the first position information and a second feature vector indicating a feature amount of the second position information based on the first position information regarding the position of the object in the first image and the second position information regarding the position of the object in the second image; second generation means for generating, as correspondence information indicating a correspondence relationship between the object in the first image and the object in the second image, information obtained by arithmetic processing using the first feature vector and the second feature vector; calculation means for calculating the confidence level in the case of obtaining the correspondence between the object in the second image and the object in the first image based on the correspondence information; comprising The information processing apparatus according to claim 2.

5. The correspondence information includes first information indicating that the object in the second image corresponds to the object in the first image, and second information indicating that the object in the second image does not correspond to the object in the first image. The calculation means calculates the confidence level based on the first information and the second information. The information processing apparatus according to claim 4.

6. The calculation means calculates, as the confidence level, a likelihood ratio that is a ratio between the probability that the object in the second image corresponds to the object in the first image as the first information and the probability that the object in the second image does not correspond to the object in the first image as the second information. The information processing apparatus according to claim 5.

7. The information processing apparatus includes correction means for correcting the second position information using the correspondence information. The information processing apparatus according to claim 4.

8. The correction means corrects the second position information using an attention mechanism that uses the correspondence information as a weight. The information processing apparatus according to claim 7.

9. Among the first element acquired at the first time and the second element acquired at the second time after the first time included in the time series data, with the first element as the reference for the correspondence between the two elements, it is determined whether the confidence level in the case of obtaining the correspondence between the second element and the first element is higher than a predetermined threshold. When it is determined that the confidence level is higher than the predetermined threshold, the second element is selected as the new reference for the correspondence between the two elements. When it is determined that the confidence level is lower than the predetermined threshold, the first element is selected as the reference for the correspondence between the two elements. Information processing method.

10. In a computer, Among the first element acquired at the first time and the second element acquired at the second time after the first time included in the time series data, with the first element as the reference for the correspondence between the two elements, it is determined whether the confidence level in the case of obtaining the correspondence between the second element and the first element is higher than a predetermined threshold. When it is determined that the confidence level is higher than the predetermined threshold, the second element is selected as a new reference for the correspondence between the two elements. When it is determined that the confidence level is lower than the predetermined threshold, the first element is selected as a reference for the correspondence between the two elements. A recording medium on which a computer program for executing an information processing method is recorded.