Information processing device, information processing method, and recording medium

Abstract

This information processing device comprises a determination means that determines whether a certainty factor for determining a correspondence between a second element and a first element is higher than a predetermined threshold value or not with the first element serving as a reference of the correspondence between the two elements included in time-series data, the first element being obtained at a first time, the second element being obtained at a second time preceded by the first time, and a selection means that selects the second element as another reference of the correspondence between the two elements if it is determined that the certainty factor is higher than the predetermined threshold value, and selects the first element as the reference of the correspondence between the two elements if it is determined that the certainty factor is lower than the predetermined threshold value.

Description

[Technical Field]

[0001] This disclosure relates to the technical fields of information processing equipment, information processing methods, and recording media. [Background technology]

[0002] For example, a device has been proposed that tracks a specific object from images captured at multiple times, and that tracks the target object while simultaneously tracking an object similar to the target object (see Patent Document 1). Other prior art documents related to this disclosure include Patent Documents 2 to 7. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] International Publication No. 2022 / 019076 [Patent Document 2] International Publication No. 2021 / 130951 [Patent Document 3] International Publication No. 2020 / 194497 [Patent Document 4] Japanese Patent Publication No. 2022-030852 [Patent Document 5] Japanese Patent Publication No. 2022-019339 [Patent Document 6] Japanese Patent Publication No. 2020-016901 [Patent Document 7] Japanese Patent Publication No. 2018-077807 [Overview of the project] [Problems that the invention aims to solve]

[0004] This disclosure aims to provide an information processing device, an information processing method, and a recording medium that improve upon the technologies described in prior art documents. [Means for solving the problem]

[0005] One embodiment of an information processing device includes: determination means for determining whether the degree of confidence when determining the correspondence between a second element and a first element, using the first element, which is included in time-series data, as a criterion for correspondence between the two elements, is higher than a predetermined threshold; and selection means for selecting the second element as a new criterion for correspondence between the two elements if the degree of confidence is determined to be higher than the predetermined threshold, and selecting the first element as a criterion for correspondence between the two elements if the degree of confidence is determined to be lower than the predetermined threshold. The time-series data is a video containing multiple images, the first element is an object in the first image captured at the first time, the second element is an object in the second image captured at the second time, the determination means determines whether the degree of confidence when determining the correspondence between the object in the second image and the object in the first image, using the object in the first image as a reference, is higher than a predetermined threshold, the selection means selects the object in the second image as a new reference if it is determined that the degree of confidence is higher than the predetermined threshold, and selects the object in the first image as a new reference if it is determined that the degree of confidence is lower than the predetermined threshold. The system further comprises: a first generation means that selects a criterion and generates a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information based on first position information relating to the position of an object in the first image and second position information relating to the position of an object in the second image; a second generation means that generates information obtained by arithmetic processing using the first feature vector and the second feature vector as correspondence information indicating the correspondence between an object in the first image and an object in the second image; and a calculation means that calculates the degree of confidence when determining the correspondence between an object in the second image and an object in the first image based on the correspondence information.

[0006] One aspect of information processing methods is: This is a video containing multiple images. The time series data includes, The first element is the object in the first image captured at the first time point. , and, As the second element, At the second time point, which is after the first time point mentioned above: Object in the second image captured Of these, The object in the first image Using this as the criterion for correspondence between two elements, The object in the second image and the object in the first image The system determines whether the degree of confidence when seeking a correspondence is higher than a predetermined threshold, and if it is determined that the degree of confidence is higher than the predetermined threshold, The object in the second image above If the new criterion between the two elements is selected, and the confidence level is determined to be lower than the predetermined threshold, The object in the first image Select the above two elements as the criterion for correspondence between them, Based on first position information relating to the position of an object in the first image and second position information relating to the position of an object in the second image, a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information are generated. The information obtained by arithmetic processing using the first feature vector and the second feature vector is generated as correspondence information indicating the correspondence between an object in the first image and an object in the second image. Based on the correspondence information, the confidence level for determining the correspondence between an object in the second image and an object in the first image is calculated.

[0007] One form of storage medium is used in computers. This is a video containing multiple images. The time series data includes, The first element is the object in the first image captured at the first time point. , and, As the second element, At the second time point, which is after the first time point mentioned above: Object in the second image captured Of these, The object in the first image Using this as the criterion for correspondence between two elements, The object in the second image and the object in the first image The system determines whether the degree of confidence when seeking a correspondence is higher than a predetermined threshold, and if it is determined that the degree of confidence is higher than the predetermined threshold, The object in the second image above If the new criterion between the two elements is selected, and the confidence level is determined to be lower than the predetermined threshold, The object in the first image Select the above two elements as the criterion for correspondence between them, Based on first position information relating to the position of an object in the first image and second position information relating to the position of an object in the second image, a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information are generated. The information obtained by processing using the first and second feature vectors is generated as correspondence information indicating the correspondence between the object in the first image and the object in the second image. Based on the correspondence information, the confidence level for determining the correspondence between the object in the second image and the object in the first image is calculated.The computer program used to execute the information processing method is recorded. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the configuration of an information processing device. [Figure 2] This is a block diagram showing another example of the configuration of an information processing device. [Figure 3] This figure shows an example of frames included in video data. [Figure 4] This is a block diagram showing the configuration of the object matching unit. [Figure 5] This is a flowchart showing the object matching operation according to the second embodiment. [Figure 6] This figure shows an example of a similarity matrix. [Figure 7] This is a block diagram showing the structure of the refinement section. [Figure 8] This is a flowchart showing the refinement operation according to the second embodiment. [Figure 9] This figure shows an example of how the state of a tracked object changes over time. [Figure 10] This is a block diagram showing another example of the configuration of an information processing device. [Figure 11] This is a block diagram showing another example of the configuration of an information processing device. [Figure 12] This figure shows an example of a facial recognition gate device. [Figure 13] This figure shows an example of an ID mapping table. [Modes for carrying out the invention]

[0009] Embodiments relating to an information processing device, an information processing method, and a recording medium will be described.

[0010] <First Embodiment> A first embodiment of the information processing device, information processing method, and recording medium will be described with reference to Figure 1. In the following, the first embodiment of the information processing device, information processing method, and recording medium will be described using the information processing device 1.

[0011] In Figure 1, the information processing device 1 comprises a determination unit 11 and a selection unit 12. The determination unit 11 determines whether the confidence level when determining the correspondence between the second element and the first element, using the first element as the criterion for correspondence between the two elements, is higher than a predetermined threshold, from among the first element acquired at the first time step and the second element acquired at the second time step after the first time step, which are included in the time-series data. The confidence level may be calculated using a score for determining whether the second element corresponds to the first element. Time-series data refers to a sequence of data acquired in chronological order and which can be decomposed into multiple elements. Specific examples of time-series data include video data, multiple images of the same object or place taken periodically or irregularly, and sound data. When the time-series data is video data, the multiple elements included in the time-series data may be multiple frames that make up the video, or objects included in each frame.

[0012] Elements included in time-series data may change over time. For example, if an element is an object included in each of the multiple frames that make up a video, at least one of the object's position and state may change over time. When matching elements that change over time, it may be possible to determine whether a second element, which is later in time than the first element, corresponds to the first element, using the first element, which is earlier in time, as the reference point. If it is determined that the second element corresponds to the first element, it may be possible to determine whether a third element, which is later in time than the second element, corresponds to the second element, using the second element as the new reference point. On the other hand, if it is determined that the second element does not correspond to the first element, the matching of the first element is often terminated, as it is assumed that there is no element that corresponds to the first element. However, elements may change temporarily and irregularly. Due to temporary irregular changes, it may be determined that the second element does not correspond to the first element. If the matching of the first element is terminated in this case, the element matching may not be performed properly.

[0013] If the determination unit 11 determines that the confidence level is higher than a predetermined threshold (specifically, if the score used to determine whether the second element corresponds to the first element determines that the second element corresponds to the first element, and the confidence level is higher than the predetermined threshold), the selection unit 12 selects the second element as the new criterion for correspondence between the two elements. On the other hand, if the determination unit 11 determines that the confidence level is lower than a predetermined threshold (specifically, if the score used to determine whether the second element corresponds to the first element determines that the second element corresponds to the first element, while the confidence level is lower than the predetermined threshold), the selection unit 12 selects the first element as the criterion for correspondence between the two elements (i.e., maintains the criterion for correspondence between the two elements). In this case, the correspondence between the third element, which occurs later in time than the second element, and the first element may be sought. With this configuration, the impact of temporary, irregular changes in elements on the correspondence can be suppressed. Therefore, the information processing device 1 can appropriately perform element correspondence. Furthermore, if the confidence level and the predetermined threshold are equal, they can be treated as being included in either case.

[0014] In the information processing device 1, the determination unit 11 may determine whether the confidence level when determining the correspondence between the second element and the first element, using the first element as the criterion for correspondence between the two elements, is higher than a predetermined threshold, from among the first element acquired at the first time step and the second element acquired at the second time step after the first time step, which are included in the time series data. The confidence level may be calculated using a score for determining whether the second element corresponds to the first element. If the selection unit 12 determines that the confidence level is higher than the predetermined threshold, it may select the second element as the new criterion for correspondence between the two elements. If the selection unit 12 determines that the confidence level is lower than the predetermined threshold, it may select the first element as the criterion for correspondence between the two elements.

[0015] Such an information processing device 1 may be realized, for example, by a computer reading a computer program recorded on a recording medium. In this case, the recording medium contains a computer program that causes the computer to execute an information processing method that determines whether the degree of confidence when determining the correspondence between a second element and a first element, using the first element of a second element acquired at a second time point after the first time point as the criterion for correspondence between the two elements, is higher than a predetermined threshold, and if it is determined that the degree of confidence is higher than the predetermined threshold, the second element is selected as the new criterion for correspondence between the two elements, and if it is determined that the degree of confidence is lower than the predetermined threshold, the first element is selected as the criterion for correspondence between the two elements.

[0016] Furthermore, the information processing device 1 may be implemented by a server device (for example, a cloud server) or by a terminal device (for example, at least one of a smartphone, a tablet device, and a notebook personal computer).

[0017] <Second Embodiment> A second embodiment of the information processing device, information processing method, and recording medium will be described with reference to Figures 2 to 9. In the following description, the second embodiment of the information processing device, information processing method, and recording medium will be described using the information processing device 2.

[0018] (1) Configuration of the information processing device 2 As shown in Figure 2, the information processing device 2 comprises an arithmetic unit 21, a storage device 22, and a communication device 23. The information processing device 2 may also include an input device 24 and an output device 25. However, the information processing device 2 does not need to include at least one of the input device 24 and the output device 25. In the information processing device 2, the arithmetic unit 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0019] The computing device 21 may include, for example, at least one of the following: a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a TPU (Tensor Processing Unit), and a quantum processor.

[0020] The storage device 22 may include, for example, at least one of RAM (Random Access Memory), ROM (Read Only Memory), a hard disk drive, a magneto-optical disk drive, an SSD (Solid State Drive), and an optical disk array. In other words, the storage device 22 may include a non-temporary recording medium. The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic unit 21. The storage device 22 may temporarily store data that the arithmetic unit 21 temporarily uses when it is executing a computer program. The storage device 22 may include video data 221. The video data 221 corresponds to an example of "time-series data" in the first embodiment described above.

[0021] The communication device 23 may be able to communicate with external devices of the information processing device 2 via a network (not shown). The communication device 23 may use either wired or wireless communication.

[0022] The input device 24 is a device capable of receiving information input to the information processing device 2 from an external source. The input device 24 may include an operating device (e.g., a keyboard, mouse, touch panel, etc.) that can be operated by the operator of the information processing device 2. The input device 24 may also include a recording medium reader capable of reading information recorded on a recording medium that can be attached to and detached from the information processing device 2, such as a USB (Universal Serial Bus) memory. When information is input to the information processing device 2 via the communication device 23 (in other words, when the information processing device 2 acquires information via the communication device 23), the communication device 23 may function as an input device.

[0023] The output device 25 is a device capable of outputting information to the outside of the information processing device 2. The output device 25 may output visual information such as characters and images, auditory information such as sounds, or tactile information such as vibrations. The output device 25 may include, for example, at least one of a display, speaker, printer, and vibration motor. The output device 25 may also be capable of outputting information to a recording medium that can be attached to or detached from the information processing device 2, such as a USB memory stick. When the information processing device 2 outputs information via the communication device 23, the communication device 23 may function as an output device.

[0024] The arithmetic unit 21 may have an object tracking unit 211, a calculation unit 215, a determination unit 216, and a selection unit 217, either as logically implemented functional blocks or as physically implemented processing circuits. The object tracking unit 211 may have an object detection unit 212, an object matching unit 213, and a refinement unit 214. At least one of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 may be implemented in a form that combines a logical functional block and a physical processing circuit (i.e., hardware). If at least a part of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 is a functional block, at least a part of the object tracking unit 211, the calculation unit 215, the determination unit 216, and the selection unit 217 may be implemented by the arithmetic unit 21 executing a predetermined computer program.

[0025] The arithmetic unit 21 may obtain (in other words, read) the predetermined computer program from the storage device 22. The arithmetic unit 21 may also read the predetermined computer program stored on a computer-readable and non-temporary recording medium using a recording medium reader (not shown) provided by the information processing device 2. The arithmetic unit 21 may also obtain (in other words, download or read) the predetermined computer program from an external device (not shown) of the information processing device 2 via the communication device 23. The recording medium used to record the predetermined computer program executed by the arithmetic unit 21 may be at least one of an optical disc, a magnetic medium, a magneto-optical disc, a semiconductor memory, and any other medium capable of storing a program.

[0026] (2) Object tracking operations performed by the object tracking unit 211 The object tracking operation performed by the object tracking unit 211 will now be described. The object tracking operation may include object detection, object matching, and refinement operations. The object detection, object matching, and refinement operations will be described in order below. The video data 221 contained in the storage device 22 may include frames FR1, FR2, and FR3, as shown in Figure 3. Frame FR1 is a frame captured at time t-τ. Frame FR2 is a frame captured at time t. Frame FR3 is a frame captured at time t+τ. Hereinafter, "τ" is the time corresponding to the imaging period. The object tracking unit 211 may also be called a tracking means because it performs object tracking operations.

[0027] (2-1) Object detection operation The object detection operation performed by the object detection unit 212 will now be described. The object detection unit 212 reads out a frame (for example, at least one of frames FR1, FR2, and FR3) included in the video data 221 and performs an object detection operation on the read frame. The object detection unit 212 may detect an object O included in a frame (in other words, an object O captured in the frame) using an existing method for detecting an object O included in the frame. However, it is preferable that the object detection unit 212 performs the object detection operation using a method that allows it to obtain information regarding the position of the object O within the frame (hereinafter referred to as "object position information PI") by detecting the object O included in the frame. The object position information PI obtained by the object detection unit 212 indicates the result of the object detection operation by the object detection unit 212 and may therefore be referred to as object detection information. In the following description, it will be assumed that the object detection unit 212 detects an object O using a method that allows it to obtain object position information PI.

[0028] The object detection unit 212 generates a heatmap (so-called scoremap) showing the center position (Key Point) KP (see Figure 3) of object O within the frame as object position information PI. More specifically, the object detection unit 212 generates a heatmap showing the center position KP of object O within the frame for each object O. Note that the heatmap showing the center position KP is a map related to position, so it may also be called a position map.

[0029] The object detection unit 212 may generate information as object position information PI, which shows the size of the object O's bounding box BB (see Figure 3) as a score map. The information showing the size of the object O's bounding box BB can be considered as essentially information showing the size of the object O. Furthermore, since the map information showing the size of the bounding box BB is also a map relating to position, it may also be called a position map.

[0030] The object detection unit 212 may generate, as object position information PI, information indicating the correction amount (Local Offset) of the detection frame BB of the object O as a score map. Note that since the map information indicating the correction amount of the detection frame BB is also a map related to the position, it may be referred to as a position map.

[0031] In the frame FR1 captured at time t - τ, there are four objects O t-τ #1, O t-τ #2, O t-τ #3 and O t-τ #4 included. In this case, the object detection unit 212 may generate, as object position information PI t-τ at least one of information indicating the center position KP of each of the four objects O t-τ #1, O t-τ #2, O t-τ #3 and O t-τ #4, information indicating the size of the detection frame BB, and information indicating the correction amount of the detection frame BB.

[0032] In the frame FR2 captured at time t, there are four objects O t #1, O t #2, O t #3 and O t #4 included. In this case, the object detection unit 212 may generate, as object position information PI t at least one of information indicating the center position KP of each of the four objects O t #1, O t #2, O t #3 and O t #4, information indicating the size of the detection frame BB, and information indicating the correction amount of the detection frame BB.

[0033] Furthermore, the object detection unit 212 may perform object detection using a computational model that outputs object position information PI when a frame is input. An example of such a computational model is a computational model using a neural network (e.g., CNN: Convolutional Neural Network). Furthermore, the parameters of the computational model may be optimized to output appropriate object position information PI. In this case, the parameters of the computational model are the object position information PI acquired by the object detection unit 212 (e.g., object position information PI t-τ and PI t The data may be updated based on a loss function relating to at least one of the following. The object detection unit 212 may also calculate the loss of the object position information PI based on the loss function.

[0034] (2-2) Object matching operation The object matching operation performed by the object matching unit 213 will be explained with reference to Figures 4 and 5. The object matching unit 213 reads the object position information PI acquired by the object detection unit 212 and performs the object matching operation using the read object position information PI. As shown in Figure 4, the object matching unit 213 includes a feature map conversion unit 2131, a feature vector conversion unit 2132, a feature conversion unit 2133, and a normalization unit 2134.

[0035] Below are the four objects O included in frame FR1. t-τ #1, O t-τ #2, O t-τ #3 and O t-τ #4 and the four objects O included in frame FR2 t #1, O t #2, O t #3 and O t The object matching operation that matches with #4 will be explained. From here on, the four objects O contained in frame FR1 t-τ #1, O t-τ #2, O t-τ #3 and O t-τ #4, as appropriate, "Object O t-τ It is called ". Also, the four objects O included in frame FR2 t #1, O t #2, O t#3 and O t #4, as appropriate, "Object O t It is called "".

[0036] In the flowchart of Figure 5, the feature map conversion unit 2131 converts the object O included in frame FR1. t-τ (That is, four objects O t-τ #1, O t-τ #2, O t-τ #3 and O t-τ Object position information PI related to #4) t-τ It is possible to obtain (step S101). The feature map conversion unit 2131 obtains object position information PI t-τ From, Feature Map CM t-τ You may generate (step S102). The feature map conversion unit 2131 converts the object O included in frame FR2 t (That is, four objects O t #1, O t #2, O t #3 and O t Object position information PI related to #4) t It is possible to obtain (step S101). The feature map conversion unit 2131 obtains object position information PI t From, Feature Map CM t You may generate a feature map CM (for example, a feature map CM t-τ and CM t ) is object position information PI (for example, object position information PI t-τ and PI t This is a feature map that shows the features of each arbitrary channel.

[0037] Furthermore, the feature map conversion unit 2131 may generate the feature map CM using a computational model that outputs a feature map CM when object position information PI is input. An example of such a computational model is a computational model using a neural network (e.g., CNN). The parameters of the computational model may be optimized to output an appropriate feature map CM (in particular, a feature map CM suitable for generating the similarity matrix AM described later).

[0038] In the flowchart of Figure 5, after the processing in step S102, the feature vector transformation unit 2132 transforms the feature map CM t-τ From there, feature vector CV t-τ The feature vector transformation unit 2132 generates the feature map CM. t From there, feature vector CV t The object matching unit 213 may generate the feature vector CV directly from the object position information PI without generating the feature map CM. The feature vector conversion unit 2132 generates the feature vector CV and may therefore be referred to as the first generation means.

[0039] In the flowchart of Figure 5, after the processing in step S103, the feature transformation unit 2133 performs the feature vector CV t-τ and feature vector CV t Using these, an affinity matrix (AM) may be generated (step S104). In the process of step S104, the feature transformation unit 2133 transforms the feature vector CV t-τ and feature vector CV t The similarity matrix AM may be generated using a computational model that outputs a similarity matrix AM when given the inputs. An example of such a computational model is one that uses a neural network (e.g., CNN).

[0040] In step S104, the normalization unit 2134 normalizes the similarity matrix AM. The normalization unit 2134 then normalizes the feature vector CV. t and feature vector CV t-τ The similarity matrix AM may be normalized by normalizing the matrix product with the matrix. The normalization unit 2134 may perform any normalization process on the similarity matrix AM, such as a normalization process using at least one of the sigmoid function and the softmax function.

[0041] The case in which the normalization unit 2134 performs normalization on the similarity matrix AM using the softmax function will be specifically explained. The normalization unit 2134 may perform normalization on the row vector components using the softmax function so that the sum of the row vector components, which are composed of multiple components in each row of the similarity matrix AM, becomes 1. The normalization unit 2134 may perform normalization on the column vector components using the softmax function so that the sum of the column vector components, which are composed of multiple components in each column of the similarity matrix AM, becomes 1. The normalization unit 2134 may use a matrix containing components obtained by multiplying the normalized row vector components and the normalized column vector components as the normalized similarity matrix AM.

[0042] Feature vector CV t The vector components of (x1, x2, ..., x n ) and feature vector CV t-τ The vector components of (y1, y2, ..., y n ) In this case, feature vector CV t and feature vector CV t-τ The first row of the similarity matrix AM obtained by calculating the Hadamard product with (x1*y1, x1*y2, ... x1*y) is composed of the elements of (x1*y1, x1*y2, ... x1*y n ) may be the case. The elements of the second row of the similarity matrix AM are (x2*y1, x2*y2, ... x2*y n ) may be the case. The elements of the nth row of the similarity matrix AM are (x n *y1, x n *y2, ...x n *y n ) is acceptable. Here, "*" indicates the element product by the Hadamard product.

[0043] Therefore, the components of each row of the similarity matrix AM are the feature vector CV. t A vector component and feature vector CV t-τ It may be the element-wise product of each vector component. Therefore, the vertical axis of the similarity matrix AM is the feature vector CV. t It can be said that this corresponds to the vector components of the similarity matrix AM. In other words, the vertical axis of the similarity matrix AM corresponds to the object O contained in frame FR2 at time t.t can be said to correspond to the detection result (e.g., the position of object O t . Each component of the similarity matrix AM is the element-wise product of a certain vector component of the feature vector CV t-τ and each vector component of the feature vector CV t . Therefore, it can be said that the horizontal axis of the similarity matrix AM corresponds to the vector components of the feature vector CV t-τ . That is, it can be said that the horizontal axis of the similarity matrix AM corresponds to the detection result (e.g., the position of object O t-τ ) of object O included in the frame FR1 at time t - τ t-τ .

[0044] In addition, the feature conversion unit 2133 may generate the similarity matrix AM from the element-wise product of the feature vector CV t-τ and the feature vector CV t and the features obtained by a convolutional neural network (CNN). In this case, each component of each row of the similarity matrix AM is the product of a certain vector component of the feature vector CV t-τ and each vector component of the feature vector CV t . Therefore, it can be said that the vertical axis of the similarity matrix AM corresponds to the vector components of the feature vector CV t-τ . That is, it can be said that the vertical axis of the similarity matrix AM corresponds to the detection result (e.g., the position of object O t-τ ) of object O included in the frame FR1 at time t - τ t-τ . Each component of each column of the similarity matrix AM is the product of a certain vector component of the feature vector CV t and each vector component of the feature vector CV t-τ . Therefore, it can be said that the horizontal axis of the similarity matrix AM corresponds to the vector components of the feature vector CV t . That is, it can be said that the horizontal axis of the similarity matrix AM corresponds to the detection result (e.g., the position of object O t ) of object O included in the frame FR2 at time t t .

[0045] The vector component corresponding to a certain object O t on the vertical axis and a certain object O on the horizontal axist-τ At the point where the corresponding vector component intersects, the components of the similarity matrix AM react (for example, become non-zero). In other words, object O on the vertical axis t Detection results and object O on the horizontal axis t-τ At the point where the detection result intersects with the feature vector CV, the components of the similarity matrix AM react. In other words, the similarity matrix AM is the same as the feature vector CV. t Includes an object O t The corresponding vector components and the feature vector CV t-τ Includes an object O t-τ The matrix may be such that the value of the component at the intersection of the corresponding vector component is the value obtained by multiplying the two vector components together (for example, a non-zero value), while the values ​​of the other components are zero.

[0046] In the similarity matrix AM shown in Figure 6, the feature vector CV t Object O contained within t #1 corresponding vector component and feature vector CV t-τ Object O contained within t-τ #1, object O t-τ #2, object O t-τ #3 and object O t-τ #4 The components of the similarity matrix AM at the intersection of the corresponding vector components are a 11 a 12 a 13 and a 14 Let's assume that.

[0047] In the similarity matrix AM, the feature vector CV t Object O contained within t #2 corresponding vector component and feature vector CV t-τ Object O contained within t-τ #1, object O t-τ #2, object O t-τ #3 and object O t-τ #4 The components of the similarity matrix AM at the intersection of the corresponding vector components are a 21 a 22 a 23 and a 24 Let's assume that.

[0048] In the similarity matrix AM, the feature vector CV t Object O contained within t #3 corresponds to the vector component and the feature vector CV. t-τ Object O contained within t-τ #1, object O t-τ #2, object O t-τ #3 and object O t-τ #4 The components of the similarity matrix AM at the intersection of the corresponding vector components are a 31 a 32 a 33 and a 34 Let's assume that.

[0049] In the similarity matrix AM, the feature vector CV t Object O contained within t #4 corresponds to the vector component and the feature vector CV. t-τ Object O contained within t-τ #1, object O t-τ #2, object O t-τ #3 and object O t-τ #4 The components of the similarity matrix AM at the intersection of the corresponding vector components are a 41 a 42 a 43 and a 44 Let's assume that.

[0050] In the similarity matrix AM, the feature vector CV is used. t Includes an object O t The corresponding vector components and the feature vector CV t-τ Includes an object O t-τ The component at the intersection of the corresponding vector component reacts (for example, it becomes a non-zero value). Therefore, the similarity matrix AM is for object O t and object O t-τ This information can be used to show the correspondence between objects O in frame FR2. In other words, the similarity matrix AM is used to show the correspondence between objects O in frame FR2. t and object O included in frame FR1 t-τ This information can be used to show the matching result with object O in frame FR1. The similarity matrix AM is used to show the matching result with object O t-τThis information can be used to track the position of object O within frame FR2. The similarity matrix AM is used for object O t and object O t-τ Since this information shows the correspondence relationship, it may also be called correspondence information. The feature conversion unit 2133 generates a similarity matrix AM, which may also be called correspondence information, so it may also be called a second generation means.

[0051] (2-3) Refine operation The refinement operation performed by the refinement unit 214 will be explained with reference to Figures 7 and 8. The refinement operation is an operation to correct the object position information PI acquired by the object detection unit 212. In Figure 7, the refinement unit 214 includes a feature map conversion unit 2141, a feature vector conversion unit 2142, a matrix calculation unit 2143, and a residual processing unit 2144. Since the refinement unit 214 performs a refinement operation to correct the object position information PI, it may also be called a correction means.

[0052] In the flowchart of Figure 8, the feature map conversion unit 2141 converts the object O included in frame FR1. t-τ (That is, four objects O t-τ #1, O t-τ #2, O t-τ #3 and O t-τ Object position information PI related to #4) t-τ It is possible to obtain (step S201). The feature map conversion unit 2141 converts object information PI t-τ From, Feature Map CM' t-τ You may generate (step S202). The feature map conversion unit 2141 converts the object O included in frame FR2 t (That is, four objects O t #1, O t #2, O t #3 and O t Object position information PI related to #4) t It is possible to obtain (step S201). The feature map conversion unit 2141 converts object information PI t From, Feature Map CM' t You may generate (step S202).

[0053] Furthermore, the feature map conversion unit 2141 of the refinement unit 214 and the feature map conversion unit 2131 of the object matching unit 213 are related to object position information PI (for example, object position information PI t-τ and PI t They share the commonality of generating a feature map (for example, feature map CM or CM') from the object matching matrix. However, the feature map conversion unit 2131 of the object matching unit 213 generates feature map CM for the purpose of generating a similarity matrix AM (i.e., for the purpose of performing object matching operations). In contrast, the feature map conversion unit 2141 of the refinement unit 214 generates feature map CM' for the purpose of correcting object position information PI using the similarity matrix AM (i.e., for the purpose of performing refinement operations). Therefore, the feature map conversion unit 2131 of the object matching unit 213 can generate a feature map CM that is more suitable for generating the similarity matrix AM. The feature map conversion unit 2141 of the refinement unit 214 can generate a feature map CM' that is more suitable for correcting object value information PI.

[0054] The feature map conversion unit 2141 converts object position information PI (for example, object position information PI t-τ and PI t Using a computational model that outputs a feature map CM' when ) is input, feature map CM' (for example, feature map CM' t-τ and CM' t At least one of the above may be generated. An example of such a computational model is a computational model using a neural network (e.g., CNN). The parameters of the computational model may be optimized to output an appropriate feature map CM' (in particular, a feature map CM' suitable for correcting object position information PI).

[0055] In the flowchart of Figure 8, after the processing in step S202, the feature vector transformation unit 2142 transforms the feature map CM'. t-τ From this, the feature vector CV' t-τ The feature vector transformation unit 2142 generates the feature map CM'. t From this, the feature vector CV' tYou may generate (step S203).

[0056] In the flowchart of Figure 8, the matrix calculation unit 2143 may obtain the similarity matrix AM generated by the object matching unit 213 (specifically, the feature transformation unit 2133) in parallel with or before / after the processing of steps S201 to S203 (step S204). The matrix calculation unit 2143 then obtains the feature vector CV'. t The feature vector CV_res may be generated using the similarity matrix AM (step S205). In the process of step S205, the matrix operation unit 2143 generates the feature vector CV' t The information obtained by calculating the matrix product of the similarity matrix AM (i.e., the matrix product) may be generated as the feature vector CV_res.

[0057] In the flowchart of Figure 8, after the processing in step S205, the feature vector conversion unit 2142 may generate a feature map CM_res from the feature vector CV_res (step S206). In the processing of step S206, the feature vector conversion unit 2142 may generate the feature map CM_res by converting the feature vector CV_res to the feature map CM_res.

[0058] In the flowchart of Figure 8, after the processing in step S206, the feature map conversion unit 2141 converts the object position information PI from the feature map CM_res. t_res The object position information PI may be generated (step S207). In the process of step S207, the feature map conversion unit 2141 converts the dimensions of the feature map CM_res to obtain the object position information PI from the feature map CM_res. t_res You may generate this.

[0059] For example, when the feature map conversion unit 2141 receives the feature map CM_res as input, it converts the object position information PI t_res Using a computational model that outputs object position information PI t_resIt may also be used to generate [this]. An example of such a computational model is a computational model using a neural network (e.g., CNN). The parameters of the computational model are appropriate object position information PI. t_res It may be optimized to output the following:

[0060] Furthermore, the feature map conversion unit 2141 converts (i) the object O within frame FR2 from the feature map CM_res. t Map information indicating the central position KP, and (ii) object O within frame FR2 t Map information showing the size of the detection frame BB, and (iii) object O within frame FR2 t Object position information PI includes map information indicating the correction amount of the detection frame BB. t_res You may generate this.

[0061] The processing in step SS207 essentially uses an attention mechanism that uses the similarity matrix AM as a weight to process the object position information PI. t_res This may be considered equivalent to the process of generating the object position information PI. In other words, the refinement unit 214 may constitute at least a part of the attention mechanism. t_res This is the refined object position information PI t It may also be used as follows. In this case, the process in step S207 is essentially an attention mechanism that uses the similarity matrix AM as a weight to process the object position information PI. t This may be considered equivalent to a process of correcting (in other words, updating, adjusting, or improving) it.

[0062] Here, object position information PI t_res This is the original object position information PI t (That is, object position information PI that has not undergone refinement operation) t The information contained in ) may have been lost. This is because the object position information PI t_resThis is because the similarity matrix AM, which indicates the part that should be paid attention to in the attention mechanism (in this case, the detection position of object O), is used as a weight. Therefore, there is a possibility that information other than the information regarding the detection position of object O within the object detection information may be lost.

[0063] The refinement unit 214 contains the original object position information PI. t Processing may be performed to suppress the loss of information contained in the data. Specifically, the residual processing unit 2144 processes the object position information PI. t_res The original object position information PI t By adding to this, the object position information PI t_ref This may be corrected (step S208).

[0064] In the processing of step S208, the residual processing unit 2144 processes the object position information PI t_res Object O contained within t Map information showing the central position KP and the original object position information PI t Object O contained within t Map information indicating the center position KP may be added. The residual processing unit 2144 processes the object position information PI. t_res Object O contained within t Map information showing the size of the detection frame BB, and the original object position information PI. t Object O contained within t The detection frame BB may be added to the map information indicating the size of the detection frame BB. The residual processing unit 2144 processes the object position information PI t_res Map information showing the correction amount for the detection frame BB included, and the original object position information PI. t Map information indicating the correction amount for the detection frame BB included in may be added.

[0065] Furthermore, the processing in step S208 essentially involves using a residual attention mechanism including a residual processing unit 2144 to process object position information PI. t_ref This process may be considered equivalent to the process of generating the result. In other words, the refinement unit 214 may constitute at least a part of the residual attention mechanism.

[0066] Object position information PI t_ref This is the original object position information PI t It contains information that was included in frame FR2. For example, object O included in frame FR2 t And object O included in frame FR3 t+τ When an object matching operation is performed to compare the object, the feature map conversion unit 2131 of the object matching unit 213 converts the object position information PI t Instead, object position information PI t_ref It may also obtain object position information PI. In other words, the feature map conversion unit 213 obtains object position information PI. t_ref Features Map CM t You may generate this.

[0067] Furthermore, the refinement section 214 contains the original object position information PI. t It is not necessary to perform the processing to suppress the loss of information contained in (i.e., the processing in step S208). In this case, the refinement unit 214 does not need to have a residual processing unit 2144. Note that the refinement unit 214 is used for object position information PI t_res and PI t_ref Based on the loss function relating to at least one of the following, the object position information PI t_res and PI t_ref The loss of at least one of the following may be calculated.

[0068] (3) Correspondence operation The process of matching object O using the similarity matrix AM generated by the object matching unit 213 (specifically, the feature transformation unit 2133) will be explained below. As an example, object O included in frame FR1 will be described below. t-τ (That is, four objects O t-τ #1, O t-τ #2, O t-τ #3 and O t-τ #4) and object O included in frame FR2 t (That is, four objects O t #1, O t #2, O t #3 and O t The correspondence operation with #4) will be explained.

[0069] In the similarity matrix AM shown in Figure 6, component a 11 a 12 a 13 and a 14 component a 11 Assume that the value of component a is the maximum. 21 a 22 a 23 and a 24 component a 22 Assume that the value of component a is the maximum. 31 a 32 a 33 and a 34 component a 33 Assume that the value of component a is the maximum. 41 a 42 a 43 and a 44 component a 44 Assume that the value of is the maximum.

[0070] The calculation unit 215 calculates object O included in frame FR2. t However, object O included in frame FR1 t-τ An index is calculated to show the likelihood of corresponding to object O. As mentioned above, the similarity matrix AM is for object O t and object O t-τ Since this information shows the correspondence with object O, each component of the similarity matrix AM is: t and object O t-τ This can be considered as a correspondence score. Here, the class that indicates "matching" is the class pos, and the class that indicates "not matching" is the class neg. The calculation unit 215 calculates the correspondence score of objects O included in frame FR2 based on the similarity matrix AM. t These can be classified as either class pos or class neg.

[0071] Component a of the similarity matrix AM 11 a 12 a 13 and a 14 component a 11 The value of is the maximum. In this case, object O included in frame FR2 t#1 is object O included in frame FR1 t-τ It is highly likely that this corresponds to #1. In this case, the calculation unit 215 calculates object O included in frame FR2. t #1 is object O included in frame FR1 t-τ The probability of being associated with #1 (in other words, object O contained in frame FR2) t You may calculate the probability that #1 belongs to class pos. The result of this calculation is “p(pos|O t It is acceptable to write it as "#1)". For example, "p(pos|O t #1)=a 11 "This may be the case. The calculation unit 215 calculates the object O included in frame FR2. t #1 is object O included in frame FR1 t-τ The probability that object O cannot be associated with #1 (in other words, object O included in frame FR2) t You may calculate the probability that #1 belongs to the class neg. The result of this calculation is “p(neg|O t It is acceptable to write it as "#1)". For example, "p(neg|O t #1) = 1 - a 11 "That's fine."

[0072] The calculation unit 215 calculates object O included in frame FR2. t #1 is object O included in frame FR1 t-τ As an indicator of the likelihood of corresponding to #1, the likelihood ratio “p(pos|O t #1) / p(neg|O t You may calculate "#1)". Note that "p(pos|O t #1)” is object O included in frame FR2 t #1 is object O included in frame FR1 t-τ This could also be called the first piece of information indicating that it corresponds to #1. “p(neg|O t #1)” is object O included in frame FR2 t #1 is object O included in frame FR1 t-τ This could also be called second information indicating that it does not correspond to #1.

[0073] By the way, the calculation unit 215 calculates the object O included in frame FR2. t Object O is included in frame FR1. t-τ An indicator that shows the likelihood of corresponding to (for example, “p(pos|O t ) / p(neg|O t )") is an object O included in frame FR2. t and object O included in frame FR1 t-τ It may also be calculated considering the relationship with. In this case, the above index is “p(pos|O t ,O t-τ ) / p(neg|O t ,O t-τ It may be written as ")". However, in this embodiment, object O t and object O t-τ The correspondence with (in other words, object O t and object O t-τ The similarity matrix AM, which shows the relationship between objects O, can be used. t and object O t-τ The pair can be treated as a single element. Therefore, according to this embodiment, the calculation cost for the calculation unit 215 to calculate the above index can be reduced.

[0074] As mentioned above, component a 21 a 22 a 23 and a 24 component a 22 The value of is the maximum. In this case, object O included in frame FR2 t #2 is object O included in frame FR1 t-τ It is highly likely that this corresponds to #2. The calculation unit 215 is for object O included in frame FR2. t #2 is object O included in frame FR1 t-τ As an indicator of the likelihood of corresponding to #2, the likelihood ratio “p(pos|O t #2) / p(neg|O t You may calculate #2).

[0075] As mentioned above, component a 31 a32 a 33 and a 34 component a 33 The value of is the maximum. In this case, object O included in frame FR2 t #3 is object O included in frame FR1 t-τ It is highly likely that this corresponds to #3. The calculation unit 215 is for object O included in frame FR2. t #3 is object O included in frame FR1 t-τ As an indicator of the likelihood of corresponding to #3, the likelihood ratio “p(pos|O t #3) / p(neg|O t You may calculate #3).

[0076] As mentioned above, component a 41 a 42 a 43 and a 44 component a 44 The value of is the maximum. In this case, object O included in frame FR2 t #4 is object O included in frame FR1 t-τ It is highly likely that this corresponds to #4. The calculation unit 215 is for object O included in frame FR2. t #4 is object O included in frame FR1 t-τ As an indicator of the likelihood of corresponding to #4, the likelihood ratio “p(pos|O t #4) / p(neg|O t You may calculate #4).

[0077] Furthermore, the calculation unit 215 calculates the object O included in frame FR2. t However, object O included in frame FR1 t-τ The log-likelihood ratio (for example, Log{p(pos|O)) is an indicator of the likelihood of corresponding to t ) / p(neg|O t The following may be calculated: )}). The above indicators (e.g., likelihood ratio, log-likelihood ratio) may also be referred to as confidence levels.

[0078] The determination unit 216 determines, based on the index (e.g., likelihood ratio) calculated by the calculation unit 215, that the object O included in frame FR2t However, object O included in frame FR1 t-τ The determination unit 216 determines whether or not it corresponds to object O included in frame FR2. t Regarding #1, the likelihood ratio “p(pos|O t #1) / p(neg|O t #1)” can be used to determine whether it is greater than the threshold th1. Likelihood ratio “p(pos|O t #1) / p(neg|O t If #1)" is greater than the threshold th1, the determination unit 216 determines that the object O included in frame FR2 t #1 can be determined to be a suitable reference for the mapping in the next frame. Likelihood ratio “p(pos|O t #1) / p(neg|O t #1) If the threshold th1 is smaller than the threshold, the determination unit 216 determines that the object O included in frame FR2 t #1 can be determined to be unsuitable as a reference source for the mapping in the next frame. Note that the likelihood ratio “p(pos|O t #1) / p(neg|O t If #1)" is equal to the threshold th1, then it can be included in either case.

[0079] If the index calculated by the calculation unit 215 is the log-likelihood ratio, the threshold th1 may be "1". This is because if the likelihood ratio is greater than 1, p(pos|O t )>p(neg|O t Therefore, it is appropriate to classify it into the pos class, which indicates that it is "associated".

[0080] The determination unit 216 determines object O included in frame FR2. t Regarding #2, the likelihood ratio “p(pos|O t #2) / p(neg|O t #2)” can be used to determine whether it is greater than the threshold th1. Likelihood ratio “p(pos|O t #2) / p(neg|O t #2) If the threshold th1 is greater than the threshold, the determination unit 216 determines that the object O included in frame FR2 t#2 can be determined to be a suitable reference for the mapping in the mapping of the next frame. Likelihood ratio “p(pos|O t #2) / p(neg|O t #2) If the threshold th1 is smaller than the threshold, the determination unit 216 determines that the object O included in frame FR2 t #2 can be determined to be unsuitable as a reference source for the mapping in the next frame. Note that the likelihood ratio “p(pos|O t #2) / p(neg|O t #2) If the threshold th1 is equal to the threshold, then it can be included in either case.

[0081] The determination unit 216 determines object O included in frame FR2. t Regarding #3, the likelihood ratio “p(pos|O t #3) / p(neg|O t #3)” can be used to determine whether it is greater than the threshold th1. Likelihood ratio “p(pos|O t #3) / p(neg|O t #3) If the threshold th1 is greater than the threshold, the determination unit 216 determines that the object O included in frame FR2 t #3 can be determined to be a suitable reference for the mapping in the mapping of the next frame. Likelihood ratio “p(pos|O t #3) / p(neg|O t #3) If the threshold th1 is smaller, the determination unit 216 determines that the object O included in frame FR2 t #3 can be determined to be unsuitable as a reference source for the mapping in the next frame. Note that the likelihood ratio “p(pos|O t #3) / p(neg|O t If #3)" is equal to the threshold th1, then it can be included in either case.

[0082] The determination unit 216 determines object O included in frame FR2. t Regarding #4, the likelihood ratio “p(pos|O t #4) / p(neg|O t #4)” can be used to determine whether it is greater than the threshold th1. Likelihood ratio “p(pos|O t #4) / p(neg|Ot #4) If the threshold th1 is greater than the threshold, the determination unit 216 determines that the object O included in frame FR2 t #4 can be determined to be a suitable reference for the mapping in the next frame. Likelihood ratio “p(pos|O t #4) / p(neg|O t #4) If the threshold th1 is smaller than the threshold, the determination unit 216 determines that the object O included in frame FR2 t #4 can be determined to be unsuitable as a reference source for the mapping in the next frame. Note that the likelihood ratio “p(pos|O t #4) / p(neg|O t If #4)" is equal to the threshold th1, then it should be included in either case.

[0083] The selection unit 217, based on the determination result of the confidence level in the log-likelihood ratio of the determination unit 216, selects the object O included in frame FR2. t and object O included in frame FR1 t-τ The correspondence is made with the O included in frame FR2. The selection unit 217 is O t The matching and calculation of confidence levels may be performed for each item. Note that the matching may be performed by the determination unit 216 instead of the selection unit 217.

[0084] For example, the determination unit 216 determines that object O included in frame FR2 t #1 is object O, which is included in frame FR1. t-τ If the selection unit 217 determines that the confidence level for #1 is high (for example, the log-likelihood ratio is higher than the threshold), then the selection unit 217 selects object O included in frame FR2. t #1 may be used as the reference source for the correspondence in the next frame. Specifically, the selection unit 217 selects object O included in frame FR2. t #1 contains object O in frame FR1 t-τ After assigning the same tracking ID as the one assigned to #1, the object matching unit 213 of the next frame obtains the necessary information using a feature vector CV. t-τ It may be used as such.

[0085] In this case, the selection unit 217 is selected for the object O included in frame FR2. t #1, object O t #1 may be selected as a reference (e.g., a reference source) for tracking its position within frame FR3 (see Figure 3). As a result, the object tracking unit 211 uses frames FR2 and FR3 to track the position of object O contained in frame FR2. t Object tracking operation may be performed for #1. In this case, the object matching unit 213 will perform object position information PI t Instead, object position information PI t_res or PI t_ref You may use the following. Note that object position information PI t The object detection unit 212 detects an object O included in frame FR2. t Object O within frame FR2 is obtained by detecting it. t This is information regarding the position of the object. Object position information PI t_res or PI t_ref This is the refined object position information PI generated by the refinement unit 214. t That is the case.

[0086] On the other hand, the determination unit 216 determines that the object O included in the frame FR2 t #1 is object O, which is included in frame FR1. t-τ If the confidence level for #1 is determined to be low (for example, the log-likelihood ratio is lower than the threshold), the selection unit 217 selects object O included in frame FR2. t #1, object O included in frame FR1 t-τ It is not necessary to associate it with #1. In this case, the selection unit 217 is the object O included in frame FR2. t #1 is a new object (i.e., object O included in frame FR1) t-τ It may be determined that it is an object different from the object O included in frame FR2. In this case, the selection unit 217 determines that it is an object O t You may assign a new tracking ID (in other words, an unused tracking ID) to #1.

[0087] In this case, the selection unit 217 is selected for the object O included in frame FR1. t-τ #1, object O t-τIt may be selected as a reference (e.g., a source) for tracking the position of #1 within frame FR3. This is because object O contained in frame FR1 is in frame FR2. t-τ This is because it does not contain the object corresponding to #1. As a result, the object tracking unit 211 uses frames FR1 and FR3 to track the object O contained in frame FR1. t-τ You may perform object tracking on #1.

[0088] For example, the determination unit 216 determines that object O included in frame FR2 t #1 is object O, which is included in frame FR1. t-τ While the confidence level is determined to be high for #1, the determination unit 216 determines that object O included in frame FR2 t #2 is object O, which is included in frame FR1. t-τ If the confidence level for #2 is determined to be low, the selection unit 217 selects object O included in frame FR2. t #1, object O t Selected as the reference point (e.g., source) for tracking the position of object O in frame FR1, and also as the reference point for tracking the position of object O in frame FR3. t-τ #2, object O t-τ This can be selected as a reference point (e.g., a source) for tracking the position within frame FR3 of #2.

[0089] As a result, the object tracking unit 211 uses frames FR2 and FR3 to track the object O contained in frame FR2. t Object tracking operation may be performed for #1. The object tracking unit 211 uses frames FR1 and FR3 to track object O included in frame FR1. t-τ You may perform object tracking operations for #2.

[0090] Furthermore, the operation of the information processing device 2 described above may be achieved by the information processing device 2 reading a computer program recorded on a recording medium. In this case, it can be said that the recording medium contains a computer program that causes the information processing device 2 to perform the above-described operation.

[0091] (Technical effects) When tracking an object using multiple images (e.g., video) captured by a camera as time-series data, the following technical problems may arise. For example, the camera may be temporarily unable to capture the target object because it is hidden by other objects. In this case, tracking of the object may be terminated because the object in one image is not included in other images captured after that image. For example, the target object may undergo irregular changes. Specifically, if the object is a person, they may suddenly crouch or change direction. In this case, even if the same object is included in one image and other images captured after that image, the object in one image and the object in the other images may not be able to be associated. In this case, the object in the other images may be recognized as a new object.

[0092] As shown in Figure 9, assume that the state of person P as the object being tracked changes. Specifically, at times t1 and t2, person P is walking. At times t3 and t4, person P is jumping. At times t5 and t6, person P is walking again. In this case, if tracking of person P is performed using an image containing person P taken at time t2 and an image containing person P taken at time t3, it is possible that person P in the image taken at time t2 and person P in the image taken at time t3 will be determined not to correspond. This is because the difference between the state of person P at time t2 (e.g., posture) and the state of person P at time t3 is relatively large. In this case, person P at time t2 and person P at time t3 may be treated as different people. In other words, the tracking of the tracking ID assigned to person P at time t2 may be terminated, and a new tracking ID may be assigned to person P at time t3.

[0093] Furthermore, if tracking of person P is performed using an image containing person P taken at time t4 and an image containing person P taken at time t5, it is possible that person P in the image taken at time t4 and person P in the image taken at time t5 may be determined not to correspond. This is because the difference between the state of person P at time t4 (e.g., posture) and the state of person P at time t5 is relatively large. In this case, person P at time t4 and person P at time t5 may be treated as different people. In other words, the tracking of the tracking ID assigned to person P at time t4 may be terminated, and a new tracking ID may be assigned to person P at time t5.

[0094] To address these technical problems, one possible approach is to use three or more images to track objects (in other words, to associate objects). However, since three or more images must be processed in a single object tracking operation, real-time processing is extremely difficult. Furthermore, if the time-series data is a 30 FPS (Frequencies Per Second) video, only about 0.1 seconds of object movement can be considered from a computational cost perspective.

[0095] For example, the determination unit 216 determines that object O included in frame FR2 t However, object O included in frame FR1 t-τ You may determine whether or not it corresponds to object O in frame FR2. t However, object O included in frame FR1 t-τ If it is determined that it corresponds to object O included in frame FR2, the selection unit 217 selects object O t This may be selected as a reference (e.g., a reference source) for tracking the position of object O within frame FR3. As a result, the object tracking unit 211 uses frames FR2 and FR3 to track object O contained in frame FR2. t Object tracking operations may be performed on object O included in frame FR2. On the other hand, object O included in frame FR2 t However, object O included in frame FR1 t-τ If it is determined that it does not correspond, the selection unit 217 selects object O included in frame FR1. t-τThis may be selected as a reference (e.g., a reference source) for tracking the position of object O within frame FR3. As a result, the object tracking unit 211 uses frames FR1 and FR3 to track object O contained in frame FR1. t-τ Object tracking operations may be performed on the object.

[0096] In the example shown in Figure 9, the determination unit 216 may determine that the person P included in the image captured at time t2 does not correspond to the person P included in the image captured at time t3. In this case, the selection unit 217 may select the person P included in the image captured at time t2 as a reference (e.g., a reference source) for tracking the location of person P in the image captured at time t4.

[0097] The object tracking unit 211 may perform object tracking using the image captured at time t2 and the image captured at time t4. The determination unit 216 may determine that the person P included in the image captured at time t2 does not correspond to the person P included in the image captured at time t4. In this case, the selection unit 217 may select the person P included in the image captured at time t2 as a reference (e.g., a reference source) for tracking the person P in the image captured at time t5.

[0098] The object tracking unit 211 may perform object tracking using the image captured at time t2 and the image captured at time t5. The determination unit 216 may determine that the person P included in the image captured at time t2 corresponds to the person P included in the image captured at time t5. In this case, the selection unit 217 may assign the same tracking ID to the person P included in the image captured at time t5 as the tracking ID assigned to the person P included in the image captured at time t2.

[0099] According to the information processing device 2, even if the object to be tracked cannot be imaged temporarily or changes erratically temporarily, the object to be tracked can be properly tracked. In addition, since the object tracking operation performed by the object tracking unit 211 is performed using two images, computational costs can be reduced and processing can be done in real time.

[0100] Furthermore, the object to be tracked is not limited to a person (e.g., person P). The object to be tracked may also be a moving object such as a vehicle. Furthermore, the information processing device 2 may be implemented by a server device (e.g., a cloud server) or by a terminal device (e.g., at least one of a smartphone, tablet device, and notebook personal computer).

[0101] (modified version) If the object to be tracked is a person (e.g., person P), a facial recognition operation may be performed in addition to the object tracking operation. In Figure 10, the information processing device 2a may be equipped with a facial recognition unit 218 for performing the facial recognition operation. The storage device 22 may contain a facial feature database 222 (hereinafter referred to as "facial feature DB222"). Existing technologies (e.g., at least one of a two-dimensional (2D) authentication method and a three-dimensional (3D) authentication method) can be applied to the facial recognition operation.

[0102] The facial recognition unit 218 uses object position information PI (for example, object position information PI) acquired by the object detection unit 212. t-τ and PI t Based on at least one of the above, the face of object O (here, a person) contained in a frame (for example, at least one of frames FR1 and FR2) may be detected. Note that existing techniques can be applied to the method of detecting a person's face from a frame (image), so a detailed explanation of that will be omitted.

[0103] If a face is detected, the face recognition unit 218 may generate a face image that includes the face region in the frame. The face recognition unit 218 may extract feature quantities from the generated face image. The face recognition unit 218 may calculate a matching score (or similarity score) based on the extracted feature quantities and the feature quantities registered in the face feature quantity DB 222. The face recognition unit 218 may compare the calculated matching score with a threshold th2. If the matching score is greater than the threshold th2, the face recognition unit 218 may determine that face recognition was successful. In this case, the face recognition unit 218 may associate the object O (in this case, a person) included in the frame with the authentication ID registered in the face feature quantity DB 222.

[0104] If the matching score is less than the threshold th2, the face recognition unit 218 may determine that face recognition has failed. If the matching score and the threshold th2 are "equal", either case may be included. Furthermore, if no face is detected in a given frame, the face recognition unit 218 does not need to perform face recognition for that frame.

[0105] <Third Embodiment> A third embodiment of the information processing device, information processing method, and recording medium will be described with reference to Figures 11 and 12. In the following description, the third embodiment of the information processing device, information processing method, and recording medium will be described using the information processing device 3.

[0106] As shown in Figure 11, the information processing device 3 comprises an arithmetic unit 31, a storage device 32, and a communication device 33. The information processing device 3 may also include an input device 34 and an output device 35. However, the information processing device 3 does not need to include at least one of the input device 34 and the output device 35. In the information processing device 3, the arithmetic unit 31, the storage device 32, the communication device 33, the input device 34, and the output device 35 may be connected via a data bus 36. The storage device 32 may include a face feature database 321 (hereinafter referred to as "face feature DB321") and an ID correspondence table 322.

[0107] Furthermore, the basic configurations of the arithmetic unit 31, storage device 32, communication device 33, input device 34, and output device 35 may be the same as those of the arithmetic unit 21, storage device 22, communication device 23, input device 24, and output device 25 in the second embodiment described above. For this reason, a description of the basic configurations of the arithmetic unit 31, storage device 32, communication device 33, input device 34, and output device 35 will be omitted.

[0108] The arithmetic unit 31 may have a face tracking unit 311 and a face recognition unit 316, either as logically implemented functional blocks or as physically implemented processing circuits. At least one of the face tracking unit 311 and the face recognition unit 316 may be implemented in a form that combines logical functional blocks and physical processing circuits (i.e., hardware). If at least a part of the face tracking unit 311 and the face recognition unit 316 are functional blocks, at least a part of the face tracking unit 311 and the face recognition unit 316 may be implemented by the arithmetic unit 31 executing a predetermined computer program.

[0109] The arithmetic unit 31 may obtain (in other words, read) the predetermined computer program from the storage device 32. The arithmetic unit 31 may also read the predetermined computer program stored on a computer-readable and non-temporary recording medium using a recording medium reader (not shown) provided by the information processing device 3. The arithmetic unit 31 may also obtain (in other words, download or read) the predetermined computer program from an external device (not shown) of the information processing device 3 via the communication device 33. At least one of the following may be used as the recording medium for recording the predetermined computer program executed by the arithmetic unit 31: an optical disc, a magnetic medium, a magneto-optical disc, a semiconductor memory, and any other medium capable of storing a program.

[0110] The information processing device 3 is assumed to be part of the facial recognition gate device 4 shown in Figure 12. However, the information processing device 3 may be a different device from the facial recognition gate device 4. In this case, the information processing device 3 may be configured to communicate with the facial recognition gate device 4 via the communication device 33. In this case, the information processing device 3 may be implemented as a server device (e.g., a cloud server) or as a terminal device (e.g., at least one of a smartphone, tablet, and notebook computer).

[0111] The facial recognition gate device 4 is equipped with a camera CAM. The facial recognition unit 316 of the information processing device 3 may perform facial recognition using a facial image generated by the camera CAM capturing the face of the person to be recognized (for example, a person attempting to pass through the facial recognition gate device 4). If facial recognition of the person to be recognized is successful, the facial recognition gate device 4 allows the person to pass through. If the facial recognition gate device 4 is a flap-type gate device, the facial recognition gate device 4 may keep the flap open. On the other hand, if facial recognition of the person to be recognized fails, the facial recognition gate device 4 does not allow the person to pass through. In this case, the facial recognition gate device 4 may keep the flap closed. Note that the facial recognition gate device 4 is not limited to a flap-type gate device, but may also be an arm-type gate device or a sliding-type gate device.

[0112] The camera CAM captures the face of the person to be authenticated multiple times as they approach the face authentication gate device 4. As a result, multiple chronologically consecutive face images may be generated. These multiple face images correspond to other examples of "time-series data" in the first embodiment described above. The face authentication unit 316 may perform face authentication using at least one of the multiple face images. Therefore, if face authentication is successful, the face authentication gate device 4 can open its flap before the person to be authenticated reaches the face authentication gate device 4. As a result, the person to be authenticated can pass through the face authentication gate device 4 without stopping at it. In other words, the face authentication gate device 4 is a so-called walk-through type face authentication gate device.

[0113] In FIG. 12, when the face authentication unit 316 performs a face authentication operation using a face image generated by the camera CAM capturing the face of person P11 (i.e., the authenticated person), person P12 may interrupt in front of person P11. In this case, if the flap of the face authentication gate device 4 is in an open state due to the successful face authentication of person P11, there is a possibility that person P12 may pass through the face authentication gate device 4. In FIG. 12, the dotted arrows indicate the traveling directions of persons P11 and P12.

[0114] The face tracking unit 311 of the arithmetic device 31 may perform a face tracking operation using a plurality of face images generated by the camera CAM capturing the authenticated person (for example, at least one of persons P11 and P12) a plurality of times. For example, assuming that the face F included in the face image at time t - τ is the face of person P11. t-τ Let's assume that the face of person P11 as face F has a unique tracking ID assigned to it. Let's assume that the tracking ID assigned to the face of person P11 is "00001". [[ID=**7]] t-τ The tracking ID is registered in the ID correspondence table 322. As shown in FIG. 13, the ID correspondence table 322 shows the correspondence relationship between the tracking ID and the authentication ID. The ID correspondence table 322 may include the collation time, which is the time when the face authentication operation was performed.

[0115] The face authentication unit 316 may perform a face authentication operation using a face image including a face to which a tracking ID is assigned. The face authentication unit 316 may extract the feature amount of a face image including a face to which a tracking ID is assigned. The face authentication unit 316 may calculate a collation score (or a similarity score) based on the extracted feature amount and the feature amount registered in the face feature amount DB 321. The face authentication unit 316 may compare the calculated collation score with a threshold th3.

[0116]

[0117] ​If the matching score is greater than the threshold th3, the face recognition unit 316 may determine that face recognition was successful. In this case, the face recognition unit 316 may associate the tracking ID (in other words, the face included in the face image) with the authentication ID registered in the face feature database 321. The face recognition unit 316 may associate the tracking ID with the authentication ID by registering the authentication ID in the ID correspondence table 322.

[0118] If the matching score is less than the threshold th3, the facial recognition unit 316 may determine that facial recognition has failed. In this case, the facial recognition unit 316 may register information indicating that there is no matching person (for example, "N / A (Not Applicable)") in the ID correspondence table 322. If the matching score and the threshold th3 are "equal", it may be treated as being included in either case.

[0119] Here, we assume that facial recognition for person P11 is successful, and that the authentication ID "00121" is associated with the tracking ID "00001".

[0120] The face tracking unit 311 includes a face matching unit 312, a calculation unit 313, a determination unit 314, and a selection unit 315. The face matching unit 312 may extract feature quantities from the face image at time t-τ (here, a face image including the face of person P11) and also extract feature quantities from the face image at time t. The face matching unit 312 may calculate a matching score based on the feature quantities of the face image at time t-τ and the feature quantities of the face image at time t. The method for calculating the matching score can be the same as the method for calculating the matching score in face recognition operation. The operation of the face matching unit 312 may be performed by the face recognition unit 316. In this case, the face tracking unit 311 does not need to have a face matching unit 312.

[0121] The calculation unit 313 determines the face F included in the face image at time t based on the matching score calculated by the face matching unit 312. t Face F is included in the face image at time t-τ. t-τ An index indicating the likelihood of a correspondence may be calculated. This index may be a likelihood ratio or a log-likelihood ratio. The determination unit 314 may compare the index calculated by the calculation unit 313 with the threshold th4.

[0122] If the calculated index is determined to be greater than the threshold th4, the determination unit 314 determines that the face F included in the face image at time t is t Face F is included in the face image at time t-τ. t-τ It can be determined that this corresponds to (the face of person P11). In this case, the selection unit 315 selects face F included in the face image at time t. t In the face image at time t-τ, face F t-τ The same tracking ID as the one assigned to the previous device may be assigned. In this case, the selection unit 315 may select the face image at time t as the criterion for tracking the face of person P11.

[0123] If the calculated index is determined to be smaller than the threshold th4, the determination unit 314 determines that the face F included in the face image at time t is t Face F is included in the face image at time t-τ. t-τ It is acceptable to determine that it does not correspond to (the face of person P11 in this case). In this case, the selection unit 314 selects face F included in the face image at time t. t In the face image at time t-τ, face F t-τ A different tracking ID (for example, an unused tracking ID) may be assigned to the device. In this case, the selection unit 314 may select the face image at time t-τ as the criterion for tracking the face of person P11.

[0124] The facial recognition gate device 4 may determine whether or not to allow the person to pass through based on the ID correspondence table 322 and the tracking ID assigned to the face included in the facial image generated by the camera CAM capturing images of the person to be authenticated (for example, at least one of people P11 and P12).

[0125] For example, if the tracking ID assigned to a face in the most recently generated face image is "00001" (i.e., if the person being authenticated is person P11), then this tracking ID is associated with the authentication ID "00121". In this case, the face authentication gate device 4 may allow the person being authenticated (i.e., person P11) to pass through. As a result, the face authentication gate device 4 may open its flap.

[0126] For example, if the tracking ID assigned to a face in the most recently generated face image is "00002" (for example, if the person being authenticated is person P12), then this tracking ID is associated with "N / A". In this case, the face authentication gate device 4 does not need to allow the person being authenticated (for example, person P12) to pass through. As a result, the face authentication gate device 4 may keep the flap closed.

[0127] (Technical effects) The facial recognition gate device 4 may determine whether to allow the person to be authenticated to pass through based on the ID correspondence table 322 and the tracking ID assigned to the face included in the most recent facial image. For example, the tracking ID assigned to the face of person P11 and the tracking ID assigned to the face of person P12 are different. Therefore, if person P12 cuts in front of person P11, even if facial recognition of person P11 is successful, if facial recognition of person P12 is not successful, the flap of the facial recognition gate device 4 will be closed. As a result, it is possible to prevent person P12 from passing through the facial recognition gate device 4 before the facial recognition operation for person P12, who cut in front of person P11, is completed.

[0128] For example, suppose the face image at time t-τ contains the face of person P11. Suppose the face image at time t does not contain the face of person P11, but does contain the face of person P12. Suppose the face image at time t+τ does not contain the face of person P12, but does contain the face of person P11.

[0129] In this case, the determination unit 314 may determine that the face included in the face image at time t (i.e., the face of person P12) does not correspond to the face included in the face image at time t-τ (i.e., the face of person P11). In this case, the selection unit 314 may select the face image at time t-τ as the reference for tracking the face of person P11. As a result, face tracking may be performed using the face image at time t-τ and the face image at time t+τ. In this case, the determination unit 314 may determine that the face included in the face image at time t+τ (i.e., the face of person P11) corresponds to the face included in the face image at time t-τ (i.e., the face of person P11). In this case, the selection unit 315 may assign the same tracking ID to the face included in the face image at time t+τ as the tracking ID assigned to the face included in the face image at time t-τ.

[0130] With this configuration, even if the camera CAM is temporarily unable to capture the face of person P11 (i.e., the person being authenticated), the face of person P11 can still be properly tracked. For example, if facial recognition for person P11 is successful before the camera CAM is unable to capture the face of person P11, then when the camera CAM becomes able to capture the face of person P11, person P11 may be allowed to pass through the facial recognition gate device 4 without having to perform the facial recognition operation for person P11 again.

[0131] <Note> The following additional information is disclosed regarding the embodiments described above.

[0132] (Note 1) A determination means for determining whether the degree of confidence when determining the correspondence between a second element and a first element, using the first element obtained at a first time point and a second element obtained at a second time point after the first time point as the basis for correspondence between the two elements, is higher than a predetermined threshold, A selection means that, if the confidence level is determined to be higher than the predetermined threshold, selects the second element as the new criterion for correspondence between the two elements, and if the confidence level is determined to be lower than the predetermined threshold, selects the first element as the criterion for correspondence between the two elements. An information processing device equipped with the following features.

[0133] (Appendix 2) The time-series data is a video including a plurality of images, The first element is an object in a first image captured at the first time among the plurality of images, The second element is an object in a second image captured at the second time among the plurality of images, The determination means determines whether or not the confidence level when obtaining the correspondence between the object in the second image and the object in the first image based on the object in the first image is higher than the predetermined threshold value, When it is determined that the confidence level is higher than the predetermined threshold value, the selection means selects the object in the second image as a new reference, and when it is determined that the confidence level is lower than the predetermined threshold value, the selection means selects the object in the first image as a reference The information processing apparatus according to Appendix 1.

[0134] (Appendix 3) The information processing apparatus includes tracking means for tracking an object in the plurality of images, The tracking means is When the object in the first image is selected as a reference by the selection means, tracking of the object in the first image is performed using the first image and a third image captured at a third time after the second time among the plurality of images, When the object in the second image is selected as a new reference by the selection means, tracking of the object in the second image is performed using the second image and the third image The information processing apparatus according to Appendix 2.

[0135] (Appendix 4) The information processing apparatus is First generation means for generating a first feature vector indicating a feature amount of the first position information and a second feature vector indicating a feature amount of the second position information based on the first position information regarding the position of the object in the first image and the second position information regarding the position of the object in the second image, A second generation means generates correspondence information indicating the correspondence between an object in the first image and an object in the second image, based on information obtained by computation using the first and second feature vectors, A calculation means for calculating the degree of confidence when determining the correspondence between an object in the second image and an object in the first image based on the correspondence information, Equipped with The information processing device described in Appendix 2 or 3.

[0136] (Note 5) The correspondence information includes first information indicating that an object in the second image corresponds to an object in the first image, and second information indicating that an object in the second image does not correspond to an object in the first image. The calculation means calculates the confidence level based on the first and second information. The information processing device described in Appendix 4.

[0137] (Note 6) The calculation means calculates the confidence level as a likelihood ratio, which is the ratio of the probability that the object in the second image corresponds to the object in the first image, based on the first information, to the probability that the object in the second image does not correspond to the object in the first image, based on the second information. The information processing device described in Appendix 6.

[0138] (Note 7) The information processing device includes correction means for correcting the second position information using the corresponding information. An information processing device as described in any one of the items 4 to 6 of the appendix.

[0139] (Note 8) The correction means corrects the second position information using an attention mechanism that uses the correspondence information as a weight. The information processing device described in Appendix 7.

[0140] (Note 9) When the object in the second image is selected as a new reference by the selection means, the first generation means generates a corrected second feature vector representing the feature quantities of the corrected second position information based on the second position information corrected by the correction means. The information processing device described in Appendix 7 or 8.

[0141] (Note 10) The system determines whether the confidence level when determining the correspondence between the second element and the first element, using the first element obtained at the first time point and the second element obtained at a second time point after the first time point as the basis for correspondence between the two elements, is higher than a predetermined threshold. If the confidence level is determined to be higher than the predetermined threshold, the second element is selected as a new criterion for correspondence between the two elements. If the confidence level is determined to be lower than the predetermined threshold, the first element is selected as the criterion for correspondence between the two elements. Information processing methods.

[0142] (Note 11) On the computer, The system determines whether the confidence level when determining the correspondence between the second element and the first element, using the first element obtained at the first time point and the second element obtained at a second time point after the first time point as the basis for correspondence between the two elements, is higher than a predetermined threshold. If the confidence level is determined to be higher than the predetermined threshold, the second element is selected as a new criterion for correspondence between the two elements. If the confidence level is determined to be lower than the predetermined threshold, the first element is selected as the criterion for correspondence between the two elements. A recording medium on which computer programs for executing information processing methods are stored.

[0143] This disclosure is not limited to the embodiments described above and may be modified as appropriate, provided that it does not contradict the gist or idea of ​​the invention as can be inferred from the claims and the specification as a whole. Information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical scope of this disclosure. [Explanation of symbols]

[0144] 1, 2, 2a, 3 Information Processing Devices 11, 216, 314 Judgment section 12, 217, 315 Selection section 21, 31 Arithmetic device 211 Object Tracking Unit 212 Object detection unit 213 Object matching unit 214 Refinement Section 215, 313 Calculation Unit 218, 316 Facial Recognition Section 311 Face Tracking Unit 312 Face Recognition Unit

Claims

1. A determination means for determining whether the degree of confidence when determining the correspondence between a first element and a second element, which are included in time-series data and are acquired at a first time point and a second element acquired at a second time point after the first time point, using the first element as the criterion for correspondence between the two elements, is higher than a predetermined threshold, A selection means that, if the confidence level is determined to be higher than the predetermined threshold, selects the second element as the new criterion for correspondence between the two elements, and if the confidence level is determined to be lower than the predetermined threshold, selects the first element as the criterion for correspondence between the two elements. Equipped with, The aforementioned time-series data is a video containing multiple images, The first element is an object in the first image captured at the first time, among the plurality of images. The second element is an object in the second image captured at the second time, among the plurality of images. The determination means determines whether the degree of confidence when determining the correspondence between an object in the second image and an object in the first image, based on an object in the first image, is higher than the predetermined threshold. If the selection means determines that the confidence level is higher than the predetermined threshold, it selects an object in the second image as a new criterion; if the confidence level is lower than the predetermined threshold, it selects an object in the first image as a criterion. A first generation means generates a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information, based on first position information relating to the position of an object in the first image and second position information relating to the position of an object in the second image. A second generation means generates correspondence information indicating the correspondence between an object in the first image and an object in the second image, based on information obtained by computation using the first and second feature vectors, A calculation means for calculating the degree of confidence when determining the correspondence between an object in the second image and an object in the first image based on the correspondence information, It also has Information processing device.

2. The information processing device includes tracking means for tracking objects in the plurality of images, The aforementioned tracking means is If an object in the first image is selected as a reference by the selection means, the object in the first image is tracked using the first image and a third image taken at a third time point after the second time point from among the plurality of images. If the object in the second image is selected as a new reference by the selection means, the object in the second image is tracked using the second image and the third image. The information processing apparatus according to claim 1.

3. The correspondence information includes first information indicating that an object in the second image corresponds to an object in the first image, and second information indicating that an object in the second image does not correspond to an object in the first image. The calculation means calculates the confidence level based on the first and second information. The information processing apparatus according to claim 1.

4. The calculation means calculates the likelihood ratio as the confidence level, which is the ratio of the probability that the object in the second image corresponds to the object in the first image (as the first piece of information) to the probability that the object in the second image does not correspond to the object in the first image (as the second piece of information). The information processing apparatus according to claim 3.

5. The information processing device includes correction means for correcting the second position information using the corresponding information. The information processing apparatus according to claim 1.

6. The correction means corrects the second position information using an attention mechanism that uses the correspondence information as a weight. The information processing apparatus according to claim 5.

7. Among time-series data which is a video containing multiple images, a first element is an object in a first image captured at a first time, and a second element is an object in a second image captured at a second time after the first time, and it is determined whether the degree of confidence when determining the correspondence between the object in the second image and the object in the first image is higher than a predetermined threshold, using the object in the first image as the criterion for correspondence between the two elements. If the confidence level is determined to be higher than the predetermined threshold, the object in the second image is selected as the new criterion between the two elements. If the confidence level is determined to be lower than the predetermined threshold, the object in the first image is selected as the criterion for correspondence between the two elements. Based on the first position information relating to the position of an object in the first image and the second position information relating to the position of an object in the second image, a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information are generated. The information obtained by the calculation process using the first feature vector and the second feature vector is generated as correspondence information indicating the correspondence between an object in the first image and an object in the second image. Based on the correspondence information, the confidence level is calculated when determining the correspondence between the object in the second image and the object in the first image. Information processing methods.

8. On the computer, In a time-series data that is a video containing multiple images, the system determines whether the confidence level when determining the correspondence between the object in the second image and the object in the first image is higher than a predetermined threshold, using the object in the first image as the criterion for correspondence between the two elements, and considering the first element as a first element, which is an object in the first image captured at a first time, and the second element as an object in the second image captured at a second time after the first time. If the confidence level is determined to be higher than the predetermined threshold, the object in the second image is selected as the new criterion between the two elements. If the confidence level is determined to be lower than the predetermined threshold, the object in the first image is selected as the criterion for correspondence between the two elements. Based on the first position information relating to the position of an object in the first image and the second position information relating to the position of an object in the second image, a first feature vector representing the feature quantities of the first position information and a second feature vector representing the feature quantities of the second position information are generated. The information obtained by the calculation process using the first feature vector and the second feature vector is generated as correspondence information indicating the correspondence between an object in the first image and an object in the second image. Based on the correspondence information, the confidence level is calculated when determining the correspondence between the object in the second image and the object in the first image. A recording medium on which computer programs for executing information processing methods are stored.

Images