Information processing device, information processing method, and computer program

Abstract

An information processing device 1 comprises: an estimation unit 11 that estimates a trajectory of an object appearing in a first frame of a moving image; an appearance determination unit 12 that determines whether or not the object appears in a second frame different from the first frame; and a boarding determination unit 13 that determines whether or not the object appearing in the second frame is on the estimated trajectory.

Description

Information processing device, information processing method, and recording medium

[0001] The present disclosure relates to the technical fields of an information processing device, an information processing method, and a recording medium.

[0002] Patent Literature 1 discloses a technology for predicting the location information of a target object by using an action value function with variables being a "state" including information on multiple candidate locations of the target object at a single time point and "action" including information on which candidate location the target object will move to at that single time point. The technology determines the "action" of the target object at the time of prediction from the "state" at the time of prediction as the target object's location information at the time of prediction. Patent Literature 2 discloses a technology for generating first and second feature vectors (CVt-τ, CVt) based on first position information (PIt-τ) regarding the position of an object (Ot-τ) in a first image (IMt-τ) captured at a first time (t-τ) and second position information (PIt) regarding the position of an object (Ot) in a second image (IMt) captured at a second time (t), and generating information obtained by arithmetic processing using the first and second feature vectors as correspondence information (AM) indicating the correspondence between the object in the first image and the object in the second image. Patent Literature 3 discloses a technology for tracking an object by extracting, based on an image area for one object determined at a time point prior to the time point, multiple image areas in an image at the time point as multiple candidate image areas for the one object, and determining, from among the multiple candidate image areas, an image area for the one object that is considered to be correct at the time point based on evidence including the degree of similarity or overlap between the candidate image area and an image area for another object determined at the previous time point or a past image area for another object that has been lost, and determining image areas for the object or lost objects in each image of a time-series image group that may contain multiple tracking targets. Patent Literature 4 discloses a technology for tracking an object in a video by outputting image features of the object using time-series images of the object, outputting an identification result that identifies the object using the image features and the object tracking result, selecting an appropriate motion model from multiple motion models based on the identification result and the tracking result, tracking the object in time series from the identified object, and calculating a tracking result.

[0003] Japanese Patent Application Laid-Open No. 2017-168029 International Publication No. 2021 / 130951 Japanese Patent Application Laid-Open No. 2018-032078 International Publication No. 2021 / 181612

[0004] An object of this disclosure is to provide an information processing device, an information processing method, and a recording medium that aim to improve upon the techniques described in prior art documents.

[0005] One aspect of the information processing device includes an estimation means for estimating the trajectory of an object appearing in a first frame of a video, an appearance determination means for determining whether the object appears in a second frame different from the first frame, and a boarding determination means for determining whether the object appearing in the second frame is on the trajectory.

[0006] One aspect of the information processing method estimates the trajectory of an object appearing in a first frame of a video, determines whether the object appears in a second frame different from the first frame, and determines whether the object appearing in the second frame is on the trajectory.

[0007] One aspect of the recording medium has recorded thereon a computer program for causing a computer to execute an information processing method that estimates the trajectory of an object appearing in a first frame of a video, determines whether the object appears in a second frame different from the first frame, and determines whether the object appearing in the second frame is on the trajectory.

[0008] FIG. 1 is a block diagram showing the configuration of an information processing device in a first embodiment. FIG. 2 is a block diagram showing the configuration of an information processing device in a second embodiment. FIG. 3 is a conceptual diagram of information processing operation of the information processing device in the second embodiment. FIG. 4 is a flowchart showing the flow of tracking operation of the information processing device in the second embodiment. FIG. 5 is a conceptual diagram of information processing operation of the information processing device in the second embodiment. FIG. 6 is a conceptual diagram of information processing operation of the information processing device in the second embodiment. FIG. 7 is a conceptual diagram of information processing operation of the information processing device in the second embodiment. FIG. 8 is a block diagram showing the configuration of an information processing device in a third embodiment. FIG. 9 is a flowchart showing the flow of tracking operation of the information processing device in the third embodiment. FIG. 10 is a conceptual diagram of information processing operation of the information processing device in the third embodiment. FIG. 11 is a block diagram showing the configuration of a tracking network. FIG. 12 is a conceptual diagram of operation of the tracking network.

[0009] Hereinafter, embodiments of an information processing device, an information processing method, and a recording medium will be described with reference to the drawings. [1: First Embodiment]

[0010] A first embodiment of an information processing device, an information processing method, and a recording medium will be described below. Hereinafter, the first embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 1 to which the first embodiment of the information processing device, the information processing method, and the recording medium is applied. [1-1: Configuration of Information Processing Device 1]

[0011] FIG. 1 is a block diagram showing the configuration of an information processing device 1 in the first embodiment. As shown in FIG. 1, the information processing device 1 includes an estimation unit 11, an appearance determination unit 12, and a boarding determination unit 13. The estimation unit 11 estimates the trajectory of an object appearing in a first frame of a video. The appearance determination unit 12 determines whether or not an object appears in a second frame different from the first frame. The determination of whether or not an object appears in a second frame different from the first frame is called appearance determination. The boarding determination unit 13 determines whether or not an object appearing in the second frame is on the estimated trajectory. The determination of whether or not an object appearing in the second frame is on the estimated trajectory is called boarding determination. [1-2: Technical Effects of the Information Processing Device 1]

[0012] The information processing device 1 in the first embodiment estimates the trajectory of an object, determines whether the object is present, and determines whether the object is on board, and therefore can accurately acquire the transition of the position of an object appearing in frames of a video. [2: Second Embodiment]

[0013] Next, a second embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the second embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 2 to which the second embodiment of the information processing device, the information processing method, and the recording medium is applied. [2-1: Configuration of the information processing device 2]

[0014] 2 is a block diagram showing the configuration of an information processing device 2 in the second embodiment. As shown in FIG. 2, the information processing device 2 includes a calculation device 21 and a storage device 22. The learning device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the learning device 2 does not necessarily have to include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0015] The arithmetic device 21 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic device 21 loads a computer program. For example, the arithmetic device 21 may load a computer program stored in the storage device 22. For example, the arithmetic device 21 may load a computer program stored in a computer-readable, non-transitory storage medium using a storage medium reading device (e.g., the input device 24 described later) not shown in the drawing provided in the learning device 2. The arithmetic device 21 may acquire (i.e., download or load) the computer program from a device (not shown) located outside the learning device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the loaded computer program. As a result, logical functional blocks for executing the operations to be performed by the learning device 2 are realized within the arithmetic device 21. In other words, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) to be performed by the learning device 2.

[0016] FIG. 2 shows an example of logical functional blocks implemented within the computing device 21 to execute information processing operations. As shown in FIG. 3 , the computing device 21 implements a detection unit 211, which is a specific example of a "detection means" described in the appendix below, an acquisition unit 212, and a tracking unit 213, which is a specific example of a "tracking means" described in the appendix below. However, the acquisition unit 212 does not have to be implemented within the computing device 21. The tracking unit 213 includes a trajectory estimation unit 2131, which is a specific example of an "estimation means" described in the appendix below, an appearance determination unit 2132, which is a specific example of a "boarding determination means" described in the appendix below, and a boarding determination unit 2133, which is a specific example of a "boarding determination means" described in the appendix below. Details of the operations of the detection unit 211, the acquisition unit 212, and the tracking unit 213 will be described later with reference to FIGS. 3 to 7 .

[0017] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the information processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0018] The communication device 23 is capable of communicating with devices external to the information processing device 2 via a communication network (not shown). The communication device 23 may be a communication interface based on standards such as Ethernet (registered trademark), Wi-Fi (registered trademark), Bluetooth (registered trademark), or USB (Universal Serial Bus).

[0019] The input device 24 is a device that accepts information input to the information processing device 2 from outside the information processing device 2. For example, the input device 24 may include an operation device (e.g., at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the information processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the information processing device 2.

[0020] The output device 25 is a device that outputs information to the outside of the information processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-2: Information Processing Operation Performed by the Information Processing Device 2]

[0021] 3 to 7, the information processing operation performed by the information processing device 2 will be described. In this embodiment, a person is targeted and a case where the person is tracked will be described. In this embodiment, a video captured by a camera with a fixed angle of view, in which the same space is included in each frame, will be described as an example. [2-2-1: Detection Operation]

[0022] FIG. 3 is a conceptual diagram of the detection operation performed by the information processing device 2. FIG. 3 illustrates an example of a frame of a video. The frame illustrated in FIG. 3 is a specific example of the "first frame" described in the appendix, which will be described later. The frame illustrated in FIG. 3 is called frame 0. FIG. 3 illustrates a case in which two people to be tracked first appear in the video in frame 0. That is, the detection unit 211 detects the two people to be tracked from frame 0. The detection unit 211 may identify the two detected people as person A and person B.

[0023] In FIG. 3A , a circle A0 in the grid pattern illustrates the position of person A in frame 0 (the position in the "first frame" is called the "initial position"), and a triangle B0 in the grid pattern illustrates the initial position of person B in frame 0. The frame illustrated in FIG. 3A also includes an occlusion region O. The occlusion region O is, for example, a region in the space included in the frame that is closer to the camera than the tracked person and contains an object that obscures the tracked person from the frame. Therefore, when the tracked person is in the occlusion region O within the frame, the tracked person will not appear in the frame. For example, the occlusion region O may be a region in which a fixed structure in space is captured. Furthermore, the occlusion region O may be a region in which a person other than the tracked person of interest is captured. Information regarding the occlusion region O may be known information or information estimated from other frames.

[0024] When the detection unit 211 detects a tracking target person, the trajectory estimation unit 2131 estimates the trajectory of the tracking target person. The trajectory estimation unit 2131 may estimate a future trajectory of the tracking target person. The trajectory estimation unit 2131 may estimate a trajectory using only a single frame. Alternatively, the trajectory estimation unit 2131 may estimate a trajectory using two or more frames. In other words, the number of frames required for the trajectory estimation unit 2131 to estimate a trajectory is not specified. The future trajectory includes the position of the tracking target person in a subsequent frame (i.e., a frame captured after the current frame) after the current frame. The subsequent frame is a specific example of a "second frame" described in the appendix below. Alternatively, the trajectory estimation unit 2131 may estimate a transition in the position of the tracking target person in subsequent frames. Alternatively, the detection unit 211 may detect an object whose trajectory is estimated by the trajectory estimation unit 2131. The trajectory estimation unit 2131 may estimate the trajectory of the person to be tracked until the person exits the frame.

[0025] For example, the trajectory estimation unit 2131 may estimate a trajectory using a module that handles time series. For example, the trajectory estimation unit 2131 may estimate a trajectory using a recurrent neural network (RNN) that employs a long short-term memory (LSTM). Alternatively, the trajectory estimation unit 2131 may estimate a trajectory using a model that generates a trajectory using a mechanism different from the RNN. Alternatively, the trajectory estimation unit 2131 may estimate a trajectory based on a predetermined rule using the acceleration of the movement of an object in a video. In the second embodiment, the trajectory estimation unit 2131 may perform estimation using any estimation method.

[0026] FIG. 3(b) shows a trajectory AT of person A estimated by the trajectory estimation unit 2131, and a trajectory BT of person B estimated by the trajectory estimation unit 2131. The white circle in FIG. 3(b) illustrates an example of the position of person A in the trajectory estimation in the subsequent frame. FIG. 3(b) illustrates that person A is hidden by an occlusion region O in the subsequent frame. The white triangle in FIG. 3(b) illustrates an example of the estimated position of person B in the subsequent frame. [2-2-2: Tracking Operation]

[0027] Fig. 4 is a flowchart showing the flow of the tracking operation performed by the information processing device 2. The operations from "start" to "end" shown in Fig. 4 may be operations for each frame. Figs. 5 and 7 are conceptual diagrams of the tracking operation performed by the information processing device 2.

[0028] As shown in FIG. 4 , the acquisition unit 212 acquires frame information (step S20). The frame information includes information indicating the position of a person (or a person-like object) as a target appearing in the corresponding frame of the video. The frame information may also include information indicating a bounding box surrounding the person (or a tracked target such as an object) appearing in the corresponding frame of the video. FIG. 5( a) illustrates frame 2 as a subsequent frame acquired after frame 0. In the example illustrated in FIG. 5( a), the frame information may include the position of person A in frame 2, indicated by a black circle A2, and the position of person B in frame 2, indicated by a black triangle B2.

[0029] The acquisition unit 212 acquires trajectory information (step S21). The trajectory information includes the future position of each tracked person. The future position is the estimated position of the tracked person in a relevant frame of the video and a subsequent frame. The trajectory information may include the past position of each tracked person in addition to the future position. The past position is the position of the tracked person in a previous frame of the video. Furthermore, the trajectory information may include the initial position of each tracked person. FIG. 5(b) illustrates an example of trajectory information. FIG. 5(b) illustrates an example of a case in which the trajectory information includes the past position of person A indicated by a black circle, the future position of person A indicated by a white circle, the past position of person B indicated by a black triangle, and the future position of person B indicated by a white triangle. The black circle A1 illustrates the position of person A in frame 1 (an example of a frame acquired after frame 0 and before frame 2), and the black triangle B1 illustrates the position of person B in frame 1.

[0030] The appearance determination unit 2132 determines whether or not a person to be tracked appears in the frame (step S22). The operation for frame 2 will be described with reference to Fig. 5. Fig. 5(a) illustrates an example in which person A and person B appear in frame 2.

[0031] If the person to be tracked appears in frame 2 (step S22: Yes), the boarding determination unit 2133 makes a determination based on frame 2 and the trajectory information. The boarding determination unit 2133 determines whether the person to be tracked who appears in frame 2 is on the trajectory (step S23). Figure 5 (c) illustrates an example in which, in frame 2, person A is at position A2 on the trajectory AT estimated based on frame 0. The boarding determination unit 2133 determines that person A who appears in frame 2 is on the trajectory AT. The tracking unit 213 tracks the person to be tracked based on whether the person to be tracked follows the estimated trajectory.

[0032] If it is determined that the person to be tracked is on the trajectory estimated based on the previous frame (step S23: Yes), the process returns to step S22 to make a determination about the next person to be tracked. Fig. 5(c) illustrates an example in which, in frame 2, person B is at position B2 on the trajectory BT estimated based on frame 0. The boarding determination unit 2133 determines that person B appearing in frame 2 is on the trajectory BT.

[0033] The tracking unit 213 determines whether or not the processing has been performed for all tracking target people appearing in the frame (step S24). If the processing has been performed for all tracking target people appearing in the frame (step S24: Yes), the tracking unit 213 updates the trajectory information (step S26).

[0034] FIG. 6A illustrates frame 3, which is acquired after frame 2. The operation related to frame 3 will be described with reference to FIG. 6 . FIG. 6A illustrates a case in which person A does not appear in frame 3 because person A is hidden by an occlusion region O. If the occlusion region O and the trajectory overlap, the appearance determination unit 2132 may determine that the tracked person is hidden by the occlusion region O. If the tracked person is not found within a predetermined range of the trajectory, the appearance determination unit 2132 may determine that the target is hidden by the occlusion region O. If the tracked person does not appear in the frame (step S22: No), the tracking unit 213 may estimate that the tracked person is on the trajectory. That is, the tracking unit 213 may estimate that person A is at position A3. In other words, the tracking unit 213 determines that a tracked person who should appear in the frame according to the estimated trajectory does not appear in the frame is hidden by the occlusion region O.

[0035] FIG. 6( b) illustrates an example in which person B appears in frame 3 (step S22: Yes) but is not on the trajectory. That is, FIG. 6( b) illustrates an example in which position B3 of person B in frame 3 is not on the trajectory BT estimated based on a frame prior to frame 3. If the boarding determination unit 2133 determines that the tracked person is not on the trajectory estimated based on the previous frame (step S23: No), the tracking unit 213 updates the current position of the tracked person (step S25). In the example illustrated in FIG. 6( b), the tracking unit 213 updates the position of person B in frame 3 included in the trajectory information to position B3.

[0036] If processing has been performed for all tracked persons appearing in frame 3 (step S24: Yes), the tracking unit 213 updates the trajectory information to include the past positions (frames 1 to 3) of person A indicated by black circles and the future positions of person A indicated by white circles, as well as the past positions (frames 1 to 3) of person B indicated by black triangles and the future positions of person B indicated by white triangles, as shown in FIG. 5(c) (step S26). In step S26, the trajectory estimation unit 2131 may update the trajectory based on the updated current position. For example, the trajectory estimation unit 2131 may update the trajectory BT illustrated in FIG. 5(b) to the trajectory BT' illustrated in FIG. 5(c).

[0037] The operation of step S25 may be performed in step S26. Alternatively, the trajectory information may be updated for each target. That is, the trajectory information may be updated at a timing other than that shown in the flowchart of Fig. 4. The flow of the operation shown in the flowchart of Fig. 4 may be understood as one example.

[0038] FIG. 7( a) illustrates frame 6, which is acquired after frame 3. FIG. 7( a) illustrates an example in which person A and person B are in the same position in frame 6. In other words, FIG. 7( a) illustrates a case in which one of person A and person B obscures the other. In this case, the appearance determination unit 2132 may determine that only one of person A and person B appears in frame 6. In other words, the appearance determination unit 2132 may determine that one of person A and person B obscures the other in frame 6, and that the other of person A and person B does not appear in frame 6. In this case, the boarding determination unit 2133 may determine that one of person A and person B is boarding the trajectory. In addition, in this case, the appearance determination unit 2132 may determine that the other of person A and person B is on the trajectory.

[0039] FIG. 7B illustrates frame 7, which is acquired after frame 6. The tracking unit 213 performs tracking based on the trajectory estimated based on the previous frame, the appearance determination result by the appearance determination unit 2132, and the boarding determination result by the boarding determination unit 2133. Therefore, in frame 7, the tracking unit 213 can find person A at position A7 on the trajectory AT. Also, in frame 7, the tracking unit 213 can find person B at position B7 on the trajectory BT. That is, even if person A and person B were in the same position in the previous frame, the tracking unit 213 will not confuse person A and person B. In this embodiment, the trajectories of each tracked person are independent, so even if the trajectories intersect, the tracked people will not be swapped. [2-3: Technical Effects of the Information Processing Device 2]

[0040] For example, in the comparative example, which tracks objects by comparing and matching objects that appear in two frames, as the number of objects increases, the processing becomes more complex, and errors such as misidentification of objects become more likely to occur. Also, in the comparative example, once a misidentification is made, the misidentification may continue to remain. In other words, the comparative example often results in a decrease in tracking accuracy.

[0041] In contrast, the information processing device 2 according to the second embodiment estimates a trajectory, performs boarding determination, and tracks based on the determination result of the boarding determination, thereby preventing the misidentification of tracking target persons. The information processing device 2 predicts the path of each target, so it can accurately track multiple targets simultaneously. Furthermore, the information processing device 2 updates the trajectory based on the determination result of the boarding determination, so it can track with high accuracy. Furthermore, the information processing device 2 detects a new tracking target person that appears in the frame, so it can accurately track each tracking target person that appears in the frame.

[0042] Furthermore, since the information processing device 2 performs appearance determination, it can handle cases where the tracking target person is hidden in an occlusion region O. Even when the tracking target person does not appear in the frame due to the occlusion region O, tracking can be continued and tracking can be prevented from being interrupted. In other words, since the information processing device 2 estimates a trajectory, robust tracking can be performed even when the target does not appear in the frame. [3: Third Embodiment]

[0043] Next, a third embodiment of the information processing device, the information processing method, and the recording medium will be described. Hereinafter, the third embodiment of the information processing device, the information processing method, and the recording medium will be described using an information processing device 3 to which the third embodiment of the information processing device, the information processing method, and the recording medium is applied. [3-1: Configuration of the information processing device 3]

[0044] The configuration of the information processing device 3 in the third embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the information processing device 3 in the third embodiment.

[0045] As shown in FIG. 8 , the information processing device 3 in the third embodiment includes a calculation device 21 and a storage device 22, similar to the information processing device 3 in the second embodiment. Furthermore, the information processing device 3 in the third embodiment may include a communication device 23, an input device 24, and an output device 25, similar to the information processing device 2 in the second embodiment. However, the information processing device 3 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The information processing device 3 in the third embodiment differs from the information processing device 2 in the second embodiment in that the detection unit 311 includes a position extraction unit 3111 and a matching unit 3112. Other features of the information processing device 3 may be the same as other features of the information processing device 2. Therefore, hereinafter, differences from the embodiments already described will be described in detail, and descriptions of other overlapping parts will be omitted as appropriate. [3-2: Information Processing Operation Performed by Information Processing Device 3]

[0046] Fig. 9 is a flowchart showing the flow of information processing operations performed by the information processing device 3. The operations from "Start" to "End" shown in Fig. 9 may be operations for each frame. Fig. 10 is a conceptual diagram of information processing operations performed by the information processing device 3. [3-2-1: Detection operations performed by the detection unit 311]

[0047] When a frame is input, the position extraction unit 3111 outputs frame information. When a frame is input, the position extraction unit 3111 outputs information indicating the position of a person (or a person-like object) appearing in the frame. The position extraction unit 3111 may also output information indicating a bounding box surrounding the person (or a person-like object) appearing in the frame.

[0048] As shown in Fig. 9 , in step S20 of the third embodiment, frame information output by the position extraction unit 3111 is acquired. Fig. 10 illustrates frame 7, which is acquired after frame 6 illustrated in Fig. 7A. In the example illustrated in Fig. 10 , the frame information may include the position in frame 7 of the person indicated by the black circle A7 and the position in frame 7 of the person indicated by the square C7 in the grid pattern.

[0049] 10 illustrates a case where person B is hidden in occlusion area O2 in frame 7. Also, FIG. 10 illustrates a case where a tracking target person appears in frame 7, and does not correspond to either trajectory AT or trajectory BT. The operation relating to frame 7 will be described below with reference to FIG. 10.

[0050] The acquisition unit 312 acquires trajectory information (step S21). The trajectory information may include, for example, information on the positions of person A and person B from frame 0 to frame 6, as illustrated in Fig. 7A, and information on their estimated positions from frame 7 onwards.

[0051] The matching unit 3112 determines whether or not there is a trajectory corresponding to the person included in the frame information, based on the position of the person included in the frame information and the position of the tracked person included in the trajectory information (step S30). The matching unit 3112 may determine whether or not a new tracked person is included in the frame information, based on the overlap rate between the bounding box included in the frame information and the bounding box included in the trajectory information. In other words, the matching unit 3112 may determine whether or not the person included in the frame information and the tracked person included in the trajectory information are the same person, based on whether or not the position of the person included in the frame information and the position of the tracked person included in the trajectory information are closer than a predetermined distance. The operations from step S30 to step S34 are performed for each person appearing in the frame.

[0052] The matching unit 3112 determines the person indicated by the square C7 in the grid pattern. The position of the person indicated by the square C7 in the grid pattern does not correspond to either the trajectory AT or the trajectory BT. If there is no corresponding trajectory (step S30: No), the matching unit 3112 determines the person indicated by the square C7 in the grid pattern as a new person to be tracked, and the trajectory estimation unit 3131 estimates the trajectory of the new person to be tracked (step S31). That is, the detection unit 311 detects the new person to be tracked based on the trajectory estimated based on a frame prior to frame 7. In this case, the frame prior to frame 7 is a specific example of the "third frame" described in the appendix below. [3-2-2: Tracking Operation Performed by the Tracking Unit 313]

[0053] The matching unit 3112 then determines the person indicated by the black circle A7. Since the position of the person indicated by the black circle A7 corresponds to the trajectory AT (step S30: Yes), the matching unit 3112 determines that the person indicated by the black circle A7 is not a new person to be tracked, and the tracking unit 313 determines that the person indicated by the black circle A7 is person A to be tracked.

[0054] The boarding determination unit 3133 determines whether or not person A, the person to be tracked who appears in frame 7, is on the trajectory AT (step S32). In other words, the boarding determination unit 3133 determines whether or not the position of the person indicated by the black circle A7 is on the trajectory AT.

[0055] In frame 7, the person A to be tracked is riding on the trajectory AT (step S32: Yes), so the tracking unit 313 does not update the current location. On the other hand, if the person A to be tracked is not riding on the trajectory AT (step S32: No), the tracking unit 313 updates the current location (step S33), similar to step S25 described above. That is, if the person to be tracked is riding on the trajectory, the tracking unit 313 tracks the person along the trajectory.

[0056] Furthermore, the boarding determination unit 3133 may compare a group of detected positions (bounding boxes) included in the trajectory with the bounding box detected in a specific frame by the detection unit 311. The boarding determination unit 3133 may determine that the detected bounding boxes are the same person if they are located near the trajectory. Furthermore, the boarding determination unit 3133 may determine that the detected bounding boxes are different people if they are not located near the trajectory. If the boarding determination unit 3133 determines that the detected bounding boxes are the same person, the tracking unit 313 may merge the corresponding trajectories.

[0057] The matching unit 3112 determines whether or not the processing has been performed for all people appearing in the frame (step S34). Note that, since the fact that a person is the target of the operations from step S30 to step S34 means that the position extraction unit 3111 has detected the person as appearing in the frame, the tracking unit 313 may reset "K" of the person to be tracked who has been the target of the operations from step S30 to step S34 to "0."

[0058] If processing has been performed for all people appearing in the frame (step S34: Yes), the tracking unit 313 determines whether tracking operations have been performed for all people to be tracked included in the trajectory information (step S35). In the example shown in Figure 10, position B7 is hidden by occlusion area O2, and the position extraction unit 3111 has not detected the person indicated by the black triangle B7. Therefore, the tracking unit 313 has not performed tracking operations for person B to be tracked included in the tracking information.

[0059] For tracking target person B for whom tracking operation is not being performed, i.e., who does not appear in the frame, the appearance determination unit 3132 adds 1 to "K" (step S36). "K" is a variable representing the number of times the tracking target person does not appear consecutively in the frame. If "K" for the tracking target person exceeds an upper limit, the tracking unit 313 may determine that the tracking target person is not a tracking target and terminate tracking of the tracking target person. The upper limit of "K" may be set based on an estimated time until the tracking target person exits the frame. In other words, if "K" exceeds the upper limit, the tracking unit 313 may determine that the tracking target person has exited the frame. Furthermore, the appearance determination unit 3132 determines that person B is at position B7 on the trajectory BT in frame 7. The processes of steps S35 to S36 are performed for each tracking target included in the tracking information.

[0060] The flow of a series of information processing operations shown in the flowchart of FIG. 9 is one example. The order of the above-described processes, such as the boarding determination operation and the appearance determination operation, may be reversed. Furthermore, the information processing device 3 may execute some of the processes in parallel. The information processing device 3 may perform each of the above-described operations from the time a frame is input until the time updated trajectory information is output.

[0061] If the tracking operation has been performed for all of the tracking target persons included in the trajectory information (step S35: Yes), the tracking unit 313 updates the trajectory information (step S26). In step S26, the tracking unit 313 may exclude from the tracking targets any tracking target person whose "K" exceeds the upper limit value. The tracking unit 313 may also exclude from the tracking targets any tracking target person whose estimated future position does not appear in the frame. For example, in the example shown in FIG. 10, the future position of tracking target person A (i.e., the white circle) is not included in the frame. In this case, the tracking unit 313 may exclude tracking target person A from the tracking targets. [3-3: Tracking Network TN]

[0062] The information processing device 3 may realize the information processing operation using a tracking network TN. The tracking network TN may be a mechanism built based on deep learning.

[0063] Fig. 11 is a block diagram showing the configuration of the tracking network TN. As shown in Fig. 11, the tracking network TN is a mechanism that outputs trajectory information when a frame is input. The tracking network TN also receives trajectory information output in accordance with the previous frame. The tracking network TN includes a detection mechanism DM and a tracking mechanism TM.

[0064] The detection mechanism DM is a network that detects a new tracking target person that appears in a video. The detection mechanism DM may include a mechanism such as a base network that extracts features from an image, a detection mechanism that detects a target using the features, and a selection mechanism that selects a new tracking target person from the detection results. The detection mechanism DM may be a mechanism that uses a convolutional neural network. The detection unit 311 may perform detection operations using the detection mechanism DM.

[0065] The tracking mechanism TM is a module that tracks the person being tracked. The tracking mechanism TM may include a mechanism that extracts features from time-series information. The tracking mechanism TM may include a mechanism that uses an RNN that employs an LSTM. The tracking unit 313 may use the tracking mechanism TM to perform tracking operations. By employing an LSTM, the tracking mechanism TM can handle information over a relatively long period of time. The tracking mechanism TM may estimate a trajectory, for example, 100 to 500 frames ahead. A tracking mechanism TM that employs an LSTM has a loop function, so that the state when processing a current frame can be inherited as the state when processing a subsequent frame. This allows the tracking mechanism TM to handle time-series information. A quantity that indicates the characteristics of the state when the tracking network TN processes a frame is called an internal feature.

[0066] FIG. 12 is a conceptual diagram of information processing operations realized by the tracking network TN. When frame 0 is input, the tracking network TN outputs trajectory information 0. The tracking network TN also outputs internal features. Frame 1, trajectory information 0, and internal features are input to the tracking network TN. That is, the tracking network TN uses the output of the action prior to the action at which the frame is input as information related to the action at which the frame is input. This allows the tracking network TN to reflect the influence of previously input information when the action at which the frame is input is performed. When frame 1, trajectory information 0, and internal features are input, the tracking network TN outputs trajectory information 1. The trajectory information 1 may be understood as information obtained by updating the trajectory information 0.

[0067] Similarly, when frame 2, trajectory information 1, and internal feature values ​​are input, the tracking network TN outputs trajectory information 2. When frame 3, trajectory information 2, and internal feature values ​​are input, the tracking network TN outputs trajectory information 3. The tracking network TN repeats the same operation. [3-4: Technical Effects of the Information Processing Device 3]

[0068] The information processing device 3 according to the third embodiment uses the detection mechanism at the first stage only to detect a newly appearing tracking target. Furthermore, the information processing device 3 tracks the detected tracking target using only the tracking mechanism at the second stage. In other words, the information processing device 3 is designed to perform tracking operations without relying on the detection mechanism, and therefore can suppress tracking interruptions even when the detection mechanism cannot take time series into account. Furthermore, the information processing device 3 estimates the trajectory of the tracking target person using a mechanism that extracts features from time series information, and can therefore estimate a trajectory over a relatively long period of time. The information processing device 3 can also handle relatively long tracking interruptions, such as occlusion. The information processing device 3 can continue tracking to the end point of the estimated trajectory, enabling long-distance tracking. Meanwhile, the information processing device 3 excludes tracking target people who have left the frame from tracking, thereby reducing unnecessary processing. Furthermore, the information processing device 3 excludes tracking target people who have not appeared in the frame for a long period of time from tracking, thereby reducing unnecessary processing.

[0069] The information processing device 3 according to this embodiment can be applied to general applications that perform multi-object tracking. For example, the information processing device 3 can be applied to gateless authentication systems, security systems, systems that use motion recognition, systems that track specific objects, etc. Examples of systems that use motion recognition include business monitoring systems and expert behavior analysis systems. Examples of specific objects include people, faces, hands, automobiles, tools, and foreign objects. [4: Supplementary Note]

[0070] The following supplementary notes are further disclosed with respect to the above-described embodiments. [Supplementary Note 1] An information processing device comprising: estimation means for estimating a trajectory of an object appearing in a first frame of a video; appearance determination means for determining whether the object appears in a second frame different from the first frame; and boarding determination means for determining whether the object appearing in the second frame is on the trajectory. [Supplementary Note 2] The information processing device according to Supplementary Note 1, further comprising detection means for detecting the object whose trajectory is estimated by the estimation means from the first frame. [Supplementary Note 3] The information processing device according to Supplementary Note 1 or 2, wherein the estimation means updates the trajectory based on at least one of a boarding determination result by the boarding determination means and a appearance determination result by the appearance determination means. [Supplementary Note 4] The information processing device according to Supplementary Note 1 or 2, further comprising tracking means for tracking the object based on the trajectory, the boarding determination result by the boarding determination means, and the appearance determination result by the appearance determination means. [Supplementary Note 5] The information processing device according to Supplementary Note 2, wherein the detection means detects the object using the trajectory estimated based on a third frame different from the first frame. [Supplementary Note 6] The information processing device according to Supplementary Note 4, wherein the tracking means tracks the object using a mechanism for extracting features from time-series information. [Supplementary Note 7] The information processing device according to Supplementary Note 4, wherein the estimation means estimates the trajectory of the object within the frames of the video, and the tracking means tracks the object within the frames of the video. [Supplementary Note 8] The information processing device according to Supplementary Note 4, wherein the appearance determination means determines that the object has not appeared over a predetermined number of frames or more, and the tracking means terminates tracking of the object. [Supplementary Note 9] An information processing method comprising: estimating a trajectory of an object appearing in a first frame of a video; determining whether the object appears in a second frame different from the first frame; and determining whether the object appearing in the second frame is on the trajectory.[Supplementary Note 10] A recording medium having recorded thereon a computer program for causing a computer to execute an information processing method of: estimating a trajectory of an object appearing in a first frame of a video; determining whether or not the object appears in a second frame different from the first frame; and determining whether or not the object appearing in the second frame is on the trajectory.

[0071] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Information processing devices, information processing methods, and recording media that involve such modifications are also included in the technical idea of ​​this disclosure.

[0072] 1, 2, 3 Information processing device 11 Estimation unit 12, 2132, 3132 Appearance determination unit 13, 2133, 3133 Boarding determination unit 211, 311 Detection unit 3111 Position extraction unit 3112 Matching unit 212, 312 Acquisition unit 213, 313 Tracking unit 2131, 3131 Trajectory estimation unit TN Tracking network DM Detection mechanism TM Tracking mechanism

Claims

1. an estimation means for estimating a trajectory of an object appearing in a first frame of a video; an appearance determination means for determining whether the object appears in a second frame different from the first frame; a boarding determination means for determining whether or not the object appearing in the second frame is on the trajectory; An information processing device comprising:

2. a detection means for detecting the object whose trajectory is estimated by the estimation means from the first frame; The information processing device according to claim 1 .

3. The estimation means updates the trajectory based on at least one of a boarding determination result by the boarding determination means and an appearance determination result by the appearance determination means.

3. The information processing device according to claim 1.

4. The vehicle further includes a tracking means for tracking the target based on the trajectory, a boarding determination result by the boarding determination means, and an appearance determination result by the appearance determination means.

3. The information processing device according to claim 1.

5. The detection means detects the object using the trajectory estimated based on a third frame different from the first frame. The information processing device according to claim 2 .

6. The tracking means tracks the object using a mechanism for extracting features from time-series information. The information processing device according to claim 4 .

7. the estimation means estimates the trajectory of the object within frames of the video; The tracking means tracks the object within the frames of the video. The information processing device according to claim 4 .

8. When the appearance determination means determines that the object has not appeared over a predetermined number of frames or more, the tracking means terminates tracking of the object. The information processing device according to claim 4 .

9. Estimating the trajectory of an object appearing in the first frame of the video; determining whether the object appears in a second frame different from the first frame; Determine whether the object appearing in the second frame is on the trajectory. A computer-implemented information processing method.

10. On the computer, Estimating the trajectory of an object appearing in the first frame of the video; determining whether the object appears in a second frame different from the first frame; Determine whether the object appearing in the second frame is on the trajectory. A computer program for executing an information processing method.