Image processing system and image processing method

Abstract

The present application provides an image processing system and an image processing method capable of improving the accuracy of the cause detection of tracking failure. In the image processing system of the present application, a plurality of frames in time series are acquired by an acquisition unit. A part detection unit initially detects a tracking target part from one frame. A forward tracking unit tracks the tracking target part in a time forward direction. A part estimation unit estimates a part in which the tracking target part is predicted to exist (an estimated part) in a frame in which tracking of the tracking target part has failed. A reverse tracking unit tracks the estimated part in a time reverse direction. A determination unit determines that the tracking of the tracking target part has failed due to an occurrence of an occlusion when the position of the estimated part and the position of the tracking target part are not consistent in a frame in which tracking of the estimated part has succeeded.

Description

Technical Field

[0001] This invention relates to an image processing system and an image processing method. Background Technology

[0002] Patent Document 1 discloses a system for anonymizing target objects within a video. The system in Patent Document 1 tracks the anonymized target object not only in a forward temporal direction within the video image, but also in a reverse temporal direction, thereby improving the tracking performance of the target object.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2021-064203 Summary of the Invention

[0004] Tracking objects within a video often fails. The inventors believe that the failure to track target objects within a video is not only due to "detection errors" but also to "occlusion." Here, occlusion refers to the phenomenon where a preceding object obscures a following object, making the following object invisible.

[0005] In the system of Patent Document 1, the possibility of occlusion as a cause of tracking failure is not considered. Therefore, the accuracy of detecting the cause of tracking failure in the system of Patent Document 1 is insufficient, and there is room for improvement in this accuracy.

[0006] The purpose of this invention is to provide an image processing system and image processing method that can improve the accuracy of detecting the cause of tracking failure, in view of this problem.

[0007] The image processing system of the present invention includes: an acquisition unit that acquires a plurality of frames in a time series; a part detection unit that initially detects and tracks a target part from one of the frames in the time series; a forward tracking unit that tracks the target part in a forward temporal direction for one or more frames after the frame in which the target part was initially detected; a part estimation unit that, in the case of tracking failure of the target part, estimates and predicts the presence of a part of the target part as an estimated part in the frame where tracking failure occurred; a reverse tracking unit that tracks the estimated part in a reverse temporal direction for one or more frames before the frame in the time series where tracking failure occurred; and a determination unit that, in the case of successful tracking of the estimated part, determines that the tracking failure of the target part was due to occlusion if the position of the estimated part in the frame where tracking success occurred is inconsistent with the position of the target part.

[0008] In another aspect of the image processing system of the present invention, if the tracking of the estimated part fails, or if the tracking of the estimated part is successful and the position of the estimated part matches the position of the tracking target part in the frame where the tracking of the estimated part is successful, the determination unit determines that the tracking failure of the tracking target part is due to a detection error.

[0009] The image processing system of the present invention, through the above-described structure, can improve the accuracy of detecting the cause of tracking failure.

[0010] Another aspect of the image processing system according to the present invention further includes an anonymization processing unit, which, when it is determined that the tracking failure of the tracking target part is due to occlusion, does not perform anonymization processing on the tracking target part in the frame where the tracking of the tracking target part fails.

[0011] The image processing system of the present invention, through the above-described structure, can improve the anonymization accuracy of objects.

[0012] In the image processing method of the present invention, a computer performs the following steps: acquiring multiple frames of a time series; initially detecting and tracking a target part from one of the multiple frames of the time series; tracking the target part in a forward temporal direction in one or more frames after the frame in which the target part was initially detected; if the tracking of the target part fails, presuming the presence of a part of the target part as a presumed part in the frame where the tracking of the target part failed; tracking the presumed part in a reverse temporal direction in one or more frames before the frame in which the tracking of the target part failed; and if the tracking of the presumed part is successful, and if the position of the presumed part in the frame where the tracking of the target part is successful is inconsistent with the position of the target part, determining that the tracking failure of the target part is due to occlusion.

[0013] The image processing method of the present invention, through the above structure, can improve the accuracy of detecting the cause of tracking failure.

[0014] Invention Effects

[0015] This invention provides an image processing system and method that can improve the accuracy of detecting the cause of tracking failure. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating an example of the structure of the image processing system according to the first embodiment.

[0017] Figure 2 This diagram illustrates an example where the tracking failure of the target region in the image processing system according to the first embodiment is determined to be due to occlusion.

[0018] Figure 3 This figure shows an example where the tracking failure of the target part in the image processing system according to the first embodiment is determined to be due to a detection error.

[0019] Figure 4 This diagram illustrates an example of anonymization processing in a frame where the tracking of a target region in the image processing system according to the first embodiment is determined to have failed due to occlusion.

[0020] Figure 5 This is a diagram illustrating an example of anonymization processing in a frame where the tracking failure of the target region in the image processing system according to the first embodiment is determined to be due to a detection error.

[0021] Figure 6 This is a flowchart illustrating an example of the operation of the image processing system according to the first embodiment. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same or corresponding elements are labeled with the same symbols, and repeated descriptions are omitted as needed for clarity.

[0023] (Specific description of the research topic)

[0024] First, the subject matter of this invention will be specifically described. When applying AI-based image analysis technology to real-world scenarios, to protect the privacy of the subjects and objects being photographed, measures such as blacking out or blurring objects that reflect personal information are necessary. To perform these processing steps, accurate detection of the object's position and edges is required. In situations where temporal information, such as in video, can be utilized, detection methods combined with tracking are effective. It is believed that the sudden failure of tracking when an object moves outside the field of view is caused by the following two reasons.

[0025] (1) Detection of errors

[0026] (2) The occurrence of occlusion

[0027] Although the accuracy of object tracking has improved year by year, for target objects such as people whose shape or posture changes significantly, tracking failures can still occur due to "(1) detection errors" caused by insufficient accuracy of tracking technology. However, for temporarily generated detection errors, countermeasures such as interpolation processing based on intermediate frames between preceding and following frames or temporary changes to the detection threshold can be applied.

[0028] On the other hand, when tracking fails due to "(2) Occlusion," data needs to be handled carefully. Without anonymizing objects that are not occluded, privacy cannot be protected. However, if objects that are not occluded are excessively blacked out, the information needed for analysis will be lost from the video. Therefore, it is necessary to correctly determine the cause of the tracking failure and take countermeasures.

[0029] However, in the background art such as Patent Document 1, the possibility that the tracking failure is caused by "(2) occlusion" is not considered. Therefore, in the background art such as Patent Document 1, the accuracy of the detection of the cause of tracking failure is insufficient, and there is room for improvement.

[0030] The object of this invention is to provide an image processing system and method that can improve the accuracy of detecting the cause of tracking failure, in view of this problem. Furthermore, the object of this invention is to provide an image processing system and method that improves the accuracy of object anonymization by improving the accuracy of detecting the cause of tracking failure.

[0031] (First Embodiment)

[0032] use Figure 1 The structure of the image processing system 10 according to the first embodiment will be described.

[0033] Figure 1 This is a block diagram illustrating an example of the structure of the image processing system 10 according to the first embodiment.

[0034] like Figure 1 As shown, the image processing system 10 is implemented by one or more computers. The image processing system 10 can be implemented as an image processing device.

[0035] The image processing system 10 includes an acquisition unit 11, a location detection unit 12, a forward tracking unit 13, a location estimation unit 14, a reverse tracking unit 15, a determination unit 16, and an anonymization processing unit 17.

[0036] The acquisition unit 11 acquires multiple frames (images) of the video, i.e., the time series.

[0037] The part detection unit 12 initially detects the target part to be tracked from one frame out of multiple frames in the time series. The target part is detected on a pixel-by-pixel basis. Furthermore, when there are multiple target parts to be tracked, each target part is processed as follows.

[0038] The forward tracking unit 13 tracks the target part in a forward temporal direction for one or more frames after the frame in which the target part is initially detected in a multi-frame time series. Here, "forward temporal direction" means "in sequence according to the passage of time".

[0039] Then, the forward tracking unit 13 determines whether the tracking of the target part is successful. Specifically, if the forward tracking unit 13 continuously detects the target part from frame T-5 to frame T-1, but suddenly fails to detect the target part in frame T, it determines that the tracking of the target part in frame T has failed. Furthermore, if the forward tracking unit 13 detects the target part in frames T-1 and T+1, but fails to detect it in frame T, it can determine that the tracking of the target part in frame T has failed. Moreover, if the forward tracking unit 13 is tracking a human part, and the position of the target part in frame T moves from the position of the target part in frame T-1 at a speed that is difficult to consider as the movement speed of a normal person, it can determine that the tracking of the target part in frame T has failed.

[0040] When tracking of a target part fails, the part estimation unit 14 estimates and predicts the presence of the target part as an estimated part in the frame where tracking of the target part failed. Specifically, the part estimation unit 14 uses information from the frames before and after the frame where tracking of the target part failed to estimate the estimated part. At this time, the part estimation unit 14 can calculate the velocity or angular velocity of the target part and use the calculation result to estimate the estimated part. When the target part is a human part, the part estimation unit 14 can calculate the passable area of ​​a person from the frame and use the calculation result to estimate the estimated part.

[0041] The reverse tracking unit 15 tracks the estimated part in reverse order of time in one or more frames before the frame in which the tracking of the target part in multiple frames of the time series fails. Here, "reverse order of time" means "in reverse order of time".

[0042] If the tracking of the estimated part is successful, and the position of the estimated part is inconsistent with the position of the target part in the frame where the tracking of the estimated part is successful, the determination unit 16 determines that the tracking of the target part failed due to "occlusion". Furthermore, the degree of "inconsistency" between the position of the estimated part and the position of the target part can be set separately. For example, the "inconsistency" between the position of the estimated part and the position of the target part may also include cases where the position of the estimated part and the position of the target part are partially consistent.

[0043] Furthermore, if the tracking of the estimated part fails, or if the tracking of the estimated part is successful and the position of the estimated part matches the position of the target part in the frame where the tracking of the estimated part is successful, the determination unit 16 determines that the tracking failure of the target part is due to "a detection error has occurred".

[0044] If the tracking failure of the target part is determined to be due to occlusion, the anonymization processing unit 17 does not perform anonymization processing on the target part in the frame where the tracking failure occurred.

[0045] If the tracking failure of the target area is determined to be due to a detection error, the anonymization processing unit 17 interpolates the frame in which the tracking of the target area failed. Furthermore, the anonymization processing unit 17 performs anonymization processing on the target area in the interpolated frame.

[0046] In addition, if the tracking of the target part is successful, the anonymization processing unit 17 performs anonymization processing on the region corresponding to the tracking target part in the frame in which the tracking of the target part is successful.

[0047] Next, use Figure 2 and Figure 3 An example of detecting the cause of tracking failure of the target part in the image processing system 10 according to the first embodiment will be described.

[0048] Figure 2 This is a diagram illustrating an example where the tracking failure of the target region in the image processing system 10 according to the first embodiment is determined to be due to occlusion.

[0049] like Figure 2 As shown in step S11, the part detection unit 12 of the image processing system 10 initially detects and tracks the target part A in the frame at time T-2. The target part A is a pixel corresponding to the left foot of a person. Here, the forward tracking unit 13 fails to track the target part A in the frame at time T after the frame at time T-2. Thus, the part estimation unit 14 estimates and predicts the existence of the target part A in the frame at time T as the estimated part B.

[0050] Next, as Figure 2As shown in step S12, the reverse tracking unit 15 of the image processing system 10 tracks the estimated region B in reverse order of time, specifically in the order of the frame before time T (time T-1) and the frame before time T-2. In this example, the reverse tracking of the estimated region B is successful in the frame at time T-1. However, the position of the estimated region B in the frame at time T-1 is inconsistent with the position of the tracking target region A. In this case, the determination unit 16 determines that the tracking failure of the tracking target region A in the frame at time T is due to occlusion. This is because it is assumed that the reverse tracking is successful even if the failure of forward tracking is not due to a detection error. Furthermore, the estimated region B is a pixel of the object (chair) that occludes the tracking target region A, and is a different object from the tracking target region A. Therefore, it is assumed that the estimated region B and the tracking target region A are tracked separately.

[0051] Figure 3 This diagram illustrates an example where the tracking failure of the target region A in the image processing system 10 according to the first embodiment is determined to be due to a detection error.

[0052] like Figure 3 As shown in step S21, the part detection unit 12 of the image processing system 10 initially detects and tracks the target part A in the frame at time T-2. Here, the forward tracking unit 13 fails to track the target part A in the frame at time T after the frame at time T. Thus, the part estimation unit 14 estimates and predicts the existence of a part of the target part A in the frame at time T as an estimated part B.

[0053] Next, as Figure 3 As shown in step S22, the reverse tracking unit 15 of the image processing system 10 tracks the estimated part B in reverse order across multiple frames of the time series, following the sequence of frames before time T (e.g., frames at time T-1 and time T-2). In this example, the tracking of the estimated part B fails in the frame at time T-1. In this case, the determination unit 16 determines that the tracking failure of the target part A in the frame at time T is due to a detection error. This is because it is assumed that in most cases, if forward tracking fails due to a detection error, reverse tracking will also fail.

[0054] Furthermore, assuming that the tracking of the presumed part B is successful in the frame at time T-1, if the position of the presumed part B is the same as the position of the tracked target part A, the determination unit 16 determines that the tracking failure of the tracked target part A is due to a detection error. This is because, if it is presumed that the tracking failure of the tracked target part A is due to a detection error, the tracked target part A and the presumed part B become the same object, and the presumed part B and the tracked target part A are tracked in the same way.

[0055] Next, use Figure 4 and Figure 5 The anonymization method of the image processing system 10 according to the first embodiment will be described in detail.

[0056] Figure 4 This diagram illustrates an example of anonymization processing in a frame where the tracking of a target region in the image processing system 10 according to the first embodiment fails due to occlusion. (See diagram for example.) Figure 4 As shown, the anonymization processing unit 17 of the image processing system 10 does not perform anonymization processing on the tracking target part in frames where the tracking failure is determined to be due to occlusion. However, the anonymization processing unit 17 performs anonymization processing such as blacking on the tracking target parts that are not occluded (i.e., the tracking target parts that are successfully tracked).

[0057] Figure 5 This diagram illustrates an example of anonymization processing in a frame where the tracking failure of the target region in the image processing system 10 according to the first embodiment is determined to be due to a detection error. For example... Figure 5 As shown, the anonymization processing unit 17 of the image processing system 10 interpolates frames in which the tracking failure of the target area is determined to be due to a detection error. For example, the anonymization processing unit 17 infers the position / pose of the target based on information from the preceding and following frames and reflects it in the original frame. Then, the anonymization processing unit 17 performs anonymization processing of the target area in the interpolated frame.

[0058] use Figure 5 The operation of the image processing system 10 according to the first embodiment will be described.

[0059] Figure 5 This is a block diagram illustrating an example of the operation of the image processing system 10 according to the first embodiment.

[0060] First, in step S101, the acquisition unit 11 of the image processing system 10 acquires multiple frames of the time series.

[0061] Next, in step S102, the part detection unit 12 initially detects and tracks the target part in one of the multiple frames of the time series.

[0062] Next, in step S103, the forward tracking unit 13 tracks the target part in a forward time direction in one or more frames after the frame in which the target part is initially detected in multiple frames of the time series.

[0063] Next, in step S104, the forward tracking unit 13 determines whether the tracking of the target part has failed.

[0064] If the tracking of the target area is determined to be successful (No in step S104), the process proceeds to step S105. In step S105, the anonymization processing unit 17 performs anonymization processing on the region corresponding to the tracked target area in the frame where forward tracking is successful. Then, the process proceeds to step S105. In step S105, the anonymization processing unit 17 performs anonymization processing on the tracked target area in the frame where forward tracking is successful. Then, the process proceeds to step S114.

[0065] On the other hand, if it is determined that the tracking of the target part has failed (as in step S104), the process proceeds to step S106. In step S106, if the tracking of the target part fails, the part estimation unit 14 estimates and predicts the existence of a part (estimated part) B of the target part in the frame where the tracking of the target part has failed.

[0066] Next, in step S107, the reverse tracking unit 15 tracks the estimated part in reverse time in one or more frames before the frame in which the tracking of the target part in the time series fails.

[0067] Next, in step S108, the determination unit 16 determines whether the tracking of the estimated part is successful. If the tracking of the estimated part is successful (yes in step S108), the process proceeds to step S109. In step S109, the determination unit 16 determines whether the position of the estimated part in the frame where the tracking of the estimated part is successful is consistent with the position of the tracking target part. If the position of the estimated part in the frame where the tracking of the estimated part is successful is inconsistent with the position of the tracking target part (no in step S109), the process proceeds to step S110.

[0068] In step S110, the determination unit 16 determines that the tracking failure of the target part is due to occlusion. Next, in step S111, the anonymization processing unit 17 does not perform anonymization processing on the target part in the frame where the tracking failure occurred. Then, the process proceeds to step S114.

[0069] Furthermore, in step S108, if the tracking of the estimated part is determined to be unsuccessful (No in step S108), the process proceeds to step S112. Similarly, in step S109, if the position of the estimated part in the frame where the tracking of the estimated part is determined to be successful matches the position of the tracking target part (Yes in step S109), the process proceeds to step S112.

[0070] In step S112, the determination unit 16 determines that the tracking failure of the target part is due to a detection error. Next, in step S113, the anonymization processing unit 17 interpolates the frame in which the tracking of the target part failed, and performs anonymization processing on the target part in the interpolated frame. Then, the process proceeds to step S114.

[0071] In step S114, the image processing system 10 determines whether the processing of all frames of the multiple frames in the time series has ended. At this time, the image processing system 10 may also determine whether the processing of tracking target parts other than the aforementioned tracking target parts has also ended. If it is determined that the processing of all frames of the multiple frames in the time series has ended (yes in step S114), the series of processes ends. On the other hand, if it is determined that the processing of all frames of the multiple frames in the time series has not ended (no in step S114), the processing returns to step S103.

[0072] As described above, the image processing system 10 according to the first embodiment can improve the accuracy of detecting the cause of tracking failure through the above structure. Moreover, by improving the accuracy of cause detection, the image processing system 10 can improve the accuracy of object anonymization.

[0073] Each structure of the image processing system 10 in the above embodiments is composed of hardware or software or both. It can be composed of one piece of hardware or software or multiple pieces of hardware or software.

[0074] Specifically, each structure of the image processing system 10 in the above embodiments comprises one or more computers. Each of the one or more computers has a processor and memory. The processor may be, for example, a microprocessor, a microprocessor unit (MPU), or a central processing unit (CPU). Multiple processors may be included. The memory comprises a combination of volatile and non-volatile memory. The memory may include storage devices separately configured from the processor. In this case, the processor may access the memory via an I / O interface not shown. The processor executes one or more programs including a set of commands for causing the computer to perform the algorithm (i.e., the image processing method) described in the accompanying drawings.

[0075] When the above program is read into a computer, it includes a set of commands (or software code) for causing the computer to perform one or more functions as described in the implementation.

[0076] Programs may be stored on non-transitory computer-readable media or tangible storage media. As non-limiting examples, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc (registered trademark) or other optical disc storage devices, magnetic tape, magnetic tape, disk storage devices, or other magnetic storage devices. Programs may be transmitted on temporary computer-readable media or communication media. As limiting but not restrictive examples, temporary computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagation signals.

[0077] The present invention has been described above with reference to the embodiments described above, but the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the structure or details of the present invention within the scope of the present invention. Furthermore, each embodiment can be appropriately combined with other embodiments.

[0078] The accompanying drawings are merely illustrative examples of one or more embodiments. Each drawing may be associated not only with a specific embodiment but also with one or more other embodiments. To enable those skilled in the art to understand, various features or steps described with reference to any of the drawings can be combined with features or steps shown in one or more other drawings to create embodiments, for example, not explicitly illustrated or described. For the purpose of illustrating illustrative embodiments, not all features or steps shown in any of the drawings are necessarily required, and some features or steps may be omitted. The order of steps described in any of the drawings may be appropriately modified.

[0079] Symbol Explanation

[0080] 10-Image processing system, 11-Acquisition unit, 12-Location detection unit, 13-Forward tracking unit, 14-Location estimation unit, 15-Reverse tracking unit, 16-Decision unit, 17-Anonymization processing unit.

Claims

1. An image processing system, characterized in that, have: The acquisition unit acquires multiple frames of the time series. A part detection unit initially detects and tracks the target part from one of the multiple frames in the time series; A forward tracking unit tracks the target part in a forward temporal direction in one or more frames following the frame in which the target part is initially detected in a plurality of frames of the time series. The part estimation unit, in the case that the tracking of the target part fails, estimates and predicts the part where the tracking of the target part exists as an estimated part in the frame where the tracking of the target part fails. A reverse tracking unit, which tracks the estimated region in reverse time for one or more frames preceding the frame in which tracking of the target region fails in the multiple frames of the time series; and The determination unit determines that the tracking failure of the target part is due to occlusion if the tracking of the estimated part is successful, and if the position of the estimated part is inconsistent with the position of the tracking target part in the frame in which the tracking of the estimated part is successful.

2. The image processing system according to claim 1, characterized in that, It also has: The anonymization processing unit, when determining that the tracking failure of the tracking target part is due to occlusion, does not perform anonymization processing on the tracking target part in the frame where the tracking failure of the tracking target part occurs.

3. The image processing system according to claim 1, characterized in that, If the tracking of the estimated part fails, or if the tracking of the estimated part is successful and the position of the estimated part matches the position of the tracking target part in the frame where the tracking of the estimated part is successful, the determination unit determines that the tracking failure of the tracking target part is due to a detection error.

4. An image processing method, characterized in that, The computer performs the following steps: Acquire multiple frames of a time series; The target region is initially detected and tracked from one frame out of multiple frames in the time series; In one or more frames following the initial detection of the target region in the multiple frames of the time series, the target region is tracked in a forward temporal direction. In the event that the tracking of the target part fails, in the frame where the tracking of the target part fails, the part where the tracking of the target part is presumed to exist is presumed to be the presumed part. In one or more frames preceding the frame in which the tracking of the target region in the multiple frames of the time series fails, the estimated region is tracked in reverse time. and If the tracking of the presumed part is successful, and if the position of the presumed part is inconsistent with the position of the tracking target part in the frame where the tracking of the presumed part is successful, it is determined that the tracking failure of the tracking target part is due to occlusion.

Images