Video processing system, video processing method, and video processing device

Abstract

Provided is a video processing system comprising: a detection means (11) for detecting a tracking target from an input video; a trajectory prediction means (12) for predicting the trajectory of the tracking target in the video; and a tracking means (13) which tracks the tracking target and which, if there is a deficient region in the video, estimates the position of the tracking target in the deficient region by using the prediction results of the trajectory prediction means. The video processing system may also comprise: a determination means (503) for determining whether a tracking target tracked by the tracking means is the same as a tracking target detected by the detection means; and a trajectory information output means (14) for attaching identification information to a detected tracking target.

Description

Image processing system, image processing method, and image processing device

[0001] The present disclosure relates to a video processing system, a video processing method, and a video processing device.

[0002] As a related technique, Patent Literature 1 discloses an object tracking device that can reliably track a fast-moving object and draw an accurate trajectory without attaching a sensor to the object. The object tracking device includes an infrared light detection unit that detects a detected blob from an infrared image, a trajectory generation unit that generates a trajectory based on the detected detected blob, and a position prediction unit that calculates a predicted position and search range of the detected blob. The object tracking device also includes a history data generation unit that generates history data for each trajectory T, a delay time determination unit that compares and determines the elapsed time and delay time of the history data, and a detection success count determination unit that determines whether the number of successful detections is equal to or greater than a threshold. The object tracking device also includes a visible image delay unit that delays a visible image by a delay time m, and a trajectory drawing unit that draws the trajectory T on the visible image.

[0003] JP 2018-78431 A

[0004] In related technology, objects are tracked using a tracking device such as the one described above. However, when receiving video via a network, packet loss due to a deterioration in the network communication environment can cause the video to be distorted. For example, frames or parts of frames can be lost. In such degraded video, it is sometimes difficult to accurately track objects. Therefore, an object tracking method suitable for video distribution via a network is proposed.

[0005] The video processing system of the present disclosure is an video processing system including: a detection means for detecting a tracking target from an input video; a trajectory prediction means for predicting a trajectory of the tracking target in the video; and a tracking means for tracking the tracking target in a missing area, if the video has a missing area, using a prediction result of the trajectory prediction means.

[0006] The video processing method disclosed herein is a video processing method that detects a tracking target from an input video, predicts a trajectory of the tracking target in the video, and, if the video has a missing area, tracks the tracking target in the missing area using the result of the trajectory prediction.

[0007] The image processing device of the present disclosure is an image processing device including: a detection means for detecting a tracking target from an input image; a trajectory prediction means for predicting a trajectory of the tracking target in the image; and a tracking means for tracking the tracking target in a missing area, if the image has a missing area, using a prediction result of the trajectory prediction means.

[0008] The present disclosure provides a suitable method for tracking an object when video is distributed over a network.

[0009] 1 is a block diagram of a video processing system according to an embodiment. 2 is a block diagram of a video processing device according to an embodiment. 3 is a flowchart of a video processing method according to an embodiment. 4 is a schematic diagram of a remote monitoring system according to an embodiment. 5 is a block diagram of a video processing system according to a first embodiment. 6 is a flowchart of a video processing method according to a first embodiment. 7 is a flowchart of a video processing method according to a first embodiment. 8 is a diagram showing an example of a storage format of trajectory reliability according to the first embodiment. 9 is a diagram showing a detection example in which related trajectory reliability decreases according to the first embodiment. 10 is a diagram showing a detection example in which trajectory reliability is constant and does not fall below a predetermined value according to the first embodiment. 11 is a diagram showing an example in which an object cannot be tracked due to a missing frame.

[0010] Embodiments Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. However, the invention according to the claims is not limited to the following embodiments. Furthermore, not all of the configurations described in the embodiments are necessarily essential as means for solving the problems. For clarity of explanation, the following description and drawings have been omitted and simplified as appropriate. In each drawing, the same elements are assigned the same reference numerals, and repeated explanations are omitted as necessary.

[0011] (Description of a video processing system according to an embodiment) Fig. 1 is a block diagram of a video processing system according to an embodiment. A video processing system 10 according to an embodiment will be described with reference to Fig. 1. The video processing system 10 can be applied to, for example, a remote monitoring system that collects video via a network and monitors the video.

[0012] As shown in FIG. 1, a video processing system 10 according to the embodiment includes a detection unit 11, a trajectory prediction unit 12, and a tracking unit 13.

[0013] The detection unit 11 detects a tracking target from the input video. The tracking target is an object set as a tracking target, and is, for example, a moving object such as a car, a bicycle, a robot, or a ship, or a person. The detection unit 11 transmits extracted information, which is information about the tracking target and is extracted by the detection process, to the trajectory prediction unit 12. For example, the detection unit 11 preferably transmits position information of the tracking target obtained by surrounding the detection target with a rectangular frame as the extracted information, but is not limited to this. The frame surrounding the detection target may be rectangular, circular, or irregular. Furthermore, the detected information may not only be position information, but also time-series position information of the tracking target, the speed of the tracking target, the motion vector of the tracking target, identification information of the tracking target, and the type of the tracking target.

[0014] The trajectory prediction unit 12 predicts the trajectory of the tracking target in the video. The trajectory can also be interpreted as a figure drawn while the position satisfies certain conditions. Based on the extracted information, the trajectory prediction unit 12 predicts the position of the tracking target in the video after the video in which the detection process was performed.

[0015] The tracking unit 13 tracks the tracking target. If there is a missing area in the video, the tracking unit 13 estimates the position of the tracking target in the missing area using the prediction result of the trajectory prediction unit.

[0016] The video processing system 10 may be configured with one device or multiple devices. Fig. 2 shows the configuration of a video processing device 20 according to an embodiment. As shown in Fig. 2, the video processing device 20 may include the detection unit 11, the trajectory prediction unit 12, and the tracking unit 13 shown in Fig. 1, or a device including the detection unit 11 and a device including the trajectory prediction unit 12 and the tracking unit 13 may communicate with each other and operate as functions of the video processing system. Furthermore, part or all of the video processing system 10 may be located in a cloud. For example, each function may be distributed across the cloud.

[0017] Fig. 3 is a flowchart of a video processing method according to an embodiment. For example, the video processing method according to the embodiment is executed by the video processing system 10 of Fig. 1 and the video processing device of Fig. 2. As shown in Fig. 3, first, a tracking target is detected from the video (step S11). Next, the trajectory of the tracking target is predicted (step S12). Next, it is determined whether or not there is a missing area in the video (step S13). If there is no missing area (No in step S13), tracking of the tracking target is continued until the tracking target goes outside the shooting range, and the process ends. If there is a missing area (Yes in step S13), the prediction result is used to estimate the position of the tracking target, and the process ends.

[0018] As described above, in the video processing system according to the embodiment, when a video has a missing area, the position of the tracking target in the missing area is estimated using the prediction result of the tracking prediction unit, thereby providing a suitable method for tracking an object when video is distributed over a network.

[0019] (Description of a remote monitoring system according to an embodiment) A remote monitoring system, which is an example of a system to which an embodiment is applied, will be described. Fig. 4 is a schematic diagram of a remote monitoring system according to an embodiment. The remote monitoring system according to the embodiment will be described with reference to Fig. 4.

[0020] The remote monitoring system 1 is a system that monitors an area captured by a camera using video. Note that video includes multiple images (also called frames) in a time series, so the terms video and image are interchangeable. In other words, the remote monitoring system can be considered a video processing system that processes video, and an image processing system that processes images.

[0021] As shown in FIG. 4 , the remote monitoring system 1 includes multiple terminals 100, a center server 200, a base station 300, and an MEC 400. The terminals 100, the base station 300, and the MEC 400 are located on the field side, and the center server 200 is located on the center side. For example, the center server 200 is located in a data center or a monitoring center located away from the field. The field side is also the edge side of the system, and the center side is also the cloud side. In this specification, the field side may refer to a device located at the field where the terminal is installed or near the field. In addition, a device close to the terminal in terms of the network hierarchy may also be referred to as a field-side device. Because the center server 200 is located farther away than the edge-side devices, it may also be referred to as the center side in contrast to the edge side. For example, the center server 200 may be located in a monitoring center or data center located several kilometers away from the field. Furthermore, since the center server 200 may be placed on the cloud using network virtualization technology, the center side may also be referred to as the cloud side. The center server 200 may be configured by one device or by multiple devices. Furthermore, part or all of the center server 200 may be placed on the cloud.

[0022] The terminal 100 and the base station 300 are communicatively connected via a network NW1. The network NW1 is, for example, a wireless network such as 4G, local 5G / 5G, LTE (Long Term Evolution), or wireless LAN. The base station 300 and the center server 200 are communicatively connected via a network NW2. The network NW2 includes, for example, a core network such as 5GC (5th Generation Core network) or EPC (Evolved Packet Core), or the Internet. It can also be said that the terminal 100 and the center server 200 are communicatively connected via the base station 300. The base station 300 and the MEC 400 are communicatively connected via any communication method, but the base station 300 and the MEC 400 may be a single device.

[0023] The terminal 100 is a terminal device connected to the network NW1 and also serves as an image generating device that generates images of the site. The terminal 100 acquires images captured by a camera 101 installed at the site and transmits the acquired images to the center server 200 via the base station 300. The camera 101 may be located outside the terminal 100 or inside the terminal 100.

[0024] The terminal 100 compresses video from the camera 101 to a predetermined bit rate and transmits the compressed video. The terminal 100 has a compression efficiency optimization function 102 that optimizes compression efficiency, and a video transmission function 103. The compression efficiency optimization function 102 performs ROI control that controls the image quality of an ROI (Region of Interest; also called a fixation region). The compression efficiency optimization function 102 reduces the bit rate by maintaining the image quality of an ROI that includes a person or object while lowering the image quality of the surrounding area. The video transmission function 103 transmits the video with controlled image quality to the center server 200.

[0025] The base station 300 is a base station device of the network NW1, and also a relay device that relays communication between the terminal 100 and the center server 200. For example, the base station 300 is a local 5G base station, a 5G gNB (next generation Node B), an LTE eNB (evolved Node B), a wireless LAN access point, or the like, but may also be another relay device.

[0026] The MEC (Multi-access Edge Computing) 400 is an edge processing device located on the edge side of the system. The MEC 400 is an edge server that controls the terminal 100 and has a compression bit rate control function 401 and a terminal control function 402 that control the bit rate of the terminal. The compression bit rate control function 401 controls the bit rate of the terminal 100 through adaptive video distribution control and QoE (quality of experience) control. For example, the compression bit rate control function 401 predicts the recognition accuracy to be obtained while suppressing the bit rate according to the communication environment of the networks NW1 and NW2, and allocates a bit rate to the camera 101 of each terminal 100 so as to improve the recognition accuracy. The terminal control function 402 controls the terminal 100 to transmit video at the allocated bit rate. The terminal 100 encodes the video so as to obtain the allocated bit rate and transmits the encoded video.

[0027] The center server 200 is a server installed on the center side of the system. The center server 200 may be one or more physical servers, or may be a cloud server built on the cloud or other virtualized server. The center server 200 is a monitoring device that monitors on-site work by recognizing the work of people from on-site camera images. The center server 200 is also a recognition device that recognizes the behavior of people in images transmitted from the terminal 100.

[0028] The center server 200 has a detection unit 11, a trajectory prediction unit 12, a tracking unit 13, and a trajectory information output unit 14. The detection unit 11 detects objects or people by inputting video transmitted from the terminal 100 into a video recognition AI (Artificial Intelligence) engine. The trajectory prediction unit 12 predicts the trajectory of the detected object or person. The tracking unit 13 tracks the object and estimates the trajectory of the object or person in the missing area. The trajectory information output unit 14 displays the video of the terminal 100 and the tracking results on a GUI (Graphical User Interface).

[0029] In particular, in remote monitoring systems, if the quality of communication between the site and the center deteriorates, the image quality of the video received at the center deteriorates, which causes the problem of not being able to accurately identify and track the target of monitoring from the video received at the center.

[0030] (Description of Video Processing System According to First Embodiment) The video processing system according to the first embodiment will be described below with reference to the drawings. The video processing system 500 is mainly constructed by the center server 200 described in FIG. 4. FIG. 5 is a block diagram of the video processing system according to the first embodiment. Note that the configuration of each device is an example, and other configurations may be used as long as the operation according to the present embodiment described below is possible. For example, some functions of the terminal 100 may be arranged in the center server 200 or another device, or some functions of the center server 200 may be arranged in the terminal 100 or another device.

[0031] 5 , the video processing system 500 according to the first embodiment includes a compressed video stream input unit 501, a decoding unit 502, a detection unit 11, a trajectory prediction unit 12, a tracking unit 13, a determination unit 503, and a trajectory information output unit 14. The video processing system 500 is obtained by adding the compressed video stream input unit 501, the decoding unit 502, the determination unit 503, and the trajectory information output unit 14 to the video processing system 10.

[0032] The compressed video stream input unit 501 is a part having a function of inputting a stream of video compressed by the terminal 100. The compressed video is sent to a decoding unit 502.

[0033] The decoding unit 502 decodes the compressed video. The decoded video is sent to the detection unit 11 as video frame information. The video frame information is also sent to the tracking unit 13. At the same time, the decoded video is judged for the presence or absence of a missing area. The decoding unit 502 provides the tracking unit 13 with information on the presence or absence of a missing area, and in some cases, information on the position of the missing area. In some cases, the decoding unit 502 may send motion vector information of the tracking target to the trajectory prediction unit 12.

[0034] The detection unit 11 detects objects and people to be tracked from the input video. For example, if a person is detected, the detection unit 11 assigns 1234 to the person as an identification number and sends it to the tracking unit 13. The detection unit 11 also calculates the detection reliability. The detection reliability is a numerical value indicating the likelihood of detection, and is expressed as a numerical value between 0 and 1, a rate, or an index.

[0035] The trajectory prediction unit 12 predicts the trajectory of the tracked object in the video. The trajectory prediction unit 12 sends the position of a rectangle predicted based on the predicted trajectory of the tracked object to the tracking unit 13. The trajectory prediction unit 12 predicts the trajectory of the tracked object for each frame. The trajectory prediction unit 12 calculates prediction reliability. The prediction reliability is a numerical value indicating the likelihood of the prediction, and is expressed as a numerical value between 0 and 1, a rate, or an index.

[0036] The trajectory prediction unit 12 can predict the trajectory of a tracking target by learning the type and behavior of the tracking target using AI. The trajectory prediction unit 12 learns the type and behavior of past tracking targets, and when a missing area appears, identifies the type of tracking target that was captured in the frame before the missing area appeared. The type of tracking target is classified into types of objects with characteristic movements, such as people, bicycles, or automobiles. The type of tracking target is identified by the detection unit 11, and the result is received by the trajectory prediction unit 12. For example, if the tracking target is a person, it learns that it moves relatively slowly from its current position to the surrounding area, and if the tracking target is a car, it learns that it moves relatively quickly along the lane. The trajectory prediction unit 12 can then predict the behavior of the tracking target in the missing area based on the learning results and the identified type of tracking target.

[0037] Furthermore, the trajectory prediction unit 12 can predict the trajectory of the tracking target using the motion vector assigned by the decoding unit. The trajectory prediction unit 12 acquires the tracking target to which the motion vector has been assigned, and when a missing area appears, identifies the position of the tracking target that was captured in the frame before the missing area appeared. The trajectory prediction unit 12 can predict the position of the tracking target in the missing area from the identified position of the tracking target and the motion vector.

[0038] The tracking unit 13 sends the tracking result corresponding to the frame to the trajectory prediction unit 12 and feeds back the prediction of the trajectory prediction unit 12. If there is no missing area, the tracking unit 13 sends the detection result and tracking result to the determination unit 503. The determination unit 503 determines whether the tracking target tracked by the tracking unit is the same as the tracking target detected by the detection unit. The determination unit 503 then returns the determination result to the tracking unit 13. If they are the same, the tracking unit 13 assigns the same identification number (ID) to the tracking target and sends the tracking result to the trajectory information output unit 14. If they are not the same, the tracking unit 13 assigns a new identification number to the tracking target and sends the tracking result to the trajectory information output unit 14. A name or symbol may be used instead of the identification number. These identification numbers, symbols, and names are used as identification information.

[0039] If there is a missing area and the tracking target is included in the missing area, the tracking unit 13 estimates the position of the tracking target in the missing area using the prediction result of the trajectory prediction unit 12. While the missing area exists, the tracking result is not sent to the determination unit 503. Then, when the missing area disappears or the tracking target exits the missing area, the tracking unit 13 sends the detection result and tracking result to the determination unit 503. The determination unit 503 determines whether the tracking target tracked by the tracking unit is the same as the tracking target detected by the detection unit 11. The determination unit 503 returns the determination result to the tracking unit 13. If they are identical, the tracking unit 13 assigns the same identification number to the tracking target and sends the tracking result to the trajectory information output unit 14. If they are not identical, the tracking unit 13 assigns a new identification number to the tracking target and sends the tracking result to the trajectory information output unit 14.

[0040] If there is no missing area, the trajectory information output unit 14 displays the detected tracking target by adding a rectangle and identification information. For example, as shown in rectangle 504 in Fig. 5, the trajectory information output unit 14 displays a rectangle surrounding a person by adding 1234 as an identification number. If there is a missing area, the trajectory information output unit 14 displays an image in which a rectangle and identification information are superimposed on the position of the estimated tracking target.

[0041] The rectangle 504 displayed in the missing area can be a picture or symbol that resembles the tracking target, or a clipped image of the image detected in the previous frame. The rectangle 504 can also be overlaid on the image by increasing its transparency, or the border color can be changed. For example, visible objects can be colored red, and predicted objects can be colored blue. The border of the rectangle 504 can also take various forms, such as a dotted line, a solid line, or a dashed line.

[0042] The trajectory information output unit 14 also calculates trajectory reliability based on the prediction reliability and the detection reliability. The trajectory reliability is a numerical value indicating the likelihood of a trajectory, and is expressed as a numerical value, a rate, or an index ranging from 0 to 1. The trajectory reliability is calculated based on the detection reliability and the prediction reliability. The trajectory reliability is determined to be constant when the tracking target is present in the missing area, and therefore does not fall below a predetermined value. Therefore, the tracking target is preserved while it is in the missing area. This reduces the possibility of terminating tracking of the tracking target while it is in the missing area.

[0043] The tracking unit 13 tracks the tracking target in every frame. If the trajectory reliability falls below a predetermined value, the tracking unit 13 stops tracking the tracking target. The video processing system 500 described above may be configured with one device or multiple devices. For example, the video processing system 500 may be realized by configuring the compressed video stream input unit 501, decoding unit 502, and detection unit 11 as one device, and the trajectory prediction unit 12, tracking unit 13, determination unit 503, and trajectory information output unit 14 as one device, with communication between the two devices.

[0044] FIG. 11 is a diagram showing an example in which an object cannot be tracked due to a missing frame. FIG. 11 shows the passage of time from the left to the center and right. As shown in FIG. 11, when there is no tracking target in the missing region of a frame, object 1001 can be correctly tracked, and the same object is tracked using the same identification number (ID456 in this case). On the other hand, as shown in the center of FIG. 11, when a missing frame occurs, object 1003 assigned ID123 is located in missing region 1002, making it impossible to detect object 1003. Therefore, as shown in the right of FIG. 11, after the loss is recovered, object 1003, which could not be detected due to the missing frame, cannot be correctly tracked. As such, there is a problem in that a missing frame results in the same object being assigned a new identification number, ID124.

[0045] To address the above-described problems, in the video processing system according to the embodiment, when there is a missing area in the video, the tracking target is tracked in the missing area using the prediction result of the tracking prediction unit. This makes it possible to track an object even if it becomes undetectable due to a missing frame in the video. Therefore, it is possible to provide a method for tracking an object that is suitable for video distribution over a network.

[0046] (Explanation of video processing method according to embodiment 1) Fig. 6 is a flowchart of the video processing method according to embodiment 1. Fig. 7 is a flowchart of the video processing method according to embodiment 1. Fig. 8 is a diagram showing an example of a storage format of trajectory reliability according to embodiment 1. The video processing method according to embodiment 1 will be described with reference to Figs. 6 to 8 .

[0047] As shown in Fig. 6, first, a tracking target is detected from an image (step S601). For example, the image is an image obtained by decoding a compressed image. A tracking target is detected from the input image. In the step of detecting the tracking target, a detection reliability indicating the reliability of the detection of the tracking target is calculated. Next, a trajectory of the tracking target is predicted (step S602). The trajectory of the tracking target in the input image is predicted. In the step of predicting the trajectory, a prediction reliability indicating the reliability of the prediction of the tracking target is calculated. Next, it is determined whether or not there is a missing area in the image (step S603).

[0048] If there is no missing area in the image (No in step S603), the tracking of the tracking target is continued until the tracking target goes out of the shooting range, and then the process ends. If there is a missing area in the image (Yes in step S603), the position of the tracking target is estimated using the prediction result (step S604). If there is a missing area in the image, the tracking target is tracked in the missing area using the prediction result of the trajectory prediction step.

[0049] Next, it is determined whether the tracked target is the same as the detected target (step S605). When the target leaves the missing area, it is determined whether the target tracked in the step of tracking the target is the same as the target detected in the step of detecting the target. If they are the same (Yes in step S605), the previous ID is assigned to the detected target (step S606). If they are not the same (No in step S605), a new ID is assigned to the detected target (step S607). In both cases, a rectangle is superimposed on the image, an identification number is assigned to the rectangle, and the process ends. In the step of assigning the identification number, a trajectory reliability indicating the reliability of the trajectory of the tracked target is calculated based on the detection reliability and the prediction reliability. The trajectory reliability is constant when the tracked target is in the missing area and is determined so as not to decrease, so it does not fall below a predetermined value. Therefore, the tracked target is preserved while it is in the missing area. This reduces the possibility of terminating tracking of the tracked target while it is in the missing area.

[0050] The step of tracking the tracked object includes constantly tracking the tracked object, and stopping tracking the tracked object when the trajectory reliability falls below a predetermined value.

[0051] As described above, in the video processing method according to the first embodiment, when there is a missing area in the video, the position of the tracking target in the missing area is estimated using the prediction result of the tracking prediction unit, thereby making it possible to track the object even if it becomes impossible to detect the object due to a missing frame in the video.

[0052] FIG. 7 describes the detection reliability, prediction reliability, and trajectory reliability in more detail. First, a tracking target is detected from the input image (step S701). Next, the position of the object is predicted. Also, prediction reliability is calculated (step S702). For the trajectory of a tracking target whose detected subsequent trajectory has been predicted, the position (rectangle) in the next frame is predicted. A tracking target whose detected subsequent trajectory has been predicted does not mean a tracking target detected in the current frame, but rather a tracking target whose trajectory has been detected up to the previous frame and whose trajectory has been predicted and stored. The prediction reliability is calculated by multiplying the stored trajectory reliability of the tracking target by a coefficient specific to the prediction. Next, it is determined whether the predicted rectangle is an object within a missing area (step S703).

[0053] A case where the predicted rectangle is within the missing area (Yes in step S703) is designated Case A. In Case A, the trajectory is updated using the prediction result (step S704). That is, for each rectangle within the missing area, the prediction result (rectangle, reliability) is used to update the trajectory associated with the prediction. In this case, since no rectangle appears in the video, it is not determined whether the detected tracking target and the predicted rectangle are the same. The prediction reliability is designated as trajectory reliability. Furthermore, since the prediction reliability is determined to be constant if the tracking target is present in the missing area, it does not fall below a predetermined value. Therefore, the trajectory reliability also does not fall below a predetermined value.

[0054] If the predicted rectangle is outside the missing area (No in step S703), it is determined whether the detected tracking target and the predicted rectangle are the same (step S705). That is, it is determined whether the tracked tracking target or the predicted tracking target is the same as the detected tracking target. For example, the Hungarian algorithm is used to solve the problem, with the IoU (Intersection over Union) (degree of overlap) of each detected tracking target and the predicted rectangle being used as a weight. Alternatively, feature quantities specific to a person, bicycle, or automobile may be compared to determine whether they are the same.

[0055] The process performed for each predicted rectangle that is not identical is referred to as Case B. In Case B, the predicted rectangle is used to update the trajectory (step S706). That is, the predicted rectangle is used to update (continue) the trajectory associated with the prediction. The prediction reliability is multiplied by a constant coefficient of 1.0 or less to obtain the updated trajectory reliability.

[0056] The process performed for each detected tracking target and predicted rectangle that are identical is referred to as Case C. In Case C, the detected tracking target is used to update the trajectory associated with the predicted rectangle (step S707). The trajectory associated with the predicted rectangle is updated (continued) using the detected tracking target. In other words, the trajectory associated with the predicted rectangle is updated using the detected tracking target. The updated trajectory reliability is calculated based on the prediction reliability and the detection reliability.

[0057] The process performed for each detected tracking target that is not identical is referred to as Case D. In Case D, a new trajectory is created (step S708). That is, a new identification number is assigned to the detected tracking target and stored. The trajectory reliability is stored following the detection reliability.

[0058] In Cases B, C, and D, trajectories whose trajectory reliability is below the threshold are discarded (step S709). That is, the trajectory of the tracking target is deleted. In other words, tracking of the tracking target is stopped. This approach prevents the number of tracking targets from increasing and diverging.

[0059] Figure 8 shows a table of trajectory ID, time t, position (rectangle), and trajectory reliability. The table is saved in the storage unit of the center server or in the system cloud. Time t is the current time, and position is the coordinate in the video. The trajectory reliability is saved using such a table.

[0060] (Explanation of an example of detection of a tracked object according to an embodiment) Fig. 9 is a diagram showing an example of detection in which the associated trajectory reliability decreases according to the first embodiment. Fig. 10 is a diagram showing an example of detection in which the trajectory reliability is constant and does not fall below a predetermined value according to the first embodiment. An example of detection of a tracked object according to the first embodiment will be described with reference to Figs. 9 and 10. In Figs. 9 and 10, ◯ indicates a state in which tracking is successful, and X indicates a state in which tracking is unsuccessful.

[0061] The left diagram of FIG. 9 shows Case D, a scene in which a tracking target is detected. Trajectory tracking begins from this point. The tracking target is assigned an identification number, ID123. The middle left diagram of FIG. 9 shows a scene in which detection of the tracking target fails due to the presence of a missing area in the frame. In this case, Case B is shown, in which an operation to identify the tracking target's trajectory is performed using the prediction result. The middle right diagram of FIG. 9 shows a scene in which the trajectory reliability falls below a threshold; in other words, Case B shows a scene in which the trajectory is truncated, i.e., tracking is stopped. The right diagram of FIG. 9 shows Case D, a scene in which a tracking target is detected. In this case, the detected target and the tracking target are not the same, and the target is recognized as a new target to be tracked, and an identification number, ID124, is assigned. In related methods, i.e., detection methods that rely solely on trajectory reliability, if detection fails, the prediction reliability, i.e., the trajectory reliability, decreases, resulting in the trajectory being truncated and a new identification number being assigned.

[0062] The left diagram of FIG. 10 shows Case D, in which a tracking target is detected. Trajectory tracking begins from this point. The tracking target is assigned an identification number, ID123. The middle left diagram of FIG. 10 shows a scene in which detection of the tracking target fails due to the presence of a missing area in the frame. In this case, Case A, in which tracking of the trajectory is performed using prediction, is shown. The middle right diagram of FIG. 10 shows a scene in which the trajectory reliability is kept constant and does not fall below the threshold. This is the case of Case A, in which tracking of the trajectory is performed using prediction. The right diagram of FIG. 10 shows Case C, in which a tracking target is detected and the detected tracking target and the tracked tracking target are determined to be the same. In this case, the detected tracking target is assigned the previous identification number, ID123. As described above, in the method disclosed herein, the prediction reliability, i.e., the trajectory reliability, is constant and does not fall below the threshold, so the identification number does not need to be updated.

[0063] In this way, the prediction reliability, i.e., the trajectory reliability, does not fall below a predetermined value, and the trajectory is stored, thereby preventing the identification number from being updated.

[0064] Each configuration in each of the above-described embodiments may be configured with hardware or software, or both, and may be configured with a single piece of hardware or software, or may be configured with multiple pieces of hardware or software. The functions of each device may be realized by a computer having a CPU (Central Processing Unit), memory, etc. For example, a program for performing the method in each embodiment may be stored in a storage device, and each function may be realized by executing the program stored in the storage device with the CPU.

[0065] These programs include instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The programs may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The programs may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0066] The present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the present disclosure.

[0067] Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to the following. (Supplementary Note 1) A video processing system comprising: detection means for detecting a tracking target from an input video; trajectory prediction means for predicting a trajectory of the tracking target in the video; and tracking means for tracking the tracking target, the tracking means being configured to, if the video has a missing area, estimate a position of the tracking target in the missing area using a prediction result of the trajectory prediction means. (Supplementary Note 2) The video processing system according to Supplementary Note 1, comprising: determination means for determining whether the tracking target tracked by the tracking means is the same as the tracking target detected by the detection means; and trajectory information output means for assigning identification information to the detected tracking target. (Supplementary Note 3) The video processing system according to Supplementary Note 2, wherein the trajectory prediction means calculates a prediction reliability indicating a reliability of the prediction of the tracking target, and the tracking means stops tracking the tracking target in accordance with the prediction reliability. (Supplementary Note 4) The video processing system according to Supplementary Note 3, wherein the detection means calculates a detection reliability indicating a reliability of detection of the tracked object, the trajectory information output means calculates a trajectory reliability indicating a reliability of the trajectory of the tracked object based on the detection reliability and the prediction reliability, and the tracking means stops tracking of the tracked object according to the trajectory reliability. (Supplementary Note 5) The video processing system according to Supplementary Note 4, wherein the trajectory reliability is not reduced while the tracked object is in the missing area. (Supplementary Note 6) The video processing system according to Supplementary Note 2, wherein the trajectory information output means superimposes the detected tracked object on the video. (Supplementary Note 7) The video processing system according to any of Supplements 1 to 6, further comprising decoding means for decoding the input video, and the decoding means provides information of the missing area to the tracking means. (Supplementary Note 8) The video processing system according to any one of Supplementary Notes 1 to 7, wherein the trajectory prediction means learns past movements of the tracked object according to its type, and when the missing area appears, identifies the type of the tracked object that was shown in a frame before the missing area appeared, and predicts the movement of the tracked object in the missing area from the result of the learning and the identified type of the tracked object.(Supplementary Note 9) The video processing system according to any one of Supplementary Notes 1 to 7, wherein the trajectory prediction means acquires the tracking target to which a motion vector has been assigned, and, when the missing area appears, identifies the position of the tracking target that appeared in the frame before the missing area appeared, and predicts the position of the tracking target in the missing area from the identified position of the tracking target and the motion vector. (Supplementary Note 10) A video processing method comprising: detecting a tracking target from an input video; predicting a trajectory of the tracking target in the video; and, when tracking the tracking target, if there is a missing area in the video, estimating the position of the tracking target in the missing area using the result of the trajectory prediction. (Supplementary Note 11) The video processing method according to Supplementary Note 10, determining whether the tracking target tracked by tracking the tracking target is the same as the tracking target detected by detecting the tracking target, and assigning identification information to the detected tracking target. (Supplementary Note 12) The video processing method according to Supplementary Note 11, wherein in predicting the trajectory, a prediction reliability indicating a reliability of the prediction of the tracked object is calculated, and tracking of the tracked object is stopped according to the prediction reliability being a predetermined value. (Supplementary Note 13) The video processing method according to Supplementary Note 12, wherein in detecting the tracked object, a detection reliability indicating a reliability of detection of the tracked object is calculated, and in assigning the identification information, a trajectory reliability indicating a reliability of the trajectory of the tracked object is calculated based on the detection reliability and the prediction reliability, and tracking of the tracked object is stopped according to the trajectory reliability. (Supplementary Note 14) The video processing method according to Supplementary Note 13, wherein the trajectory reliability is not reduced while the tracked object is in the missing region. (Supplementary Note 15) The video processing method according to Supplementary Note 11, wherein in assigning the identification information, the detected tracked object is superimposed on the video. (Supplementary Note 16) The video processing method according to any one of Supplementary Notes 10 to 15, further comprising: decoding the input video; and providing information about the missing region for tracking the tracking target, wherein the decoding of the video provides information about the missing region for tracking the tracking target.(Supplementary Note 17) The video processing method according to any one of Supplementary Notes 10 to 16, wherein the trajectory prediction involves learning past behaviors of the tracked target according to its type, identifying the type of the tracked target that appeared in frames before the missing area appeared when the missing area appears, and predicting the behavior of the tracked target in the missing area from the learning result and the identified type of tracked target. (Supplementary Note 18) The video processing method according to any one of Supplementary Notes 10 to 16, wherein the trajectory prediction involves acquiring the tracked target to which a motion vector has been assigned, identifying the position of the tracked target that appeared in frames before the missing area appeared when the missing area appears, and predicting the position of the tracked target in the missing area from the identified position of the tracked target and the motion vector. (Supplementary Note 19) A video processing device comprising: detection means for detecting the tracked target from an input video; trajectory prediction means for predicting a trajectory of the tracked target in the video; and tracking means for tracking the tracked target, the tracking means being configured to estimate the position of the tracked target in the missing area using the prediction result of the trajectory prediction means when the video has a missing area. (Supplementary Note 20) The video processing device according to Supplementary Note 19, comprising: a determination means for determining whether a tracking target tracked by the tracking means is the same as the tracking target detected by the detection means; and a trajectory information output means for assigning identification information to the detected tracking target. (Supplementary Note 21) The video processing device according to Supplementary Note 20, wherein the trajectory prediction means calculates a prediction reliability indicating a reliability of prediction of the tracking target, and the tracking means stops tracking the tracking target in accordance with the prediction reliability. (Supplementary Note 22) The video processing device according to Supplementary Note 21, wherein the detection means calculates a detection reliability indicating a reliability of detection of the tracking target, and the trajectory information output means calculates a trajectory reliability indicating a reliability of the trajectory of the tracking target based on the detection reliability and the prediction reliability, and the tracking means stops tracking the tracking target in accordance with the trajectory reliability. (Supplementary Note 23) The video processing device according to Supplementary Note 22, wherein the trajectory reliability is not reduced while the tracking target is present in the defective region. (Supplementary Note 24) The image processing device according to Supplementary Note 20, wherein the trajectory information output means superimposes the detected tracking target on the image.(Supplementary Note 25) The video processing device according to any of Supplements 19 to 24, comprising decoding means for decoding the input video, wherein the decoding means provides information about the missing area to the tracking means. (Supplementary Note 26) The video processing device according to any of Supplements 19 to 25, wherein the trajectory prediction means learns past behaviors of the tracking target according to its type, and, when the missing area appears, identifies the type of the tracking target that appeared in a frame before the missing area appeared, and predicts the behavior of the tracking target in the missing area from the result of the learning and the identified type of tracking target. (Supplementary Note 27) The video processing device according to any of Supplements 19 to 25, wherein the trajectory prediction means acquires the tracking target to which a motion vector has been assigned, and, when the missing area appears, identifies the position of the tracking target that appeared in a frame before the missing area appeared, and predicts the position of the tracking target in the missing area from the identified position of the tracking target and the motion vector.

[0068] REFERENCE SIGNS LIST 1 Remote monitoring system 10 Video processing system 11 Detection unit 12 Trajectory prediction unit 13 Tracking unit 14 Trajectory information output unit 20 Video processing device 100 Terminal 101 Camera 102 Compression efficiency optimization function 200 Center server 300 Base station 400 MEC 401 Compression bit rate control function 500 Video processing system 501 Compressed video stream input unit 502 Decoding unit 503 Determination unit 504 Rectangle 1001 Object 1002 Missing area 1003 Object 1004 Object

Claims

1. A detection means for detecting a tracking target from an input image; a trajectory prediction means for predicting a trajectory of the tracked object in the video; a tracking means for tracking the tracking object, the tracking means being configured to, when the image has a missing area, estimate a position of the tracking object in the missing area using a prediction result of the trajectory prediction means.

2. a determination means for determining whether the tracking target tracked by the tracking means is the same as the tracking target detected by the detection means; The image processing system according to claim 1 , further comprising: a trajectory information output unit that assigns identification information to the detected tracking target.

3. The trajectory prediction means calculates a prediction reliability indicating a reliability of the prediction of the tracked object, The image processing system according to claim 2 , wherein the tracking means stops tracking the tracked object in response to the prediction reliability.

4. The detection means calculates a detection reliability indicating a reliability of detection of the tracked object, the trajectory information output means calculates a trajectory reliability indicating a reliability of the trajectory of the tracked object based on the detection reliability and the prediction reliability; The image processing system according to claim 3 , wherein the tracking means stops tracking the tracking target in response to the trajectory reliability.

5. The image processing system according to claim 4 , wherein the trajectory reliability is not decreased while the tracking target is in the missing region.

6. The trajectory prediction means learns the motion of the object to be tracked according to the type of the object to be tracked, When the defective area appears, a type of the tracking target that was shown in the frame before the defective area appeared is identified; The image processing system according to claim 1 , further comprising: a predictor configured to predict a motion of the tracking target in the missing region based on a result of the learning and the type of the tracking target identified.

7. The trajectory prediction means acquires the tracking target to which a motion vector has been assigned, When the defective area appears, the position of the tracking target that was captured in the frame before the defective area appeared is identified; The image processing system according to claim 1 , further comprising: predicting a position of the tracking target in the missing region from the identified position of the tracking target and the motion vector.

8. Detects the target to be tracked from the input video, predicting a trajectory of the tracked object in the video; When tracking the tracking object, if there is a missing area in the image, a position of the tracking object in the missing area is estimated using a result of the trajectory prediction.

9. determining whether or not a tracked object tracked by tracking the tracked object is the same as a tracked object detected by detecting the tracked object; The image processing method according to claim 8 , further comprising the step of providing identification information to the detected tracking target.

10. A detection means for detecting a tracking target from an input image; a trajectory prediction means for predicting a trajectory of the tracked object in the video; a tracking means for tracking the tracking target, the tracking means being configured to, when the image has a missing area, estimate a position of the tracking target in the missing area using a prediction result of the trajectory prediction means.