Evaluation system, evaluation method, and program

The evaluation system addresses the challenge of accurately assessing tracking system performance by using error-specific coefficients to calculate an evaluation value, reflecting the algorithm's performance in real-world conditions.

JP7582170B2Active Publication Date: 2024-11-13TOYOTA JIDOSHA KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2021199462
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2021-12-08
Publication Date
2024-11-13
Estimated Expiration
2041-12-08

AI Technical Summary

Technical Problem

Existing tracking systems lack a comprehensive method to evaluate their performance accurately across different usage environments, as evaluation metrics like MOTA may not reflect the system's actual performance in specific conditions.

Method used

An evaluation system and method that assesses the quality of tracking algorithms by identifying and weighting various types of estimation errors, such as false positives, missed detections, and object swaps, using specific coefficients for each error type to calculate an evaluation value that reflects the algorithm's performance accurately.

Benefits of technology

The system provides a more nuanced evaluation of tracking algorithms by considering the impact of different types of errors, allowing for a more accurate assessment of their performance in real-world scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007582170000008
    Figure 0007582170000008
  • Figure 0007582170000009
    Figure 0007582170000009
  • Figure 0007582170000010
    Figure 0007582170000010
Patent Text Reader

Abstract

To provide an evaluation system capable of appropriately evaluating quality of an algorithm tracking a movable object by estimating a position of the object.SOLUTION: An estimation result determination unit 120 determines, for each object, a correct estimation result or one of a plurality of false estimation types, which indicate types of false estimation results using ground truth data that corresponds to a video image and output data indicating the result of the estimation made on the video image by an algorithm. An evaluation value calculation unit 160 adds false estimation coefficients that correspond to the respective false estimation types and are provided so as to become higher in accordance with a degree of impact of the false estimation type by the number of objects that correspond to the false estimation type and thus calculates an evaluation value of the algorithm based on the total value of added values of the false estimation coefficients obtained for each of the false estimation types.SELECTED DRAWING: Figure 4
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an evaluation system, an evaluation method, and a program. [Background technology]

[0002] There is a technology for tracking movable objects using video data. In relation to this technology, Patent Document 1 discloses a system and method for predictive anomalous behavior detection. The system in Patent Document 1 receives surveillance data, such as video data, and creates and updates a number of predictive models. The system also receives video data related to a moving object and generates a prediction of the moving object's future location based on the generated predictive models. The predicted movement is scored by a scoring engine to determine whether the predicted movement is unsafe or otherwise undesirable. Non-Patent Document 1 also discloses a method for evaluating a tracking system. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2012-518846 [Non-patent literature]

[0004] [Non-Patent Document 1] Jonathon Luiten et. al, "HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking", International Journal of Computer Vision (2021) 129:548-578, https: / / doi.org / 10.1007 / s11263-020-01375-2 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, when a tracking system is used in a specific usage environment such as traffic control, it is required to evaluate whether the tracking system performs well in that usage environment. On the other hand, Patent Document 1 does not disclose a method for evaluating a tracking system. In addition, the technology in Non-Patent Document 1 lists evaluation methods such as MOTA (Multi-Object Tracking Accuracy). However, depending on the usage environment of the tracking system, even if the value (evaluation value) of the evaluation formula listed in Non-Patent Document 1 is high, the performance of the tracking system is not necessarily good. Therefore, it is desirable to appropriately evaluate the quality of an algorithm (tracking system) that estimates the position of a movable object and tracks the object.

[0006] The present invention provides an evaluation system, an evaluation method, and a program that are capable of appropriately evaluating the quality of an algorithm that estimates the position of a movable object and tracks the object. [Means for solving the problem]

[0007] The evaluation system of the present invention is an evaluation system that evaluates the quality of an algorithm that estimates the position of a movable object in a video and tracks the object, and has an estimation result determination unit that determines for each object whether the estimation result is correct or one of a number of error estimation types, which are types of erroneous estimation results, using ground truth data corresponding to the video and output data indicating the estimation result of the algorithm that performed estimation on the video, and an evaluation value calculation unit that adds up error estimation coefficients that correspond to each of the multiple error estimation types and are set to increase according to the degree of influence of the error estimation type, by the number of objects corresponding to the error estimation type, and calculates an evaluation value of the algorithm based on the total value of the added values ​​of the error estimation coefficients obtained for each of the multiple error estimation types.

[0008] In addition, the evaluation method of the present invention is a method for evaluating the quality of an algorithm that estimates the position of a movable object in a video and tracks the object, and uses ground truth data corresponding to the video and output data indicating the estimation result of the algorithm that performed estimation on the video to determine for each object whether the estimation result is correct or one of a number of error estimation types, which are types of erroneous estimation results, and adds up error estimation coefficients that correspond to each of the multiple error estimation types and are set to increase according to the degree of influence of the error estimation type, by the number of objects corresponding to the error estimation type, and calculates an evaluation value of the algorithm based on the total value of the added values ​​of the error estimation coefficients obtained for each of the multiple error estimation types.

[0009] In addition, the program of the present invention is a program that realizes an evaluation method for evaluating the quality of an algorithm that estimates the position of a movable object in a video and tracks the object, and causes a computer to execute the following steps: determining for each object whether the estimation result is correct or one of a plurality of error estimation types, which are types of erroneous estimation results, using ground truth data corresponding to the video and output data indicating the estimation result of the algorithm that performed estimation on the video; and adding up error estimation coefficients that correspond to each of the plurality of error estimation types and are set to increase according to the degree of influence of the error estimation type, by the number of objects corresponding to the error estimation type, and calculating an evaluation value of the algorithm based on the total value of the added values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types.

[0010] In the present invention, the error estimation coefficient is set to be higher as the degree of influence of the corresponding error estimation type is greater. Therefore, the more erroneous estimations with a large degree of influence are made, the lower the evaluation may be. And, it can be said that the quality of an algorithm that makes many erroneous estimations with a large degree of influence is low. Therefore, the present invention makes it possible to appropriately evaluate the quality of an algorithm.

[0011] In addition, preferably, among the multiple error estimation types, the error estimation coefficient corresponding to a first error estimation type relating to the algorithm being unable to estimate the object included in the correct answer data is set higher than the error estimation coefficients corresponding to the other error estimation types. Being configured in this way, the present invention makes it possible to more appropriately evaluate the quality of an algorithm.

[0012] In addition, preferably, among the multiple types of incorrect estimation, the incorrect estimation coefficient corresponding to a second incorrect estimation, which is an incorrect estimation type relating to the algorithm estimating that multiple objects included in the correct answer data are swapped between one timing and the next timing, is set to be higher than the incorrect estimation coefficient corresponding to a third incorrect estimation, which is an incorrect estimation type relating to the algorithm estimating that one object included in the correct answer data is a different object between one timing and the next timing. Being configured in this way, the present invention makes it possible to more appropriately evaluate the quality of an algorithm.

[0013] In a preferred embodiment, the image forming apparatus further comprises a coefficient calculation section for calculating the error estimation coefficient for each estimation timing. By being configured in this manner, the present invention can change the error estimation coefficient in response to changes in the situation, thereby making it possible to more appropriately evaluate the quality of the estimation algorithm.

[0014] In addition, preferably, when erroneous estimations of the same erroneous estimation type are made consecutively for a certain object, the coefficient calculation unit calculates the erroneous estimation coefficient so as to make the erroneous estimation coefficient corresponding to the erroneous estimation type for the object higher. By configuring the present invention in this way, it becomes possible to more appropriately evaluate the quality of an estimation algorithm.

[0015] In addition, preferably, the coefficient calculation unit calculates the error estimation coefficient so as to make the error estimation coefficient corresponding to the first error estimation higher when a first error estimation, which is an error estimation type relating to the algorithm being unable to estimate the object included in the correct answer data, is made consecutively. By configuring the present invention in this way, it becomes possible to more appropriately evaluate the quality of an estimation algorithm.

[0016] In addition, preferably, the coefficient calculation section calculates the error estimation coefficient such that the error estimation coefficient corresponding to the error estimation type related to the object is increased as the speed of the object increases. By configuring the present invention in this way, it becomes possible to more appropriately evaluate the quality of an estimation algorithm.

[0017] In addition, preferably, the coefficient calculation unit calculates the error estimation coefficient so that the greater the number of other objects of the same type as an object whose distance from the object is equal to or less than a predetermined threshold, the lower the error estimation coefficient corresponding to the type of error estimation related to the object. By configuring the present invention in this way, it becomes possible to more appropriately evaluate the quality of an estimation algorithm. Effect of the Invention

[0018] According to the present invention, it is possible to provide an evaluation system, an evaluation method, and a program capable of appropriately evaluating the quality of an algorithm that estimates the position of a movable object and tracks the object. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a diagram showing an evaluation system according to a first embodiment. [Diagram 2] FIG. 11 is a diagram for explaining types of erroneous estimation according to the present embodiment. [Diagram 3]FIG. 11 is a diagram for explaining types of erroneous estimation according to the present embodiment. [Figure 4] FIG. 1 is a diagram illustrating a configuration of an evaluation device according to a first embodiment. [Diagram 5] 4 is a flowchart showing an evaluation method executed by the evaluation system according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating a configuration of an evaluation device according to a second embodiment. [Figure 7] 13 is a flowchart showing an evaluation method executed by the evaluation system according to the second embodiment. [Figure 8] 13 is a flowchart showing an evaluation method executed by the evaluation system according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] (Embodiment 1) Hereinafter, an embodiment of the present invention will be described with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In addition, in each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0021] 1 is a diagram showing an evaluation system 1 according to a first embodiment. The evaluation system 1 includes a tracking engine 20, a video storage device 40, and an evaluation device 100. The tracking engine 20, the video storage device 40, and the evaluation device 100 may be communicatively connected via a wired or wireless network 2.

[0022] The tracking engine 20 is, for example, a computer. The tracking engine 20 operates an estimation algorithm that estimates the position of a movable object (moving object) and tracks the moving object. That is, the estimation algorithm detects an object from a video. The tracking engine 20 inputs a video stored in a video storage device 40 (described later) to the estimation algorithm and operates the estimation algorithm. The estimation algorithm estimates the position and type of an object for each image frame of the input video and outputs the estimation result. The tracking engine 20 transmits the estimation result of the object output from the estimation algorithm to the evaluation device 100 as output data. Note that the estimation algorithm does not need to perform estimation processing on each and every image frame. For example, estimation processing may be performed on even-numbered image frames.

[0023] Alternatively, the tracking engine 20 may transmit output data (estimation results) for each image frame, or may transmit output data (estimation results) for the entire video sequence after the estimation algorithm estimates and tracks objects for the entire video sequence.

[0024] The video storage device 40 is, for example, a storage device. The video storage device 40 stores in advance video used to evaluate the quality of the estimation algorithm. The video storage device 40 stores video and correct answer data in association with each image frame. In other words, the correct answer data is associated with the image frames constituting the video. The correct answer data indicates the position of an object photographed in the corresponding image frame and the type of the object at that position. The type of object is, for example, a pedestrian, a bicycle, a motorcycle, an automobile, a large vehicle, etc. In addition, the position of the object may be a position on the image frame (position of a pixel), or may be position information in the real environment (for example, an intersection, etc.) photographed by the image frame. The position information may be acquired, for example, by a GNSS (Global Navigation Satellite System).

[0025] The tracking engine 20 may be realized by the evaluation device 100 described later. That is, the evaluation device 100 may operate the estimation algorithm. Similarly, the video storage device 40 may be realized by the evaluation device 100 described later. That is, the evaluation device 100 may store the video associated with the correct answer data.

[0026] The evaluation device 100 is, for example, a computer. The evaluation device 100 is, for example, a server. The evaluation device 100 may be realized by, for example, cloud computing. The evaluation device 100 (evaluation system 1) evaluates the quality of an estimation algorithm using a moving image. Specifically, the evaluation device 100 determines, for each object, either a correct estimation (correct estimation) or multiple types of erroneous estimation (erroneous estimation types) using correct answer data corresponding to the moving image and output data indicating the estimation result of the estimation algorithm. A correct estimation (correct estimation result) is an estimation result in which an estimation is made correctly for an object. On the other hand, an erroneous estimation type is a type of an erroneous estimation result. In other words, an erroneous estimation type is a type of an estimation result (erroneous estimation) in which an erroneous estimation is made for an object. In this embodiment, there are multiple erroneous estimation types. Details of the erroneous estimation types will be described later.

[0027] Furthermore, for each of a plurality of types of erroneous estimation, the evaluation device 100 adds up the erroneous estimation coefficients corresponding to the erroneous estimation type by the number of corresponding objects. The evaluation device 100 calculates a total value of the sum of the erroneous estimation coefficients obtained for each of the plurality of types of erroneous estimation. Then, the evaluation device 100 calculates an evaluation value of the estimation algorithm based on the total value. Note that this evaluation value may be a value such that the evaluation of the algorithm is lowered as the total value increases. Details of the evaluation value will be described later.

[0028] 2 and 3 are diagrams for explaining types of erroneous estimation according to the present embodiment. FIG. 2 shows a specific example of an erroneous estimation type. As shown in FIG. 2, in the embodiment, the estimation result includes a correct estimation (true estimation) and erroneous estimation types. A "true estimation (TP: True Positive)" corresponds to a state in which the estimation result in the output of the estimation algorithm corresponds to an object present in the correct answer data. In addition, the erroneous estimation types include "false detection (false tracking)", "oversight", "new false detection", and "substitution".

[0029] A "false positive (FP)" (fourth erroneous estimation) corresponds to a state in which the estimation result at the output of an estimation algorithm does not correspond to any object present in the ground truth data. In other words, a "false positive" corresponds to a state in which an estimation algorithm estimates (detects) an object at a position where the object does not actually exist. In yet another way, a "false positive" is a type of erroneous estimation in which an estimation algorithm estimates the presence of an object that does not actually exist. In yet another way, a "false positive" is a type of erroneous estimation in which an actually existing object is tracked to a different position. Therefore, when an erroneous estimation of a "false positive" occurs, the estimation result by the estimation algorithm is given an ID (Identifier) ​​for a fictitious object or an object that actually exists at a different position at a position that does not exist in the ground truth data.

[0030] A "false negative (FN)" (first erroneous estimation) corresponds to a state in which an object present in the correct answer data does not correspond to any of the estimation results in the output of the estimation algorithm. In other words, a "false negative" corresponds to a state in which an object present in the correct answer data is not estimated in the output of the estimation algorithm. In other words, a "false negative" corresponds to a state in which the estimation algorithm is unable to estimate (detect) an object at the position of an object that actually exists. In yet other words, a "false negative" is a type of erroneous estimation in which the estimation algorithm is unable to estimate (detect) an object included in the correct answer data. Therefore, when a "false negative" erroneous estimation occurs, an ID is not assigned to the object present in the correct answer data in the estimation result by the estimation algorithm.

[0031] A "new erroneous detection (IDSWn: ID Switch New; new ID assignment)" (third erroneous estimation) corresponds to a state in which an estimation algorithm estimates an object present in the correct answer data as a different object in the previous estimation process and the current estimation process. In other words, a "new erroneous detection" corresponds to a state in which an object present in the correct answer data is estimated (detected) as a different object in the estimation result at a certain timing and the estimation result at the next timing in the output of the estimation algorithm. In other words, a "new erroneous detection" corresponds to a state in which an estimation algorithm estimates (detects) an object as a different object at a certain timing and the next timing, when in fact it is. In further words, a "new erroneous detection" is a type of erroneous estimation related to an estimation algorithm estimating that an object included in the correct answer data is a different object at a certain timing and the next timing. Therefore, when an erroneous estimation of a "new erroneous detection" occurs, in the estimation result by the estimation algorithm, a different ID is assigned to an object present in the correct answer data at a certain timing and the next timing.

[0032] "ID Switch Change (IDSWc)" (second erroneous estimation) is an erroneous estimation type related to the estimation algorithm estimating multiple objects included in the correct answer data by switching them between a certain timing and the next timing. In other words, "switching" corresponds to a state in which multiple objects present in the correct answer data are estimated (detected) in a switched manner in the output of the estimation algorithm between the estimation result at a certain timing and the estimation result at the next timing. Therefore, when an erroneous estimation of "switching" occurs, in the estimation result by the estimation algorithm, an ID assigned to each of multiple objects present in the correct answer data at a certain timing is assigned to another object among the multiple objects at the next timing. For example, when object A and object B exist in the correct answer data, an ID assigned to the position of object A at a certain timing is assigned to the position of object B at the next timing. Similarly, an ID assigned to the position of object B at a certain timing is assigned to the position of object A at the next timing.

[0033] FIG. 3 illustrates multiple types of erroneous estimations and correct estimations in a video. FIG. 3 illustrates an estimation result obtained by performing an estimation process on multiple objects included in a video captured near a road intersection. In FIG. 3, object P (P1 to P4) indicates an object included in the correct answer data. Meanwhile, rectangle R (R11, etc.) indicates a position where the estimation algorithm estimates that an object exists. In the example of FIG. 3, estimation is performed for each image frame. Then, t (t=1 to 6) corresponds to each image frame and indicates the timing of estimation in the corresponding image frame.

[0034] In the example shown in FIG. 3, for object P1, at timing t=1 (image frame), the estimation algorithm correctly performs estimation as shown by rectangle R11. At this time, "ID: #01" is assigned to rectangle R11 corresponding to this object P1. Also, for the moving object P1, at timings t=2 to 4 (image frames), the estimation algorithm correctly performs estimation as shown by rectangles R12 to R14, respectively. At this time, "ID: #01" assigned at t=1 is still assigned to rectangles R12 to R14 corresponding to object P1.

[0035] On the other hand, for object P1, the estimation algorithm was unable to make an estimation at the timing (image frame) of t=5. In other words, at t=5, a "missed-notice (FN)" erroneous estimation occurred for object P1. Similarly, at the timing (image frame) of t=6, the estimation algorithm was unable to make an estimation. In other words, at t=6, a "missed-notice (FN)" erroneous estimation occurred for object P1.

[0036] In contrast, at the timing (image frame) of t=5, the estimation algorithm estimates that object P1 is present in a position where object P1 is not present, as shown by rectangle R15. In other words, at t=5, an erroneous estimation of "false detection (FP)" (false tracking) occurs for object P1. Similarly, at the timing (image frame) of t=6, the estimation algorithm estimates that object P1 is present in a position where object P1 is not present, as shown by rectangle R16. In other words, at t=6, an erroneous estimation of "false detection (FP)" occurs for object P1. At this time, the rectangles R15 to R16 continue to be assigned "ID: #01" that was assigned at t=1.

[0037] In the example shown in FIG. 3, the estimation algorithm correctly estimates object P2 at timing t=1 (image frame), as shown by rectangle R21. At this time, "ID: #02" is assigned to rectangle R21 corresponding to object P2. Also, for moving object P2, the estimation algorithm correctly estimates object P2 at timing t=2 (image frame), as shown by rectangle R22. At this time, "ID: #02" assigned at t=1 is still assigned to rectangle R22 corresponding to object P2.

[0038] In contrast, at the timing (image frame) of t=3, the estimation algorithm estimates that object P2 is present in a position where object P2 is not present, as shown by rectangle R23. In other words, at t=3, an erroneous estimation of "false detection (FP)" occurs for object P2. Similarly, at the timing (image frame) of t=4, the estimation algorithm estimates that object P2 is present in a position where object P2 is not present, as shown by rectangle R24. In other words, at t=4, an erroneous estimation of "false detection (FP)" occurs for object P2. At this time, "ID: #02" that was assigned at t=1 continues to be assigned to rectangles R23 to R24.

[0039] On the other hand, for object P2, at the timing (image frame) of t=3, the estimation algorithm estimates that it is a different object from object P2 estimated at t=1 to 2, as shown by rectangle R2'3. That is, at t=3, an erroneous estimation of "new erroneous detection (IDSWn)" occurs for object P2. At this time, rectangle R2'3 is assigned "ID: #12", which is different from "ID: #02" assigned at t=1 and corresponds to object P2. At the timing (image frame) of t=4, the estimation algorithm estimates that it is the same object (object P2) estimated at t=3, as shown by rectangle R2'4. In this case, since the same object P2 is estimated to be the same object in the previous estimation process and the current estimation process, this estimation result is a correct estimation (TP). At this time, rectangle R2'4 is assigned "ID: #12", which is the same ID assigned to rectangle R2'3 at t=3.

[0040] In the example shown in FIG. 3, objects P3 and P4 are moving close to each other. For object P3, the estimation algorithm correctly estimates the object at t=1 (image frame), as shown by rectangle R31. At this time, "ID: #03" is assigned to rectangle R31 corresponding to object P3. For moving object P3, the estimation algorithm correctly estimates the object at t=2 (image frame), as shown by rectangle R32. At this time, "ID: #03" assigned at t=1 is still assigned to rectangle R32 corresponding to object P3.

[0041] Furthermore, for object P4, at timing t=1 (image frame), the estimation algorithm correctly performs estimation, as shown by rectangle R41. At this time, "ID: #04" is assigned to rectangle R41 corresponding to this object P4. Furthermore, for the moving object P4, at timing t=2 (image frame), the estimation algorithm correctly performs estimation, as shown by rectangle R42. At this time, "ID: #04" assigned at t=1 is still assigned to rectangle R42 corresponding to object P4.

[0042] In contrast, at the timing (image frame) of t=3, the estimation algorithm estimates that object P4 exists at the position of object P3, as shown by rectangle R43. Also, at the timing (image frame) of t=3, the estimation algorithm estimates that object P3 exists at the position of object P4, as shown by rectangle R33. That is, at t=3, the estimation algorithm erroneously estimates object P3 as object P4, and erroneously estimates object P4 as object P3. Therefore, at t=3, an erroneous estimation of "swap (IDSWc)" occurs for object P3 and object P4. At this time, "ID: #04" that was assigned to object P4 at t=1 to 2 is assigned to rectangle R43. Also, "ID: #03" that was assigned to object P3 at t=1 to 2 is assigned to rectangle R33.

[0043] Here, as shown in FIG. 2, the error estimation coefficient for "false detection (FP)" is α. The error estimation coefficient for "false detection (FN)" is β. The error estimation coefficient for "new false detection (IDSWn)" is γ. The error estimation coefficient for "replacement (IDSWc)" is δ. Here, the error estimation coefficient functions as a weight for each of multiple error estimation types in the evaluation formula used to evaluate the estimation algorithm. The error estimation coefficient is set to be different for each of the multiple error estimation types. The more an error estimation with a large error estimation coefficient occurs among the multiple error estimation types, the lower the evaluation value of the evaluation formula for the estimation algorithm may be.

[0044] Here, when the estimation algorithm is used in traffic control such as an autonomous vehicle control system, the estimation algorithm estimates the position of an object such as a pedestrian and tracks the object such as a pedestrian. Then, the moving direction and speed of the object such as a pedestrian can be estimated using the estimation results of the estimation algorithm at multiple timings (image frames). Specifically, the speed of the object may be calculated using the moving distance calculated from the difference (distance) between the positions of the object in each image frame and the frame rate. That is, the speed of the object may be calculated using the moving distance of the object and the time from when an image frame is acquired to when the next image frame is acquired. More specifically, the position of a pixel on each image frame may be associated with the position in the real environment where the image was captured using the angle of view of a camera that captured the moving image. Then, the position (real position) of the object in the real environment may be estimated from the position of a pixel corresponding to the bottom end of the object (such as the foot of a pedestrian). Then, the difference between the real position of the object in one image frame and the real position of the object in the next image frame may be calculated as the moving distance of the object. Then, the speed of the object may be estimated based on the moving distance and the frame rate. The moving direction of the object may also be calculated by calculating the direction from the actual position of the object in the previous image frame to the actual position of the object in the current image frame.

[0045] In addition, by estimating the moving direction and speed of an object such as a pedestrian, it is possible to predict the possibility that the object will arrive at a specific position (e.g., an intersection or a roadway) and the arrival time if it does arrive. Therefore, using the estimation results of the estimation algorithm, it is possible to control a traveling vehicle to avoid colliding with the object.

[0046] When the estimation algorithm is used in a traffic control environment in this way, the risks of each of the above-mentioned types of erroneous estimation occurring may differ from one another. In this embodiment, the higher the risk of each type of erroneous estimation, the higher the value of the corresponding erroneous estimation coefficient is set. In other words, the higher the erroneous estimation coefficient is set, the greater the degree (seriousness) of the influence (impact) of the corresponding erroneous estimation type. In other words, the erroneous estimation coefficient is set according to the degree (seriousness) of the influence (impact) of the corresponding erroneous estimation type.

[0047] When a "false detection" occurs, control such as collision avoidance may be performed assuming that an object that does not actually exist is present. In this case, excessive (unnecessary) control may be performed, but in terms of traffic control, this control is on the safe side. Therefore, the risk of a "false detection" occurring is small. In other words, the degree of impact when a "false detection" occurs is small. Therefore, the error estimation coefficient α corresponding to a "false detection" may be set relatively low.

[0048] When a "missed sight" occurs, it is estimated that an object does not exist in a position where the object actually exists, so there is a possibility that control to avoid a collision with the object will not be performed. In this case, in terms of traffic control, there is a possibility that necessary control to avoid a collision will not be performed. Therefore, the risk of a "missed sight" is high. In other words, the degree of impact when a "missed sight" occurs is high. Therefore, the misestimation coefficient β corresponding to a "missed sight" can be set relatively high.

[0049] When "switching" occurs, it is estimated that some object is present at the position where an object actually exists, so the degree of impact (risk) is smaller than in the case of "overlooking". However, if the IDs are switched midway, as explained below, there is a possibility that the moving direction and speed of each of the multiple objects cannot be estimated with high accuracy.

[0050] For example, in the example of FIG. 3, from the actual movement trajectory of the object P3 from t=1 to 3, the object P3 may be predicted to actually move in the direction indicated by the arrow A3. However, when the estimation result of the estimation algorithm is used, from the movement trajectories of the rectangles R31, R32, and R33 to which "ID: #03" is added, the object corresponding to this "ID: #03" may be predicted to move in the direction indicated by the dashed arrow A3'. Similarly, from the actual movement trajectory of the object P4 from t=1 to 3, the object P4 may be predicted to actually move in the direction indicated by the arrow A4. However, when the estimation result of the estimation algorithm is used, from the movement trajectories of the rectangles R41, R42, and R43 to which "ID: #04" is added, the object corresponding to this "ID: #04" may be predicted to move in the direction indicated by the dashed arrow A4'.

[0051] Similarly, as described below, the moving speeds of the objects P3 and P4 may not be predicted with high accuracy. In particular, when there is a large difference between the actual moving speed (kph) of the object P3 and the actual moving speed (kph) of the object P4, the moving speeds of both objects may not be predicted with high accuracy.

[0052] Specifically, in the example of FIG. 3, both objects P3 and P4 move at a roughly constant speed, but object P3 moves faster than object P4. In this case, the difference (movement distance) between the position of object P3 at t=3 and the position of object P3 at t=2 is actually roughly the same as the difference (movement distance) between the position of object P3 at t=2 and the position of object P3 at t=1. On the other hand, the difference between the position of rectangle R33 and the position of rectangle R32 may be smaller than the difference between the position of rectangle R32 and the position of rectangle R31. Therefore, the speed of the object corresponding to "ID:#03" (object P3) may be estimated to have suddenly slowed down. Therefore, the speed of the object corresponding to the subsequent "ID:#03" may also be predicted to be slower than the actual speed of object P3.

[0053] Similarly, the difference between the position of object P4 at t=3 and the position of object P4 at t=2 is approximately the same as the difference between the position of object P4 at t=2 and the position of object P4 at t=1. On the other hand, the difference between the position of rectangle R43 and the position of rectangle R42 may be greater than the difference between the position of rectangle R42 and the position of rectangle R41. Therefore, there is a risk that the speed of the object corresponding to "ID:#04" (object P4) will be estimated to have suddenly increased. Therefore, there is a risk that the speed of the object corresponding to the subsequent "ID:#04" will also be predicted to be faster than the actual speed of object P4.

[0054] As described above, in traffic control, by estimating the moving direction and moving speed of an object, it is possible to predict how many seconds it will take for the object to arrive at a specific location such as an intersection. This makes it possible to perform collision avoidance control. However, if the moving direction and moving speed of an object cannot be estimated with high accuracy, the arrival time described above may not be predicted with high accuracy. As a result, there is a risk that collision avoidance control may not be performed appropriately. Therefore, the degree of impact (risk) when a "switch" occurs is somewhat large.

[0055] When a "new false positive" occurs, it is estimated that some object is present at the location where an object actually exists, so the degree of impact (risk) is smaller than in the case of a "missed detection". However, because an object cannot be tracked correctly, there is a risk that the direction and speed of movement of the object cannot be determined with high accuracy, as in the case of a "replacement". On the other hand, after a new ID is assigned, there is a possibility that a correct estimation can be made, so the degree of impact (risk) of a "new false positive" is smaller than the degree of impact (risk) of a "replacement".

[0056] For example, in the example of FIG. 3, different IDs are assigned to object P2 at t=2 and t=3. Therefore, the moving direction and moving speed of object P2 cannot be estimated from the difference between the position of object P2 (rectangle R2'3) at t=3 and the position of object P2 (rectangle R22) at t=2. On the other hand, the same ID (ID: #12) is assigned to object P2 at t=3 and t=4. Therefore, it is possible to appropriately estimate the moving direction and moving speed of object P2 from the difference between the position of object P2 (rectangle R2'4) at t=4 and the position of object P2 (rectangle R2'3) at t=3. Also, it is assumed that the moving direction and moving speed of object P2 are approximately constant. In this case, the moving direction and moving speed of object P2 (ID: #02) estimated from the moving trajectory at t=1-2 may be approximately the same as the moving direction and moving speed of object P2 (ID: #12) estimated from the moving trajectory at t=3-4. Therefore, the degree of impact when a "new false detection" occurs is smaller than the degree of impact of a "replacement". Therefore, the degree of impact (risk) when a "new false detection" occurs is somewhat small.

[0057] From the above, the error estimation coefficients α, β, γ, and δ are set as shown in the following formula (1): In the first embodiment, it is assumed that the error estimation coefficients are constant (unchanged from the initial values).

number

[0058] That is, since the degree of influence of "missed detection (FN)" is the largest, the error estimation coefficient β corresponding to "missed detection (FN)" (first error estimation) is set higher than the error estimation coefficients (α, γ, δ) corresponding to the other types of error estimation. Also, the degree of influence of "replacement (IDSWc)" is larger than "new false detection (IDSWn)". Therefore, the error estimation coefficient δ corresponding to "replacement (IDSWc)" (second error estimation) is set higher than the error estimation coefficient γ corresponding to "new false detection (IDSWn)". Also, γ+δ=1.

[0059] In the first embodiment, the evaluation formula used to evaluate the estimation algorithm in the image frame acquired at the timing t is expressed as the following formula (2).

number

[0060] In equation (2), GT(t) indicates the number of actual objects included in the ground truth data corresponding to the image frame at time t. FP(t) indicates the number of objects (rectangles) erroneously estimated to be "false positive (FP)" for the image frame at time t. FN(t) indicates the number of objects erroneously estimated to be "missed (FN)" for the image frame at time t. IDSWn(t) indicates the number of objects (rectangles) erroneously estimated to be "new false positive (IDSWn)" for the image frame at time t. IDSWc(t) indicates the number of objects (rectangles) erroneously estimated to be "replaced (IDSWc)" for the image frame at time t.

[0061] The numerator of the fraction in the second term on the right side of formula (2) corresponds to the sum of the values ​​(α×FP(t), β×FN(t), γ×IDSWn(t), and δ×IDSWc(t)) obtained by multiplying the number of objects corresponding to each of the multiple error estimation types by the error estimation coefficient. Here, it can be said that the value obtained by multiplying the number of objects corresponding to each of the multiple error estimation types by the error estimation coefficient is a value obtained by adding the error estimation coefficient corresponding to the number of objects corresponding to each of the multiple error estimation types. For example, "β×FN(t)" corresponds to a value (Σβ) obtained by adding the error estimation coefficient β by the number of objects FN(t) corresponding to "oversight (FN)". Therefore, this sum is the sum of the sum of the error estimation coefficients obtained for each of the multiple error estimation types by adding the error estimation coefficient corresponding to the number of objects corresponding to each of the multiple error estimation types. Then, as shown in formula (2), the larger this sum is, the smaller the evaluation value F(t) is. In other words, the larger this sum is, the lower the evaluation of the estimation algorithm is.

[0062] Here, in Non-Patent Document 1, the evaluation value of the estimation algorithm is calculated by setting all the erroneous estimation coefficients in Equation (2) to 1. Therefore, in Non-Patent Document 1, the evaluation value is calculated without considering the risk of each of the multiple erroneous estimation types, that is, the degree of impact (severity) of the erroneous estimation. Therefore, even if the technology in Non-Patent Document 1 is applied, it does not necessarily mean that the estimation algorithm will not make erroneous estimations with a high risk (degree of impact) just because it is highly evaluated. Therefore, there is a risk that the quality of the estimation algorithm cannot be appropriately evaluated. In other words, for example, when a certain estimation algorithm has a large number of objects corresponding to "failure to notice (FN)" and another estimation algorithm has a large number of objects corresponding to "false positive (FP)", the quality of both estimation algorithms may be evaluated to be about the same.

[0063] In contrast, in the first embodiment, the error estimation coefficient is set to be different for each of the multiple error estimation types. Specifically, the error estimation coefficient is set to be higher as the degree of influence of the corresponding error estimation type becomes greater. Therefore, the more the error estimation with a large degree of influence is performed, the smaller the evaluation value may be. In other words, the more the error estimation with a large degree of influence is performed, the lower the evaluation of the estimation algorithm may be. Here, it can be said that the quality of an estimation algorithm that performs many error estimations with a large degree of influence is low. Therefore, in the first embodiment, it is possible to appropriately evaluate the quality of the estimation algorithm. Note that "a high error estimation coefficient" is not limited to simply having a large numerical value of the error estimation coefficient. The numerical value of the error estimation coefficient itself may become smaller as the degree of influence becomes greater. In this case, "a high error estimation coefficient" corresponds to a small numerical value of the error estimation coefficient. In addition, in this case, the smaller the "total value of the sum of the error estimation coefficients obtained for each of the multiple error estimation types by adding the corresponding error estimation coefficients by the number of objects corresponding to each of the multiple error estimation types" is, the lower the evaluation of the estimation algorithm may be.

[0064] In addition, in the first embodiment, the error estimation coefficient β corresponding to "missed occurrence (FN)" (first error estimation) among the multiple error estimation types is set higher than the error estimation coefficients corresponding to the other error estimation types. As described above, the degree of influence of "missed occurrence (FN)" is the largest among the multiple error estimation types. Therefore, by setting the error estimation coefficient β higher than the error estimation coefficients corresponding to the other error estimation types, it is possible to more appropriately evaluate the quality of the estimation algorithm. In other words, it is possible to lower the evaluation of an estimation algorithm that frequently performs "missed occurrence (FN)" which has a large degree of influence.

[0065] In addition, in the first embodiment, the error estimation coefficient δ corresponding to "replacement (IDSWc)" (second error estimation) is set higher than the error estimation coefficient γ corresponding to "new error detection (IDSWn)" (third error estimation). As described above, the degree of influence of "replacement (IDSWc)" is greater than the degree of influence of "new error detection (IDSWn)". Therefore, by setting the error estimation coefficient δ higher than the error estimation coefficient γ, it becomes possible to more appropriately evaluate the quality of the estimation algorithm. In other words, it is possible to lower the evaluation of an estimation algorithm that frequently performs "replacement (IDSWc)", which has a greater degree of influence than "new error detection (IDSWn)".

[0066] Fig. 4 is a diagram showing a configuration of the evaluation device 100 according to the first embodiment. As shown in Fig. 4, the evaluation device 100 has, as main hardware components, a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF; Interface). The control unit 102, the storage unit 104, the communication unit 106, and the interface unit 108 are connected to each other via a data bus or the like. Note that the tracking engine 20 and the video storage device 40 may also have the hardware configuration of the evaluation device 100 shown in Fig. 4.

[0067] The control unit 102 is a processor such as a CPU (Central Processing Unit). The control unit 102 has a function as a calculation device that performs control processing and calculation processing. The control unit 102 may have multiple processors. The storage unit 104 is a storage device such as a memory or a hard disk. The storage unit 104 is, for example, a ROM (Read Only Memory) or a RAM (Random Access Memory). The storage unit 104 has a function for storing a control program and a calculation program executed by the control unit 102. That is, the storage unit 104 (memory) stores one or more instructions. The storage unit 104 also has a function for temporarily storing processing data and the like. The storage unit 104 may include a database. The storage unit 104 may also have multiple memories.

[0068] The communication unit 106 performs processing required for communicating with other devices such as the tracking engine 20 or the video storage device 40 via a network. The communication unit 106 may include a communication port, a router, a firewall, etc. The interface unit 108 (IF; Interface) is, for example, a user interface (UI). The interface unit 108 has an input device such as a keyboard, a touch panel, or a mouse, and an output device such as a display or a speaker. The interface unit 108 may be configured such that the input device and the output device are integrated, such as a touch screen (touch panel). The interface unit 108 accepts data input operations by a user (operator) and outputs information to the user.

[0069] The evaluation device 100 according to the first embodiment has, as its components, a correct answer data acquisition unit 112, an output data acquisition unit 114, an estimation result determination unit 120, a parameter storage unit 130, an evaluation value calculation unit 160, and an evaluation value output unit 162. Each of the above-mentioned components can be realized, for example, by executing a program under the control of the control unit 102. More specifically, each of the components can be realized by the control unit 102 executing a program (instruction) stored in the storage unit 104. Each of the components may be realized by recording a necessary program in an arbitrary non-volatile recording medium and installing it as necessary. Each of the components may be realized by any combination of hardware, firmware, and software, without being limited to being realized by software by a program. Each of the components may be realized by using an integrated circuit programmable by a user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, a program consisting of each of the above-mentioned components may be realized by using this integrated circuit. The same applies to the second embodiment described later.

[0070] The correct data acquisition unit 112 acquires correct data corresponding to the video on which the estimation algorithm has performed estimation processing and tracking processing from the video storage device 40. The correct data acquisition unit 112 may acquire correct data for each image frame constituting the video. The output data acquisition unit 114 acquires output data resulting from the estimation processing and tracking processing of the estimation algorithm from the tracking engine 20. The output data acquisition unit 114 may acquire output data for each image frame constituting the video.

[0071] The estimation result determination unit 120 uses the acquired correct answer data and output data to determine the estimation result for each image frame. Specifically, the estimation result determination unit 120 collates the correct answer data with the output data to determine whether each object is a correct estimation result or one of a plurality of erroneous estimation types. More specifically, the estimation result determination unit 120 determines whether the estimation result for each object (rectangle) is one of the above-mentioned "correct estimation (TP)", "false detection (FP)", "missed detection (FN)", "new false detection (IDSWn)", and "replacement (IDSWc)". More specifically, the estimation result determination unit 120 determines the estimation result by determining whether the type and position of the object included in the correct answer data are correctly estimated in the output data. In addition, the estimation result determination unit 120 determines the estimation result by determining whether the same ID is assigned to the same object included in the correct answer data in a series of image frames.

[0072] In the example of Fig. 3, for the image frame at t=1, the position of rectangle R11 corresponds to the position of object P1, so the position of object P1 included in the correct answer data is correctly estimated. Therefore, the estimation result determination unit 120 determines that the estimation result of object P1 is a "correct estimation (TP)". Similarly, the estimation result determination unit 120 also determines that the estimation results of other objects P2 to P4 are also "correct estimation (TP)".

[0073] Furthermore, for the image frame at t=2, the position of the rectangle R12 corresponds to the position of the object P1, so the position of the object P1 included in the correct answer data is correctly estimated. Furthermore, the ID (#01) given to the rectangle R12 is the same as the ID given to the rectangle R11 at t=1. Therefore, the estimation result determination unit 120 determines that the estimation result of the object P1 is a "correct estimation (TP)". Similarly, the estimation result determination unit 120 also determines that the estimation results of the other objects P2 to P4 are also "correct estimation (TP)". Furthermore, for the image frames at t=3 to 4, the estimation result determination unit 120 also determines that the estimation result of the object P1 is a "correct estimation (TP)" in the same manner as above.

[0074] Also, in the image frame at t=3, the position of rectangle R2'3 corresponds to the position of object P2. However, the ID (#12) assigned to rectangle R2'3 is different from the ID (#02) assigned to rectangle R22 corresponding to object P2 at t=2. Therefore, the estimation result determination unit 120 determines that the estimation result of rectangle R2'3 corresponding to object P2 is "new erroneous detection (IDSWn)". On the other hand, in the image frame at t=4, the position of rectangle R2'4 corresponds to the position of object P2, and the ID (#12) assigned to rectangle R2'4 is the same as the ID (#12) assigned to rectangle R2'3 corresponding to object P2 at t=3. Therefore, the estimation result determination unit 120 determines that the estimation result of object P2 (rectangle R2'4) is "correct estimation (TP)".

[0075] Furthermore, for the image frame at t=3, it is estimated that an object P2 exists at the position of a rectangle R23, but no object exists at the position of the rectangle R23. Therefore, the estimation result determination unit 120 determines that the estimation result of the rectangle R23 near the object P2 is a "false positive (FP)". The same is true for the image frame at t=4.

[0076] Furthermore, in the image frame at t=3, the position of rectangle R33 corresponds to the position of object P4. Here, the ID (#03) assigned to rectangle R33 is different from the ID (#04) assigned to rectangle R42 corresponding to object P4 at t=2, and is the same as the ID (#03) assigned to rectangle R32 corresponding to object P3 at t=2. Therefore, the inference result determination unit 120 determines that the inference result of object P4 is "switched (IDSWc)".

[0077] Similarly, in the image frame at t=3, the position of rectangle R43 corresponds to the position of object P3. Here, the ID (#04) assigned to rectangle R43 is different from the ID (#03) assigned to rectangle R32 corresponding to object P3 at t=2, and is the same as the ID (#04) assigned to rectangle R42 corresponding to object P4 at t=2. Therefore, the inference result determination unit 120 determines that the inference result of object P3 is "switched (IDSWc)".

[0078] In addition, for the image frame at t=5, it is estimated that an object P1 exists at the position of a rectangle R15, but no object exists at the position of the rectangle R15. Therefore, the estimation result determination unit 120 determines that the estimation result of the rectangle R15 near the object P1 is a "false positive (FP)". The same is true for the image frame at t=6.

[0079] Also, in the image frame at t=5, there is no rectangle at the position corresponding to the object P1. Therefore, the estimation result determination unit 120 determines that the estimation result of the object P1 is “missed detection (FN).” The same is true for the image frame at t=6.

[0080] Furthermore, the estimation result determination unit 120 counts the number of objects (rectangles) corresponding to the correct estimation and each incorrect estimation type for each image frame. Specifically, the estimation result determination unit 120 counts the number of objects determined to be "correct estimation (TP)" for each image frame. Similarly, the estimation result determination unit 120 counts the number of objects determined to be "missed detection (FN)" for each image frame. Similarly, the estimation result determination unit 120 counts the number of objects determined to be "replaced (IDSWc)" for each image frame. Furthermore, the estimation result determination unit 120 counts the number of rectangles determined to be "falsely detected (FP)" for each image frame. Similarly, the estimation result determination unit 120 counts the number of rectangles determined to be "newly detected false detection (IDSWn)" for each image frame.

[0081] In the example of Fig. 3, in the image frame at t = 3, the estimation result determination unit 120 counts the number of "correct estimations (TP)" as 1. Similarly, in the image frame at t = 3, the estimation result determination unit 120 counts the number of "false detections (FP)" as 1, the number of "new false detections (IDSWn)" as 1, and the number of "replacements (IDSWc)" as 2. Also, in the image frame at t = 5, the estimation result determination unit 120 counts the number of "false detections (FP)" as 1, and the number of "false negatives (FN)" as 1.

[0082] The parameter storage unit 130 stores parameters used when calculating the evaluation value. In the first embodiment, the parameter storage unit 130 stores the above-mentioned error estimation coefficients α, β, γ, and δ. The parameter storage unit 130 may also temporarily store the evaluation value F(t) calculated for each image frame by a process described later.

[0083] The evaluation value calculation unit 160 calculates an evaluation value of the estimation algorithm. Specifically, the evaluation value calculation unit 160 calculates the evaluation value using the evaluation formula shown in formula (2). The evaluation value calculation unit 160 calculates an evaluation value for each image frame (timing t). Then, when the evaluation values ​​for all image frames have been calculated, the evaluation value calculation unit 160 calculates an overall evaluation value F, which is an evaluation value for the entire video of the estimation algorithm, by adding up the evaluation values ​​calculated for all image frames as shown in the following formula (3). For example, when the estimation algorithm performs estimation processing (tracking processing) on ​​a 5-minute video with a frame rate of 10 fps, the evaluation value calculation unit 160 calculates the overall evaluation value F by adding up the evaluation values ​​obtained for 10 x 60 x 5 = 3000 image frames.

number

[0084] The evaluation value output unit 162 outputs the evaluation value calculated by the evaluation value calculation unit 160. The evaluation value output unit 162 may, for example, cause the evaluation value to be displayed on a display of the interface unit 108. The evaluation value output unit 162 may also, for example, control the communication unit 106 to cause the evaluation value to be displayed on a display of another device (such as a user terminal). The evaluation value output unit 162 may output the evaluation value by voice or the like. The evaluation value output unit 162 may output an overall evaluation value F, or may output an evaluation value F(t) for each image frame.

[0085] Fig. 5 is a flowchart showing an evaluation method executed by the evaluation system 1 according to the first embodiment. The process shown in Fig. 5 may be mainly executed by the evaluation device 100. First, the tracking engine 20 performs an estimation process using an estimation algorithm (step S100). Note that after this estimation process (S100) is executed for all image frames, the processes from S101 onwards may be executed.

[0086] The evaluation device 100 sets t=1 (step S101) and processes the t-th image frame (image frame at timing t) (step S102). That is, the evaluation device 100 first processes the first image frame. The evaluation device 100 then performs the processes of S104 to S150 described below on the first image frame, and if there is an unprocessed image frame (YES in S160), it increments t by 1 (step S162) and processes the second image frame (S102). Thereafter, the evaluation device 100 similarly performs the processes of S104 to S150 on the t-th image frame.

[0087] The correct data acquisition unit 112 acquires correct data for the t-th image frame (step S104). The output data acquisition unit 114 acquires output data related to the estimation process executed for the t-th image frame (step S106). The estimation result determination unit 120 determines the estimation result for the t-th image frame as described above (step S110). Specifically, the estimation result determination unit 120 determines whether the object and the rectangle are “correct estimation (TP)”, “false detection (FP)”, “false detection (FN)”, “new false detection (IDSWn)”, or “replacement (IDSWc)” for the t-th image frame. The estimation result determination unit 120 counts the number of objects for each type of false estimation as described above (step S112). The evaluation value calculation unit 160 calculates the evaluation value of the estimation algorithm for the t-th image frame using the evaluation formula shown in Equation (2) (step S150).

[0088] Then, the evaluation device 100 determines whether or not there is an unprocessed image frame (step S160). If there is an unprocessed image frame (YES in S160), the evaluation device 100 increments t by 1 (step S162). Then, the process flow returns to S102, and the evaluation device 100 processes the next image frame (S102). Then, the processes of S104 to S150 are repeated.

[0089] On the other hand, if there are no unprocessed image frames (NO in S160), the evaluation value calculation unit 160 sums up the evaluation values ​​obtained for all image frames as shown in equation (3) (step S164), and the evaluation value output unit 162 outputs the evaluation value of the entire video (step S166).

[0090] (Embodiment 2) Next, a second embodiment will be described. The configuration of the evaluation system 1 according to the second embodiment is substantially similar to the configuration of the evaluation system 1 according to the first embodiment shown in Fig. 1, and therefore the description will be omitted. The evaluation device 100 according to the second embodiment differs from the evaluation device 100 according to the first embodiment in that the error estimation coefficient differs for each object and can change for each image frame.

[0091] 6 is a diagram showing a configuration of the evaluation device 100 according to the second embodiment. As in the first embodiment, the evaluation device 100 according to the second embodiment has, as main hardware components, a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108. The evaluation device 100 according to the second embodiment also has, as components, a correct answer data acquisition unit 112, an output data acquisition unit 114, an estimation result determination unit 120, a parameter storage unit 130, a coefficient calculation unit 240, an evaluation value calculation unit 260, and an evaluation value output unit 162. The coefficient calculation unit 240 has a successive erroneous estimation calculation unit 242, an object group calculation unit 244, and a speed calculation unit 246.

[0092] The functions of the correct data acquisition unit 112, the output data acquisition unit 114, the estimation result determination unit 120, the parameter storage unit 130, and the evaluation value output unit 162 are substantially similar to those in the first embodiment, and therefore description thereof will be omitted. The parameter storage unit 130 may store constants used in calculating the erroneous estimation coefficient, such as a in equation (4) and c in equation (6) described later. The parameter storage unit 130 may also temporarily store the erroneous estimation coefficient calculated for each object (rectangle) for each image frame by processing described later.

[0093] The coefficient calculation unit 240 calculates an erroneous estimation coefficient, which is a variable, for each image frame. That is, the coefficient calculation unit 240 calculates an erroneous estimation coefficient for each estimation timing. Furthermore, the coefficient calculation unit 240 calculates an erroneous estimation coefficient for each object (rectangle) determined to be one of a plurality of erroneous estimation types in an image frame. Note that the processes of the continuous erroneous estimation calculation unit 242, the object group calculation unit 244, and the speed calculation unit 246 for a certain object may be executed in parallel with each other, or may be executed sequentially. When executed in parallel with each other, the coefficient calculation unit 240 may set the sum or average value of the values ​​calculated by these processes as the erroneous estimation coefficient for the object. The case of sequential execution will be described with reference to FIG. 8 described later.

[0094] The state of the object corresponding to each erroneous estimation type may differ for each object even if the erroneous estimation type is the same. And, even if the erroneous estimation type is the same, the degree of the influence of the erroneous estimation may differ depending on the state of the object. Furthermore, the state of the object corresponding to each erroneous estimation type changes for each image frame. And, the degree of the influence of the erroneous estimation also changes depending on the change in the state of the object. Also, the degree of the influence of the erroneous estimation of each erroneous estimation type may change depending on the occurrence of the erroneous estimation in the previous image frame. In this way, the degree of the influence of the erroneous estimation of each erroneous estimation type is not necessarily constant throughout the video. Therefore, the coefficient calculation unit 240 according to the second embodiment calculates the erroneous estimation coefficient for each image frame. Thereby, the erroneous estimation coefficient can be changed depending on the change in the situation. Therefore, it is possible to more appropriately evaluate the quality of the estimation algorithm.

[0095] The consecutive error estimation calculation unit 242 calculates an error estimation coefficient so that, when an error estimation of the same error estimation type is continuously made for a corresponding object, the error estimation coefficient corresponding to the error estimation type for the object is increased. Specifically, the consecutive error estimation calculation unit 242 calculates an error estimation coefficient so that the error estimation coefficient corresponding to the error estimation type for the object is increased as the number of consecutive error estimations of the same error estimation type for the corresponding object increases.

[0096] More specifically, when a certain object is erroneously estimated as a "missed no attention (FN)", the consecutive error estimation calculation unit 242 calculates the error estimation coefficient by using the following formula (4). Note that the consecutive error estimation calculation unit 242 may also calculate error estimation coefficients corresponding to other types of error estimation in the same manner as formula (4).

number

[0097] Here, i is the index of the object that is "missed" in the image frame. i (t-1) indicates the type of erroneous estimation determined for object i in the image frame at timing (t-1). Also, a is a multi-additive constant and is a constant greater than 0. Note that a may differ depending on the type of erroneous estimation to which Equation 4 is applied. For example, the value of a when calculating β may be greater than the value of a when calculating α. Also, β0 is the initial value of the erroneous estimation coefficient β. Also, β i (t) is the value of the misestimation coefficient β corresponding to object i in the image frame at time t.

[0098] As shown in formula (4), if object i was determined to be a "missed miss (FN)" at the previous timing (t-1) and is also determined to be a "missed miss (FN)" at the current timing t, the previous value of β is multiplied by (1+a) to become the current value of β. Here, (1+a) is greater than 1. Therefore, the error estimation coefficient β is higher than the previous value. Furthermore, each time object i is determined to be a "missed miss (FN)" in succession, the error estimation coefficient β is multiplied by (1+a). Therefore, the more times the object i is determined to be a "missed miss (FN)" in succession, the higher the error estimation coefficient β becomes. Note that formula (4) is an example showing the calculation of the consecutive error estimation calculation unit 242. The consecutive error estimation calculation unit 242 may calculate the error estimation coefficient using another formula that increases the error estimation coefficient when the same error estimation is made consecutively for a certain object.

[0099] In the example of FIG. 3, object P1 is determined to be a false negative (FN) at t=5, and is also determined to be a false negative (FN) at the next timing t=6. Therefore, the misestimation coefficient corresponding to object P1 is β i (5) = β0, and at t = 6, β i (6) is calculated as β0×(1+a).

[0100] Even if an erroneous estimation is made for a certain object for only a moment, if a correct estimation is made at the next timing, the object can be detected and tracked appropriately. Therefore, the degree of the impact is not so large. On the other hand, if erroneous estimation occurs continuously, there is a risk that the object cannot be detected and tracked appropriately during that time. And, in traffic control, etc., if a period during which an object cannot be detected and tracked appropriately becomes long, there is a risk that collision avoidance control for that object cannot be performed appropriately during that period. Therefore, the degree of the impact (risk) increases. Therefore, it can be said that the quality of an estimation algorithm that performs such an estimation process is low. In other words, it is desirable for an estimation algorithm not to make erroneous estimations for a certain object continuously.

[0101] In contrast, the evaluation device 100 according to the second embodiment calculates an error estimation coefficient so that, when an error estimation of the same type is made consecutively for a certain object, the error estimation coefficient corresponding to the error estimation type for the object is increased. This makes it possible to increase the error estimation coefficient when the degree of influence increases due to consecutive erroneous estimations. Therefore, it is possible to lower the evaluation of the quality of an estimation algorithm that performs an estimation process that increases the degree of influence, such as consecutive erroneous estimations. In other words, it is possible to raise the evaluation of the quality of an estimation algorithm that is unlikely to cause consecutive erroneous estimations. Therefore, it is possible to more appropriately evaluate the quality of an estimation algorithm.

[0102] Furthermore, if a certain object is repeatedly erroneously estimated as "missed no" (FN), the object may not be detected in the subsequent image frames, and therefore the object may not be tracked. In other words, the object may be completely missed. In that case, there is a risk that collision avoidance control for the object cannot be performed at all. Thus, the degree of impact is quite large. Therefore, the quality of an estimation algorithm that repeatedly erroneously estimates "missed no" (FN) may be considered to be quite low. For example, some estimation algorithms may repeatedly erroneously estimate "missed no" (FN) for pedestrians wearing clothing of a certain color.

[0103] In contrast, the evaluation device 100 according to the second embodiment calculates an error estimation coefficient so as to increase the error estimation coefficient β when an erroneous estimation (first erroneous estimation) of "missed detection (FN)" is made consecutively. This makes it possible to increase the error estimation coefficient β when the degree of influence increases considerably due to consecutive erroneous estimations of "missed detection (FN)". Therefore, it is possible to lower the evaluation of the quality of an estimation algorithm that makes consecutive erroneous estimations of "missed detection (FN)". This makes it possible to more appropriately evaluate the quality of an estimation algorithm.

[0104] The object group calculation unit 244 calculates the error estimation coefficient so that the more other objects whose distance from a certain object is equal to or less than a predetermined threshold, the lower the error estimation coefficient corresponding to the error estimation type for the object. Here, "other objects" are objects of the same type as the object. In other words, when there are many other objects of the same type around an object, the object group calculation unit 244 lowers the error estimation coefficient below the initial value. For example, when a certain object is a "pedestrian" and there are many other "pedestrians" around the object, the error estimation coefficient for the object becomes lower.

[0105] Specifically, for example, when a "missed detection (FN)" erroneous estimation is made for a certain object i, the object group calculation unit 244 calculates the erroneous estimation coefficient by using the following formula (5). Note that the object group calculation unit 244 may also calculate the erroneous estimation coefficients corresponding to other erroneous estimation types in the same manner as formula (5).

number

[0106] Here, n is the number of other objects of the same type as the object i whose distance from the object i is equal to or less than a threshold. From formula (5), the larger n is, the lower the error estimation coefficient β is. Note that formula (5) is an example showing the calculation of the object group calculation unit 244. The object group calculation unit 244 may calculate the error estimation coefficient using another formula such that the error estimation coefficient becomes lower as the number n of other objects is larger.

[0107] As described above, the evaluation device 100 according to the second embodiment calculates the error estimation coefficient so that the more the number of other objects whose distance from a certain object is equal to or less than a predetermined threshold, the lower the error estimation coefficient corresponding to the error estimation type for the object. Here, when an object of the same type as a certain object exists around the object, the plurality of objects can be treated as an "object group". Therefore, even if one or more of the objects cannot be appropriately detected (for example, "missed"), the degree of influence is not so large because it is only necessary to perform a collision avoidance control for the object group. In other words, it can be said that the degree of influence when such an error estimation occurs is smaller than when an error estimation occurs for an object that does not have the same type of object around it. In other words, it can be said that there is not much need to particularly lower the evaluation of the quality of an estimation algorithm that makes such an error estimation. Therefore, the evaluation device 100 according to the second embodiment can more appropriately evaluate the quality of an estimation algorithm.

[0108] The speed calculation unit 246 calculates an error estimation coefficient according to the speed of the corresponding object. Specifically, the speed calculation unit 246 calculates an error estimation coefficient such that the faster the speed of the object, the higher the error estimation coefficient corresponding to the error estimation type related to the object. More specifically, the speed calculation unit 246 calculates an error estimation coefficient for an error estimation of "missed detection (FN)", for example, using the following formula (6). Note that the speed calculation unit 246 may also calculate error estimation coefficients corresponding to other error estimation types in the same manner as formula (6).

number

[0109] where v i (t) is the velocity of object i at time t. i (t) is calculated from the distance that object i moves from timing (t-1) to timing t and the time between the image frame at timing (t-1) and the image frame at timing t. In other words, v i (t) is calculated from the moving distance of object i and the frame rate. Furthermore, c is a constant greater than 0. Note that c may differ depending on the type of erroneous estimation to which formula 6 is applied. For example, the value of c may differ between when β is calculated and when α is calculated.

[0110] From equation (6), the velocity v of object i is i The faster (t) is, the higher the error estimation coefficient β is. Note that equation (6) is an example showing the calculation of the speed calculation unit 246. The speed calculation unit 246 may calculate the error estimation coefficient using another equation such that the error estimation coefficient becomes higher as the speed of the object becomes faster.

[0111] When an erroneous estimation is made for a fast object, the object moves a long distance while it cannot be detected properly. Therefore, the degree of the impact is large because it becomes difficult to properly control the collision avoidance for the object. In particular, the faster the object is, the longer the moving distance while it cannot be detected properly, so the degree of the impact is further increased. Note that when an erroneous estimation of a "missed no" (FN) for a fast object is made, the object moves a long distance while the estimation algorithm cannot detect the object at all (misses it). Therefore, the degree of the impact of an erroneous estimation of a "missed no" (FN) for a fast object is particularly large. In other words, it is desirable for an estimation algorithm not to generate erroneous estimations for fast objects.

[0112] In contrast, the evaluation device 100 according to the second embodiment calculates the error estimation coefficient so that the faster the speed of the object, the higher the error estimation coefficient corresponding to the type of error estimation for the object is. This makes it possible to increase the error estimation coefficient when the degree of influence increases due to an error estimation made on a fast-moving object. Therefore, it is possible to lower the evaluation of the quality of an estimation algorithm that performs an estimation process that increases the degree of influence, such as an error estimation made on a fast-moving object. In other words, it is possible to higher the evaluation of the quality of an estimation algorithm that is less likely to make an error estimation even on a fast-moving object. Therefore, it is possible to more appropriately evaluate the quality of the estimation algorithm.

[0113] The evaluation value calculation unit 260 calculates an evaluation value of the estimation algorithm. Here, in the first embodiment, the erroneous estimation coefficient is constant for each erroneous estimation type. In contrast, in the second embodiment, the erroneous estimation coefficient is calculated for each object (rectangle). That is, even if the same type of erroneous estimation is performed on multiple objects, the erroneous estimation coefficient may differ for each object.

[0114] Therefore, in the second embodiment, the evaluation value calculation unit 260 calculates the evaluation value using the evaluation formula shown in the following formula (7). As in the first embodiment, the evaluation value calculation unit 260 calculates the evaluation value for each image frame (timing t).

number

[0115] The second term of the numerator of the fraction in the second term on the right-hand side of equation (7) (Σβ i (t)) is the error estimation coefficient β for object i that is “missed (FN)” in the image frame at time t. iThis shows that (t) is added to the number of objects (FN(t)) corresponding to the “missed detection (FN)”. In other words, the second term of the numerator of the fraction in the second term on the right side of equation (7) corresponds to the sum of the error estimation coefficient and the number of objects corresponding to that error estimation type.

[0116] The first term of the numerator of the fraction in the second term on the right-hand side of equation (7) (Σα h In (t), h is the index of the rectangle (object) that is erroneously estimated as a “false positive (FP)”. h (t) is the error estimation coefficient calculated for the rectangle h that is erroneously estimated as a "false positive (FP)" in the image frame at time t. Therefore, the first term of the numerator of the fraction in the second term on the right side of equation (7) is the error estimation coefficient α h This indicates that (t) is added to the number of objects (rectangles) (FP(t)) corresponding to the "false positive (FP)". In other words, the first term of the numerator of the fraction in the second term on the right side of equation (7) corresponds to the sum of the mis-estimation coefficient and the number of objects (rectangles) corresponding to that mis-estimation type.

[0117] The third term of the numerator of the fraction in the second term on the right-hand side of equation (7) (Σγ j In (t), j is the index of the rectangle (object) that is erroneously estimated as “new false positive (IDSWn)”. Also, γ j (t) is the error estimation coefficient calculated for the rectangle j in which the “new error detection (IDSWn)” was erroneously estimated in the image frame at time t. Therefore, the third term of the numerator of the fraction in the second term on the right side of equation (7) is the error estimation coefficient γ j This indicates that (t) is added to the number of objects (rectangles) (IDSWn(t)) corresponding to the “new false positive (IDSWn)”. In other words, the third term of the numerator of the fraction in the second term on the right side of equation (7) corresponds to the sum of the false estimation coefficient and the number of objects (rectangles) corresponding to that false estimation type.

[0118] The fourth term of the numerator of the fraction in the second term on the right-hand side of Equation (7) (Σδ k In (t), k is the index of the object that is mis-predicted as “replacement (IDSWc)”. k (t) is the misestimation coefficient calculated for object k whose “replacement (IDSWc)” was misestimated in the image frame at time t. Therefore, the fourth term of the numerator of the fraction in the second term on the right side of equation (7) is the misestimation coefficient δ k This indicates that (t) is added to the number of objects (IDSWc(t)) corresponding to the “replacement (IDSWc).” In other words, the fourth term of the numerator of the fraction in the second term on the right side of equation (7) corresponds to the sum of the error estimation coefficient and the number of objects corresponding to that error estimation type.

[0119] Therefore, the numerator of the fraction in the second term on the right side of equation (7) corresponds to the sum of the error estimation coefficients corresponding to the number of objects corresponding to each of the multiple error estimation types. As shown in equation (7), the larger this sum is, the smaller the evaluation value F(t) is. In other words, the larger this sum is, the lower the evaluation of the estimation algorithm is. When the evaluation values ​​have been calculated for all image frames, the evaluation value calculation unit 260 sums up the evaluation values ​​calculated for all image frames as in equation (3) above to calculate an overall evaluation value F, which is the evaluation value of the estimation algorithm for the entire video.

[0120] 7 and 8 are flowcharts showing an evaluation method executed by the evaluation system 1 according to the second embodiment. The processes shown in Fig. 7 and Fig. 8 can be mainly executed by the evaluation device 100. First, similar to S100 in Fig. 5, the tracking engine 20 performs an estimation process of an estimation algorithm (step S200). The evaluation device 100 sets t=1 (step S201) and processes the t-th image frame (step S202).

[0121] The correct data acquisition unit 112 acquires correct data for the t-th image frame (step S204), similar to S104 in FIG. 5. The output data acquisition unit 114 acquires output data related to the estimation process executed for the t-th image frame (step S206), similar to S106 in FIG. 5. The estimation result determination unit 120 determines the estimation result for the t-th image frame (step S210), similar to S110 in FIG. 5. The estimation result determination unit 120 counts the number of objects for each erroneous estimation type, as described above, similar to S112 in FIG. 5. The coefficient calculation unit 240 calculates the corresponding erroneous estimation coefficient for each object (rectangle) (step S220). The process of S220 will be described later with reference to FIG. 8.

[0122] When the erroneous estimation coefficient is calculated in S220, the evaluation value calculation unit 160 calculates the evaluation value of the estimation algorithm for the t-th image frame using the evaluation formula shown in the above formula (7) (step S250). Then, the evaluation device 100 judges whether or not there is an unprocessed image frame (step S260). If there is an unprocessed image frame (YES in S260), the evaluation device 100 increments t by one (step S262). Then, the process flow returns to S202, and the evaluation device 100 processes the next image frame (S202). Then, the processes of S204 to S250 are repeated. On the other hand, if there is no unprocessed image frame (NO in S260), the evaluation value calculation unit 160 sums up the evaluation values ​​obtained for all image frames as shown in the above formula (3) (step S264). Then, the evaluation value output unit 162 outputs the evaluation value of the entire moving image (step S266).

[0123] The process of S220 will be described with reference to Fig. 8. Fig. 8 shows a process for an object erroneously estimated as "missed detection (FN)". Note that the same process as Fig. 8 may be performed for other types of erroneous estimation. Also, calculation of erroneous estimation coefficients for multiple types of erroneous estimation may be performed in parallel.

[0124] The coefficient calculation unit 240 determines an object i to be processed from objects in which a "missed no attention (FN)" has been erroneously estimated in the image frame to be processed (step S222). The object to be processed may be determined for each object ID. The coefficient calculation unit 240 sets an initial value of the erroneous estimation coefficient (step S224). Specifically, the coefficient calculation unit 240 extracts an initial value (β0) of the erroneous estimation coefficient β related to the "missed no attention (FN)" from the parameter storage unit 130, and sets the extracted initial value.

[0125] The consecutive error estimation calculation unit 242 judges whether the erroneous estimation judgment in the previous image frame (t-1) and the erroneous estimation judgment in the current image frame (t) for the object i to be processed are both "missed no attention (FN)" (step S230). If the erroneous estimation judgment of "missed no attention (FN)" for the object i in the previous image frame is also judged (YES in S230), the consecutive error estimation calculation unit 242 calculates the erroneous estimation coefficient β i The misestimation coefficient β i (t) (step S232). Specifically, the continuous error estimation calculation unit 242 calculates the error estimation coefficient β i On the other hand, if the previous image frame is not determined to be a "missed detection (FN)" erroneous estimation (NO in S230), the process of S232 is skipped. That is, the continuous erroneous estimation calculation unit 242 calculates β i Let (t) = β0.

[0126] The object group calculation unit 244 calculates the distance between the object i to be processed and other objects of the same type (step S234). Note that the distance between the objects may be calculated, for example, by estimating the real positions of the object i to be processed and the other objects from pixels corresponding to each of them, and calculating the distance between the estimated real positions. Alternatively, the distance between the objects may be calculated, for example, by using the distance between the pixel corresponding to the object i to be processed and the pixel corresponding to the other objects in the image frame.

[0127] The object group calculation unit 244 determines whether there is another object whose distance from the object i to be processed is equal to or less than a threshold (step S236). The distance threshold is, for example, about 5 m in a real environment, but is not limited to this. The threshold can be appropriately set to a degree that one or more objects whose distance from the object i is within the threshold can be recognized as an object group together with the object i.

[0128] If there are other objects whose distance from the object i to be processed is equal to or less than the threshold (YES in S236), the object group calculation unit 244 calculates the error estimation coefficient according to the number of other objects (step S238). Specifically, the object group calculation unit 244 calculates the error estimation coefficient β using the formula (5). i When the process of S232 is performed, the object group calculation unit 244 calculates β0 in the formula (5) by multiplying β i It may be replaced with the value of (t).

[0129] On the other hand, if there is no other object whose distance from the object i to be processed is equal to or less than the threshold (NO in S236), the process of S238 is skipped. i Let (t) remain the value of the error estimation coefficient calculated by the continuous error estimation calculation unit 242.

[0130] The speed calculation unit 246 calculates an error estimation coefficient according to the speed of the object i to be processed (step S240). Specifically, the speed calculation unit 246 calculates (estimates) the speed of the object i to be processed. Then, the speed calculation unit 246 calculates the error estimation coefficient β i When at least one of the processes in S232 and S238 has been performed, the speed calculation unit 246 calculates β0 in the formula (6) by multiplying β i It may be replaced with the value of (t).

[0131] The coefficient calculation unit 240 determines whether or not there is an unprocessed object in the image frame to be processed (step S242). In the example of FIG. 8, the coefficient calculation unit 240 determines whether or not there is an unprocessed object among the objects corresponding to "missed image (FN)". If there is an unprocessed object (YES in S242), the process flow returns to S222. Then, an object to be processed is determined from the unprocessed objects (S222), and the processes of S224 to S240 are repeated. On the other hand, if there is no unprocessed object (NO in S242), the process flow proceeds to S250.

[0132] (Application examples) In addition, by using the evaluation device 100 according to the present embodiment, in the estimation algorithm, a threshold value of the degree of coincidence for determining the identity between the observed value and the estimated value can be determined so that the degree of influence on erroneous estimation in traffic control and the like is reduced. In other words, by determining a threshold value that will give a high evaluation result by the evaluation device 100, an estimation algorithm that will be highly evaluated can be obtained. For example, if the threshold value is set high, a "false positive (FP)" may occur easily. On the other hand, if the threshold value is set low, a "missed detection (FN)" may occur easily. Therefore, for example, when the threshold value in the estimation algorithm is set to 70%, 80%, and 90%, the estimation algorithm is evaluated using the evaluation device 100, and the threshold value with the highest evaluation value is adopted.

[0133] Furthermore, the evaluation device 100 according to the present embodiment can be used to detect an abnormality in a sensor used to track an object in real time. That is, the evaluation device 100 acquires its own position information from an object such as an autonomous vehicle, and uses the position information as ground truth data. Meanwhile, in parallel with this, the estimation algorithm estimates and tracks the position of the object (such as an autonomous vehicle), and uses the estimation result as output data. Then, the evaluation device 100 performs the evaluation process of the estimation algorithm as described above using the ground truth data and the output data. Then, the evaluation result at that time is compared with the evaluation result in normal times, and if the evaluation is lower than a predetermined threshold value, it can be determined that the sensor is abnormal.

[0134] (Modification) The present invention is not limited to the above embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, the order of each step (process) in the above-mentioned flowchart can be changed as appropriate. Also, one or more steps (processes) in the above-mentioned flowchart can be omitted. For example, the process of S212 in FIG. 7 can be omitted.

[0135] In addition, in the second embodiment, it is not necessary for all of the successive error estimation calculation unit 242, the object group calculation unit 244, and the speed calculation unit 246 constituting the coefficient calculation unit 240 to perform processing. That is, in the flowchart of Fig. 8, any of the processing of S230 to S232, the processing of S234 to S238, and the processing of S240 may not be performed.

[0136] In the above embodiment, the estimation algorithm is used in a traffic control environment, but the environment in which the estimation algorithm is used is not limited to traffic control. The evaluation device 100 according to the present embodiment is applicable to estimation algorithms that perform any object detection other than traffic control.

[0137] The above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiments when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or the tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disk (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, the transitory computer-readable medium or the communication medium includes electrical, optical, acoustic, or other forms of propagating signals. [Explanation of symbols]

[0138] 1. Rating System 2 Network 20 Tracking Engine 40 Video storage device 100 Evaluation Device 112 Correct Data Acquisition Section 114 Output data acquisition unit 120 Estimation result judgment unit 130 Parameter storage section 160 Evaluation value calculation unit 162 Evaluation value output unit 240 Coefficient calculation section 242 Continuous Error Estimation Calculation Unit 244 Object Group Calculation Unit 246 Speed ​​calculation section 260 Evaluation value calculation unit

Claims

1. 1. An evaluation system for evaluating the quality of an algorithm for estimating a position of a movable object in a video and tracking the object, comprising: an inference result determination unit that determines, for each object, which of a plurality of erroneous inference types is at least a type of erroneous inference result, using correct answer data corresponding to the video and output data indicating an inference result of the algorithm that performed an inference on the video; an evaluation value calculation unit that adds up error estimation coefficients corresponding to each of the plurality of error estimation types and set so as to increase according to the degree of influence of the corresponding error estimation type, by the number of objects corresponding to the corresponding error estimation type, and calculates an evaluation value of the algorithm based on a total value of the addition values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types; having Among the plurality of error estimation types, the error estimation coefficient corresponding to a second error estimation, which is an error estimation type related to the algorithm estimating that a plurality of the objects included in the correct answer data are interchanged between one timing and the next timing, is set to be higher than the error estimation coefficient corresponding to a third error estimation, which is an error estimation type related to the algorithm estimating that one of the objects included in the correct answer data is a different object between one timing and the next timing. Rating system.

2. Among the plurality of error estimation types, the error estimation coefficient corresponding to a first error estimation type relating to the algorithm being unable to estimate the object included in the correct answer data is set higher than the error estimation coefficients corresponding to the other error estimation types. The evaluation system according to claim 1 .

3. A system for evaluating the quality of an algorithm for estimating a position of a movable object in a video and tracking the object, comprising: an inference result determination unit that determines, for each object, which of a plurality of erroneous inference types is at least a type of erroneous inference result, using correct answer data corresponding to the video and output data indicating an inference result of the algorithm that performed an inference on the video; an evaluation value calculation unit that adds up error estimation coefficients corresponding to each of the plurality of error estimation types and set so as to increase according to the degree of influence of the corresponding error estimation type, by the number of objects corresponding to the corresponding error estimation type, and calculates an evaluation value of the algorithm based on a total value of the addition values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types; a coefficient calculation unit that calculates the error estimation coefficient for each timing at which the position of the object in the video is estimated; An evaluation system having:

4. the coefficient calculation unit calculates the error estimation coefficient so as to increase the error estimation coefficient corresponding to the error estimation type regarding a certain object when the same error estimation type is continuously made regarding the certain object; The evaluation system according to claim 3 .

5. the coefficient calculation unit calculates the error estimation coefficient so as to increase the error estimation coefficient corresponding to the first error estimation when a first error estimation, which is an error estimation type related to the algorithm being unable to estimate the object included in the correct answer data, is made consecutively; The evaluation system according to claim 4.

6. the coefficient calculation unit calculates the error estimation coefficient such that the error estimation coefficient corresponding to the error estimation type related to the object is increased as the speed of the object increases; The evaluation system according to any one of claims 3 to 5.

7. the coefficient calculation unit calculates the error estimation coefficient such that the error estimation coefficient corresponding to the error estimation type related to an object is lower as the number of other objects of the same type as an object that are at a distance from the object that is equal to or smaller than a predetermined threshold value increases. The evaluation system according to any one of claims 3 to 6.

8. 1. A method for evaluating the quality of an algorithm for estimating a position of a movable object in a video sequence and tracking the object, comprising the steps of: determining, for each object, which of a plurality of erroneous estimation types is a type of at least an erroneous estimation result, using correct answer data corresponding to the video and output data indicating an estimation result of the algorithm that performed estimation on the video; adding up error estimation coefficients corresponding to each of the plurality of error estimation types and increasing in accordance with the degree of influence of the corresponding error estimation type by the number of objects corresponding to the corresponding error estimation type, and calculating an evaluation value of the algorithm based on a total value of the added values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types; Among the plurality of error estimation types, the error estimation coefficient corresponding to a second error estimation, which is an error estimation type related to the algorithm estimating that a plurality of the objects included in the correct answer data are interchanged between one timing and the next timing, is set to be higher than the error estimation coefficient corresponding to a third error estimation, which is an error estimation type related to the algorithm estimating that one of the objects included in the correct answer data is a different object between one timing and the next timing. Evaluation method.

9. A method for evaluating the quality of an algorithm for estimating the position of a movable object in a video sequence and tracking said object, comprising: determining, for each object, which of a plurality of erroneous estimation types is a type of at least an erroneous estimation result, using correct answer data corresponding to the video and output data indicating an estimation result of the algorithm that performed estimation on the video; adding up error estimation coefficients corresponding to each of the plurality of error estimation types and increasing in accordance with the degree of influence of the corresponding error estimation type by the number of objects corresponding to the corresponding error estimation type, and calculating an evaluation value of the algorithm based on a total value of the added values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types; calculating the error estimation coefficient for each timing at which the position of the object in the video is estimated; Evaluation method.

10. A program for implementing a method for evaluating the quality of an algorithm that estimates a position of a movable object in a video and tracks the object, comprising: determining, for each object, which of a plurality of erroneous estimation types is at least a type of erroneous estimation result, using correct answer data corresponding to the video and output data indicating an estimation result of the algorithm that performed estimation on the video; a step of adding up error estimation coefficients, which correspond to each of the plurality of error estimation types and are set to increase according to the degree of influence of the corresponding error estimation type, by the number of objects corresponding to the corresponding error estimation type, and calculating an evaluation value of the algorithm based on a total value of the addition values ​​of the error estimation coefficients obtained for each of the plurality of error estimation types; on the computer, Among the plurality of error estimation types, the error estimation coefficient corresponding to a second error estimation, which is an error estimation type related to the algorithm estimating that a plurality of the objects included in the correct answer data are interchanged between one timing and the next timing, is set to be higher than the error estimation coefficient corresponding to a third error estimation, which is an error estimation type related to the algorithm estimating that one of the objects included in the correct answer data is a different object between one timing and the next timing. program.

Citation Information

Patent Citations

  • Systems and methods for predicting abnormal behavior

    JP2012518846A