Evaluation system, evaluation method and computer-readable medium

Abstract

This invention provides an evaluation system, an evaluation method, and a computer-readable medium. In this evaluation system, an inference result determination unit uses positive solution data corresponding to a moving image and output data representing the inference result of an algorithm that performs inference on the moving image, thereby determining, for each object, whether the inference result is correct or one of multiple types of incorrect inference. An evaluation value calculation unit adds up the error inference coefficients, which are set to correspond to each of the multiple error inference types and increase according to the degree of influence of that error inference type, according to the number of objects corresponding to that error inference type, and calculates the evaluation value of the algorithm based on the sum of the summed error inference coefficients obtained for each of the multiple error inference types.

Description

Technical Field

[0001] This invention relates to an evaluation system, evaluation method, and procedure. Background Technology

[0002] A technique for tracking moving objects using video data is disclosed. Regarding this technique, Japanese Patent Application Publication No. 2012-518846 discloses a system and method for predictive abnormal behavior detection. In the system disclosed in Japanese Patent Application Publication No. 2012-518846, surveillance data such as video data is received, and multiple prediction models are created and updated. Furthermore, the system receives video data associated with a moving object and generates a prediction of the moving object's future location based on the generated prediction models. The predicted motion is scored by a scoring engine to determine whether the predicted motion is unsafe or otherwise undesirable. In addition, Luiten2021 (JonathonLuiten et al., “HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking”, International Journal of Computer Vision (2021) 129: 548-578, https: / / doi.org / 10.1007 / s11263-020-01375-2) discloses a method for evaluating tracking systems. Summary of the Invention

[0003] When a tracking system is used in specific environments such as traffic control, it is necessary to evaluate whether the tracking system performs well in that environment. However, Japanese Patent Publication No. 2012-518846 does not disclose a method for evaluating tracking systems. Furthermore, evaluation methods such as MOTA (Multi-Object Tracking Accuracy) are listed in the technologies covered by Luiten2021. However, depending on the usage environment of the tracking system, even if the value of the evaluation formula listed in Luiten2021 is high (evaluation value), the performance of the tracking system may not necessarily be good. Therefore, it is desirable to appropriately evaluate the quality of the algorithm (tracking system) that infers the position of a movable object and performs object tracking.

[0004] This invention provides an evaluation system, evaluation method, and program that can appropriately evaluate the quality of algorithms that infer the position of a movable object and perform object tracking.

[0005] The evaluation system involved in this invention is an evaluation system for evaluating the quality of an algorithm that infers the position of a movable object in a dynamic image and performs tracking of the object. The evaluation system includes: an inference result determination unit that uses positive solution data corresponding to the dynamic image and output data representing the inference result of the algorithm that performs inference on the dynamic image, thereby determining, for each object, whether it is a correct inference result or one of a plurality of incorrect inference types that is an incorrect inference result; and an evaluation value calculation unit that adds up the number of objects corresponding to the multiple incorrect inference types, each set in a manner that increases according to the degree of influence of the incorrect inference type, and calculates the evaluation value of the algorithm based on the sum of the sums of the incorrect inference coefficients obtained for each of the multiple incorrect inference types.

[0006] Furthermore, the evaluation method involved in this invention is an evaluation method for evaluating the quality of an algorithm that infers the position of a movable object in a dynamic image and performs tracking of the object. In this evaluation method, positive solution data corresponding to the dynamic image and output data representing the inference result of the algorithm that performed the inference for the dynamic image are used to determine, for each object, either a correct inference result or one of a plurality of incorrect inference types. The incorrect inference coefficients, which are set to correspond to each of the plurality of incorrect inference types and increase according to the degree of influence of the incorrect inference type, are added together according to the number of objects corresponding to the incorrect inference type. The evaluation value of the algorithm is calculated based on the sum of the sums of the incorrect inference coefficients obtained for each of the plurality of incorrect inference types.

[0007] Furthermore, the program involved in this invention is a program for evaluating the quality of an algorithm that infers the position of a movable object in a moving image and performs tracking of the object. The program causes a computer to perform the following steps: using positive solution data corresponding to the moving image and output data representing the inference result of the algorithm performed on the moving image, to determine, for each object, either a correct inference result or one of a plurality of incorrect inference types; adding error inference coefficients, each corresponding to one of the plurality of incorrect inference types and set in a manner that increases according to the degree of influence of that error inference type, according to the number of objects corresponding to that error inference type, and calculating an evaluation value for the algorithm based on the sum of the summed values ​​of the error inference coefficients obtained for each of the plurality of incorrect inference types.

[0008] In this invention, the error inference coefficient is set such that the greater the influence of the corresponding error inference type, the higher the error inference coefficient. Therefore, the more error inferences with a greater influence are implemented, the lower the evaluation may be. Moreover, it can be said that an algorithm that implements more error inferences with a greater influence has lower quality. Therefore, this invention can appropriately evaluate the quality of the algorithm.

[0009] Furthermore, preferably, the error inference coefficient corresponding to the first error inference among the plurality of error inference types is set higher than the error inference coefficients corresponding to the other error inference types, wherein the first error inference is an error inference type related to the case where the algorithm cannot infer the object contained in the correct solution data.

[0010] The present invention can be configured in such a way that the quality of the algorithm can be further and appropriately evaluated.

[0011] Furthermore, preferably, the error inference coefficient corresponding to the second error inference among the plurality of error inference types is set higher than the error inference coefficient corresponding to the third error inference, wherein the second error inference is an error inference type related to the case where the algorithm replaces and infers the plurality of objects contained in the correct solution data at a certain time and the next time, and the third error inference is an error inference type related to the case where the algorithm infers a one of the objects contained in the correct solution data as a different object at a certain time and the next time.

[0012] The present invention can be configured in such a way that the quality of the algorithm can be further and appropriately evaluated.

[0013] Furthermore, preferably, it also includes a coefficient calculation unit that calculates the erroneous inference coefficients for each inference time.

[0014] This invention can be configured in such a way that the error inference coefficients can be varied according to changes in conditions. Therefore, it is possible to more appropriately evaluate the quality of the inference algorithm.

[0015] Furthermore, preferably, when the coefficient calculation unit makes consecutive erroneous inferences of the same type about a certain object, it calculates the erroneous inference coefficient in a manner that makes the erroneous inference coefficient corresponding to the erroneous inference type related to that object higher.

[0016] The present invention can be configured in such a way that the quality evaluation of the inference algorithm can be implemented more appropriately.

[0017] Furthermore, preferably, when the coefficient calculation unit makes a first incorrect inference consecutively, it calculates the incorrect inference coefficient in a manner that makes the incorrect inference coefficient corresponding to the first incorrect inference higher, wherein the first incorrect inference is an incorrect inference type related to the case where the algorithm cannot infer the object contained in the correct solution data.

[0018] The present invention can be configured in such a way that the quality evaluation of the inference algorithm can be implemented more appropriately.

[0019] Furthermore, preferably, the coefficient calculation unit calculates the error inference coefficient in such a way that the faster the object moves, the higher the error inference coefficient corresponding to the error inference type associated with that object becomes.

[0020] The present invention can be configured in such a way that the quality evaluation of the inference algorithm can be implemented more appropriately.

[0021] Furthermore, preferably, the coefficient calculation unit calculates the error inference coefficient in such a way that the more other objects of the same type as the object that are at a distance of less than a predetermined threshold from the object, the lower the error inference coefficient corresponding to the error inference type associated with the object.

[0022] The present invention can be configured in such a way that the quality evaluation of the inference algorithm can be implemented more appropriately.

[0023] According to the present invention, an evaluation system, evaluation method, and procedure are provided that can appropriately evaluate the quality of an algorithm for inferring the position of a movable object and performing object tracking.

[0024] The above and other objects, features and advantages of this disclosure will be more fully understood from the detailed description given below and the accompanying drawings, which are for illustrative purposes only, and should therefore not be construed as limiting the disclosure. Attached Figure Description

[0025] Figure 1 A diagram illustrating the evaluation system involved in Implementation Method 1.

[0026] Figure 2 This diagram is used to illustrate the types of incorrect inferences involved in this embodiment.

[0027] Figure 3 This diagram is used to illustrate the types of incorrect inferences involved in this embodiment.

[0028] Figure 4This diagram illustrates the structure of the evaluation device according to Embodiment 1.

[0029] Figure 5 This is a flowchart illustrating the evaluation method performed using the evaluation system described in Implementation 1.

[0030] Figure 6 This diagram illustrates the structure of the evaluation device involved in Embodiment 2.

[0031] Figure 7 A flowchart illustrating the evaluation method performed by the evaluation system according to Embodiment 2.

[0032] Figure 8 A flowchart illustrating the evaluation method performed by the evaluation system according to Embodiment 2. Detailed Implementation

[0033] (Implementation Method 1)

[0034] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. For clarity of description, the following description and drawings have been appropriately omitted and simplified. Furthermore, in the various drawings, the same symbols are used for the same elements, and repeated descriptions have been omitted as necessary.

[0035] Figure 1 Figure 1 illustrates the evaluation system 1 according to Embodiment 1. The evaluation system 1 includes a tracking engine 20, a motion image storage device 40, and an evaluation device 100. The tracking engine 20, the motion image storage device 40, and the evaluation device 100 can be connected together in a manner that allows them to communicate with each other via a wired or wireless network 2.

[0036] The tracking engine 20 is, for example, a computer. The tracking engine 20 runs an inference algorithm that infers the position of a movable object (moving object) and tracks the moving object. That is, the inference algorithm detects objects from a moving image. The tracking engine 20 inputs the moving image stored in the moving image storage device 40 (described later) into the inference algorithm, thereby running the inference algorithm. For the input moving image, the inference algorithm infers the position and type of the object for each image frame and outputs the inference result. The tracking engine 20 sends the object inference result output from the inference algorithm as output data to the evaluation device 100. Furthermore, the inference algorithm does not need to perform inference processing on each image frame individually. For example, inference processing can be performed on the even-numbered image frames.

[0037] Alternatively, the tracking engine 20 may send output data (inference results) for each image frame. Or, the tracking engine 20 may send output data (inference results) related to the entire dynamic image after the inference algorithm has performed object inference and tracking for the entire dynamic image.

[0038] The dynamic image storage device 40 is, for example, a storage device. The dynamic image storage device 40 pre-stores dynamic images used when evaluating the quality of the inference algorithm. The dynamic image storage device 40 stores these images in a manner that establishes a correspondence between the dynamic images and the forward solving data and each image frame. In other words, a correspondence is established between the forward solving data and the image frames constituting the dynamic images. The forward solving data represents the location of an object captured in the corresponding image frame and the type of object at that location. The type of object can be, for example, a pedestrian, bicycle, motorcycle, car, large vehicle, etc. Furthermore, the location of the object can be either its location on the image frame (pixel position) or its location information in the actual environment captured by the image frame (e.g., an intersection). The location information can also be obtained, for example, through GNSS (Global Navigation Satellite System).

[0039] Alternatively, the tracking engine 20 can also be implemented using the evaluation device 100 described later. That is, the evaluation device 100 can also run inference algorithms. Similarly, the motion image storage device 40 can also be implemented using the evaluation device 100 described later. That is, the evaluation device 100 can also store motion images that correspond to the correct solution data.

[0040] Evaluation device 100 is, for example, a computer. Evaluation device 100 is, for example, a server. Evaluation device 100 can also be implemented, for example, through cloud computing. Evaluation device 100 (evaluation system 1) uses dynamic images to evaluate the quality of the inference algorithm. Specifically, evaluation device 100 uses positive solution data corresponding to the dynamic image and output data representing the inference result of the inference algorithm to determine, for each object, whether it is a correct inference (correct inference) or one of several types of incorrect inferences (incorrect inference types). A correct inference (correct inference result) is an inference result in which the inference was correctly performed for the object. On the other hand, an incorrect inference type is a type of incorrect inference result. That is, an incorrect inference type is a type of inference result in which the incorrect inference was performed for the object (incorrect inference). In this embodiment, multiple types of incorrect inference may occur. Details regarding the types of incorrect inference will be described later.

[0041] Furthermore, for each of the multiple types of incorrect inferences, the evaluation device 100 adds an incorrect inference coefficient corresponding to that type of incorrect inference to the number of corresponding objects. The evaluation device 100 calculates the sum of the incorrect inference coefficients obtained for each of the multiple types of incorrect inferences. Then, based on the sum, the evaluation device 100 calculates an evaluation value for the inference algorithm. Alternatively, this evaluation value can be a value in which a larger sum indicates a lower evaluation of the algorithm. Details regarding the evaluation value will be described later.

[0042] Figure 2 as well as Figure 3 This is a diagram used to illustrate the types of incorrect inferences involved in this embodiment. Figure 2 Specific examples to indicate the types of incorrect inferences. For example... Figure 2 As shown, in this implementation, the inference results include correct inferences (True Positive Inferences) and incorrect inference types. A "True Positive Inference (TP)" corresponds to a state where the inference result in the output of the inference algorithm corresponds to an object present in the correct solution data. Furthermore, incorrect inference types include "error detection (error tracking)," "omission," "new error detection," and "replacement."

[0043] "False Positive (FP)" (fourth false inference) corresponds to the state where the inference algorithm's output does not correspond to any object present in the correct solution data. In other words, "false detection" corresponds to the state where the inference algorithm infers (detects) an object at a location where it does not actually exist. Further, "false detection" is a type of false inference associated with the inference algorithm inferring the existence of an object that does not actually exist. Even further, "false detection" is a type of false inference associated with tracking an object that actually exists in another location. Therefore, when a false inference of "false detection" occurs, an ID (Identifier) ​​associated with a hypothetical object or an object that actually exists in another location is added to the inference result obtained by the inference algorithm at the location where the object does not exist in the correct solution data.

[0044] "False Negative (FN)" (first-order incorrect inference) corresponds to the state where no object existing in the correct solution data corresponds to any inference result in the output of the inference algorithm. In other words, "false negative" corresponds to the state where an object existing in the correct solution data is not inferred in the output of the inference algorithm. In other words, "false negative" corresponds to the state where the inference algorithm cannot infer (detect) an object at the actual location of the object. Further, "false negative" is a type of incorrect inference related to the inference algorithm's inability to infer (detect) objects contained in the correct solution data. Therefore, when a false negative "false negative" occurs, no ID is added to the object existing in the correct solution data in the inference result obtained by the inference algorithm.

[0045] "New Error Detection (IDSWn: ID Switch New; Add New ID)" (Third Error Inference) corresponds to the state where the inference algorithm infers an object existing in the correct solution data as a different object in the previous inference process and the current inference process. In other words, "New Error Detection" corresponds to the state where an object existing in the correct solution data is inferred (detected) as a different object in the inference algorithm's output at a certain time interval and the inference result at the next time interval. In other words, "New Error Detection" corresponds to the state where the inference algorithm actually infers (detects) an object as a different object at a certain time interval and the next time interval. Further, "New Error Detection" is a type of error inference related to the situation where the inference algorithm infers an object contained in the correct solution data as a different object at a certain time interval and the next time interval. Therefore, when a "New Error Detection" error inference occurs, an ID that is different at a certain time interval and the next time interval is added to an object existing in the correct solution data in the inference result obtained by the inference algorithm.

[0046] "ID Switch Change (IDSWc)" (Second Error Inference) refers to a type of error inference related to the inference algorithm replacing and inferring multiple objects contained in the correct solution data at a certain time interval and the next time interval. In other words, "replacement" corresponds to the state where multiple objects existing in the correct solution data are replaced and inferred (detected) in the inference algorithm's output at a certain time interval and the inference result at the next time interval. Therefore, when an error inference of "replacement" occurs, in the inference result obtained by the inference algorithm, for each of the multiple objects existing in the correct solution data, the ID added at a certain time interval will be added to another object among the multiple objects at the next time interval. For example, when object A and object B exist in the correct solution data, the ID added to the position of object A at a certain time interval will be added to the position of object B at the next time interval. Similarly, the ID added to the position of object B at a certain time interval will be added to the position of object A at the next time interval.

[0047] exist Figure 3 This paper illustrates several types of incorrect inferences and correct inferences in dynamic images. Figure 3 The example illustrates the inference results obtained by performing inference processing on multiple objects contained in a dynamic image obtained near a road intersection. Figure 3 In this context, objects P (P1 to P4) represent objects contained in the positive solution data. On the other hand, rectangles R (R11, etc.) represent locations where the inference algorithm deduces the presence of objects. Furthermore, in... Figure 3 In the example, it is assumed that inference is performed for each image frame. Furthermore, t (t = 1 to 6) corresponds to each image frame and represents the timing of the inference in that image frame.

[0048] exist Figure 3 In the example shown, regarding object P1, at time t=1 (image frame), the inference algorithm correctly performed the inference as shown in rectangle R11. At this time, "ID: #01" was added to rectangle R11 corresponding to object P1. Furthermore, regarding the moving object P1, even at time points t=2 to 4 (image frames), the inference algorithm correctly performed the inference as shown in rectangles R12 to R14 respectively. At this time, "ID: #01" was added again to rectangles R12 to R14 corresponding to object P1, as added at t=1.

[0049] On the other hand, regarding object P1, the inference algorithm cannot make inferences at a timing of t=5 (image frame). That is, at t=5, an incorrect inference of "missed (FN)" occurs regarding object P1. Similarly, even at a timing of t=6 (image frame), the inference algorithm cannot make inferences. That is, even at t=6, an incorrect inference of "missed (FN)" occurs regarding object P1.

[0050] In response, at time t=5 (image frame), the inference algorithm, as shown in rectangle R15, infers that object P1 exists at the location where it does not exist. That is, at t=5, an erroneous inference of "Error Detection (FP)" (error tracking) occurs regarding object P1. Similarly, at time t=6 (image frame), the inference algorithm, as shown in rectangle R16, infers that object P1 exists at the location where it does not exist. That is, even at t=6, an erroneous inference of "Error Detection (FP)" occurs regarding object P1. At this point, "ID: #01", which was added at t=1, is added again to rectangles R15 and R16.

[0051] In addition, Figure 3 In the example shown, regarding object P2, at time t=1 (image frame), the inference algorithm correctly performed the inference as shown in rectangle R21. At this time, "ID: #02" was added to rectangle R21 corresponding to object P2. Furthermore, regarding the moving object P2, even at time t=2 (image frame), the inference algorithm correctly performed the inference as shown in rectangle R22. At this time, "ID: #02" was added to rectangle R22 corresponding to object P2, continuing to be added to it at t=1.

[0052] In contrast, at a timing of t=3 (image frame), the inference algorithm, as shown in rectangle R23, infers that object P2 exists at the location where it does not exist. That is, at t=3, an incorrect inference of "Fault Detection (FP)" is generated regarding object P2. Similarly, even at a timing of t=4 (image frame), the inference algorithm, as shown in rectangle R24, infers that object P2 exists at the location where it does not exist. That is, even at t=4, an incorrect inference of "Fault Detection (FP)" is generated regarding object P2. At this point, "ID: #02", which was added at t=1, is added again to rectangles R23-R24.

[0053] On the other hand, regarding object P2, at time t=3 (image frame), the inference algorithm, as shown in rectangle R2'3, infers an object different from the object P2 inferred at t=1-2. That is, at t=3, a erroneous inference of "New Error Detection (IDSWn)" occurs regarding object P2. At this time, an "ID: #12" is added to rectangle R2'3, different from the "ID: #02" corresponding to object P2 added at t=1. Furthermore, at time t=4 (image frame), the inference algorithm, as shown in rectangle R2'4, infers an object identical to the object (object P2) inferred at t=3. In this case, since the same object P2 was inferred as the same object in both the previous and current inference processes, the inference result is a correct inference (TP). At this time, an "ID: #12" is added to rectangle R2'4, identical to the ID added to rectangle R2'3 at t=3.

[0054] In addition, Figure 3 In the example shown, objects P3 and P4 are approaching and moving towards each other. Regarding object P3, at time t = 1 (image frame), the inference algorithm correctly infers as shown in rectangle R31. At this time, "ID: #03" is added to rectangle R31 corresponding to object P3. Furthermore, regarding the moving object P3, even at time t = 2 (image frame), the inference algorithm correctly infers as shown in rectangle R32. At this time, "ID: #03" is again added to rectangle R32 corresponding to object P3, the same "ID: #03" added at t = 1.

[0055] Furthermore, regarding object P4, at time t=1 (image frame), the inference algorithm correctly performed the inference as shown in rectangle R41. At this time, "ID: #04" was added to rectangle R41 corresponding to object P4. Moreover, regarding the moving object P4, even at time t=2 (image frame), the inference algorithm correctly performed the inference as shown in rectangle R42. At this time, "ID: #04" was added to rectangle R42 corresponding to object P4, continuing to be added to it at t=1.

[0056] In contrast, at a timing of t=3 (image frame), the inference algorithm, as shown in rectangle R43, infers that object P4 exists at the location of object P3. Furthermore, at a timing of t=3 (image frame), the inference algorithm, as shown in rectangle R33, infers that object P3 exists at the location of object P4. That is, at t=3, the inference algorithm incorrectly infers object P3 as object P4 and object P4 as object P3. Therefore, at t=3, an incorrect inference of "replacement (IDSWc)" occurs regarding both object P3 and object P4. At this time, the "ID: #04" added to object P4 at t=1~2 is added to rectangle R43. Additionally, the "ID: #03" added to object P3 at t=1~2 is added to rectangle R33.

[0057] Here, as Figure 2 As shown, the error inference coefficient related to "Error Detection (FP)" is set as α. The error inference coefficient related to "Omission (FN)" is set as β. The error inference coefficient related to "New Error Detection (IDSWn)" is set as γ. The error inference coefficient related to "Replacement (IDSWc)" is set as δ. Here, the error inference coefficients play a role as weights for each of the multiple error inference types in the evaluation formula used to evaluate the inference algorithm. Moreover, the error inference coefficients are set differently for each of the multiple error inference types. The larger the error inference coefficient among the multiple error inference types, the lower the evaluation value of the inference algorithm's evaluation formula will be.

[0058] Here, when the inference algorithm is used in traffic control systems such as autonomous vehicle control systems, the inference algorithm infers the position of objects such as pedestrians and tracks them. Furthermore, it can use the inference results from multiple time points (image frames) to infer the movement direction and speed of objects such as pedestrians. Specifically, the object's speed can be calculated using the movement distance calculated based on the difference (distance) between the object's positions in each image frame and the frame rate. That is, the object's speed can also be calculated using the object's movement distance and the time from acquiring one image frame to acquiring the next. More specifically, the field of view of the camera capturing the dynamic image can be used to establish a correspondence between the pixel positions in each image frame and their positions in the actual environment of the captured image. Furthermore, the inference algorithm can also infer the object's position in the actual environment (actual position) based on the pixel position corresponding to, for example, the lower end of the object (a pedestrian's foot). Moreover, the inference algorithm can also calculate the object's movement distance using the difference between the actual position of the object associated with a certain image frame and the actual position of the object associated with the next image frame. Furthermore, the inference algorithm can also infer the object's velocity based on the movement distance and frame rate. Additionally, the object's movement direction can be calculated by measuring the direction from the object's actual position in the previous image frame to its actual position in the current image frame.

[0059] Furthermore, by inferring the direction and speed of movement of objects such as pedestrians, it is possible to predict the probability of an object reaching a specific location (such as an intersection or roadway) and the arrival time if it does arrive. Therefore, the inference results of the inference algorithm can be used to implement control to avoid collisions between moving vehicles and objects.

[0060] When the inference algorithm is used in such a manner in an environment such as traffic control, the risks associated with each of the aforementioned types of incorrect inferences may differ. Furthermore, in this embodiment, the higher the risk of each type of incorrect inference, the higher the corresponding incorrect inference coefficient is set. In other words, the higher the incorrect inference coefficient is set, the greater the degree (severity) of the impact (impact) caused by the corresponding type of incorrect inference. That is, the incorrect inference coefficient is set according to the degree (severity) of the impact caused by the corresponding type of incorrect inference.

[0061] In the event of a "false detection," an object that does not actually exist might be identified as existing, and collision avoidance controls might be implemented. While this would constitute excessive (redundant) control, in traffic management, it would be considered a safety-oriented approach. Therefore, the risk associated with a "false detection" is relatively low. In other words, the impact of a "false detection" is relatively small. Consequently, the error inference coefficient α corresponding to "false detection" can be set relatively low.

[0062] Since an "omission" leads to the inference that an object does not exist at the actual location of the object, there is a possibility that collision avoidance controls for that object have not been implemented. In this case, in traffic management, there is a possibility that necessary collision avoidance controls have not been implemented. Therefore, the risk of an "omission" is relatively high. In other words, the impact of an "omission" is relatively large. Therefore, the error inference coefficient β corresponding to an "omission" can be set relatively high.

[0063] Since a "replacement" implies the presence of an object at the actual location of the object, the impact (risk) is less than that of an "omission." However, if the ID is replaced midway, as explained below, there is a possibility that the movement direction and speed of multiple objects cannot be accurately inferred.

[0064] For example, in Figure 3 In the example, based on the actual movement trajectory of object P3 from t=1 to 3, object P3 should actually be predicted to move in the direction indicated by arrow A3. However, when using the inference algorithm, based on the movement trajectories of rectangles R31, R32, and R33 marked with "ID: #03", the object corresponding to that "ID: #03" might be predicted to move in the direction indicated by dashed arrow A3'. Similarly, based on the actual movement trajectory of object P4 from t=1 to 3, object P4 should actually be predicted to move in the direction indicated by arrow A4. However, when using the inference algorithm, based on the movement trajectories of rectangles R41, R42, and R43 marked with "ID: #04", the object corresponding to that "ID: #04" might be predicted to move in the direction indicated by dashed arrow A4'.

[0065] Similarly, as explained below, the movement speeds of objects P3 and P4 may not be accurately predicted. In particular, when there is a significant difference between the actual movement speed (speed per hour) of object P3 and the actual movement speed (speed per hour) of object P4, there is a possibility that the movement speeds of both cannot be accurately predicted.

[0066] Specifically, in Figure 3 In the example, suppose that although both objects P3 and P4 are moving at roughly the same speed, object P3 moves faster than object P4. In this case, the difference (distance moved) between the position of object P3 at t=3 and the position of object P3 at t=2 is actually roughly the same as the difference (distance moved) between the position of object P3 at t=2 and the position of object P3 at t=1. On the other hand, the difference between the positions of rectangle R33 and rectangle R32 may be smaller than the difference between the positions of rectangle R32 and rectangle R31. Therefore, the speed of the object corresponding to "ID: #03" (object P3) may be inferred to be drastically slower. Therefore, the speed of subsequent objects corresponding to "ID: #03" may also be predicted to be slower than the actual speed of object P3.

[0067] Similarly, the differences between the position of object P4 at t=3 and the position of object P4 at t=2 are roughly the same as the differences between the position of object P4 at t=2 and the position of object P4 at t=1. On the other hand, the difference between the position of rectangle R43 and the position of rectangle R42 may be greater than the difference between the position of rectangle R42 and the position of rectangle R41. Therefore, the speed of the object corresponding to "ID: #04" (object P4) may be inferred to be rapidly increasing. Therefore, the speed of subsequent objects corresponding to "ID: #04" may also be predicted to be faster than the actual speed of object P4.

[0068] As mentioned above, under traffic control, it is possible to predict when an object will arrive at a specific location, such as an intersection, within a few seconds based on its direction and speed of movement. This allows for collision avoidance control. However, when the direction and speed of movement cannot be accurately predicted, the arrival time may not be accurately predicted. Therefore, collision avoidance control may not be properly implemented. Consequently, the impact (risk) of a "replacement" scenario is slightly greater.

[0069] Since a "new error detection" infers the presence of certain objects at locations where objects actually exist, its impact (risk) is less than that of an "omission." However, because an object cannot be accurately tracked, similar to the "replacement" case, it's possible that the object's direction and speed of movement may not be accurately inferred. On the other hand, since the inference is more likely to be performed correctly after a new ID is added, the impact (risk) of a "new error detection" is less than that of a "replacement."

[0070] For example, in Figure 3In the example, object P2 has different IDs at t=2 and t=3. Therefore, the movement direction and speed of object P2 cannot be inferred based on the difference between the position of object P2 (rectangle R2'3) at t=3 and the position of object P2 (rectangle R2'2) at t=2. On the other hand, object P2 has the same ID (ID: #12) at t=3 and t=4. Therefore, the movement direction and speed of object P2 can be appropriately inferred based on the difference between the position of object P2 (rectangle R2'4) at t=4 and the position of object P2 (rectangle R2'3) at t=3. Furthermore, let's assume that the movement direction and speed of object P2 are approximately fixed. In this case, the movement direction and speed of object P2 (ID: #02) inferred from the movement trajectory at t=1 to 2 can be approximately the same as the movement direction and speed of object P2 (ID: #12) inferred from the movement trajectory at t=3 to 4. Therefore, the impact of a "new error detection" is less than the impact of a "replacement". Thus, the impact (risk) of a "new error detection" is slightly smaller.

[0071] Based on the above, the error inference coefficients α, β, γ, and δ can be set according to the following equation (1). Furthermore, in Embodiment 1, the error inference coefficients are set to be fixed (unchanged from their initial values).

[0072] Mathematical formula 1:

[0073] β>(γ+δ)>α

[0074] 1 > δ > γ = 1 - δ > 0

[0075] …(1)

[0076] That is, since the impact of "omission (FN)" is the greatest, the error inference coefficient β corresponding to "omission (FN)" (first error inference) is set higher than the error inference coefficients (α, γ, δ) corresponding to other error inference types. Furthermore, the impact of "substitution (IDSWc)" is greater than that of "new error detection (IDSWn)". Therefore, the error inference coefficient δ corresponding to "substitution (IDSWc)" (second error inference) is set higher than the error inference coefficient γ corresponding to "new error detection (IDSWn)". Moreover, γ + δ = 1.

[0077] Furthermore, in Implementation 1, the evaluation formula used to evaluate the inference algorithm in the image frame obtained at time t can be expressed as the following equation (2).

[0078] Mathematical formula 2:

[0079]

[0080] In equation (2), GT(t) represents the number of actual objects contained in the forward data corresponding to the image frame at time t. Furthermore, FP(t) represents the number of objects (rectangles) incorrectly identified as "error detection (FP)" for the image frame at time t. Additionally, FN(t) represents the number of objects incorrectly identified as "omission (FN)" for the image frame at time t. Furthermore, IDSWn(t) represents the number of objects (rectangles) incorrectly identified as "new error detection (IDSWn)" for the image frame at time t. Finally, IDSWc(t) represents the number of objects (rectangles) incorrectly inferred as "replacement (IDSWc)" for the image frame at time t.

[0081] The numerator of the fraction in the second term on the right-hand side of equation (2) corresponds to the sum of the values ​​obtained by multiplying the number of corresponding objects by the error inference coefficient for each of the multiple error inference types (α×FP(t), β×FN(t), γ×IDSWn(t), and δ×IDSWc(t)). In this sense, the value obtained by multiplying the number of corresponding objects by the error inference coefficient for each of the multiple error inference types is the sum of the error inference coefficients corresponding to the number of objects corresponding to each of the multiple error inference types. For example, “β×FN(t)” corresponds to the value (Σβ) obtained by adding the error inference coefficient β to the number of objects corresponding to “omission (FN)” (FNt). Therefore, this summation is the sum of the error inference coefficients obtained by adding the corresponding error inference coefficients for each of the multiple error inference types. Moreover, as shown in equation (2), the larger this summation is, the smaller the evaluation value F(t). In other words, the larger the total value, the lower the evaluation of the inference algorithm.

[0082] Here, in Luiten2021, all the error inference coefficients in Equation (2) are set to 1, thereby calculating the evaluation value of the inference algorithm. Therefore, in Luiten2021, the evaluation value is calculated without considering the risk of each of the multiple error inference types, that is, the degree (severity) of the impact of the error inference. Therefore, even if the inference algorithm has a high evaluation even with the application of Luiten2021 technology, it does not necessarily mean that there are no error inferences with a high risk (degree of impact). Therefore, it may be impossible to properly evaluate the quality of the inference algorithm. That is to say, for example, when the number of objects corresponding to "missed (FN)" is large in one inference algorithm and the number of objects corresponding to "false detection (FP)" is large in another inference algorithm, the quality of the two inference algorithms may be evaluated as equal.

[0083] In contrast, in Implementation 1, the error inference coefficient is set differently for each of the multiple error inference types. Specifically, the error inference coefficient is set such that the greater the influence of the corresponding error inference type, the higher the error inference coefficient. Therefore, the more error inferences with a greater influence are made, the lower the evaluation value may be. In other words, the more error inferences with a greater influence are made, the lower the evaluation of the inference algorithm may be. Here, it can be said that the quality of the inference algorithm is lower if more error inferences with a greater influence are made. Therefore, in Implementation 1, the quality of the inference algorithm can be appropriately evaluated. In addition, the term "higher error inference coefficient" is not limited to a large value of the error inference coefficient. The value of the error inference coefficient itself may decrease as the influence increases. In this case, "higher error inference coefficient" corresponds to the case where the value of the error inference coefficient is small. Furthermore, in this case, the smaller the sum of the error inference coefficients obtained by adding the corresponding error inference coefficients according to the number of objects corresponding to each of the multiple error inference types, the lower the evaluation of the inference algorithm may be.

[0084] Furthermore, in Implementation 1, the error inference coefficient β corresponding to "omission (FN)" (first error inference) among the plurality of error inference types is set higher than the error inference coefficients corresponding to other error inference types. As described above, the impact of "omission (FN)" is the greatest among the plurality of error inference types. Therefore, by setting the error inference coefficient β higher than the error inference coefficients corresponding to other error inference types, the quality of the inference algorithm can be evaluated more appropriately. In other words, the evaluation of inference algorithms that have performed more extensively on "omission (FN)" errors can be reduced.

[0085] Furthermore, in Implementation 1, the error inference coefficient δ corresponding to "Replacement (IDSWc)" (second error inference) is set higher than the error inference coefficient γ corresponding to "New Error Detection (IDSWn)" (third error inference). As described above, the impact of "Replacement (IDSWc)" is greater than that of "New Error Detection (IDSWn)". Therefore, by setting the error inference coefficient δ higher than the error inference coefficient γ, the quality of the inference algorithm can be evaluated more appropriately. In other words, the evaluation of inference algorithms that perform "Replacement (IDSWc)" with a greater impact than "New Error Detection (IDSWn)" can be reduced.

[0086] Figure 4 This is a diagram illustrating the structure of the evaluation device 100 according to Embodiment 1. Figure 4 As shown, the evaluation device 100 includes a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 (IF). The control unit 102, storage unit 104, communication unit 106, and interface unit 108 are interconnected via a data bus or the like. Additionally, the tracking engine 20 and the dynamic image storage device 40 may also be included. Figure 4 The hardware structure of the evaluation device 100 shown.

[0087] The control unit 102 is a processor, such as a CPU (Central Processing Unit). The control unit 102 functions as a computing device that performs control processing and arithmetic processing. Furthermore, the control unit 102 may have multiple processors. The storage unit 104 is a storage device, such as a memory or hard disk. The storage unit 104 is, for example, a ROM (Read Only Memory) or RAM (Random Access Memory). The storage unit 104 has the function of storing control programs and arithmetic programs executed by the control unit 102. That is, the storage unit 104 (memory) stores more than one command. In addition, the storage unit 104 has the function of temporarily storing processing data. The storage unit 104 may contain a database. Furthermore, the storage unit 104 may have multiple memories.

[0088] The communication unit 106 performs the processing required for communication with other devices such as the tracking engine 20 or the dynamic image storage device 40 via a network. The communication unit 106 may include a communication port, router, firewall, etc. The interface unit 108 (IF; Interface) is, for example, a user interface (UI). The interface unit 108 has input devices such as a keyboard, touch panel, or mouse, and output devices such as a display or speaker. The interface unit 108 may also be configured such that the input and output devices are integrated, for example, a touch screen (touch panel). The interface unit 108 accepts data input operations performed by the user (operator) and outputs information to the user.

[0089] In the evaluation apparatus 100 according to Embodiment 1, the structural elements include a positive solution data acquisition unit 112, an output data acquisition unit 114, an inference result determination unit 120, a parameter storage unit 130, an evaluation value calculation unit 160, and an evaluation value output unit 162. Each of these structural elements can be implemented, for example, by executing a program under the control of the control unit 102. More specifically, each structural element can be implemented by the control unit 102 executing a program (command) stored in the storage unit 104. Alternatively, each structural element can be implemented by pre-recording the required program on any non-volatile recording medium and installing it as needed. Furthermore, each structural element is not limited to being implemented by software formed by a program; it can also be implemented by any combination of hardware, firmware, and software. Additionally, each structural element can be implemented using a user-programmable integrated circuit, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, the program composed of the aforementioned structural elements can also be implemented using this integrated circuit. These situations are also the case in Embodiment 2 described later.

[0090] The forward resolution data acquisition unit 112 acquires forward resolution data corresponding to the dynamic image that has undergone inference processing and tracking processing by the inference algorithm from the dynamic image storage device 40. The forward resolution data acquisition unit 112 may also acquire forward resolution data for each image frame constituting the dynamic image. The output data acquisition unit 114 acquires output data generated by the inference processing and tracking processing of the inference algorithm from the tracking engine 20. The output data acquisition unit 114 may also acquire output data for each image frame constituting the dynamic image.

[0091] The inference result determination unit 120 uses the acquired correct solution data and output data to determine the inference result for each image frame. Specifically, the inference result determination unit 120 compares the correct solution data and output data, and determines whether the inference result is correct or one of several types of incorrect inference for each object. More specifically, the inference result determination unit 120 determines which of the following categories the inference result for each object (rectangle) is correct: "TP", "FP", "FN", "IDSWn", and "IDSWc". More specifically, the inference result determination unit 120 determines the inference result by determining whether the type and position of the object contained in the correct solution data are correctly inferred in the output data. In addition, the inference result determination unit 120 determines the inference result by determining whether the same ID is added in a series of image frames for the same object contained in the correct solution data.

[0092] exist Figure 3 In the example, for the image frame at t=1, since the position of rectangle R11 corresponds to the position of object P1, the position of object P1 contained in the positive solution data is correctly inferred. Therefore, the inference result determination unit 120 determines that the inference result for object P1 is "correct inference (TP)". Similarly, the inference result determination unit 120 determines that the inference results for other objects P2 to P4 are also "correct inference (TP)".

[0093] Furthermore, regarding the image frame at t=2, since the position of rectangle R12 corresponds to the position of object P1, the position of object P1 contained in the correct solution data is correctly inferred. Further, the ID (#01) added to rectangle R12 is the same as the ID added to rectangle R11 at t=1. Therefore, the inference result determination unit 120 determines that the inference result for object P1 is "correct inference (TP)". Similarly, the inference result determination unit 120 determines that the inference results for other objects P2 to P4 are also "correct inference (TP)". Furthermore, regarding the image frames at t=3 to 4, the inference result determination unit 120 also determines, in the same way as above, that the inference result for object P1 is "correct inference (TP)".

[0094] Furthermore, regarding the image frame at t=3, the position of rectangle R2'3 corresponds to the position of object P2. However, the ID (#12) added to rectangle R2'3 is different from the ID (#02) added to rectangle R22 corresponding to object P2 at t=2. Therefore, the inference result determination unit 120 determines that the inference result of rectangle R2'3 corresponding to object P2 is "New Error Detection (IDSWn)". On the other hand, regarding the image frame at t=4, the position of rectangle R2'4 corresponds to the position of object P2, and the ID (#12) added to rectangle R2'4 is the same as the ID (#12) added to rectangle R2'3 corresponding to object P2 at t=3. Therefore, the inference result determination unit 120 determines that the inference result of object P2 (rectangle R2'4) is "Correct Inference (TP)".

[0095] Furthermore, regarding the image frame at t=3, although it is inferred that object P2 exists at the position of rectangle R23, no object actually exists at the position of rectangle R23. Therefore, the inference result determination unit 120 determines that the inference result of rectangle R23 near object P2 is "false detection (FP)". The same applies to the image frame at t=4.

[0096] Furthermore, regarding the image frame at t=3, the position of rectangle R33 corresponds to the position of object P4. Here, the ID (#03) added to rectangle R33 is different from the ID (#04) added to rectangle R42 corresponding to object P4 at t=2, but the same as the ID (#03) added to rectangle R32 corresponding to object P3 at t=2. Therefore, the inference result determination unit 120 determines that the inference result for object P4 is "replacement (IDSWc)".

[0097] Similarly, for the image frame at t=3, the position of rectangle R43 corresponds to the position of object P3. Here, the ID (#04) added to rectangle R43 is different from the ID (#03) added to rectangle R32 corresponding to object P3 at t=2, but the same as the ID (#04) added to rectangle R42 corresponding to object P4 at t=2. Therefore, the inference result determination unit 120 determines that the inference result for object P3 is "replacement (IDSWc)".

[0098] Furthermore, regarding the image frame at t=5, although it is inferred that object P1 exists at the position of rectangle R15, no object exists at the position of rectangle R15. Therefore, the inference result determination unit 120 determines that the inference result of rectangle R15 near object P1 is "error detection (FP)". The same applies to the image frame at t=6.

[0099] Furthermore, regarding the image frame at t=5, there is no rectangle at the position corresponding to object P1. Therefore, the inference result determination unit 120 determines that the inference result for object P1 is "omission (FN)". The same applies to the image frame at t=6.

[0100] Furthermore, the inference result determination unit 120 counts the number of objects (rectangles) corresponding to correct inferences and various types of incorrect inferences for each image frame. Specifically, the inference result determination unit 120 counts the number of objects determined to be "correct inference (TP)" for each image frame. Similarly, the inference result determination unit 120 counts the number of objects determined to be "missed (FN)" for each image frame. Similarly, the inference result determination unit 120 counts the number of objects determined to be "replacement (IDSWc)" for each image frame. In addition, the inference result determination unit 120 counts the number of rectangles determined to be "error detection (FP)" for each image frame. Similarly, the inference result determination unit 120 counts the number of rectangles determined to be "new error detection (IDSWn)" for each image frame.

[0101] exist Figure 3 In the example, in the image frame t=3, the inference result determination unit 120 counts the number of "correct inferences (TP)" as 1. Similarly, in the image frame t=3, the inference result determination unit 120 counts the number of "error detections (FP)" as 1, the number of "new error detections (IDSWn)" as 1, and the number of "replacements (IDSWc)" as 2. Furthermore, in the image frame t=5, the inference result determination unit 120 counts the number of "error detections (FP)" as 1 and the number of "omissions (FN)" as 1.

[0102] The parameter storage unit 130 stores the parameters used in calculating the evaluation value. In Embodiment 1, the parameter storage unit 130 stores the error inference coefficients α, β, γ, and δ described above. Furthermore, the parameter storage unit 130 may also temporarily store the evaluation value F(t) calculated for each image frame through the processing described later.

[0103] The evaluation value calculation unit 160 calculates the evaluation value of the inference algorithm. Specifically, the evaluation value calculation unit 160 uses the evaluation formula shown in equation (2) to calculate the evaluation value. The evaluation value calculation unit 160 calculates the evaluation value for each image frame (timing t). Then, when the evaluation value is calculated for all image frames, the evaluation value calculation unit 160 sums up the evaluation values ​​calculated for all image frames as shown in equation (3) below, and calculates the overall evaluation value F of the inference algorithm for the entire dynamic image. For example, when the inference algorithm performs inference processing (tracking processing) for a 5-minute dynamic image with a frame rate of 10fps, the evaluation value calculation unit 160 sums up the evaluation values ​​obtained from 10×60×5=3000 image frames, and calculates the overall evaluation value F.

[0104] Mathematical formula 3:

[0105]

[0106] The evaluation value output unit 162 outputs the evaluation value calculated by the evaluation value calculation unit 160. The evaluation value output unit 162 can also display the evaluation value on the display of the interface unit 108, for example. Furthermore, the evaluation value output unit 162 can also control the communication unit 106, thereby causing the display of other devices (such as user terminals) to display the evaluation value. The evaluation value output unit 162 can also output the evaluation value via voice or the like. The evaluation value output unit 162 can output either the overall evaluation value F or the evaluation value F(t) for each image frame.

[0107] Figure 5 Here is a flowchart illustrating the evaluation method performed by the evaluation system 1 according to Implementation 1. Figure 5 The processing shown can be performed primarily by the evaluation device 100. First, inference processing implemented by the inference algorithm is carried out by the tracking engine 20 (step S100). Furthermore, after this inference processing (S100) is performed on all image frames, processing after S101 can be performed.

[0108] In the evaluation device 100, the t-th image frame (the image frame at time t) is set as the processing target when t=1 (step S101) (step S102). That is, in the evaluation device 100, the first image frame is first set as the processing target. Then, the evaluation device 100 performs the processing of S104 to S150 described later with respect to the first image frame, and if there is an unprocessed image frame ("Yes" in S160), t is incremented by one (step S162), thereby setting the second image frame as the processing target (S102). In the evaluation device 100, the processing of S104 to S150 is performed on the t-th image frame in the same manner thereafter.

[0109] The correct solution data acquisition unit 112 acquires correct solution data for the t-th image frame (step S104). The output data acquisition unit 114 acquires output data related to the inference processing performed for the t-th image frame (step S106). The inference result determination unit 120 determines the inference result related to the t-th image frame in the manner described above (step S110). Specifically, in the inference result determination unit 120, for the t-th image frame, it determines "correct inference (TP)," "error detection (FP)," "omission (FN)," "new error detection (IDSWn)," or "replacement (IDSWc)" for objects and rectangles. As described above, the inference result determination unit 120 counts the number of objects for each type of error inference (step S112). In the evaluation value calculation unit 160, for the t-th image frame, the evaluation value of the inference algorithm is calculated using the evaluation formula shown in equation (2) (step S150).

[0110] Then, the evaluation device 100 determines whether there are any unprocessed image frames (step S160). If there are unprocessed image frames (S160 "Yes"), the evaluation device 100 increments t by one (step S162). Then, the processing flow returns to S102, and the evaluation device 100 sets the next image frame as the processing target (S102). Then, the processing of S104 to S150 is repeated.

[0111] On the other hand, if there are no unprocessed image frames (No in S160), as shown in Equation (3), the evaluation value calculation unit 160 sums up the evaluation values ​​obtained for all image frames (step S164). Then, the evaluation value output unit 162 outputs the overall evaluation value of the dynamic image (step S166).

[0112] (Implementation Method 2)

[0113] Next, implementation method 2 will be described. Furthermore, regarding the structure of the evaluation system 1 involved in implementation method 2, since it is related to... Figure 1 The structure of the evaluation system 1 shown in Embodiment 1 is substantially the same, so the description is omitted. The evaluation device 100 in Embodiment 2 differs from the evaluation device 100 in Embodiment 1 in that the error inference coefficient is different for each object and can vary for each image frame.

[0114] Figure 6 The diagram illustrates the structure of the evaluation device 100 according to Embodiment 2. Similar to Embodiment 1, the evaluation device 100 according to Embodiment 2 includes a control unit 102, a storage unit 104, a communication unit 106, and an interface unit 108 as its main hardware components. Furthermore, the evaluation device 100 according to Embodiment 2 includes, as structural elements, a correct solution data acquisition unit 112, an output data acquisition unit 114, an inference result determination unit 120, a parameter storage unit 130, a coefficient calculation unit 240, an evaluation value calculation unit 260, and an evaluation value output unit 162. The coefficient calculation unit 240 includes a continuous error inference calculation unit 242, an object group calculation unit 244, and a speed calculation unit 246.

[0115] Furthermore, the functions of the correct solution data acquisition unit 112, the output data acquisition unit 114, the inference result determination unit 120, the parameter storage unit 130, and the evaluation value output unit 162 are essentially the same as those described in Embodiment 1, and therefore will not be described further. Additionally, the parameter storage unit 130 may also store the constants used in calculating the error inference coefficients, such as a in equation (4) and c in equation (6) as described later. Furthermore, the parameter storage unit 130 may also temporarily store the error inference coefficients calculated for each object (rectangle) in each image frame through the processing described later.

[0116] The coefficient calculation unit 240 calculates the error inference coefficient as a variable for each image frame. That is, the coefficient calculation unit 240 calculates the error inference coefficient for the timing of each inference. Furthermore, the coefficient calculation unit 240 calculates the error inference coefficient for each object (rectangle) in the image frame that is determined to be any one of multiple error inference types. In addition, the processing of the continuous error inference calculation unit 242, the object group calculation unit 244, and the speed calculation unit 246 for a certain object can be executed either in parallel or sequentially. When executed in parallel, the coefficient calculation unit 240 can also set the sum or average of the values ​​calculated by these processes as the error inference coefficient related to that object. Regarding the sequential execution, the method described later will be used. Figure 8 Let me explain.

[0117] For the state of an object corresponding to each type of error inference, even if the error inference type is the same, it may differ for each object. Furthermore, even with the same type of error inference, the degree of its impact may vary depending on the object's state. Moreover, the state of the object corresponding to each type of error inference changes for each image frame. Furthermore, the degree of impact of the error inference also changes according to the change in the object's state. Additionally, the degree of impact of each type of error inference may change depending on the occurrence of error inference in previous image frames. In this way, the degree of impact of each type of error inference is not necessarily fixed throughout the entire moving image. Therefore, the coefficient calculation unit 240 according to Embodiment 2 calculates the error inference coefficients for each image frame. Thus, the error inference coefficients can be changed according to changes in the situation. Therefore, in Embodiment 2, the quality evaluation of the inference algorithm can be implemented more appropriately.

[0118] When the consecutive error inference calculation unit 242 performs consecutive error inferences of the same type on a corresponding object, it calculates the error inference coefficient by increasing the error inference coefficient corresponding to that error inference type associated with the object. Specifically, the consecutive error inference calculation unit 242 calculates the error inference coefficient in such a way that the more times the consecutive error inferences of the same type are performed on the corresponding object, the higher the error inference coefficient corresponding to that error inference type associated with the object.

[0119] More specifically, when the continuous error inference calculation unit 242 performs an error inference of "omission (FN)" for a certain object, for example, it will use the following equation (4) to calculate the error inference coefficient. In addition, the continuous error inference calculation unit 242 can also use the same method as equation (4) to calculate the error inference coefficient corresponding to other error inference types.

[0120] Mathematical formula 4:

[0121] ifc == FN:

[0122] β i (t)=β i (t-1)×(1+a)

[0123] else

[0124] β i (t)=β0 ...(4)

[0126] Here, i is the index of the object in the image frame that has been "missed (FN)". Furthermore, in equation (4), c represents the type of incorrect inference determined for object i in the image frame at time (t-1). Additionally, a is a weighting constant, and is a constant greater than 0. Furthermore, a can vary depending on the type of incorrect inference applied in equation (4). For example, the value of a when calculating β can be greater than the value of a when calculating α. Furthermore, β0 is the initial value of the incorrect inference coefficient β. Additionally, β... i (t) represents the value of the error inference coefficient β corresponding to object i in the image frame at time t.

[0127] As shown in equation (4), for object i, if it was judged as "omission (FN)" at the previous timing (t-1) and is also judged as "omission (FN)" at the current timing t, the value of β obtained by multiplying the previous value of β by (1+a) becomes the value of β in the current time. Here, (1+a) is greater than 1. Therefore, the error inference coefficient β is higher than the previous one. Furthermore, whenever object i is judged as "omission (FN)" consecutively, the error inference coefficient β is multiplied by (1+a). Therefore, the more times it is judged as "omission (FN)" consecutively, the higher the error inference coefficient β is. In addition, equation (4) is an example of the operation of the continuous error inference operation unit 242. When the continuous error inference operation unit 242 makes the same error inference consecutively for a certain object, other formulas such as the error inference coefficient increasing can also be used to calculate the error inference coefficient.

[0128] exist Figure 3 In the example, at time t=5, object P1 is judged as "missed (FN)", and even at the next time t=6, object P1 is still judged as "missed (FN)". Therefore, the error inference coefficient corresponding to object P1 is calculated as β at t=5. i (5) = β0, which is calculated as β at t = 6. i (6)=β0×(1+a).

[0129] Even if an incorrect inference is made for a single object for a fleeting moment, if a correct inference is made at the next timing, the object can still be properly detected and tracked. Therefore, the impact is not significant. On the other hand, if incorrect inferences occur consecutively, object detection and tracking may not be properly performed during that period. Moreover, in situations such as traffic control, if the period during which object detection and tracking cannot be properly performed becomes longer, collision avoidance control for that object may not be properly implemented during that period. Therefore, the impact (risk) increases. Thus, it can be said that inference algorithms that undergo such inference processing have lower quality. Conversely, it is desirable for inference algorithms to avoid making consecutive incorrect inferences for a single object.

[0130] In contrast, the evaluation device 100 according to Embodiment 2 calculates the error inference coefficient by increasing the error inference coefficient corresponding to the error inference type associated with the object when consecutive error inferences of the same type are performed regarding a certain object. Therefore, the error inference coefficient can be increased when the degree of influence increases due to consecutive error inferences. Thus, the quality evaluation of the inference algorithm that performs inference processing that increases the degree of influence by making consecutive error inferences can be reduced. Conversely, the quality evaluation of the inference algorithm that is less likely to produce consecutive error inferences can be improved. Therefore, the quality evaluation of the inference algorithm can be performed more appropriately.

[0131] Furthermore, if a series of incorrect "missed (FN)" inferences are made regarding a particular object, the object may not be detected in subsequent image frames, and therefore cannot be tracked. In other words, the object may be completely missed. In this case, collision avoidance control for that object may be completely impossible to implement. Therefore, the impact becomes quite significant. Thus, it can be said that the quality of an inference algorithm that makes consecutive "missed (FN)" incorrect inferences is quite low. For example, for pedestrians wearing clothing of a specific color, the inference algorithm may sometimes make consecutive "missed (FN)" incorrect inferences.

[0132] In contrast, the evaluation device 100 according to Embodiment 2 calculates the error inference coefficient by increasing the error inference coefficient β when the erroneous inference of "omission (FN)" is made continuously (first error inference). Therefore, when the impact of continuous error inference of "omission (FN)" is greatly amplified, the error inference coefficient β can be increased. Thus, the quality evaluation of the inference algorithm that makes continuous error inference of "omission (FN)" can be reduced. Therefore, the evaluation device 100 according to Embodiment 2 can more appropriately evaluate the quality of the inference algorithm.

[0133] The object group calculation unit 244 calculates the error inference coefficient in such a way that the more other objects that are within a predetermined threshold distance of a certain object, the lower the error inference coefficient corresponding to the error inference type associated with that object. Here, "other objects" refers to objects of the same type as the object. In other words, the object group calculation unit 244 lowers the error inference coefficient below the initial value when there are many other objects of the same type around the object. For example, when a certain object is a "pedestrian" and there are many other "pedestrians" around the object, the error inference coefficient for that object will be lower.

[0134] Specifically, for example, if the object group calculation unit 244 makes an erroneous inference of "omission (FN)" for a certain object i, it will use the following equation (5) to calculate the erroneous inference coefficient. In addition, the object group calculation unit 244 also uses the same method as equation (5) to calculate the erroneous inference coefficient corresponding to other types of erroneous inference.

[0135] Mathematical formula 5:

[0136]

[0137] Here, n is the number of other objects of the same type as object i that are within a distance of a threshold from object i. According to equation (5), the larger n is, the lower the false inference coefficient β is. Equation (5) represents an example of the operation of the object group operation unit 244. The object group operation unit 244 can also use other formulas, such as the larger the number of other objects n, the lower the false inference coefficient, to calculate the false inference coefficient.

[0138] As described above, the evaluation device 100 of Embodiment 2 calculates the error inference coefficient by reducing the error inference coefficient corresponding to the type of error inference associated with an object as the number of other objects whose distance from a certain object is below a predetermined threshold increases. Here, when objects of the same type as an object exist around that object, these multiple objects can be treated as an "object group." Therefore, even if one object in the object group cannot be properly detected (e.g., "omission"), the impact is not significant since collision avoidance control can be implemented for the object group. In other words, the impact of such an error inference is smaller compared to an error inference occurring when no objects of the same type exist in the vicinity. That is, it is less necessary to specifically reduce the quality evaluation of the inference algorithm that made such an error inference. Therefore, the evaluation device 100 of Embodiment 2 can more appropriately evaluate the quality of the inference algorithm.

[0139] The speed calculation unit 246 calculates the error inference coefficient based on the speed of the corresponding object. Specifically, the speed calculation unit 246 calculates the error inference coefficient in such a way that the faster the object's speed, the higher the error inference coefficient corresponding to the error inference type associated with that object. More specifically, for example, regarding the error inference of "omission (FN)," the speed calculation unit 246 uses the following equation (6) to calculate the error inference coefficient. In addition, the speed calculation unit 246 can also use the same method as equation (6) to calculate the error inference coefficient corresponding to other error inference types.

[0140] Mathematical formula 6:

[0141] β i (t)=β0+C×v i (t)

[0142] …(6)

[0143] Here, v i (t) represents the velocity of object i at time t. i (t) is calculated based on the distance object i moves from time (t-1) to time t, and the time between the image frame at time (t-1) and the image frame at time t. In other words, v i (t) is calculated based on the moving distance of object i and the frame rate. Furthermore, in equation (6), c is a constant greater than 0. Additionally, c can vary depending on the type of error inference applied to equation 6. For example, the value of c can be different when calculating β versus when calculating α.

[0144] According to equation (6), the velocity v of object i i The faster (t) is, the higher the error inference coefficient β is. Furthermore, equation (6) represents an example of the operation of the speed calculation unit 246. The speed calculation unit 246 can also use other formulas, such as "the faster the object's speed, the higher the error inference coefficient," to calculate the error inference coefficient.

[0145] If an incorrect inference is made regarding a fast-moving object, the object will travel a longer distance during the period when it cannot be properly detected. Therefore, the impact is significant because it is difficult to properly control collision avoidance for that object. In particular, the faster the object's speed, the longer the distance it travels during the period when it cannot be properly detected, thus amplifying the impact. Furthermore, if an incorrect inference of "missed (FN)" is made regarding a fast-moving object, the object will travel a longer distance during the period when the inference algorithm cannot detect it at all (missed). Therefore, the impact of incorrect inference of "missed (FN)" for fast-moving objects is particularly significant. Conversely, it is desirable for the inference algorithm to avoid making incorrect inferences regarding fast-moving objects.

[0146] In contrast, the evaluation device 100 according to Embodiment 2 calculates the error inference coefficient by increasing the error inference coefficient corresponding to the type of error inference associated with the object as the object's speed increases. Therefore, the error inference coefficient can be increased when the impact of an error inference against a faster object is amplified. Consequently, the quality evaluation of inference algorithms that perform inference processing that increases the impact of an error inference against a faster object can be reduced. Conversely, the quality evaluation of inference algorithms that are less prone to making error inferences even against faster objects can be improved. Therefore, the evaluation device 100 according to Embodiment 2 can more appropriately evaluate the quality of inference algorithms.

[0147] The evaluation value calculation unit 260 calculates the evaluation value of the inference algorithm. In Embodiment 1, the error inference coefficient is fixed for each type of error inference. In contrast, in Embodiment 2, the error inference coefficient is calculated for each object (rectangle). That is, even if the same type of error inference is made for multiple objects, the error inference coefficient may be different for each object.

[0148] Therefore, in Embodiment 2, the evaluation value calculation unit 260 will use the evaluation formula shown in Equation (7) to calculate the evaluation value. Similar to Embodiment 1, the evaluation value calculation unit 260 calculates the evaluation value for each image frame (timing t).

[0149] Mathematical expression 7:

[0150]

[0151] The second term of the numerator of the fraction of the second term on the right side of equation (7) (Σβ) i (t) shows the error inference coefficient β associated with the object i that was subject to "omission (FN)" in the image frame at time t, based on the number of objects (FN(t)) corresponding to "omission (FN)". i (t) is the case of addition. In other words, the second term of the numerator of the fraction of the second term on the right side of equation (7) corresponds to the value obtained by adding the error inference coefficients according to the number of objects corresponding to the type of error inference.

[0152] The first term (Σα) of the numerator of the second term on the right side of equation (7) h In (t)), h is the index of the rectangle (object) to which "error detection (FP)" was performed. Furthermore, α h (t) is the error inference coefficient calculated for the rectangle h in which "error detection (FP)" is performed in the image frame at time t. Therefore, the first term of the numerator of the second term on the right-hand side of equation (7) shows the error inference coefficient α associated with the rectangle h in the image frame at time t, based on the number (FP(t)) of objects (rectangles) corresponding to "error detection (FP)". h (t) is the case of addition. In other words, the first term of the numerator of the fraction of the second term on the right side of equation (7) corresponds to the value obtained by adding the error inference coefficients according to the number of objects (rectangles) corresponding to the type of error inference.

[0153] The third term (Σγ) is the numerator of the fraction of the second term on the right side of equation (7). j In (t)), j is the index of the rectangle (object) to which the "New Error Detection (IDSWn)" error inference was performed. Furthermore, γ j(t) is the error inference coefficient calculated for rectangle j in which "New Error Detection (IDSWn)" was performed in the image frame at time t. Therefore, the third term of the numerator of the second term on the right-hand side of Equation (7) shows the error inference coefficient γ associated with rectangle j in the image frame at time t, based on the number (IDSWn(t)) of objects (rectangles) corresponding to "New Error Detection (IDSWn)". j (t) is the case of addition. In other words, the third term of the numerator of the fraction of the second term on the right side of equation (7) corresponds to the value obtained by adding the error inference coefficients according to the number of objects (rectangles) corresponding to the type of error inference.

[0154] The fourth term (Σδ) is the numerator of the fraction of the second term on the right side of equation (7). k In (t)), k is the index of the object to which the erroneous inference of "replacement (IDSWc)" was performed. Furthermore, δ k (t) is the error inference coefficient calculated for the object k in which "replacement (IDSWc)" was incorrectly inferred in the image frame at time t. Therefore, the fourth term of the numerator of the fraction of the second term on the right-hand side of equation (7) shows the error inference coefficient δ associated with the object k in the image frame at time t, according to the number of objects (IDSWc(t)) corresponding to "replacement (IDSWc)". k (t) is the case of addition. In other words, the fourth term of the numerator of the fraction of the second term on the right side of equation (7) corresponds to the value obtained by adding the error inference coefficients according to the number of objects corresponding to the type of error inference.

[0155] Therefore, the numerator of the fraction in the second term on the right-hand side of equation (7) corresponds to the sum of the values ​​obtained by adding the corresponding error inference coefficients according to the number of objects corresponding to each of the multiple error inference types. Moreover, as shown in equation (7), the larger this summation value is, the smaller the evaluation value F(t). That is, the larger this summation value is, the lower the evaluation of the inference algorithm. Furthermore, when the evaluation value is calculated for all image frames, as in equation (3) above, the evaluation value calculation unit 260 sums up the evaluation values ​​calculated for all image frames, and then calculates the overall evaluation value F of the inference algorithm for the entire dynamic image.

[0156] Figure 7 as well as Figure 8 Here is a flowchart illustrating the evaluation method performed by the evaluation system 1 according to embodiment 2. Figure 7 as well as Figure 8 The processing shown can be mainly performed by the evaluation device 100. First, with Figure 5 Similarly, in step S100, the inference process of the inference algorithm is performed by the tracking engine 20 (step S200). In the evaluation device 100, the t-th image frame is set as the processing object as t=1 (step S201) (step S202).

[0157] For the correct solution data acquisition unit 112, and Figure 5 Similarly, in S104, positive resolution data is obtained for the t-th image frame (step S204). The output data acquisition unit 114 and... Figure 5 Similarly, in step S106, output data related to the inference processing performed with respect to the t-th image frame is obtained (step S206). The inference result determination unit 120 and... Figure 5 Similarly, in step S110, the inference result related to the t-th image frame is determined (step S210). The inference result determination unit 120 and... Figure 5 Similarly, as described above, step S112 counts the number of objects for each type of incorrect inference (step S212). The coefficient calculation unit 240 calculates the corresponding incorrect inference coefficient for each object (rectangle) (step S220). Regarding the processing of S220, the following will be used... Figure 8 As will be described later.

[0158] When the error inference coefficient is calculated in S220, the evaluation value calculation unit 160 calculates the evaluation value of the inference algorithm for the t-th image frame using the evaluation formula shown in equation (7) above (step S250). Then, the evaluation device 100 determines whether there are any unprocessed image frames (step S260). If there are unprocessed image frames (yes in S260), the evaluation device 100 increments t by one (step S262). Then, the processing flow returns to S202, and the evaluation device 100 sets the next image frame as the processing target (S202). Then, the processing of S204 to S250 is repeated. On the other hand, if there are no unprocessed image frames (no in S260), the evaluation value calculation unit 160 sums the evaluation values ​​obtained for all image frames as shown in equation (3) above (step S264). Then, the evaluation value output unit 162 outputs the evaluation value of the entire dynamic image (step S266).

[0159] use Figure 8 The processing of S220 will be explained below. Figure 8 The diagram illustrates the handling of objects subjected to erroneous inferences (FN). Additionally, for other types of erroneous inferences, similar procedures can be performed. Figure 8 The same processing applies. Furthermore, the calculation of error inference coefficients associated with multiple error inference types can be performed in parallel.

[0160] The coefficient calculation unit 240 determines the object i of the processing object from the objects in the image frame of the processing object that have undergone "missed (FN)" erroneous inference (step S222). The object of the processing object can be determined based on the ID of each object. The coefficient calculation unit 240 sets the initial value of the erroneous inference coefficient (step S224). Specifically, the coefficient calculation unit 240 extracts the initial value (β0) of the erroneous inference coefficient β related to "missed (FN)" from the parameter storage unit 130 and sets the extracted initial value.

[0161] The continuous error inference calculation unit 242 determines, for the object i being processed, whether both the error inference determination in the previous image frame (t-1) and the error inference determination in the current image frame (t) are "missed (FN)" (step S230). If the error inference is determined to be "missed (FN)" for object i even in the previous image frame ("Yes" in S230), the continuous error inference calculation unit 242 will use the error inference coefficient β from the previous image frame. i The way in which (t-1) increases the error inference coefficient β i (t) is calculated (step S232). Specifically, the continuous error inference calculation unit 242 uses the above equation (4) to calculate the error inference coefficient β. i (t) is calculated. On the other hand, if an erroneous inference was not determined as "missed (FN)" in the previous image frame ("No" in S230), the processing in S232 will be skipped. That is, the continuous error inference calculation unit 242 is set to β. i (t)=β0.

[0162] The object group calculation unit 244 calculates the distance between object i of the processing object and other objects of the same type (step S234). Alternatively, the distance between objects can be calculated by inferring the actual positions of each object based on the pixels corresponding to object i and other objects, and then calculating the distance between the inferred actual positions. Alternatively, the distance between objects can be calculated in the image frame using the distance between the pixels corresponding to object i of the processing object and the pixels corresponding to other objects.

[0163] The object group calculation unit 244 determines whether there are other objects whose distance from object i is less than or equal to a threshold (step S236). While the distance threshold is, for example, around 5m in a real-world environment, it is not limited to this. The threshold can be appropriately set to a level that allows for the identification of one or more objects whose distance from object i is within the threshold as part of an object group with object i.

[0164] If there are other objects whose distance from the object i being processed is below a threshold ("Yes" in S236), the object group calculation unit 244 calculates the error inference coefficient based on the number of other objects (step S238). Specifically, the object group calculation unit 244 uses equation (5) to calculate the error inference coefficient β. i (t) is calculated. Furthermore, when the processing in S232 is implemented, the object group operation unit 244 can also replace β0 in equation (5) with β obtained in the processing in S232. i The value of (t).

[0165] On the other hand, if there are no other objects whose distance from the processed object i is below the threshold ("No" in S236), then the processing in S238 is skipped. That is, the object group operation unit 244 will... i (t) is set to keep the value of the error inference coefficient calculated by the continuous error inference operation unit 242 unchanged.

[0166] The speed calculation unit 246 calculates the error inference coefficient based on the speed of the object i being processed (step S240). Specifically, the speed calculation unit 246 calculates (infers) the speed of the object i being processed. Then, the speed calculation unit 246 uses equation (6) to calculate the error inference coefficient β. i (t) is calculated. Furthermore, if at least one of the processes S232 and S238 is performed, the speed calculation unit 246 can also replace β0 in equation (6) with β obtained before the processing in S238. i The value of (t).

[0167] The coefficient calculation unit 240 determines whether there are any unprocessed objects in the image frame of the object being processed (step S242). Figure 8 In the example, the coefficient calculation unit 240 determines whether there are any unprocessed objects among the objects corresponding to "omission (FN)". If there are unprocessed objects ("Yes" in S242), the processing flow returns to S222. Then, the object to be processed is determined from the unprocessed objects (S222), and the processing of S224 to S240 is repeated. On the other hand, if there are no unprocessed objects ("No" in S242), the processing flow proceeds to S250.

[0168] (Application Examples)

[0169] Furthermore, using the evaluation device 100 according to this embodiment, in the inference algorithm, a threshold for the consistency of the identification of the observed value and the inferred value can be determined in a way that minimizes the impact of erroneous inferences in situations such as traffic control. In other words, an inference algorithm with a higher evaluation can be obtained by determining a threshold that increases the evaluation result evaluated by the evaluation device 100. For example, if the threshold is set too high, "false detection (FP)" may easily occur. On the other hand, if the threshold is set too low, "missed detection (FN)" may easily occur. Therefore, for example, regarding cases where the threshold in the inference algorithm is set to 70%, 80%, or 90%, the evaluation device 100 can be used to evaluate the inference algorithm and a threshold with a higher evaluation value can be adopted.

[0170] Furthermore, the evaluation device 100 according to this embodiment can be used to detect anomalies in sensors used for object tracking in real time. Specifically, the evaluation device 100 obtains its own position information from an object such as an autonomous vehicle and sets this position information as positive resolution data. Simultaneously, an inference algorithm infers and tracks the position of the object (autonomous vehicle, etc.) and sets the inference result as output data. Then, the evaluation device 100 uses the positive resolution data and the output data to perform the evaluation processing of the inference algorithm as described above. The evaluation result at this time is then compared with the evaluation result under normal conditions, and if the evaluation is lower than a predetermined threshold, it can be determined that the sensor is abnormal.

[0171] (Modified example)

[0172] Furthermore, the present invention is not limited to the embodiments described above, and appropriate modifications can be made without departing from the spirit of the invention. For example, the order of the various steps (processes) in the flowchart described above can be appropriately changed. In addition, one or more steps (processes) in the flowchart described above can be omitted. For example, steps (processes) can also be omitted. Figure 7 The processing of S212.

[0173] Furthermore, in Embodiment 2, it is not necessary to perform processing on all of the continuous error inference calculation unit 242, object group calculation unit 244, and speed calculation unit 246 that constitute the coefficient calculation unit 240. In other words, in Figure 8 In the flowchart, the processes S230 to S232, S234 to S238, and any of the processes in S240 may not be implemented.

[0174] Furthermore, although the inference algorithm is assumed to be used in a traffic-controlled environment in the above embodiment, the application environment of the inference algorithm is not limited to traffic control. The evaluation device 100 according to this embodiment can be applied to inference algorithms for object detection in any environment other than traffic control.

[0175] The program described above includes a set of commands (or software code) used, when read into a computer, to cause the computer to perform one or more functions described in the embodiments. The program may also be stored on a non-transitory computer-readable medium or a physical storage medium. By way of example, and not limitation, a computer-readable medium or a physical storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM (read-only optical disc), digital versatile disc (DVD), Blu-ray disc or other optical disc storage, cassette tape, magnetic tape, disk storage, or other magnetic storage devices. The program may also be transmitted on a transient computer-readable medium or a communication medium. By way of example, and not limitation, a transient computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of transmission signals.

[0176] It will be apparent from the disclosure described herein that embodiments of this disclosure may be varied in many ways. Such variations should not be considered as departing from the spirit and scope of this disclosure, and all such modifications will be apparent to those skilled in the art and will be included within the scope of the claims.

Claims

1. An evaluation system for evaluating the quality of an algorithm that infers the position of a movable object in a moving image and performs tracking of said object, said evaluation system having: The inference result determination unit uses positive solution data corresponding to the dynamic image and output data representing the inference result of the algorithm that performed the inference on the dynamic image to determine, for each object, either a correct inference result or one of a plurality of incorrect inference types that are incorrect inference results. The evaluation value calculation unit adds up the error inference coefficients, which are set to correspond to each of the multiple error inference types and increase according to the degree of influence of the error inference type, based on the number of objects corresponding to each error inference type. The unit then calculates the evaluation value of the algorithm based on the sum of the summed error inference coefficients obtained for each of the multiple error inference types. The error inference coefficient corresponding to the second error inference among the multiple error inference types is set higher than the error inference coefficient corresponding to the third error inference, wherein, The second erroneous inference is a type of erroneous inference related to the case where the algorithm replaces and infers multiple objects contained in the correct solution data at a certain time and the next time. The third erroneous inference is a type of erroneous inference related to the case where the algorithm infers one object contained in the correct solution data as a different object at a certain time and the next time.

2. The evaluation system as described in claim 1, wherein, The error inference coefficient corresponding to the first error inference among the plurality of error inference types is set higher than the error inference coefficients corresponding to the other error inference types, wherein the first error inference is an error inference type related to the case where the algorithm cannot infer the object contained in the correct solution data.

3. The evaluation system as described in claim 1 or 2, wherein, It also has a coefficient calculation unit that calculates the erroneous inference coefficients for each inference timing.

4. The evaluation system as described in claim 3, wherein, When the coefficient calculation unit makes consecutive erroneous inferences of the same type for a certain object, it calculates the erroneous inference coefficient in a manner that makes the erroneous inference coefficient corresponding to the erroneous inference type associated with that object higher.

5. The evaluation system as described in claim 4, wherein, When the coefficient calculation unit makes a first incorrect inference consecutively, it calculates the incorrect inference coefficient in a manner that makes the incorrect inference coefficient corresponding to the first incorrect inference higher, wherein the first incorrect inference is an incorrect inference type related to the case where the algorithm cannot infer the object contained in the correct solution data.

6. The evaluation system as described in claim 3, wherein, The coefficient calculation unit calculates the error inference coefficient in such a way that the faster the object moves, the higher the error inference coefficient corresponding to the error inference type associated with that object becomes.

7. The evaluation system as described in claim 3, wherein, The coefficient calculation unit calculates the error inference coefficient in such a way that the more other objects of the same type as the object that are within a predetermined threshold distance from the object, the lower the error inference coefficient corresponding to the error inference type associated with the object.

8. An evaluation method for evaluating the quality of an algorithm that infers the position of a movable object in a moving image and performs tracking of said object, wherein in the evaluation method, Using the correct solution data corresponding to the dynamic image and the output data representing the inference result of the algorithm that performed the inference on the dynamic image, for each object, a determination is made as either a correct inference result or one of a plurality of incorrect inference types that are incorrect inference results. The error inference coefficients, each corresponding to one of the multiple error inference types and set to increase according to the degree of influence of that error inference type, are summed according to the number of objects corresponding to that error inference type. The evaluation value of the algorithm is then calculated based on the sum of the error inference coefficients obtained for each of the multiple error inference types. The error inference coefficient corresponding to the second error inference among the multiple error inference types is set higher than the error inference coefficient corresponding to the third error inference, wherein, The second erroneous inference is a type of erroneous inference related to the case where the algorithm replaces and infers multiple objects contained in the correct solution data at a certain time and the next time. The third erroneous inference is a type of erroneous inference related to the case where the algorithm infers one object contained in the correct solution data as a different object at a certain time and the next time.

9. A computer-readable medium having stored thereon a program for evaluating the quality of an algorithm that infers the position of a movable object in a moving image and performs tracking of said object, said program causing a computer to perform the following steps: The step involves using positive solution data corresponding to the dynamic image and output data representing the inference result of the algorithm that performed the inference on the dynamic image, thereby determining, for each object, whether it is a correct inference result or one of a plurality of incorrect inference types that are incorrect inference results. The steps involve summing the error inference coefficients, which correspond to each of the multiple error inference types and are set in a manner that increases according to the degree of influence of the error inference type, based on the number of objects corresponding to that error inference type, and calculating the evaluation value of the algorithm based on the sum of the summed error inference coefficients obtained for each of the multiple error inference types. The error inference coefficient corresponding to the second error inference among the multiple error inference types is set higher than the error inference coefficient corresponding to the third error inference, wherein, The second erroneous inference is a type of erroneous inference related to the case where the algorithm replaces and infers multiple objects contained in the correct solution data at a certain time and the next time. The third erroneous inference is a type of erroneous inference related to the case where the algorithm infers one object contained in the correct solution data as a different object at a certain time and the next time.

Images