Detection program, detection method, and information processing apparatus

Abstract

To prevent degradation in detection accuracy of an object detection model for an object of which the appearance changes in video data.SOLUTION: An information processing apparatus inputs each frame of video data to an object detection model to acquire a detection position on an image of an object in the frame and a confidence score. The information processing apparatus acquires a first result determining that an object exists in the frame if the confidence score is equal to or higher than a first threshold and a second result determining that an object exists in the frame if the confidence score is equal to or higher than a second threshold higher than the first threshold. The information processing apparatus acquires a third result in which a period for which an object exists in the video data is specified on the basis of the first result and the second result. The information processing apparatus sets the first threshold and the second threshold so as to reduce a difference between the third result and correct answer data.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a detection program and the like. [Background technology]

[0002] Deep Learning (DL) models such as YOLO (You Only Look Once) are used for object detection. The DL model provides the object label, the object's position in the image, and a confidence score. The higher the confidence score, the more likely the object has been correctly detected.

[0003] In conventional technology, when a DL model is used for inference, a threshold is set for the reliability score, and detection results with reliability scores exceeding the threshold are adopted. Note that in conventional technology, the threshold is set high to suppress false positives, and low to suppress non-detections. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2022-545030 [Patent Document 2] Japanese Patent Publication No. 2020-86545 [Patent Document 3] Special Publication No. 2021-506017 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the above-mentioned conventional technology, the threshold value for the reliability score is difficult to set because the attempts to suppress false detections increase the number of undetected results, and the attempts to suppress undetected results increase the number of false detections. In the following description, the threshold value for the reliability score will be referred to simply as the threshold value where appropriate.

[0006] Figure 21 is a diagram for explaining the relationship between the threshold value of the DL model and the detection result. The vertical axis of graph G1 shown in Figure 21 corresponds to "Precision," and the horizontal axis corresponds to "Recall." Line l1 is called the Precision Recall Curve, and is drawn by changing the threshold value (the relationship between Precision and Recall is determined according to the threshold value).

[0007] FIG. 22 is a diagram for explaining Precision and Recall. In table Ta1 in FIG. 22, "Correct: Positive" indicates that the correct data is positive. "Correct: Negative" indicates that the correct data is negative. "Prediction: Positive" indicates that the prediction result (detection result) of the DL model is positive. "Prediction: Negative" indicates that the prediction result of the DL model is negative.

[0008] "Correct answer: positive" and "Prediction: positive" are defined as "TP (True Positive)". "Correct answer: positive" and "Prediction: negative" are defined as "FN (False Negative)". "Correct answer: negative" and "Prediction: positive" are defined as "FP (False Positive)". "Correct answer: negative" and "Prediction: negative" are defined as "TP (True Positive)".

[0009] Here, Precision is defined as "TP / (TP+FP)". The larger the Precision value, the more false positives are suppressed. Recall is defined as "TP / (TP+FN)". The larger the Recall value, the more false negatives are suppressed.

[0010] Returning to the explanation of Figure 21, for example, if the threshold (conf-thres) is set to "0.9", Precision = "0.98" and Recall = "0.3". On the other hand, if the threshold is set to "0.1", Precision = "0.2" and Recall = "0.9". In other words, line l1 in Figure 21 shows that increasing the threshold in an attempt to suppress false positives increases the number of undetected cases, and decreasing the threshold in an attempt to suppress undetected cases increases the number of false positives.

[0011] In one aspect, the present invention aims to provide a detection program, a detection method, and an information processing device that can set a threshold value in an object detection model that suppresses false detections and non-detections. [Means for solving the problem]

[0012] In the first proposal, a computer performs the following process. The computer inputs each frame of video data into an object detection model and obtains the detected position of the object in the image in the frame and a confidence score. The computer obtains a first result that determines that an object is present in the frame if the confidence score is equal to or greater than a first threshold, and a second result that determines that an object is present in the frame if the confidence score is equal to or greater than a second threshold that is higher than the first threshold. The computer obtains a third result that identifies a period in the video data in which the object is present based on the first and second results. The computer sets the first and second thresholds so that the difference between the third result and the correct answer data is small. [Effects of the Invention]

[0013] A threshold can be set in the object detection model to suppress false positives and false negatives. [Brief explanation of the drawings]

[0014] [Figure 1] FIG. 1 is a diagram for explaining the process of creating training data. [Figure 2] FIG. 2 is a diagram showing an example in which two different confidence score thresholds, one large and one small, are applied to training data. [Figure 3] FIG. 3 is a diagram (1) for explaining the process of calculating the threshold value of the reliability score according to the first embodiment. [Figure 4] FIG. 4 is a diagram (2) for explaining the process of calculating the threshold value of the reliability score according to the first embodiment. [Figure 5] FIG. 5 is a diagram (3) for explaining the process of calculating the threshold value of the reliability score according to the first embodiment. [Figure 6] FIG. 6 is a diagram for explaining the Jaccard coefficient. [Figure 7] FIG. 7 is a diagram illustrating the process of the application phase according to the first embodiment. [Figure 8] FIG. 8 is a diagram illustrating a configuration of a system according to the first embodiment. [Figure 9] FIG. 9 is a functional block diagram illustrating a configuration of the information processing device according to the first embodiment. [Figure 10] FIG. 10 is a flowchart illustrating a processing procedure of the learning phase of the information processing device according to the first embodiment. [Figure 11] FIG. 11 is a flowchart illustrating a processing procedure of the application phase of the information processing device according to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining the process of creating the correct answer data. [Figure 13] FIG. 13 is a diagram (1) for explaining the process of calculating the threshold value of the reliability score according to the second embodiment. [Figure 14] FIG. 14 is a diagram (2) for explaining the process of calculating the threshold value of the reliability score according to the second embodiment. [Figure 15] FIG. 15 is a diagram (3) for explaining the process of calculating the threshold value of the reliability score according to the second embodiment. [Figure 16] FIG. 16 is a diagram illustrating the process of the application phase according to the second embodiment. [Figure 17] FIG. 17 is a functional block diagram illustrating a configuration of an information processing device according to the second embodiment. [Figure 18]FIG. 18 is a flowchart illustrating a processing procedure of the learning phase of the information processing device according to the second embodiment. [Figure 19] FIG. 19 is a flowchart illustrating a processing procedure of the application phase of the information processing device according to the second embodiment. [Figure 20] FIG. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing apparatus of this embodiment. [Figure 21] FIG. 21 is a diagram for explaining the relationship between the threshold value of the DL model and the detection result. [Figure 22] FIG. 22 is a diagram for explaining Precision and Recall. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the detection program, detection method, and information processing device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0016] An example of processing by the information processing device according to the first embodiment will be described. The information processing device according to the first embodiment will be referred to as "information processing device 100." For example, the information processing device 100 executes "processing in a learning phase" and "processing in an application phase."

[0017] First, a description will be given of the processing in the learning phase executed by the information processing device 100. The learning phase includes a "process for creating training data" and a "process for calculating a threshold value for a reliability score."

[0018] The process of creating training data for the learning phase will now be described. FIG. 1 is a diagram illustrating the process of creating training data. For example, information processing device 100 has video data table 141, object detection model M1, and training data table 142. Video data table 141 stores video data. The video data includes image frames (still images) in chronological order. Each image frame is assigned a frame number in chronological order to identify the image frame.

[0019] The object detection model M1 is a model that takes an image frame as input and outputs the label, object position, and confidence score of the object included in the image frame. The object detection model is a DL model, etc.

[0020] The information processing device 100 obtains an output result by inputting an image frame registered in the video data table 141 into the object detection model M1. When the reliability score of the output result is equal to or greater than a threshold Th0, the information processing device 100 associates the output result (a combination of label, position, and reliability score) with the object detection model M1 and registers the result in the training data table 142. A very low value is set in advance for the threshold Th0. For example, the value of the threshold Th0 is "0.01".

[0021] The information processing device 100 repeatedly executes the above process for each image frame registered in the video data table 141. As a result, pairs of labels, positions, and reliability scores are registered in chronological order in the training data table 142. In the following description, the pairs of labels, positions, and reliability scores registered in the training data table 142 will be referred to as "training data." Note that the information processing device 100 may associate the frame numbers of the corresponding image frames with the training data and register them in the training data table 142.

[0022] The process of creating training data has been described above. Next, the process of calculating the threshold value of the reliability score will be described. In the following description, the threshold value of the reliability score may be simply referred to as the threshold value.

[0023] FIG. 2 is a diagram showing an example in which two different reliability score thresholds are applied to training data. For example, the relationship between the percentage of "correct answers" and "incorrect answers" when two different reliability score thresholds Th1 and Th2 are applied to each training data in the training data table 142 will be described. Note that a correct answer means that the object to be detected is included in training data whose reliability score is equal to or greater than the threshold. An incorrect answer means that the object to be detected is not included in training data whose reliability score is equal to or greater than the threshold.

[0024] In graph G2 of Figure 2, the horizontal axis corresponds to the frame number (time) of the training data, and the vertical axis corresponds to the magnitude of the threshold. For example, let's say threshold Th1 is "0.8" and threshold Th2 is "0.1." At threshold Th2, undetected results are suppressed, but many "correct results" are included. On the other hand, at threshold Th1, false detections are suppressed, but the proportion of "correct results" is high.

[0025] The threshold value Th1 corresponds to the "second threshold value." The threshold value Th2 corresponds to the "first threshold value."

[0026] 3, 4, and 5 are diagrams for explaining the process of calculating the threshold value of the reliability score according to the first embodiment. First, FIG. 3 will be explained. FIG. 3 shows the deviation of detection when two threshold values ​​Th1 and Th2, which are different in magnitude, are applied. The horizontal axis of graph G3 is the time axis. However, the magnitude relationship between the threshold values ​​is set to "threshold value Th1>threshold value Th2".

[0027] Line 3-1 of graph G3 indicates the detection state of an object when threshold value Th1 is set. When line 3-1 is Low, it indicates that the object is in an undetected state (the reliability score is less than threshold value Th1). When line 3-1 is High, it indicates that the object is in a detected state (the reliability score is equal to or greater than threshold value Th1).

[0028] Line 3-2 of graph G3 indicates the detection state of an object when threshold value Th2 is set. When line 3-2 is Low, it indicates that the object is in an undetected state (the reliability score is less than threshold value Th2). When line 3-2 is High, it indicates that the object is in a detected state (the reliability score is equal to or greater than threshold value Th2).

[0029] The line 3-3 of the graph G3 indicates the correct data. When the line 3-3 is low, it indicates that an object is actually undetected. When the line 3-3 is high, it indicates that an object is actually detected.

[0030] Comparing line 3-1 and line 3-2, fluctuations in the detection state occur in the time periods p3-1, p3-2, p3-3, and p3-4.

[0031] Causes of blurring include a person holding an object, an object moving, a camera moving, the whole image being visible, a part being visible, and changes due to the environment (sunny, rainy, cloudy, morning, night).

[0032] We now move on to the explanation of FIG. 4. The information processing device 100 applies two different thresholds, Th1 and Th2, to the time-series training data 142-1. The information processing device 100 identifies a first detection period in which the reliability score of the time-series training data 142-1 is equal to or greater than the threshold Th1, and a second detection period in which the reliability score of the time-series training data 142-1 is equal to or greater than the threshold Th2. The information processing device 100 generates a detection result 20 based on the first detection period and the second detection period. The first detection period corresponds to the "second result." The second detection period corresponds to the "first result."

[0033] The process of generating the detection result 20 by the information processing device 100 will be described with reference to Fig. 5. In Fig. 5, the horizontal axis of the graph G4 is the time axis.

[0034] Line 4-1 in graph G4 indicates a first detection period for an object when threshold value Th1 is set for time-series training data 142-1. A period when line 4-1 is Low indicates that the object is in an undetected state. A period when line 4-1 is High indicates that the object is in a detected state. In graph G4, the first detection periods are times t6 to t7 and t8 to t9.

[0035] The line 4-2 of the graph G4 indicates the detection state of an object when the threshold value Th2 is set. The period when the line 4-2 is Low indicates that the object is not detected. The period when the line 4-2 is High indicates that the object is detected. In the graph G4, the second detection period is from time t1 to t2, t3 to t4, and t5 to t6. 10 This becomes:

[0036] The line 4-3 of the graph G4 indicates correct data. When the line 4-3 is Low, it indicates that an object is actually undetected. When the line 4-3 is High, it indicates that an object is actually detected. In the graph G4, the period in which an object actually appears in the time-series training data 142-1 is from time t5 to t 10 It can be shown that

[0037] The information processing device 100 compares the first detection period with the second detection period, and if the first detection period is included in a consecutive second detection period, the information processing device 100 determines the consecutive second detection period as the object detection period. The object detection period corresponds to the "third result."

[0038] For example, the information processing device 100 excludes the second detection period t1 to t2 from the object detection period because the first detection period is not included in the second detection period t1 to t2. The information processing device 100 excludes the second detection period t3 to t4 from the object detection period because the first detection period is not included in the second detection period t3 to t4. The information processing device 100 excludes the second detection period t5 to t6 from the object detection period. 10 In this case, the first detection periods t6 to t7 and t8 to t9 are included, so the second detection period t5 to t 10Let be the object detection period.

[0039] The information processing device 100 executes the above process and detects an object during the object detection period t5 to t 10 The information processing device 100 obtains a detection result 20 by identifying the object detected in the detection result 20. The information processing device 100 evaluates two thresholds Th1 and Th2, each having a different magnitude, based on the correct answer data 143 (for example, line 4-3) and the detection result 20. For example, the information processing device 100 calculates the difference between the object detection period of the detection result 20 and the period during which an object indicated in the correct answer data 143 actually appears as the evaluation result (evaluation result 25 in FIG. 4 ).

[0040] The information processing device 100 sets the thresholds Th1 and Th2 so that the difference that becomes the evaluation result becomes small.

[0041] The information processing device 100 uses a grid search to set the thresholds Th1 and Th2 so that the difference that becomes the evaluation result is small. For example, the information processing device 100 prepares a plurality of pairs of thresholds Th1 and Th2 in advance, and repeatedly executes the above process for each pair of thresholds Th1 and Th2. The information processing device 100 identifies, from the plurality of pairs, the pair of thresholds Th1 and Th2 that minimizes the difference, and sets the identified pair.

[0042] The information processing device 100 may set the thresholds Th1 and Th2 using a technique other than grid search so as to reduce the difference that results in the evaluation. For example, the information processing device 100 may update and set the thresholds Th1 and Th2 using Newton's method so as to reduce the difference that results in the evaluation.

[0043] The information processing device 100 may calculate a Jaccard coefficient as the evaluation result. FIG. 6 is a diagram for explaining the Jaccard coefficient. In FIG. 6, set A corresponds to the detection results to which different thresholds (threshold Th1, threshold Th2) are applied. For example, the elements included in set A are the image frames corresponding to the object detection period of the detection result 20 in FIG. 5. Set B corresponds to the correct answer data 143. For example, the elements of set B are the image frames corresponding to the period during which an object was actually detected.

[0044] The information processing device 100 calculates the Jaccard coefficient (J(A, B)) by equation (1).

[0045] J(A,B)=(Set A∩Set B) / (Set A∪Set B)...(1)

[0046] When the information processing device 100 uses the Jaccard coefficient, it sets the thresholds Th1 and Th2 so that the Jaccard coefficient is large.

[0047] The learning phase processing executed by the information processing device 100 has been described above.

[0048] Next, a description will be given of the process of the application phase executed by the information processing device 100. Fig. 7 is a diagram for explaining the process of the application phase according to the embodiment 1. The information processing device 100 uses the thresholds Th1 and Th2 set in the learning phase.

[0049] The information processing device 100 acquires video data (time-series image frames) 144 from a camera or the like, and inputs the time-series image frames to the object detection model M1 to obtain time-series output results. The output results associate object labels, object positions, and confidence scores. The output results may also include frame numbers of the image frames.

[0050] The information processing device 100 compares the reliability score of the time-series output result with a threshold Th1 and identifies a first detection period in which the reliability score is equal to or greater than the threshold Th1. The information processing device 100 also compares the reliability score of the time-series output result with a threshold Th2 and identifies a second detection period in which the reliability score is equal to or greater than the threshold Th2.

[0051] The information processing device 100 compares the first detection period with the second detection period, and if the first detection period is included in a consecutive second detection period, the information processing device 100 determines the consecutive second detection period as the object detection period. The information processing device 100 outputs an output result corresponding to the object detection period as the detection result 21. The information processing device 100 may output an output result and an image frame corresponding to the object detection period as the detection result 21.

[0052] The processing of the application phase executed by the information processing device 100 has been described above.

[0053] As described above, the information processing device 100 according to the first embodiment inputs each image frame of video data into the object detection model M1 to obtain a reliability score, and obtains a first detection period and a second detection period based on the reliability score and the thresholds Th1 and Th2. The information processing device 100 identifies an object detection period based on the first detection period and the second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the correct answer data. This makes it possible to set thresholds in the object detection model M1 that suppress false detections and non-detections.

[0054] Next, an example of a system including the information processing device 100 of the first embodiment will be described. FIG. 8 is a diagram showing the configuration of a system according to the first embodiment. As shown in FIG. 8, this system includes a camera 10 and the information processing device 100. The camera 10 and the information processing device 100 are connected to each other via a network 30.

[0055] The camera 10 captures video data and transmits the captured video data to the information processing device 100. The video data includes image frames in time series.

[0056] Next, a description will be given of an example of the configuration of the information processing device 100 according to the present embodiment 1. Fig. 9 is a functional block diagram showing the configuration of the information processing device according to the present embodiment 1. As shown in Fig. 9, the information processing device 100 includes a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0057] The communication unit 110 executes data communication between the camera 10, an external device, etc. via the network 30. The communication unit 110 is a network interface card (NIC) or the like. For example, the communication unit 110 acquires video data from the camera 10.

[0058] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the information processing device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0059] The display unit 130 is a display device that displays information output from the control unit 150.

[0060] The storage unit 140 includes an image data table 141, a training data table 142, correct answer data 143, image data 144, and an object detection model M1. The storage unit 140 is a memory or the like.

[0061] Video data used in the learning phase is stored in the video data table 141. Other explanations regarding the video data table 141 are the same as those regarding the video data table 141 in FIG.

[0062] Time-series training data is stored in the training data table 142. Other explanations regarding the training data table 142 are the same as those regarding the training data table 142 in FIG.

[0063] The correct answer data 143 indicates the actual detection state of a certain object. For example, as described with reference to FIG. 3 etc., the correct answer data 143 is set to include a period during which a certain object is actually in an undetected state and a period during which the certain object is actually in a detected state. For example, an administrator or the like sets the correct answer data 143 in advance by referring to the video data in the video data table 141.

[0064] The video data 144 is video data captured by the camera 10 or the like. The video data 144 is used in the application phase. Other explanations regarding the video data 144 are the same as those regarding the video data 144 in FIG.

[0065] The object detection model M1 is a model that takes an image frame as input and outputs the label, object position, and confidence score of an object included in the image frame. The object detection model M1 is a DL model or the like.

[0066] Next, we move on to explaining the control unit 150. The control unit 150 has an acquisition unit 151, a training data creation unit 152, a threshold setting unit 153, and an object detection unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0067] The acquisition unit 151 acquires the video data 144 from the camera 10. The acquisition unit 151 stores the acquired video data 144 in the storage unit 140.

[0068] The training data creation unit 152 creates training data from the video data (time-series image frames) in the video data table 141. The training data creation unit 152 obtains an output result by inputting the image frames into the object detection model M1. If the reliability score of the output result is equal to or greater than the threshold value Th0, the training data creation unit 152 registers the output result in the training data table 142 as training data.

[0069] Other processes related to the training data creation unit 152 are the same as those described in connection with FIG. 1 for creating training data.

[0070] The threshold setting unit 153 acquires the first detection period and the second detection period based on the reliability scores of the time-series training data in the training data table 142 and the thresholds Th1 and Th2. The threshold setting unit 153 identifies the object detection period based on the first detection period and the second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the correct answer data 143.

[0071] Other aspects of the threshold setting unit 153 are the same as those of the process for calculating the threshold of the reliability score described with reference to FIGS.

[0072] The object detection unit 154 uses the thresholds Th1 and Th2 set by the threshold setting unit 153 in the learning phase to detect objects from the video data 144 captured by the camera 10.

[0073] The object detection unit 154 acquires video data (time-series image frames) 144 and inputs the time-series image frames to the object detection model M1 to obtain time-series output results. The object detection unit 154 compares the reliability score of the time-series output results with a threshold Th1 and identifies a first detection period in which the reliability score is equal to or greater than the threshold Th1. The object detection unit 154 also compares the reliability score of the time-series output results with a threshold Th2 and identifies a second detection period in which the reliability score is equal to or greater than the threshold Th2.

[0074] The object detection unit 154 compares the first detection period with the second detection period, and if a consecutive second detection period includes the first detection period, the consecutive second detection period is set as the object detection period. The object detection unit 154 outputs an output result corresponding to the object detection period to the display unit 130 or the like as the detection result 21. The object detection unit 154 may output the output result and image frame corresponding to the object detection period as the detection result 21.

[0075] Other aspects of the object detection unit 154 are the same as those of the application phase process described with reference to FIG.

[0076] Next, an example of a processing procedure of the information processing device 100 according to the present embodiment 1 will be described. Fig. 10 is a flowchart showing a processing procedure of the learning phase of the information processing device according to the present embodiment 1. As shown in Fig. 10, the training data creation unit 152 of the information processing device 100 inputs image frames from the video data table 141 to the object detection model M1 and obtains an output result (step S101).

[0077] The training data creating unit 152 stores the output results whose reliability scores are equal to or greater than the threshold Th0 as training data in the training data table 142 (step S102).

[0078] The threshold setting unit 153 of the information processing device 100 identifies a first detection period based on the reliability scores of the time-series training data and a threshold Th1 (step S103).The threshold setting unit 153 identifies a second detection period based on the reliability scores of the time-series training data and a threshold Th2 (step S104).

[0079] The threshold setting unit 153 identifies the object detection period based on the first detection period and the second detection period (step S105). The threshold setting unit 153 sets the thresholds Th1 and Th2 so that the difference between the object detection period and the correct answer data 143 is small (step S106).

[0080] Fig. 11 is a flowchart showing a processing procedure of the application phase of the information processing device according to Example 1. As shown in Fig. 11, the acquisition unit 151 of the information processing device 100 acquires video data from the camera 10 (step S201).

[0081] The object detection unit 154 of the information processing device 100 inputs video data to the object detection model M1 and obtains an output result (step S202). The object detection unit 154 identifies a first detection period based on the reliability score of the time-series output result and a threshold value Th1 (step S203). The object detection unit 154 identifies a second detection period based on the reliability score of the time-series output result and a threshold value Th2 (step S204).

[0082] The object detection unit 154 identifies the object detection period based on the first detection period and the second detection period (step S205), and outputs the detection result (step S206).

[0083] As described above, the information processing device 100 according to the first embodiment inputs each image frame of video data into the object detection model M1 to obtain a reliability score, and obtains a first detection period and a second detection period based on the reliability score and the thresholds Th1 and Th2. The information processing device 100 identifies an object detection period based on the first detection period and the second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the correct answer data. This makes it possible to set thresholds in the object detection model M1 that suppress false detections and non-detections.

[0084] The information processing device 100 identifies the first detection period and the second detection period based on the reliability score when the video data is input to the object detection model M1 and the thresholds Th1 and Th2 set in the learning phase. The information processing device 100 identifies the object detection period based on the first detection period and the second detection period. This allows for accurate object detection of the video data. It also prevents a decrease in the detection accuracy of the object detection model for objects whose appearance changes in the video data. [Example]

[0085] According to the information processing device 100 of the first embodiment described above, it is possible to perform object detection with higher accuracy than the conventional technology. However, if a momentary drop in the reliability score occurs, a period of non-detection may occur, and the accuracy may decrease, so there is room for improvement.

[0086] The information processing device according to the second embodiment utilizes the temporal continuity in video data and corrects the detection result using a time threshold. The information processing device according to the second embodiment is referred to as "information processing device 200." For example, the information processing device 200 executes "learning phase processing" and "application phase processing."

[0087] First, a description will be given of the processing in the learning phase executed by the information processing device 200. The learning phase includes a "process for creating training data," a "process for creating correct answer data," and a "process for calculating a threshold value for a reliability score."

[0088] The process in which the information processing device 200 creates training data is similar to the process in which the information processing device 100 creates training data described in the first embodiment.

[0089] The process of generating supervised data by the information processing device 200 will be described. Fig. 12 is a diagram for explaining the process of generating supervised data. In the example shown in Fig. 12, time-series image frames f1-1, f1-2, f1-3, and f1-4 are shown.

[0090] Ball 60 is detected in image frames f1-1 and f1-2. After that, ball 60 moves behind object 61 in image frames f1-3 and f1-4, and is not detected. In this case, the period during which the object is detected using the correct answer data is the period from the time corresponding to image frame f1-1 to the time corresponding to image frame f1-2.

[0091] The information processing device 200 utilizes the fact that the target is video data and has temporal continuity to assign information that an object exists even though it is not shown to the image frames f1-3 and f1-4. As a result, the period for detecting an object using the correct answer data is the period from the time corresponding to image frame f1-1 to the time corresponding to image frame f1-4.

[0092] For example, the user checks the correct answer data, identifies image frames f1-3 and f1-4 in which an object (ball 60) is present but not shown, and inputs these to the information processing device 200.

[0093] The process of creating correct answer data has been described above. Next, the process of calculating the threshold value of the reliability score will be described. The threshold value of the reliability score may be simply referred to as the threshold value.

[0094] 13, 14, and 15 are diagrams for explaining the process of calculating the threshold value of the reliability score according to the second embodiment. First, FIG. 13 will be explained. The information processing device 200 applies two different threshold values ​​Th1 and Th2 to the time-series training data 242-1. The information processing device 200 identifies a first detection period in which the reliability score of the time-series training data 242-1 is equal to or greater than the threshold value Th1, and a second detection period in which the reliability score of the time-series training data 242-1 is equal to or greater than the threshold value Th2.

[0095] Here, the information processing device 200 corrects the first detection period and the second detection period using the time threshold, and then generates the detection result 22 based on the corrected first detection period and the corrected second detection period.

[0096] 14 and 15, the process in which the information processing device 200 generates the detection result 22 will be described. First, Fig. 14 will be described. In Fig. 14, the horizontal axis of graph G5 is the time axis.

[0097] The line 5-1 of the graph G5 indicates a first detection period for an object when a threshold value Th1 is set for the time-series training data 242-1. The period when the line 5-1 is Low indicates that the object is in an undetected state. The period when the line 5-1 is High indicates that the object is in a detected state. In the graph G5, the first detection period is from time t8 to t9, t 10 ~t 11 This becomes:

[0098] The line 5-2 of the graph G5 indicates the detection state of an object when the threshold value Th2 is set. The period when the line 5-2 is Low indicates that the object is not detected. The period when the line 5-2 is High indicates that the object is detected. In the graph G, the second detection period is from time t1 to t2, t3 to t4, t5 to t6, t7 to t8, and t9 to t10. 12 , t 13 ~t 14 This becomes:

[0099] The line 5-3 of the graph G5 indicates correct data (correct data corrected by the process of FIG. 12). When the line 5-3 is Low, it indicates that an object is actually not detected. When the line 5-3 is High, it indicates that an object is actually detected or that an object is present even though it is not shown. In the graph G5, in the time series training data 242-1, the period when an object actually appears (or a period when an object is present even though it is not shown) is from time t5 to t 14 It can be shown that

[0100] The following describes a process for correcting the first detection period by the information processing device 200. When the time from the end time of a previous first detection period to the start time of the immediately following first detection period is less than the time threshold, the information processing device 200 sets the period from the start time of the previous first detection period to the end time of the immediately following first detection period as a new first detection period.

[0101] For example, the first detection period t8 to t9 and the first detection period t 10 ~t 11 Regarding the time t9 to the time t10 In this case, the information processing device 200 determines whether the time until the first detection period t8 to t9 and the first detection period t 10 ~t 11 Instead of the first detection period t8 to t 11 As a result of the information processing device 200 performing the above process, the first detection period based on the threshold value Th1 becomes the one indicated by the line 6-1.

[0102] The following describes a process of correcting the second detection period by the information processing device 200. When the time from the end time of a previous second detection period to the start time of the next second detection period is less than the time threshold, the information processing device 200 sets the time from the start time of the previous second detection period to the end time of the next first detection period as a new second detection period.

[0103] For example, for the second detection period t1-t2 and the first detection period t3-t4, the time from time t2 to time t3 is assumed to be less than the time threshold value. In this case, the information processing device 200 newly sets the second detection period t1-t4 instead of the second detection period t1-t2 and the second detection period t3-t4.

[0104] For example, the second detection period t5 to t6 and the second detection period t7 to t 12 Regarding the above, the time from time t6 to time t7 is set to be less than the time threshold value. 12 and the second detection period t 13 ~t 14 With respect to time t 12 From time t 13 In this case, the information processing device 200 determines whether the time from the second detection period t5 to t6 and the second detection period t7 to t 12 , the second detection period t 13 ~t 14 Instead, the second detection period t5 to t 14 Set a new value.

[0105] As a result of the information processing device 200 performing the above process, the second detection period based on the threshold value Th2 becomes as shown by the line 6-2.

[0106] Next, we move on to the description of FIG. 15. The information processing device 200 compares the corrected first detection period indicated by line 6-1 with the corrected first detection period indicated by line 6-2, and if a first detection period is included in consecutive second detection periods, the information processing device 200 determines the consecutive second detection periods as the object detection period. The process of identifying the object detection period is the same as the process described in FIG. 5 of the first embodiment. In the example shown in FIG. 15, the object detection period is the period from time t5 to t 14 This becomes:

[0107] The information processing device 100 executes the above process and detects an object during the object detection period t5 to t 14 13 ), the information processing device 200 calculates the difference between the object detection period of the detection result 22 and the period during which an object indicated in the supervised data 243 actually appears.

[0108] The information processing device 200 sets the thresholds Th1 and Th2 so that the difference that becomes the evaluation result becomes small. For example, the information processing device 200 sets the thresholds Th1 and Th2 using grid search, Newton's method, or the like, in the same way as in the first embodiment. Furthermore, the information processing device 200 may calculate a Jaccard coefficient and set the thresholds Th1 and Th2 so that the Jaccard coefficient becomes large.

[0109] The learning phase processing executed by the information processing device 200 has been described above.

[0110] Next, a description will be given of the process of the application phase executed by the information processing device 200. Fig. 16 is a diagram for explaining the process of the application phase according to the second embodiment. The information processing device 200 uses the thresholds Th1 and Th2 and the time threshold set in the learning phase.

[0111] The information processing device 200 acquires video data (time-series image frames) 244 from a camera or the like, and inputs the time-series image frames 244 to the object detection model M1 to obtain time-series output results. The output results associate object labels, object positions, and confidence scores. The output results may also include frame numbers of the image frames.

[0112] The information processing device 200 compares the reliability scores of the time-series output results with a threshold Th1, and identifies a first detection period in which the reliability score is equal to or greater than the threshold Th1. Furthermore, if the time from the end time of the previous first detection period to the start time of the immediately following first detection period is less than the time threshold, the information processing device 200 determines the period from the start time of the previous first detection period to the end time of the immediately following first detection period as a new first detection period (corrected first detection period). This processing is similar to the processing described with reference to FIG. 14.

[0113] The information processing device 200 compares the reliability scores of the time-series output results with a threshold Th2, and identifies a second detection period in which the reliability score is equal to or greater than the threshold Th2. Furthermore, if the time from the end time of the previous second detection period to the start time of the immediately following second detection period is less than the time threshold, the information processing device 200 determines the period from the start time of the previous second detection period to the end time of the immediately following second detection period as a new second detection period (corrected second detection period). This processing is similar to the processing described with reference to FIG. 14.

[0114] The information processing device 200 compares the corrected first detection period with the corrected second detection period, and if the first detection period is included in consecutive second detection periods, the information processing device 200 determines the consecutive second detection periods as the object detection period. The information processing device 200 outputs an output result corresponding to the object detection period as the detection result 27. The information processing device 200 may output an output result and an image frame corresponding to the object detection period as the detection result 27.

[0115] The processing of the application phase executed by the information processing device 200 has been described above.

[0116] As described above, the information processing device 200 according to the second embodiment inputs each image frame of video data into the object detection model M1 to obtain a confidence score, and obtains a first detection period and a second detection period based on the confidence score and the thresholds Th1 and Th2. The information processing device 200 corrects the first detection period and the second detection period using a time threshold, identifies an object detection period based on the first detection period and the second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the correct answer data. This prevents a period of non-detection even if a momentary drop in the confidence score occurs, and prevents a decrease in accuracy.

[0117] Next, an example of a system including an information processing device 200 according to the second embodiment will be described. The information processing device 200 is connected to a camera 10 via a network 30, similar to the system according to the first embodiment described with reference to Fig. 8. The information processing device 200 receives video data from the camera 10.

[0118] Next, a description will be given of an example of the configuration of the information processing device 200 according to the present embodiment 2. Fig. 17 is a functional block diagram showing the configuration of the information processing device according to the present embodiment 2. As shown in Fig. 17, the information processing device 200 includes a communication unit 210, an input unit 220, a display unit 230, a storage unit 240, and a control unit 250.

[0119] The communication unit 210 executes data communication with the camera 10, external devices, etc. via the network 30. The communication unit 210 is an NIC, etc. For example, the communication unit 210 acquires video data from the camera 10.

[0120] The input unit 220 is an input device that inputs various types of information to the control unit 250 of the information processing device 200. For example, the input unit 220 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0121] The display unit 130 is a display device that displays information output from the control unit 250.

[0122] The storage unit 240 includes an image data table 241, a training data table 242, correct answer data 243, image data 244, and an object detection model M1. The storage unit 240 is a memory or the like.

[0123] Video data used in the learning phase is stored in the video data table 241. Other explanations regarding the video data table 241 are the same as those regarding the video data table 141 in FIG.

[0124] Time-series training data is stored in the training data table 242. Other explanations regarding the training data table 242 are the same as those regarding the training data table 242 in FIG.

[0125] The correct answer data 243 indicates the actual detection state of a certain object. For example, as described with reference to Fig. 14 etc., the correct answer data 243 sets a period during which a certain object is actually in an undetected state and a period during which the certain object is actually in a detected state. In addition, the correct answer data creation unit 253, which will be described later, adds information that a certain object exists even though it is not captured.

[0126] The video data 244 is video data captured by the camera 10 or the like. The video data 244 is used in the application phase. Other explanations regarding the video data 244 are the same as those regarding the video data 244 in FIG.

[0127] The object detection model M1 is a model that takes an image frame as input and outputs the label, object position, and confidence score of an object included in the image frame. The object detection model M1 is a DL model or the like.

[0128] Next, we move on to explaining the control unit 250. The control unit 250 has an acquisition unit 251, a training data creation unit 252, a correct answer data creation unit 253, a threshold setting unit 254, and an object detection unit 255. The control unit 250 is a CPU, a GPU, or the like.

[0129] The acquisition unit 251 acquires the video data 244 from the camera 10. The acquisition unit 251 stores the acquired video data 244 in the storage unit 140.

[0130] The training data creation unit 252 creates training data from the video data (time-series image frames) in the video data table 241. The training data creation unit 252 obtains an output result by inputting the image frames to the object detection model M1. If the reliability score of the output result is equal to or greater than the threshold value Th0, the training data creation unit 252 registers the output result in the training data table 242 as training data.

[0131] The processing of the training data creating unit 252 is similar to the processing of the training data creating unit 152 described in the first embodiment.

[0132] The supervised data creating unit 253 analyzes the supervised data 243 and executes the processing described in Fig. 12. For example, the supervised data creating unit 253 utilizes the fact that the supervised data 243 is video data and has temporal continuity, and sets information that a target object exists in the corresponding image frame even though it is not shown.

[0133] The process of the correct answer data creating unit 253 is the same as the process described with reference to FIG.

[0134] The threshold setting unit 254 acquires the first detection period and the second detection period based on the reliability scores of the time-series training data in the training data table 241, the thresholds Th1 and Th2.

[0135] The threshold setting unit 254 corrects the first detection period and the second detection period using the time threshold. The process by which the threshold setting unit 254 corrects the first detection period and the second detection period using the time threshold is the same as the process described in FIG.

[0136] The threshold setting unit 254 identifies the object detection period based on the corrected first detection period and the corrected second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the corrected answer data 243.

[0137] Other aspects of the threshold setting unit 254 are similar to the process of calculating the threshold of the reliability score described with reference to FIG. 15 and the like.

[0138] The object detection unit 255 uses the thresholds Th1 and Th2 set by the threshold setting unit 254 in the learning phase to detect objects from the video data 244 captured by the camera 10.

[0139] The object detection unit 255 acquires video data (time-series image frames) 244 and inputs the time-series image frames 244 into the object detection model M1 to obtain time-series output results. The object detection unit 255 compares the reliability score of the time-series output results with a threshold Th1 and identifies a first detection period in which the reliability score is equal to or greater than the threshold Th1. The object detection unit 255 also compares the reliability score of the time-series output results with a threshold Th2 and identifies a second detection period in which the reliability score is equal to or greater than the threshold Th2.

[0140] The object detection unit 255 corrects the first detection period and the second detection period using the time threshold. The process by which the object detection unit 255 corrects the first detection period and the second detection period using the time threshold is the same as the process described in FIG.

[0141] The object detection unit 255 compares the corrected first detection period with the corrected second detection period, and if the first detection period is included in a consecutive second detection period, the consecutive second detection period is set to be the object detection period. The object detection unit 255 outputs an output result corresponding to the object detection period as the detection result 27 to the display unit 230 or the like. The object detection unit 255 may output the output result and image frame corresponding to the object detection period as the detection result 27.

[0142] Other aspects of the object detection unit 255 are the same as those of the application phase process described with reference to FIG.

[0143] Next, an example of a processing procedure of the information processing device 200 according to the present embodiment 2 will be described. Fig. 18 is a flowchart showing a processing procedure of the learning phase of the information processing device according to the present embodiment 2. As shown in Fig. 18, the training data generation unit 252 of the information processing device 200 inputs image frames of the video data table 241 to the object detection model M1 and obtains an output result (step S301).

[0144] The training data generation unit 252 stores the output results whose reliability scores are equal to or greater than the threshold Th0 as training data in the training data table 242 (step S302).

[0145] The supervised data creating unit 253 of the information processing device 200 analyzes the supervised data 243 and adds information to the supervised data 243 that the target object exists although it is not shown in the image (step S303).

[0146] The threshold setting unit 254 of the information processing device 200 identifies a first detection period based on the reliability scores of the time-series training data and a threshold Th1 (step S304).The threshold setting unit 254 identifies a second detection period based on the reliability scores of the time-series training data and a threshold Th2 (step S305).

[0147] The threshold setting unit 254 corrects the first detection period and the second detection period based on the time threshold (step S306).

[0148] The threshold setting unit 254 identifies the object detection period based on the corrected first detection period and the corrected second detection period (step S307). The threshold setting unit 254 sets the thresholds Th1, Th2, and the time threshold so that the difference between the object detection period and the corrected data 243 is small (step S308).

[0149] Fig. 19 is a flowchart showing a processing procedure of the application phase of the information processing device according to the embodiment 2. As shown in Fig. 19, the acquisition unit 251 of the information processing device 200 acquires video data from the camera 10 (step S401).

[0150] The object detection unit 255 of the information processing device 100 inputs video data into the object detection model M1 and obtains an output result (step S402). The object detection unit 255 identifies a first detection period based on the reliability score of the time-series output result and a threshold value Th1 (step S403). The object detection unit 255 identifies a second detection period based on the reliability score of the time-series output result and a threshold value Th2 (step S404).

[0151] The object detection unit 255 corrects the first detection period and the second detection period based on the time threshold (step S405). The object detection unit 255 identifies the object detection period based on the corrected first detection period and the corrected second detection period (step S406). The object detection unit 255 outputs the detection result (step S407).

[0152] As described above, the information processing device 200 according to the second embodiment inputs each image frame of video data into the object detection model M1 to obtain a confidence score, and obtains a first detection period and a second detection period based on the confidence score and the thresholds Th1 and Th2. The information processing device 200 corrects the first detection period and the second detection period using a time threshold, identifies an object detection period based on the first detection period and the second detection period, and sets the thresholds Th1 and Th2 based on the object detection period and the correct answer data. This prevents a period of non-detection even if a momentary drop in the confidence score occurs, and prevents a decrease in accuracy.

[0153] The information processing device 200 identifies the first detection period and the second detection period based on the reliability score when the video data is input to the object detection model M1 and the thresholds Th1 and Th2 set in the learning phase. The information processing device 200 corrects the first detection period and the second detection period using the time thresholds, and then identifies the object detection period based on the first detection period and the second detection period. This enables object detection of the video data to be performed with high accuracy.

[0154] The information processing device 100 (200) described in this embodiment can be used in various business situations. For example, the information processing device 100 can be applied to an autonomous driving technology to detect pedestrians as obstacles in front of a vehicle, pedestrians, and vehicles and pedestrians at intersections, thereby improving the safety of the autonomous driving technology.

[0155] Furthermore, the information processing device 100 can be applied to the medical field for patient monitoring, tracking of medical instruments, surgical assistance for surgeons to accurately position surgical instruments, and the like.

[0156] Next, a description will be given of an example of the hardware configuration of a computer that realizes the same functions as the above-described information processing devices 100 and 200. Fig. 20 is a diagram showing an example of the hardware configuration of a computer that realizes the same functions as the information processing device of this embodiment.

[0157] 20, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 10, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0158] The hard disk drive 207 stores an acquisition program 207a, a training data creation program 207b, a supervised data creation program 207c, a threshold setting program 207d, and an object detection program 207e. The CPU 201 reads out each of the programs 207a to 207e and loads them into the RAM 206.

[0159] The acquisition program 207a functions as the acquisition process 206a. The training data creation program 207b functions as the training data creation process 206b. The supervised answer data creation program 207c functions as the supervised answer data creation process 206c. The threshold setting program 207d functions as the threshold setting process 206d. The object detection program 207e functions as the object detection process 206e.

[0160] The processing of the acquisition process 206a corresponds to the processing of the acquisition units 151 and 251. The processing of the training data creation process 206b corresponds to the processing of the training data creation units 152 and 252. The processing of the supervised data creation process 206c corresponds to the processing of the supervised data creation unit 253. The processing of the threshold setting process 206d corresponds to the processing of the threshold setting units 153 and 254. The processing of the object detection process 206e corresponds to the processing of the object detection units 154 and 255.

[0161] It should be noted that each of the programs 207a to 207e does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207e.

[0162] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0163] (Supplementary Note 1) Each frame of video data is input to an object detection model, and the detected position of the object in the image in that frame and a confidence score are obtained; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct data becomes small. A detection program that causes a computer to execute a process.

[0164] (Appendix 2) The detection program described in Appendix 1, characterized in that the first result includes a plurality of first periods of the frame in which the reliability score is equal to or greater than a first threshold, the second result includes a plurality of second periods of the frame in which the reliability score is equal to or greater than a second threshold, and the process of obtaining the third result obtains, as the third result, a first period among the plurality of first periods that is a period that includes the second period.

[0165] (Appendix 3) The detection program described in Appendix 1, further comprising causing a computer to execute a process of detecting an object from each frame of the video data based on the confidence score obtained by inputting each frame of the video data into the object detection model and the first threshold and the second threshold set by the setting process.

[0166] (Note 4) If the time from the end time of a previous first period to the start time of a first period immediately following the previous first period is less than a predetermined time threshold, the period from the start time of the previous first period to the end time of the first period immediately following the previous first period is set as a new first period; The detection program described in Appendix 1 is characterized in that, when the time from the end time of a previous second period to the start time of the second period immediately following the previous second period is less than the predetermined time threshold, the computer is further caused to execute a process of setting the period from the start time of the previous second period to the end time of the second period immediately following the previous second period as a new second period.

[0167] (Appendix 5) A detection program as described in Appendix 1, characterized in that the detection program further causes a computer to execute a process of analyzing a time series of frames included in the correct answer data, identifying frames among the time series in which the target object is not shown but the target object is present, and adding information indicating the presence of an object to the identified frames.

[0168] (Appendix 6) Each frame of video data is input to an object detection model, and the detected position of the object in the image in that frame and a confidence score are obtained; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct data becomes small. A detection method characterized in that the processing is performed by a computer.

[0169] (Appendix 7) The detection method described in Appendix 6, characterized in that the first result includes a plurality of first periods of the frame in which the reliability score is equal to or greater than a first threshold, the second result includes a plurality of second periods of the frame in which the reliability score is equal to or greater than a second threshold, and the process of obtaining the third result obtains, as the third result, a first period among the plurality of first periods that is a period that includes the second period.

[0170] (Appendix 8) The detection method described in Appendix 6, further comprising having a computer execute a process of detecting an object from each frame of the video data based on the confidence score obtained by inputting each frame of the video data into the object detection model and the first threshold and the second threshold set by the setting process.

[0171] (Supplementary Note 9) When the time from the end time of a previous first period to the start time of a first period immediately following the previous first period is less than a predetermined time threshold, the period from the start time of the previous first period to the end time of the immediately following first period is set as a new first period; The detection method described in Appendix 6, further characterized in that if the time from the end time of a previous second period to the start time of the second period immediately following the previous second period is less than the predetermined time threshold, the computer is caused to execute a process of setting the period from the start time of the previous second period to the end time of the second period immediately following the previous second period as a new second period.

[0172] (Appendix 10) The detection method described in Appendix 6, characterized in that the computer further executes a process of analyzing a time series of frames included in the correct answer data, identifying frames among the time series in which the target object is not shown but the target object is present, and adding information about the presence of an object to the identified frames.

[0173] (Appendix 11) Each frame of video data is input to an object detection model, and a detected position of the object in the image in the frame and a confidence score are obtained; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct data becomes small. An information processing device having a control unit that executes processing.

[0174] (Appendix 12) The information processing device described in Appendix 11, characterized in that the first result includes a plurality of first periods of the frame in which the reliability score is equal to or greater than a first threshold, the second result includes a plurality of second periods of the frame in which the reliability score is equal to or greater than a second threshold, and the process of obtaining the third result obtains, as the third result, a first period among the plurality of first periods that is a period that includes the second period.

[0175] (Appendix 13) The information processing device described in Appendix 11, characterized in that the control unit further executes a process of detecting an object from each frame of the video data based on the confidence score obtained by inputting each frame of the video data into the object detection model and the first threshold value and the second threshold value set by the setting process.

[0176] (Supplementary Note 14) When a time from an end time of a previous first period to a start time of a first period immediately following the previous first period is less than a predetermined time threshold, the control unit sets a period from the start time of the previous first period to an end time of the immediately following first period as a new first period; The information processing device described in Appendix 11 is characterized in that, when the time from the end time of a previous second period to the start time of the second period immediately following the previous second period is less than the predetermined time threshold, the device further executes a process of setting the period from the start time of the previous second period to the end time of the second period immediately following the previous second period as a new second period.

[0177] (Appendix 15) The information processing device described in Appendix 11, characterized in that the control unit further performs a process of analyzing a time series of frames included in the correct answer data, identifying frames in the time series in which the target object is not shown but the target object is present, and adding information that the object is present in the identified frames. [Explanation of symbols]

[0178] 100, 200 Information processing device 110,210 Communications Department 120,220 Input section 130,230 Display section 140,240 storage section 141 Video Data Table 142 Training Data Table 143 Correct data 144 Video Data 150,250 Control unit 151,251 Acquisition Department 152,252 Training data creation department 153,254 Threshold setting section 154,255 Object detection unit 253 Correct Data Creation Department

Claims

1. Input each frame of the video data into an object detection model, and obtain a detected position of the object in the image for that frame and a confidence score; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct answer data is small. A detection program that causes a computer to execute a process.

2. The detection program described in claim 1, characterized in that the first result includes a plurality of first periods of the frame in which the reliability score is equal to or greater than a first threshold, the second result includes a plurality of second periods of the frame in which the reliability score is equal to or greater than a second threshold, and the process of obtaining the third result obtains, as the third result, a first period among the plurality of first periods that is a period that includes the second period.

3. The detection program of claim 1, further comprising causing a computer to execute a process of detecting an object from each frame of the video data based on the confidence score obtained by inputting each frame of the video data into the object detection model and the first threshold and the second threshold set by the setting process.

4. when the time from the end time of a previous first period to the start time of a first period immediately following the previous first period is less than a predetermined time threshold, setting the period from the start time of the previous first period to the end time of the immediately following first period as a new first period; The detection program described in claim 1, further comprising causing a computer to execute a process of setting the period from the start time of the previous second period to the end time of the second period immediately following the previous second period as a new second period if the time from the end time of the previous second period to the start time of the second period immediately following the previous second period is less than the predetermined time threshold.

5. The detection program according to claim 1, further comprising a step of causing a computer to execute a process of analyzing a time series of frames included in the correct answer data, identifying frames among the time series in which the target object is not shown but the target object is present, and adding information about the presence of an object to the identified frames.

6. Input each frame of the video data into an object detection model, and obtain a detected position of the object in the image for that frame and a confidence score; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct answer data is small. A detection method characterized in that the processing is performed by a computer.

7. Input each frame of the video data into an object detection model, and obtain a detected position of the object in the image for that frame and a confidence score; obtain a first result that an object is determined to exist in the frame when the confidence score is equal to or greater than a first threshold, and a second result that an object is determined to exist in the frame when the confidence score is equal to or greater than a second threshold that is higher than the first threshold; obtaining a third result that identifies a period in which the object is present in the video data based on the first result and the second result; The first threshold and the second threshold are set so that the difference between the third result and the correct answer data is small. An information processing device having a control unit that executes processing.

Images