Event determination method, device and program using, text moving image classification model

Abstract

To provide an event determination method, a device, and a program capable of determining presence / absence of an event in moving image data, even in an event for which definition is difficult, without using a large number of pieces of teacher data.SOLUTION: An event determination device which is connected to a text moving image classification model and a time series analysis model through a network, first designates, a moving image for which determination is performed, an event which the user wants to detect, and a text moving image classification model which the user uses, on an input screen 20 provided on the input means, then generates a keyword in a detection event, and in the moving image, calculates an existence ratio of the keyword for acquiring time series probability data 29. Then, the device analyzes the time series probability data 29 by using a moving image in which the detection event is reproduced and the time series analysis model which was learned using the keyword, for determining presence of the event for which estimation is difficult and definition is doubtful such as, stumbling or tottering.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a method, a determination device, and a determination program for determining whether or not a detection event to be detected exists in video data using a text video classification model. [Background technology]

[0002] Conventionally, there are known methods for detecting actions such as falling from video data obtained by a camera, etc. For example, a posture determination device described in Patent Document 1 is a device that determines the posture of a passenger riding in a vehicle such as a microbus using a camera installed in the vehicle, and calculates feature amounts based on the positions of bone points such as the head and shoulders of the passenger detected from the video data, and inputs these feature amounts to an inference model generated by machine learning by a posture determination probability calculation unit to calculate the probability that the passenger is in a falling posture, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-134088 A Summary of the Invention [Problem to be solved by the invention]

[0004] As with the posture determination device described in Patent Document 1, when the definition is clear, such as a falling posture, and regular determination is possible from existing features, determination is possible using a conventional inference model. However, for phenomena that are difficult to define, such as "staggering," it is difficult to determine using a conventional inference model.

[0005] When using an inference model to judge such difficult-to-define phenomena, it is necessary to prepare training data for each event, but creating a model generally requires a large amount of training data, and it is often difficult to collect a large amount of data on the events to be detected. In addition, when adding more events to be recognized, it is necessary to collect training data for each event and have the inference model learn from it.

[0006] In view of the above problems, the present invention aims to provide an event determination method, an event determination device, and an event determination program that can determine the presence or absence of an event in video data, even for events that are unclearly defined and difficult to determine in a regular manner from existing features, without requiring a large amount of training data. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the event determination method of the present invention is an event determination method for determining the presence or absence of a detection event, which is an event to be detected within video data, and is characterized in that it includes a keyword creation step for creating a plurality of keywords related to the detection event, a keyword probability calculation step for inputting the video data and the keywords into a text video classification model that has been pre-trained using video data and text data, and obtaining as output time series probability data that shows the probability of the presence of the keyword for each specified time period in the video data in a time series, and a time series determination step for determining the presence or absence of the detection event in the video data from the time series probability data.

[0008] In the event determination method of the present invention, in order to determine whether an event to be detected exists in video data, first, a plurality of keywords related to the detection event and a pre-trained text video classification model are used. This text video classification model is used to obtain time series probability data indicating the probability of the existence of keywords at each predetermined time in the video data in a time series. Next, in a time series determination step, the presence or absence of the detection event in the video data is determined from the time series probability data.

[0009] In the event determination method of the present invention, a detected event is specified by a plurality of keywords, and the presence or absence of an event is determined using the time series probability of the plurality of keywords, making it possible to flexibly determine even events that are difficult to define. In addition, by using time series probability data extracted using a text video classification model that has acquired a wide range of associations between text and videos, it becomes easy to determine the presence or absence of a detected event in video data from the time series probability data. Here, the text video classification model refers to a model that can classify events by specifying the event with keywords described in text. This text video classification model makes it unnecessary to learn a large amount of video data related to the event to be classified. Note that keywords are also called prompts.

[0010] In the event determination method of the present invention, the time series determination step may input the time series probability data into a time series analysis model that has been pre-trained using the detected event and the keyword, and determine the presence or absence of the detected event in the video data as an output.

[0011] This configuration makes it easier to make a judgment in the time series judgment step, and since the time series probability data can be used for pre-learning of the time series analysis model, the teacher data can be minimized. In addition, the text video classification model may be re-trained using data related to the detected event. Here, the time series analysis model refers to a model that can classify data that changes over time.

[0012] In the event determination method of the present invention, the video data may be pre-processed prior to the keyword probability calculation step by tracking and cropping a detection target to be determined as a detection event. With this configuration, people or objects for which an event is to be detected are tracked and cropped in the video data, so that the movements of these people or objects can be grasped reliably, improving the accuracy of determination.

[0013] In the event determination method of the present invention, in the keyword creation step, an event label indicating the detected event is created as the keyword, the result of performing the keyword probability calculation step and the time series determination step is set as a first determination result, the event label is replaced with the keyword or the keyword is added to the event label, and the result of performing the keyword creation step, keyword probability calculation step and time series determination step again is set as an nth determination result (n is an integer of 2 or more), and the keywords derived from the first to nth determination results may be set as recommended keywords for the detected event.

[0014] By performing such a procedure in the keyword creation step, appropriate keywords for detecting a detection event can be created using a text video classification model and a time series analysis model.

[0015] In addition, in the event determination method of the present invention, the pre-learning in the time series analysis model may be a time series model learning step in which event shooting data, which is video data in which the detected event is filmed, and the keyword are input into the text video classification model, and the probability of the presence of the keyword for each specified time period in the event shooting data is obtained as output as learning time series probability data, and an event label indicating the detected event and the learning time series probability data are used as teacher data.

[0016] By performing pre-learning in the time series analysis model as such a time series model learning step, the event label indicating the detected event is associated with the probability of existence of each keyword in a time series, thereby improving the accuracy of detecting the detected event.

[0017] In the event determination method of the present invention, in the keyword creation step, the keyword may be selected from the text data in the text video classification model. With this configuration, in the keyword probability calculation step, the existence probability of each keyword can be reliably calculated.

[0018] In addition, in order to achieve the above-mentioned object, the event determination device of the present invention is an event determination device that determines the presence or absence of a detection event, which is an event to be detected in video data, and is equipped with an input means for accepting the video data to be determined, a plurality of keywords related to the detection event, a connection means capable of connecting to a text video classification model that has been pre-trained using video data and text data, and a time series analysis model that has been pre-trained using the detection event and the keywords, at least the input means, a display means for displaying the determination result of the time series analysis model, and a control means for controlling the input means, the display means, and the connection means, wherein when the video data and the keyword are input by the input means, the control means inputs the video data and the keyword to the time series analysis model via the connection means, obtains time series probability data showing the probability of existence of the keyword for each specified time in the video data in a time series, inputs the time series probability data to the time series analysis model via the connection means, and displays a determination of the presence or absence of the detection event in the video data on the display means.

[0019] An event determination program according to the present invention is a program for executing each of the above event determination methods using a computer. Effect of the Invention

[0020] According to the present invention, it is possible to provide an event determination method, device, and program that can determine the presence or absence of an event in video data, even if the event is difficult to define, without requiring a large amount of training data. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is an explanatory diagram showing a configuration of an event determination device according to an embodiment of the present invention; [Diagram 2] FIG. 4 is an explanatory diagram showing an input screen in the event discrimination device of the present embodiment. [Diagram 3] FIG. 4 is an explanatory diagram showing a learning screen in the event discrimination device of the present embodiment. [Figure 4]FIG. 13 is an explanatory diagram showing a screen showing a determination result of “stumbling” in the event determination device of the present embodiment. [Diagram 5] FIG. 4 is an explanatory diagram showing a menu screen in the event discrimination device of the present embodiment. [Figure 6] 3A to 3C are explanatory diagrams showing an example of moving image data to be judged by the event discrimination device of the present embodiment. [Figure 7] FIG. 13 is an explanatory diagram illustrating a worker's silhouette when he stumbles in video data. [Figure 8] FIG. 4 is an explanatory diagram showing a screen showing a determination result of “stagger” in the event determination device of the present embodiment. [Figure 9] FIG. 13 is an explanatory diagram illustrating a schematic silhouette of a worker when the worker becomes unsteady in video data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Next, an event determination device, an event determination method, and an event determination program according to an embodiment of the present invention will be described with reference to Fig. 1 to Fig. 9. The event determination device 1 of this embodiment is mainly composed of a computer, and includes a computer main body 2, a display 3, a mouse 4, and a keyboard 5. The event determination device 1 is connected to a network 6 such as the Internet, and can be connected to a text video classification model 7 and a time series analysis model 8 via the network 6.

[0023] The computer main body 2 is equipped with a computer system (all not shown) including a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a hard disk, storage means such as a memory, and connection means with various networks. The storage means of the computer main body 2 stores an event determination program and various data for executing the event determination method using the computer as the event determination device 1 of this embodiment.

[0024] The event determination program in this embodiment is executed by a CPU or the like, and various processes described below are performed. The computer system includes so-called cloud computing. The computer program is a concept that includes not only a program stored in the event determination device 1, but also a program stored in a server installed in a location separate from the event determination device 1, or a program that utilizes API (Application Programming Interface) linkage.

[0025] Next, the functional configuration of the event determination device 1 will be described with reference to Fig. 1. The event determination device 1 includes, as its functional parts, an input means 9, a display means 10, a connection means 11, and a control means 12. In addition, the event determination device 1 can be connected to a pre-trained text video classification model 7 and a time series analysis model 8 via the connection means 11.

[0026] In this embodiment, the input means 9 includes an input screen 20 shown in Fig. 2. The input screen 20 includes a video input unit 21 capable of inputting video data V to be judged, a detection event input unit 22 capable of inputting a detection event which is an event such as an action or state to be detected, a classification model input unit 23 for selecting a text video classification model 7 to be used for the judgment, a keyword input unit 24 for accepting a keyword related to the event to be detected, a probability calculation button 25 for inputting the video data V input to the video input unit 21 and the created keyword to obtain an output of the text video classification model 7, and a calculation result display unit 26 for displaying the output of the text video classification model 7.

[0027] The video data V to be detected is input into the video input unit 21 by dragging and dropping the video to be detected, or by clicking the reference button 21a and specifying the path of a file stored in the event determination device 1, or by inputting the URL of a video published on the network, or by other operations.

[0028] An event to be detected, such as a behavior such as "stumbling" or "staggering," is input to the detection event input unit 22. In this case, by clicking the reference button 22a, events that have been judged in the past may be displayed in a pop-up menu or the like so that they can be selected.

[0029] A text moving image classification model 7 to be used in the judgment is input to the classification model input unit 23. In this classification model input unit 23, by clicking a reference button 23a, available text moving image classification models 7 are displayed in a pop-up menu or the like, so that an arbitrary text moving image classification model 7 can be selected.

[0030] A keyword related to the behavior to be detected is created and input into the keyword input unit 24. The operator may input any keyword into the keyword input unit 24. Alternatively, the operator may click the determined keyword call button 27 to display and select a keyword that has been determined for the event to be detected. Alternatively, the operator may click the model call button 28 to display and select text used for classification in the text video classification model 7 used for determination.

[0031] After entering the necessary information in the video input section 21, the detection event input section 22, the classification model input section 23, and the keyword input section 24, clicking the probability calculation button 25 will cause the probability of each keyword in the specified video data V to be calculated in the calculation result display section 26 and displayed in a graph as time-series probability data 29.

[0032] The input means 9 also includes a learning screen 30 shown in Fig. 3. The learning screen 30 is a screen for learning the time series analysis model 8, and includes a time series model input unit 31 for selecting the time series analysis model 8 for learning, an event label input unit 32 for inputting an event label which is an event to be learned, a teacher data input unit 33 capable of inputting learning time series probability data which becomes teacher data, a learning data input unit 34 for inputting the learning time series probability data which is the learning target, and a learning button 35 to be clicked when learning is performed.

[0033] Each of these input sections is provided with reference buttons 31a to 34a to assist in input. Each of the input sections and reference buttons has the same configuration as each of the input screen 20 in Fig. 2, so detailed description will be omitted.

[0034] Fig. 4 shows a determination result screen 40 that is displayed as a result of clicking an event determination button 53 on a menu screen 50 in Fig. 5 described later. The determination result screen 40 includes a video display section 41 that displays the name or storage location of the detected video, an event label display section 42 that displays the detected event, and a determination result display section 43 that displays the determination result as the output of the time series analysis model 8. The determination result screen 40 displays a line type that indicates which keyword the graph displayed on the determination result display section 43 indicates. In the actual display, this line type is displayed by changing the color of the line.

[0035] The display means 10 in this embodiment is a functional unit that displays the input screen 20, which is the input means 9, the learning screen 30, and the judgment result screen 40, etc., on the display 3 of the event judgment device 1 or on another display terminal such as another tablet terminal.

[0036] The connection means 11 in this embodiment is a functional unit that connects the event determination device 1 to the text video classification model 7 that has been pre-trained using video data and text data, and the time series analysis model 8 that has been pre-trained using detected events and keywords, and utilizes both models in the determination process.

[0037] The control means 12 in this embodiment is a functional unit that controls the input means 9, the display means 10, and the connection means 11. The control means 12 is realized by hardware including a computer main body 2 and a plurality of programs including an event determination program.

[0038] An example of the text video classification model 7 that can be used in the event determination device 1 of this embodiment is XCLIP, which is published by OpenAI. This XCLIP is a model that extends CLIP, a large-scale image classification model that can classify images by linking them with language in zero shots, to videos, and details are described in the following literature. (Expanding Language-Image Pretrained Models for General Video Recognition / 4 Aug 2022 / Bolin Ni, Houwen Peng et al. https: / / arxiv.org / pdf / 2208.02816.pdf).

[0039] The text video classification model 7 may be not only this XCLIP, but also any model capable of classifying events by specifying the events with keywords described in text. For example, XCLIP is a model that extends a language image pre-learning model to video recognition, which performs contrastive learning on images and text (descriptive sentences) using a large amount of data extracted from images and texts published on the Internet, etc., but is not limited to such models. Various models can be used, such as a natural language processing model trained using a large amount of text data and videos (including images). Such a model makes it possible to estimate the similarity between text and videos by zero-shot learning (a technology for predicting unknown things that have not been learned).

[0040] Furthermore, the time series analysis model 8 that can be used in the event determination device 1 of this embodiment is a model called TSF, which enables a random forest, which is a decision tree ensemble method, to be applied to time series data. Details of this TSF are described in the following document. (A Time Series Forest for Classification and Feature Extraction / 18 Feb 2013 / Houtao Deng et al. https: / / arxiv.org / pdf / 1302.2277.pdf).

[0041] As this time series analysis model 8, not only TSF but also a model capable of classifying time series data, such as a deep learning model or an ensemble model that combines multiple models, can be used.

[0042] Next, the event determination method of the present embodiment will be described with reference to the drawings. The event determination method of the present embodiment includes a keyword creation step, a keyword probability calculation step, a time series model learning step, and a time series determination step.

[0043] In this embodiment, as shown in Fig. 6, whether or not an incident that could be considered a near miss by worker A occurs is determined from video data V captured by a fixed camera at the work site where worker A passes through. Actions that qualify as near misses include actions such as "stumbling" and "unsteadiness" that do not result in an accident but are potentially factors that could lead to an accident. These actions are not clearly defined actions such as "falling" but have a vague concept.

[0044] In the event determination method of this embodiment, when the event determination device 1 is started and the event determination program is started, a menu screen 50 shown in Fig. 5 is displayed. When a video / keyword input button 51 on the menu screen 50 is clicked, the input screen 20 shown in Fig. 2 is displayed.

[0045] A keyword creation step is executed on the input screen 20. This keyword creation step is a step for creating a plurality of keywords related to the detection event. The detection event is an event that the person wishing to make a judgment wants to detect, for example, an action of the worker A during work that corresponds to a near miss.

[0046] In the keyword creation step, multiple related keywords are created for a detection event having an ambiguous concept. These keywords are used to identify the detection event, and can be used to divide the detection event into multiple elements. For example, as shown in Figure 2, if the detection event is a person's "stumbling," five keywords can be listed: falling, standing up, walking, lying down, and bending.

[0047] When the operator creates the keyword at his / her own discretion, the operator can input the wording he / she wants to use as the keyword into the keyword input unit 24 from the keyboard 5. At this time, the keyword may be created using LLM (Large Language Models). In addition, if a judgment has already been made on the detected event, clicking the judgment call button 27 makes it possible to use the proven keyword.

[0048] On the other hand, even if there is no past record of judgment on the detection event to be judged this time, by clicking the model call button 28, it is possible to display the text used for classification in the text video classification model 7 inputted in the classification model input unit 23. This makes it possible to create keywords whose existence probability can be calculated reliably.

[0049] Next, the keyword probability calculation step will be described. In the keyword probability calculation step, multiple keywords are associated with the detection event and input to the text video classification model 7 in the video to be detected, and time-series probability data 29 indicating the existence probability of each keyword for each predetermined time in the video data V is obtained as the output.

[0050] 2, after inputting the necessary information into the video input section 21, the detection event input section 22, the classification model input section 23, and the keyword input section 24, the probability calculation button 25 is clicked. This operation causes the control means 12 to input the input video data V and keywords to the text video classification model 7 via the connection means 11, and as an output, the probability of each keyword in the specified video data V is calculated in the calculation result display section 26 and displayed in a graph as time-series data.

[0051] In this embodiment, as pre-processing of the video data V, as shown in Fig. 6(A) to (C), a worker A who is a detection target at the work site is detected and tracked prior to the keyword probability calculation step, and the video of the worker A is cropped as shown in Fig. 7 and Fig. 9. When multiple workers A exist in one video data V, the video is cropped for each worker A.

[0052] Next, return to the menu screen 50 and execute the time series model learning step by clicking the time series model learning button 52. When the time series model learning button 52 is clicked, the learning screen 30 shown in FIG.

[0053] In the time series model learning step, the time series analysis model 8 to be learned is input to a time series model input unit 31 on a learning screen 30. In Fig. 3, TSF is input as an example of this time series analysis model 8. In addition, "stumbling" is input as an event label, which is an event to be learned, to an event label input unit 32, and event shooting data, which will be the teacher data, is input to a teacher data input unit 33.

[0054] The event image data serving as the teacher data is video data that reproduces the event label "stumbling," and the position (time) at which the stumbling occurs in the event image data is stored in the event image data, and so-called annotation is performed. In this embodiment, as shown in Fig. 3, video data stored in a computer main body 2 of the event determination device 1 is input.

[0055] In the time series model learning step, learning time series probability data to be learned is input to the learning data input unit 34 on the learning screen 30. In Fig. 3, as an example of the learning time series probability data, the learning time series probability data stored in the computer main body 2 of the event determination device 1 is input. In this embodiment, the learning time series probability data is obtained by inputting event shooting data that reproduces the event label "stumbling" in advance together with keywords into the text video classification model 7, and using the learning time series probability data that is the output as the learning data.

[0056] In this way, after inputting the necessary information on the learning screen 30, the learning button 35 is clicked to learn the TFS, which is the time series analysis model 8. Since the event label is an event to be detected in the video data V, the relationship between the event related to the event label and the probability of each keyword is learned by this learning.

[0057] Next, the time series judgment step will be described. After the time series model learning step is performed, return to the menu screen 50 and click the event judgment button 53 to execute the time series judgment step. When the event judgment button 53 is clicked, the judgment result screen 40 shown in FIG. 4 is displayed.

[0058] 4 shows the judgment result when the detection event, which is the event to be detected, is "stumbling." In the judgment result display section 43, the judgment result is displayed below the time-series probability data 29. The white parts indicate that a stumbling has not occurred, and the black parts indicate that a stumbling has occurred.

[0059] Here, when the state of worker A in the time-series probability data 29 displayed on the judgment result display unit 43 in Fig. 4 is checked using the video data V, it turns out to be in the state shown in Fig. 7. In Fig. 7, no stumbling occurs in (1), (2), and (5), but stumbling occurs in (3) and (4).

[0060] In (3) and (4) of Fig. 7, Worker A has lost his balance, but he has not fallen or been bent over. Therefore, it is difficult to determine whether Worker A has stumbled using a conventional inference model. In this embodiment, the detection event is determined using the probability of multiple keywords, so that it is possible to accurately determine even an event that is difficult to define.

[0061] 8 shows the judgment result when the detection event to be detected is "wandering." In the judgment result display section 43, the judgment result is displayed below the time-series probability data 29. The white parts indicate that wandering is not occurring, and the black parts indicate that wandering is occurring.

[0062] Here, when the state of worker A in the time-series probability data 29 displayed on the judgment result display unit 43 in Fig. 8 is checked using the video data V, it turns out to be in the state shown in Fig. 9. In Fig. 9, no stumbling occurs in (1), (4), and (5), but stumbling occurs in (2) and (3).

[0063] When the detected event is "staggering," the difference from a normal walking state is smaller than when it is "stumbling," but the event determination method of this embodiment makes it possible to determine with a high degree of probability whether or not "staggering" is present.

[0064] Next, another embodiment of the keyword creation step will be described. In the input screen 20 shown in Fig. 2, the event label "stumbling" is input as a keyword in the keyword input section 24, and the keyword probability calculation step and the time series judgment step are performed to obtain a first judgment result. The judgment result is a judgment result in which the keyword is only "stumbling" in the judgment result screen 40 shown in Fig. 4.

[0065] Next, returning to the input screen 20 shown in Fig. 2, another keyword is input in place of "stumble" in the keyword input section 24, or another keyword is added to "stumble", and a keyword probability calculation step and a time series judgment step are performed to derive a second judgment result. This operation is performed n times (n is an integer of 2 or more) to verify the judgment results, and the keyword with a high probability of judgment is set as the recommended keyword for the detected event. By verifying the first judgment result to the nth judgment result, it is possible to create a keyword with a high probability of judgment in the keyword creation step.

[0066] In the above embodiment, the event determination device 1 is realized by a desktop computer, but it is not limited thereto and can be realized by various devices such as a tablet terminal, a smartphone, etc. Also, in the above embodiment, the event determination device 1 can be connected to the text video classification model 7 and the time series analysis model 8 via the network 6, but it is not limited thereto and these models may be stored in the storage means of the computer main body 2.

[0067] In the above embodiment, the time series model learning step is performed after the keyword probability calculation step, but these steps can be performed in any order. Although the text video classification model 7 uses XCLIP, other known models may be used as long as they classify text and video. Similarly, the time series analysis model 8 may use a known time series analysis model other than TSF. Furthermore, the event determination program of the present invention may be stored not only in a computer, but also in a storage medium such as a CD-ROM or DVD-ROM.

[0068] In addition, in this embodiment, the time series analysis model 8 is used in the time series judgment step, but this is not limiting. A threshold value may be set for the probability of each keyword in the time series probability data, and the presence or absence of a detected event may be judged based on the relationship between the probability calculated for each keyword and the threshold value. In this case, the threshold value for each keyword may be stored in a data table or the like, or may be set by the operator at the time of judgment.

[0069] In this embodiment, only one event is input to the detection event input unit 22 in Fig. 2, but this is not limiting, and multiple events (for example, tripping, unsteadiness, or falling) may be input and it may be determined whether or not each event has occurred as a determination result. In this way, by simultaneously determining multiple events, it is possible to improve the efficiency of the determination.

[0070] In addition, in this embodiment, a "near miss", specifically "stumbling" or "unsteadiness" has been used as an example of a detection event, but the detection event is not limited to this, and even if a simple event such as "falling" is used as the detection event, the event can naturally be accurately determined. [Explanation of symbols]

[0071] 1...Event determination device 2. Computer body 3. Display 4. Mouse 5. Keyboard 6. Network 7…Text-video classification model 8. Time series analysis model 9. Input method 10...Display means 11. Connection means 12...Control means 20... Input screen 30…Learning screen 40…Judgment result screen 50...Menu screen

Claims

1. An event determination method for determining the presence or absence of a detection event that is an event to be detected in video data, comprising: a keyword creation step of creating a plurality of keywords related to the detected event; a keyword probability calculation step of inputting the video data and the keyword into a text video classification model that has been pre-trained using video data and text data, and obtaining, as an output, time-series probability data indicating the existence probability of the keyword for each predetermined time period in the video data in a time series; An event determination method comprising a time series determination step of determining the presence or absence of the detected event in the video data from the time series probability data.

2. The event determination method according to claim 1, The time series determination step is an event determination method characterized in that the time series probability data is input into a time series analysis model that has been pre-trained using the detected event and the keyword, and the presence or absence of the detected event in the video data is determined as an output.

3. The event determination method according to claim 1, The event determination method, characterized in that the video data is subjected to pre-processing, prior to the keyword probability calculation step, in which a detection target that is to be determined as the detection event is tracked and cropped.

4. The event determination method according to claim 1, In the keyword creation step, an event label indicating the detected event is created as the keyword, and a result of the keyword probability calculation step and the time series determination step is set as a first determination result; The event label is replaced with the keyword, or the keyword is added to the event label, and the keyword creation step, the keyword probability calculation step, and the time series determination step are performed again, and the result is defined as an n-th determination result (n is an integer of 2 or more); An event determination method, comprising the steps of: determining keywords derived from the first through n-th determination results as recommended keywords for the detected event;

5. The event determination method according to claim 2, The event determination method is characterized in that the pre-learning in the time series analysis model is a time series model learning step in which event shooting data, which is video data in which the detected event is captured, and the keyword are input into the text video classification model, and the probability of the presence of the keyword for each specified time period in the event shooting data is obtained as output as learning time series probability data, and an event label indicating the detected event and the learning time series probability data are used as teacher data.

6. The event determination method according to claim 1, The event determination method, wherein in the keyword creation step, the keywords are selected from the text data in the text video classification model.

7. An event determination device that determines the presence or absence of a detection event that is an event to be detected in video data, comprising: an input means for receiving video data to be judged and a plurality of keywords related to the detected event; A connection means capable of connecting to a text video classification model pre-trained with video data and text data, and a time series analysis model pre-trained with the detected events and the keywords; At least the input means and a display means for displaying the judgment result of the time series analysis model; a control means for controlling the input means, the display means, and the connection means; The control means, when the video data and the keyword are input by the input means, inputs the video data and the keyword into the time series analysis model via the connection means, obtains time series probability data which indicates the probability of the presence of the keyword at each specified time in the video data in a time series, inputs the time series probability data into the time series analysis model via the connection means, and displays a determination of the presence or absence of the detected event in the video data on the display means.

8. An event determination program for executing the event determination method according to any one of claims 1 to 6 by using a computer.

Images