Class definition support system, class definition support device, and class definition support method

Abstract

This aims to enable even inexperienced users to properly perform the tasks of creating annotations and class definition information necessary for training recognition models. [Solution] The system comprises a user terminal where the user inputs recognition target information that describes the content of the event to be recognized, and a model generation server that acquires class definition information based on the recognition target information. The model generation server causes the language model of the LLM server to perform a process to collect recognition target information necessary for acquiring class definition information while engaging in natural language dialogue with the user, and acquires the class definition information generated by the language model. Specifically, the model generation server acquires instruction text in natural language input by the user at the user terminal, sends the instruction text to the LLM server to instruct it to execute the class definition process, and acquires class definition information based on the instruction text from the LLM server.

Description

Technical Field

[0001] The present invention relates to a class definition support system, a class definition support device, and a class definition support method for assisting a user's operation to obtain class definition information regarding an event to be recognized.

Background Art

[0002] In manufacturing and logistics workplaces, etc., it is possible to improve work efficiency by analyzing the working conditions of workers and reviewing how the work is carried out based on the analysis results. When analyzing such working conditions of workers, using a recognition model generated by machine learning is expected to improve the accuracy of work analysis and reduce the burden on the user related to work analysis. In such a situation, in machine learning, a large amount of labeled data is required to improve the accuracy of object recognition and action recognition.

[0003] As a technique for performing work analysis using such a recognition model, conventionally, an annotation screen is displayed on a terminal device, and in the annotation screen, the user is made to specify an object to be recognized for a frame image and an action to be recognized for an interval video, and data used for learning to generate an object recognition model (object detection model) and an action recognition model (action classification model) is created (see Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to conventional technology, user operations for annotation can be made more efficient by improving the GUI, such as specifying the image area (object image) of an object on a frame image of a video taken at the site, or inputting the division of a section in the video taken at the site where a predetermined action was confirmed.

[0006] On the other hand, when extracting action images (image regions of human action scenes) from captured images, it is necessary to properly define classes, which are effective combinations of multiple objects related to the action to be recognized, that is, the person as the subject of the action and the objects that are the target of the action (including people). However, when the correlation between multiple objects related to the action becomes complex, properly defining classes requires expertise from the user, and it has been difficult for inexperienced users to properly define classes.

[0007] Therefore, the main objective of the present invention is to provide a class definition support system, a class definition support device, and a class definition support method that enable even inexperienced users to appropriately perform the task of creating class definition information necessary for annotation and training in order to generate a recognition model. [Means for solving the problem]

[0008] The class definition support system of the present invention is a class definition support system that supports user operations for obtaining class definition information relating to an event to be recognized, comprising: a terminal device in which the user performs an operation to input information relating to an event to be recognized that describes the content of the event to be recognized; and a server device connected to the terminal device via a network and which obtains the class definition information based on the information relating to the event to be recognized, wherein the server device is configured to cause a language model to perform a process to collect the information relating to the event to be recognized necessary for obtaining the class definition information while engaging in natural language dialogue with the user, and to obtain the class definition information generated by the language model.

[0009] Furthermore, the class definition support device of the present invention is a class definition support device equipped with a processor that performs processing to support user operations for acquiring class definition information relating to an event to be recognized, and is connected via a network to a terminal device in which the user performs operations to input information describing the content of the event to be recognized, and the processor is configured to acquire the class definition information generated by the language model by having a language model perform processing to collect the information necessary for acquiring the class definition information while engaging in natural language dialogue with the user.

[0010] Furthermore, the class definition support method of the present invention is a class definition support method that causes a processor to execute a process to support user operations for obtaining class definition information relating to an event to be recognized, wherein, in obtaining the class definition information based on recognition target information that describes the content of the event to be recognized, the method is configured to cause a language model to perform a process to collect the recognition target information necessary for obtaining the class definition information while engaging in natural language dialogue with the user, and to obtain the class definition information generated by the language model. [Effects of the Invention]

[0011] According to the present invention, the user's work in creating class definition information is appropriately supported, so even users unfamiliar with the process of creating class definition information can perform it properly. [Brief explanation of the drawing]

[0012] [Figure 1] Overall configuration diagram of the model generation system according to this embodiment. [Figure 2] Block diagram showing the general configuration of the model generation server. [Figure 3] This sequence diagram outlines the processing related to class definition performed on the model generation server, user terminal, and LLM server. [Figure 4] An explanatory diagram showing an example of class definition information. [Figure 5]Explanatory diagram showing an overview of object image extraction processing and object feature extraction processing performed by the model generation server [Figure 6] Explanatory diagram showing an overview of all combination extraction processing, valid combination extraction processing, action image extraction processing, and action feature extraction processing performed by the model generation server [Figure 7] Explanatory diagram showing an overview of clustering processing and representative image extraction processing performed by the model generation server [Figure 8] Explanatory diagram showing a screen of the class definition mode displayed on the user terminal [Figure 9] Explanatory diagram showing a screen of the class definition mode when a model is selected and displayed on the user terminal [Figure 10] Explanatory diagram showing a screen of the object recognition mode displayed on the user terminal [Figure 11] Explanatory diagram showing a screen of the action recognition mode displayed on the user terminal [Figure 12] Explanatory diagram showing a screen of the inference mode displayed on the user terminal [Figure 13] Flowchart showing the procedure of processing performed by the model generation server [Figure 14] Flowchart showing the procedure of input video processing performed by the model generation server [Figure 15] Flowchart showing the procedure of class definition relay processing performed by the model generation server [Figure 16] Flowchart showing the procedure of object candidate extraction processing and object recognition annotation processing performed by the model generation server [Figure 17] Flowchart showing the procedure of action candidate extraction processing and action recognition annotation processing performed by the model generation server

Mode for Carrying Out the Invention

[0013] A first invention made to solve the above problems is a class definition support system that supports a user's operation for acquiring class definition information regarding an event to be recognized, including a terminal device through which a user inputs recognition target information that describes the content of the event to be recognized, and a server device connected to the terminal device via a network and acquiring the class definition information based on the recognition target information. The server device is configured to cause a language model to perform a process of collecting the recognition target information necessary for acquiring the class definition information while conducting a natural language dialogue with the user, and to acquire the class definition information generated by the language model.

[0014] According to this, since the work of the user creating the class definition information is appropriately supported, even an inexperienced user can appropriately perform the work of creating the class definition information.

[0015] Further, a second invention is configured such that the server device acquires, as the class definition information regarding the action class, information defining an object class corresponding to the action class.

[0016] According to this, based on the class definition information, an action image used for learning an action recognition model can be appropriately extracted from a captured image.

[0017] Further, a third invention is configured such that the server device acquires, as the class definition information, information regarding a valid combination of an action class and an object class that is the target thereof.

[0018] According to this, a more appropriate action image can be extracted from a captured image.

[0019] Further, a fourth invention is configured such that the server device causes the terminal device to display a screen in which the class definition information representing the correspondence between the object class and the action class is visualized in a tabular format.

[0020] According to this, class definition information is visualized and presented to the user in a tabular format, making it easy for the user to determine whether the acquired class definition information is appropriate or not.

[0021] Furthermore, the fifth invention is configured such that the server device is connected via a network to a language model device equipped with the language model, and acquires the class definition information in cooperation with the language model device.

[0022] According to this, class definition information can be obtained inexpensively by using a language modeling device operated by another company.

[0023] Furthermore, the sixth invention is configured such that the server device acquires instruction text in natural language entered by a user in the terminal device from the terminal device, transmits the instruction text to the language model device to instruct it to execute class definition processing, and acquires the class definition information based on the instruction text from the language model device.

[0024] According to this, appropriate class definition information can be obtained by using a language modeling device operated by another company.

[0025] Furthermore, the seventh invention is configured such that the server device transmits the captured image along with the instruction text to the language model device to instruct it to execute the class definition process, and obtains the class definition information based on the instruction text and the captured image from the language model device.

[0026] According to this, the language model device can collect information missing from the instruction text from the captured image, thereby generating more appropriate class definition information and further reducing the burden on the user who inputs the instruction text.

[0027] Furthermore, the eighth invention is a class definition support device equipped with a processor that performs processing to support user operations for acquiring class definition information relating to an event to be recognized, the device being connected via a network to a terminal device in which the user performs operations to input information describing the content of the event to be recognized, and the processor, in acquiring the class definition information based on the information to be recognized, causes a language model to perform processing to collect the information to be recognized necessary for acquiring the class definition information while engaging in natural language dialogue with the user, and acquires the class definition information generated by the language model.

[0028] According to this, similar to the first invention, the user's work in creating class definition information is appropriately supported, so even users unfamiliar with the process of creating class definition information can perform it properly.

[0029] Furthermore, the ninth invention is a class definition support method that causes a processor to execute a process to support user operations for obtaining class definition information relating to an event to be recognized, wherein, in obtaining the class definition information based on recognition target information that describes the content of the event to be recognized, the method is configured to cause a language model to perform a process to collect the recognition target information necessary for obtaining the class definition information while engaging in natural language dialogue with the user, and to obtain the class definition information generated by the language model.

[0030] According to this, similar to the first invention, the user's work in creating class definition information is appropriately supported, so even users unfamiliar with the process of creating class definition information can perform it properly.

[0031] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0032] Figure 1 is an overall diagram of the model generation system according to this embodiment.

[0033] This system (class definition support system, object recognition annotation system, action recognition annotation system) comprises a model generation server 1 (server device, class definition support device, object recognition annotation device, action recognition annotation device), a user terminal 2 (terminal device), and an LLM server 3 (language model device). The model generation server 1 and user terminal 2 are connected via a network. The model generation server 1 and LLM server 3 are also connected via a network.

[0034] Model generation server 1 performs processes such as acquiring training images (object images, action images) necessary for generating recognition models (object recognition models, action recognition models), annotation to assign recognition classes (object classes, action classes) to the training images, machine learning to generate recognition models using the training images, and verifying the accuracy of the generated recognition models.

[0035] User terminal 2 displays an operation screen where the user performs operations necessary for generating recognition models (object recognition models, action recognition models) based on the control of model generation server 1. On the operation screen, the user performs operations such as inputting training videos and verification videos, inputting processing conditions related to the processing performed by model generation server 1, and checking the accuracy of the generated recognition model.

[0036] LLM Server 3 includes a language model. The language model is used to obtain information necessary for generating training images (class definition information) through interaction with the user. LLM Server 3 includes a model in which processing functions for data other than language are added to the language model, for example, a multimodal LLM (Large Language Model). Alternatively, the server may include a model that has both image processing and language processing functions, for example, a VLM (Vision-Language Model).

[0037] The model generation server 1 may consist of a single information processing device, or it may consist of multiple information processing devices. For example, it may consist of a front-end server that handles the GUI for annotation, and a back-end server that performs processing such as annotation and training for generating object recognition models and action recognition models.

[0038] Next, we will describe the general configuration of model generation server 1. Figure 2 is a block diagram showing the general configuration of model generation server 1.

[0039] The model generation server 1 comprises a communication unit 11, a storage unit 12, and a processor 13.

[0040] The communication unit 11 communicates with the user terminal 2 and the LLM server 3.

[0041] The memory unit 12 stores programs and other data that are executed by the processor 13.

[0042] Furthermore, the memory unit 12 stores training videos (frame images) and verification videos (frame images). The training videos and verification videos are recordings of the worker's work. The training videos are used for training to generate object recognition models and action recognition models. The verification videos are used to verify the accuracy of the generated object recognition models and action recognition models.

[0043] The memory unit 12 also stores object images and action images. Object images are extracted from the frame image as rectangular image regions surrounding the detected objects. Object images are used as training images to generate an object recognition model. Action images are extracted from the frame image as rectangular image regions containing multiple objects related to a person's actions. Action images are used as training images to generate an action recognition model.

[0044] Furthermore, the memory unit 12 stores class definition information, object recognition annotation information, and action recognition annotation information. Class definition information is information that defines the event classes to be recognized. Specifically, class definition information is information regarding the correspondence between object classes and action classes, that is, information regarding valid combinations of action classes and the object classes they target. Class definition information is used in the process of extracting action images from frame images. Object recognition annotation information is information regarding the object classes assigned to object images and is used for training to generate object recognition models. Action recognition annotation information is information regarding the action classes assigned to action images and is used for training to generate action recognition models.

[0045] The object class represents the type of object to be recognized (e.g., person, driver, tablet device). The action class represents the type of action to be recognized (e.g., driver fastening action, tablet device touch operation).

[0046] Furthermore, the memory unit 12 stores an object recognition model and an action recognition model. The object recognition model recognizes objects contained in the video to be processed and is generated by the processor 13. The action recognition model recognizes actions contained in the video to be processed and is generated by the processor 13.

[0047] The processor 13 performs various processes by executing programs stored in the memory unit 12. In this embodiment, the processor 13 performs processes such as video acquisition, condition setting, GUI processing, class definition relay processing, object detection, object candidate extraction, object recognition annotation, object recognition model learning, object recognition inference, action candidate extraction, action recognition annotation, action recognition model learning, and action recognition inference. Note that the processor 13 can distinguish and execute the object detection process and the object recognition inference process by using different recognition models for each.

[0048] In the input video processing, processor 13 acquires a training video in which the user has performed input operations on user terminal 2 and obtains frame images that make up that training video. In addition, in the input video processing, processor 13 acquires a verification video in which the user has performed input operations on user terminal 2 and obtains frame images that make up that verification video.

[0049] In this embodiment, a video file is input to the user terminal 2 in response to user operations, and the video file is sent from the user terminal 2 to the model generation server 1. However, the video file may also be sent to the model generation server 1 from an external device (server, online storage, etc.). Alternatively, the video file may be input to the model generation server 1 offline via a suitable storage medium on which the video file is stored.

[0050] In the condition setting process, the processor 13 sets processing conditions related to object candidate extraction, object recognition annotation, object recognition inference, action candidate extraction, action recognition annotation, and action recognition inference, in response to user input operations on the user terminal 2.

[0051] In GUI processing, the processor 13 generates screens to be displayed on the user terminal 2 and acquires input information corresponding to the user's operations on the screen. Specifically, the class definition mode screen 101 (see Figure 8), the object recognition mode screen 201 (see Figure 10), the action recognition mode screen 301 (see Figure 11), and the inference mode screen 401 (see Figure 12) are displayed on the user terminal 2.

[0052] In the class definition relay process, the processor 13 relays the communication between the user terminal 2 and the LLM server 3 regarding the class definition, and while engaging in natural language dialogue with the user, it causes the language model to perform the process of collecting recognition target information necessary for obtaining class definition information. The class definition information generated by the LLM server 3 is presented to the user on the user terminal 2, and when the user obtains class definition information that satisfies them and the user instructs the completion of the class definition process, the class definition information is stored in the storage unit 12.

[0053] In the object detection process, processor 13 detects objects from the frame image using an object detection model generated by machine learning. During this process, object classification is performed, and the object class for the detected object is predicted. Furthermore, in the object detection process, a zero-shot learning object detection model is used for the first run, and a few-shot learning object detection model is used for subsequent runs. Other general-purpose object detection methods may also be used in the object detection process.

[0054] In the object candidate extraction process, the processor 13 extracts object images that are the target of the user's labeling operation as candidate images. Specifically, the processor 13 extracts object images (rectangular image regions surrounding objects) from the frame image based on the detection results of the object detection process, extracts feature quantities for each object image, then classifies the object images into multiple clusters (groups of object images) based on the feature quantities for each object image, and extracts representative images for each cluster as candidate images. At this time, the object images are classified into object classes, and the object candidate extraction process is performed for each object class. In addition, by classifying object images into clusters based on feature quantities, object images with similar features are grouped into one cluster.

[0055] In the object recognition annotation process, the processor 13 presents representative images for each cluster extracted in the object candidate extraction process to the user as candidate images. In response to the user's operation to select a candidate image corresponding to a specified object class, the processor 13 assigns the label of the specified object class to all object images included in the cluster to which the selected candidate image (representative image) belongs.

[0056] In the object recognition model training process, processor 13 generates an object recognition model by performing machine learning using the labeled object images obtained in the object recognition annotation process.

[0057] In the object recognition inference process, processor 13 performs inference using the object recognition model to derive object recognition results from the verification video. The inference results from the object recognition inference process are presented to the user, who can then verify the accuracy of the object recognition model based on the results.

[0058] In the action candidate extraction process, the processor 13 extracts action images that are the target of the user's labeling operation as candidate images. Specifically, the processor 13 extracts valid combinations of a person (subject of the action) and an object (object of the action) corresponding to a given action based on class definition information, then extracts action images that include the image regions of the person and object that make up the valid combination, extracts feature quantities for each action image, then classifies the action images into multiple clusters (groups of action images) based on the feature quantities for each object image, and extracts representative images for each cluster as candidate images. At this time, the action images are classified into action classes, and the action candidate extraction process is performed for each action class.

[0059] In the behavior recognition annotation process, the processor 13 presents representative images for each cluster extracted in the behavior candidate extraction process to the user as candidate images. In response to the user's operation to select a candidate image corresponding to a specified behavior class, the processor 13 assigns the label of the specified behavior class to all behavior images included in the cluster to which the selected candidate image (representative image) belongs.

[0060] In the behavior recognition model learning process, processor 13 generates a behavior recognition model by performing machine learning using the labeled behavior images obtained in the behavior recognition annotation process.

[0061] In the behavior recognition inference process, processor 13 performs inference to derive behavior recognition results from the verification video using the behavior recognition model. The inference results from the behavior recognition inference process are presented to the user, and the user can verify the accuracy of the behavior recognition model based on the inference results.

[0062] Next, we will explain the processes related to class definition performed on the model generation server 1, user terminal 2, and LLM server 3. Figure 3 is a sequence diagram showing an overview of the processes related to class definition. In the example shown in Figure 3, the action of tightening screws on an engine using a screwdriver in a factory is the subject of class definition.

[0063] In this embodiment, the model generation server 1, the user terminal 2, and the LLM server 3 work together to generate class definition information. The class definition information is information regarding the correspondence between object classes and action classes, that is, information regarding effective combinations of action classes and the object classes they target.

[0064] Model generation server 1 performs class definition relay processing. In class definition relay processing, communication regarding class definitions between user terminal 2 and LLM server 3 is relayed.

[0065] First, the LLM server 3 receives the training video (frame images) acquired from the user terminal 2 and stored in the memory unit 12.

[0066] Furthermore, user terminal 2 processes input of instruction text and learning videos in response to user operations. At this time, the user performs an operation to input instruction text that includes a sentence describing the content of the action to be recognized (recognition target information). The input instruction text is sent to LLM server 3 via model generation server 1.

[0067] Next, when the LLM server 3 receives training videos from the model generation server 1 and instruction text from the user terminal 2 via the model generation server 1, it performs dialogue processing and class definition processing.

[0068] In the dialogue process, natural language response text (response information) is generated in response to the user's input of natural language instruction text (instruction information). Through repeated natural language dialogue with the user, the information necessary for class definition is collected. If any necessary information for class definition is missing, a response text is generated asking for the missing information.

[0069] In the class definition process, the content of the action to be recognized, as instructed by the user in the instruction text, is interpreted, and the information necessary for class definition is extracted from the training video, and the class definition information is output. At this time, the training video and instruction text are input to the multimodal LLM, and the response text and class definition information are output from the multimodal LLM. The response text and class definition information are sent to the user terminal 2 via the model generation server 1.

[0070] Next, user terminal 2 processes the output of the response text and class definition information provided by LLM server 3. During the output process, the response text and class definition information are displayed on the screen and presented to the user. At this point, the user determines whether the presented class definition information is at a satisfactory level.

[0071] The above procedure is repeated until class definition information that satisfies the user is obtained and the user instructs the user terminal 2 to complete the class definition process. When the user instructs the user to complete the class definition process, the class definition information is stored in the storage unit 12 of the model generation server 1.

[0072] Furthermore, the LLM server 3 is instructed in advance (prompted) to perform class definition processing in response to a class definition request from the model generation server 1. In other words, when training videos or instruction texts are sent to the LLM server 3, it is instructed in advance to perform class definition processing based on those training videos and instruction texts. The LLM server 3 is also instructed in advance to specify the output format of the class definition information. For example, it may be instructed to output in tabular format (see Figure 4).

[0073] In this embodiment, information necessary for class definition is collected through natural language interaction between the user and the LLM server 3, and appropriate class definition information is obtained. This appropriately supports the user's class definition work, and even novice users can obtain appropriate class definition information.

[0074] In this embodiment, class definition processing is performed on the LLM server 3 based on the interaction between the user and the LLM server 3, and on the video provided by the user. However, various processes can be performed by utilizing the functions of the LLM server 3. For example, the LLM server 3 may perform processing to extract problematic actions that become bottlenecks in the work, based on the interaction between the user and the LLM server 3, and on the video provided by the user.

[0075] Next, we will explain class definition information. Figure 4 is an explanatory diagram showing an example of class definition information. In Figure 4, possible combinations of a person's actions and the objects they interact with are represented by the symbol "○".

[0076] As shown in Figures 4(A) and 4(B), the class definition information presents the user with a table showing the correspondence between object classes and action classes, that is, valid combinations of actions and the objects they target. Note that the class definition information stored in the memory unit 12 may also represent the relationship between object classes and action classes using binary values, 0 and 1.

[0077] The example shown in Figure 4(A) is for work in the engine assembly process. In this case, the action classes for screw tightening and touching are set for the actions to be recognized. In addition, the object classes for person, screwdriver, engine, and tablet terminal are set for the objects to which the actions are performed. In the screw tightening action class, since a screwdriver is used to tighten screws on the engine, the corresponding object classes are screwdriver and engine. In the touching action class, since the worker performs a touch operation on the tablet terminal, the corresponding object class is tablet.

[0078] The example shown in Figure 4(B) is for work at a truck berth. In this case, the following action classes are set for the actions to be recognized: loading, unloading, moving, and talking. In addition, the following object classes are set for the objects that are the targets of the actions: person, truck, cart (cage cart), and cardboard box (luggage). In the loading action class, since loading is performed on a truck, the corresponding object class is truck. In the unloading action class, since loading is performed on a cart, the corresponding object class is cart. In the moving action class, since carts and cardboard boxes are moved, the corresponding object classes are cart and cardboard box. In the talking action class, since conversation is performed on a person, the corresponding object class is person.

[0079] Next, we will explain the object image extraction process and object feature extraction process performed on the model generation server 1. Figure 5 is an explanatory diagram showing an overview of the object image extraction process and object feature extraction process. Figure 5 is an example of work at the truck berth.

[0080] In the model generation server 1, the object candidate extraction process, that is, the process of extracting object images to be presented to the user as candidate object images (candidate images) to be used for training to generate an object recognition model, involves both an object image extraction process and an object feature extraction process.

[0081] In the object image extraction process, rectangular image regions containing objects are extracted from the frame image as object images based on the detection results of the object detection process. In the example shown in Figure 5, the image regions of the person, cart, truck, and cardboard box are extracted as object images. Note that the object image extraction process may also utilize the object detection results obtained by the LLM server 3 during the class definition process.

[0082] In object feature extraction processing, feature quantities are extracted from object images using a feature extraction model. Feature extraction models such as convolutional neural networks (CNNs) are used in this process. Furthermore, the features extracted during object feature extraction are referenced during the clustering process of object images by object class.

[0083] In object detection processing, objects are detected from frame images using an object detection model. Initially, a zero-shot learning object detection model is used, while subsequent attempts use an object detection model tuned with few-shot learning. A zero-shot learning object detection model can perform object detection even without explicitly training the target class beforehand. Note that any model that can perform object detection without pre-specifying the target object can be used instead of a zero-shot learning model. In few-shot learning tuning, the model is updated with a small amount of training data, based on either the zero-shot learning object detection model or the previously tuned model. For example, few-shot learning is performed using object recognition annotation information obtained in the object recognition annotation process, i.e., information about the object class attached to the object image. This gradually reduces false positives in the object detection process, improving the accuracy of the object detection process, and further improving the accuracy of the object image extraction process performed based on the detection results of the object detection process.

[0084] Next, we will explain the total combination extraction process, effective combination extraction process, behavioral image extraction process, and behavioral feature extraction process performed on the model generation server 1. Figure 6 is an explanatory diagram showing an overview of the total combination extraction process, effective combination extraction process, behavioral image extraction process, and behavioral feature extraction process. The example shown in Figure 6 is for the operation of a truck berth.

[0085] In the model generation server 1, the action candidate extraction process, that is, the process of extracting action images to be presented to the user as candidates (candidate images) for action images (images containing scenes of human action) used for training to generate an action recognition model, performs the following processes: total combination extraction process, valid combination extraction process, object image extraction process, and object feature extraction process.

[0086] In the total combination extraction process, all combinations of people (the subject of the action) and objects (the object of the action) are extracted based on the object recognition annotation information obtained in the object recognition annotation process (see Figure 16), that is, information on which object class labels (person, driver, etc.) are assigned to object images extracted from frame images.

[0087] In the valid combination extraction process, valid combinations of a person (subject of the action) and an object (object of the action) that match the class definition are extracted from all combinations of a person (subject of the action) and an object (object of the action), based on class definition information (see Figure 4), that is, information about the correspondence between action classes and the object classes that are the targets of the action.

[0088] Here, we will explain an example of a class definition for operations at a truck berth (see Figure 4(B)). For example, if the object is a truck, it corresponds to a loading action, but if the object is a cart, it does not correspond to a loading action. On the other hand, if the object is a truck, it does not correspond to an unloading action, but if the object is a cart, it corresponds to an unloading action.

[0089] In the action image extraction process, a rectangular image region containing both the person (the subject of the action) and the object (the object of the action) that can be combined in accordance with a given action is extracted from the frame image as an action image (an image containing a person's action scene). In the example shown in Figure 6, a person is moving a cart and cardboard boxes. In this case, a rectangular image region containing the person and the cardboard boxes that can be combined in accordance with the action of "movement" is extracted as an action image. Also, a rectangular image region containing the person and the cart that can be combined in accordance with the action of "movement" is extracted as an action image. Furthermore, in the action image extraction process, an action class is set for the extracted action images.

[0090] In the behavioral feature extraction process, feature quantities are extracted from behavioral images using a feature extraction model. For example, a convolutional neural network (CNN) is used in this process. The features extracted in the behavioral feature extraction process are then referenced during the clustering process of behavioral images by behavioral class.

[0091] Next, we will explain the clustering process and representative image extraction process performed on the model generation server 1. Figure 7 is an explanatory diagram showing an overview of the clustering process and representative image extraction process. In the example shown in Figure 7, the feature space is represented in two dimensions. Also, the feature quantities of the images extracted from the frame images (object images, behavior images) are represented by rectangles.

[0092] Model generation server 1 performs clustering processes for object classes and clustering processes for behavior classes, respectively. In the object class clustering process, each object image is classified into a cluster based on the features obtained for each object image in the object feature extraction process, and this process is performed for each object class. In the behavior class clustering process, each behavior image is classified into a cluster based on the features obtained for each behavior image in the behavior feature extraction process, and this process is performed for each behavior class.

[0093] Furthermore, clustering is performed separately for each class (object class, action class). Specifically, in clustering for object classes, a cluster (group of object images) is created for each object class (e.g., driver, engine, etc.). In clustering for action classes, a cluster is created for each action class (e.g., screw tightening, touch operation, etc.).

[0094] Furthermore, the clustering process generates a number of clusters corresponding to the number of candidate images specified by the user. In other words, object images and behavioral images are classified into a number of clusters corresponding to the number of candidate images specified by the user.

[0095] Next, the model generation server 1 performs representative image extraction processing. In this representative image extraction processing, a representative image for each cluster is extracted from the images (object images, behavior images) contained in the cluster based on predetermined extraction conditions. In addition, one representative image is extracted from each cluster during the representative image extraction processing. The example shown in Figure 7 is when the number of candidate images (number of clusters) is set to 6, and 6 clusters are created, and 6 representative images for each cluster are presented to the user as candidate images. The number of candidate images (number of clusters) can be specified by the user. Furthermore, the number of candidate images (number of clusters) can be set individually for each class.

[0096] In this embodiment, images (object images, behavior images) included in a cluster are assigned a rank based on their position (distance) from the cluster's centroid, and the user can specify this rank as an extraction criterion when extracting a representative image. Specifically, the images are assigned a rank (centroid distance rank) in order of increasing distance from the cluster's centroid. For example, if the user specifies a rank of 1, the image closest to the centroid is extracted as the representative image. If the user specifies a rank of 2, the image second closest to the centroid is extracted as the representative image. The cluster's centroid is the average of the feature quantities of each image included in the cluster, and represents the image with the most typical features within the cluster.

[0097] When extracting representative images, the extraction conditions can be set individually for each class. Specifically, in clustering processes related to object classes, extraction conditions are set for each object class (e.g., driver, engine, etc.). Similarly, in clustering processes related to behavior classes, extraction conditions are set for each behavior class (e.g., screw tightening, touch operation, etc.).

[0098] In this embodiment, representative images are extracted from the images (object images, behavior images) included in each cluster based on the centroid distance ranking. However, the extraction criteria for extracting representative images are not limited to the centroid distance ranking. For example, representative images may be randomly selected from each cluster.

[0099] In this embodiment, one representative image is extracted from each cluster, but multiple representative images (for example, two) may be extracted from a single cluster.

[0100] Next, we will describe the class definition mode screen 101 displayed on user terminal 2. Figure 8 is an explanatory diagram showing the class definition mode screen 101.

[0101] The Class Definition mode screen 101 has tabs 102 (mode selection section) for "Class Definition," "Object Recognition," "Action Recognition," and "Inference." The Class Definition mode screen 101 is displayed first, and also appears when the user operates the "Class Definition" tab 102 on other screens 201, 301, and 401 (see Figures 10, 11, and 12). When the user operates the "Object Recognition" tab 102, the screen transitions to the Object Recognition mode screen 201 (see Figure 10). When the user operates the "Action Recognition" tab 102, the screen transitions to the Action Recognition mode screen 301 (see Figure 11). When the user operates the "Inference" tab 102, the screen transitions to the Inference mode screen 401 (see Figure 12).

[0102] Furthermore, the class definition mode screen 101 is provided with a video input unit 103. The video input unit 103 allows the user to input learning videos of an operator's work. Specifically, learning videos can be input by, for example, moving the icon of a video file (a file containing the learning video) onto the video input unit 103 using a mouse operation (drag and drop operation), or by selecting a video file on a file selection screen (not shown) that appears when the mouse is right-clicked on the video input unit 103 using a mouse operation (click operation).

[0103] The input training videos are uploaded from user terminal 2 to model generation server 1, and then transferred to LLM server 3 for use in class definition processing. Furthermore, the training videos are used in machine learning for generating recognition models (object recognition models, action recognition models) performed on model generation server 1.

[0104] Furthermore, in this embodiment, a video file (a file containing learning videos) is input, but files containing various types of data other than videos may also be input. For example, files containing still images, audio, text, etc., may be input. These types of data other than videos can also be used in the class definition processing on the LLM server 3.

[0105] Furthermore, the class definition mode screen 101 is equipped with an interaction unit 104. In the interaction unit 104, interaction takes place between the user and the LLM server 3, and through this interaction, information necessary for class definition is collected from the user. In the example shown in Figure 8, the action of tightening screws on an engine using a screwdriver in a factory becomes the subject of class definition.

[0106] The dialogue unit 104 displays message boxes 111 and 112, an input box 113, and a "Send" button 114. The user can enter instruction text (prompt) in the input box 113. When the user enters instruction text and operates the "Send" button 114, the entered instruction text is sent from the user terminal 2 to the LLM server 3 via the model generation server 1.

[0107] Message box 111 displays the instruction text previously entered by the user. Message box 112 displays the response text generated by the LLM server 3 in response to the instruction text. Message boxes 111 and 112 scroll as the dialogue progresses, with instruction text and response text being exchanged. This dialogue can be repeated until the user determines that a suitable class definition has been obtained.

[0108] Furthermore, the class definition mode screen 101 is provided with a definition result output unit 105. The definition result output unit 105 displays the class definition information generated by the LLM server 3. In the example shown in Figure 8, the correspondence between action classes and object classes is displayed as class definition information in a table format, with action classes in the rows and object classes in the columns. In addition, prompts regarding the output conditions are input to the LLM server 3 when the definition result output unit 105 outputs.

[0109] Furthermore, the class definition mode screen 101 is provided with a "Class Definition Complete" button 106. The user confirms that the class definition information displayed on the definition result output unit 105 is correct, and then operates the "Class Definition Complete" button 106. As a result, the model generation server 1 confirms the class definition information and stores it in the storage unit 12.

[0110] Next, we will explain the screen 101 of the class definition mode displayed on the user terminal 2 when selecting a model. Figure 9 is an explanatory diagram showing the screen 101 of the class definition mode when selecting a model.

[0111] The screen 101 in class definition mode is provided with a model selection unit 107. The model selection unit 107 is also provided on the screen 201 in object recognition mode (see Figure 10), the screen 301 in action recognition mode (see Figure 11), and the screen 401 in inference mode (see Figure 12).

[0112] When the user operates the model selection unit 107, a dropdown list 121 is displayed. In the dropdown list 121, the user can select a recognition model (object recognition model, action recognition model) that has been created in the past. In the example shown in Figure 9, the serial number and creation date and time are displayed for each recognition model as identification information. The dropdown list 121 also has a "Create New" item, and the user can create a new recognition model by selecting the "Create New" item.

[0113] In the dropdown list 121 of the model selection unit 107, when the user selects a recognition model (object recognition model, action recognition model), the selected recognition model, the training videos used to generate the selected recognition model, and the class definition information created when generating the selected recognition model are read out.

[0114] This enables the additional training and retraining of previously created recognition models (object recognition models, action recognition models). Furthermore, previously created recognition models can be used in class definitions. Inference processing (object recognition inference processing, action recognition inference processing) can also be performed using previously created recognition models. Therefore, the accuracy of previously created recognition models can be improved when targeting the same work site as the current one. Additionally, when reusing a previously created recognition model for a different work site (work process) to the current work site (work process), the accuracy of the recognition model can be improved, making the repurposing of recognition models easier.

[0115] Next, we will explain screen 201 (annotation screen) of the object recognition mode displayed on user terminal 2. Figure 10 is an explanatory diagram showing screen 201 of the object recognition mode. On screen 201 of the object recognition mode, the user performs operations related to annotation and training for generating the object recognition model.

[0116] The object recognition mode screen 201 is provided with a learning count display unit 202. The learning count display unit 202 displays the number of learning sessions, that is, the number of times learning (including additional learning and retraining) has been performed in the past for the creation and updating of the object recognition model.

[0117] Furthermore, the object recognition mode screen 201 is equipped with an "Object Candidate Extraction" button 203. When the user operates the "Object Candidate Extraction" button 203, a process (object candidate extraction process) is executed to extract object images as candidate images for all object classes (people, drivers, etc.) that are the target of the user's labeling operation. In the case of additional training and retraining, when the user operates the "Object Candidate Extraction" button 203, the object candidate extraction process is executed based on the updated class definition information and training videos.

[0118] Furthermore, the object recognition mode screen 201 is provided with a class designation section 204. When the user operates the class designation section 204, a dropdown list (not shown) is displayed. The user can select an object class (person, driver, etc.) from the dropdown list.

[0119] Furthermore, the object recognition mode screen 201 is provided with an image number specification section 205. The image number specification section 205 consists of a slider bar. By moving the slider, the user can increase or decrease the number of candidate images displayed in the representative image display section. In the example shown in Figure 10, there are 10 candidate images, 10 clusters are generated, and 10 representative images for each cluster are displayed as candidate images. Note that if one representative image is selected as a candidate image for each cluster, the number of candidate images will match the number of clusters generated in the clustering process. Also, if multiple (e.g., two) representative images are extracted from one cluster, the number of clusters generated will be the number of candidate images divided by the number of representative images extracted from one cluster. In this case, if the division does not result in an equal distribution of clusters, a larger number of candidate images may be extracted from a particular cluster.

[0120] In the clustering process, the object images may be sorted into optimal clusters based on the distribution of features in each object image, and the optimal number of candidate images corresponding to the number of clusters may be presented to the user. Alternatively, the optimal number of candidate images (number of clusters) may be presented to the user as an initial value in the image number specification unit 205.

[0121] Furthermore, the object recognition mode screen 201 is provided with an extraction condition specification unit 206. When the user operates the extraction condition specification unit 206, a dropdown list (not shown) is displayed. In the dropdown list, the user can specify the extraction conditions for extracting a representative image from the cluster. Specifically, the user can specify the centroid distance rank (a rank assigned in order of proximity to the centroid) as the extraction condition. In the example shown in Figure 10, since the centroid distance rank specified by the user is 1, the object image closest to the centroid is extracted as the representative image (candidate image).

[0122] Furthermore, the object recognition mode screen 201 is provided with a candidate image display unit 207. In the candidate image display unit 207, object images (thumbnails) that are assumed to belong to the object class specified in the class designation unit 204 are displayed in a row as candidate images. The candidate images are representative images for each cluster classified based on the feature quantities of the object images. In addition, the candidate images are extracted based on the object class provisionally set for the object images.

[0123] The candidate image display unit 207 is provided with a checkbox 208 (correct / incorrect input unit) for each candidate image. The user visually inspects each candidate image and determines whether each candidate image is appropriate as an object image corresponding to the object class specified in the class specification unit 204. The user then performs an operation (labeling operation) to check the checkbox 208 corresponding to the candidate image that is determined to be appropriate as an object image corresponding to the specified object class. In the example shown in Figure 10, the object class of "person" is specified in the class specification unit 204, and the user performs the operation of checking the checkbox 208 corresponding to the candidate image containing a person in the candidate image display unit 207.

[0124] At this time, the model generation server 1 targets the cluster to which the candidate images that the user has checked in checkbox 208 (in this case, three checkboxes are checked), that is, the candidate images (representative images) that belong to the specified object class, and performs a process (labeling process) to assign the label of the specified object class to all object images included in the cluster.

[0125] In the candidate image display unit 207, object images that are assumed to belong to the object class specified in the class designation unit 204 are displayed as candidate images based on the provisional object class set for each object image. However, since there may be errors in the provisional object class set for each object image, the candidate image display unit 207 also displays inappropriate candidate images that do not belong to the object class specified in the class designation unit 204. Therefore, by visually inspecting each candidate image and checking only the checkbox 208 corresponding to the candidate image that is judged to be appropriate as an object image belonging to the object class specified in the class designation unit 204, inappropriate object images that do not belong to the specified object class can be excluded from the object images for training.

[0126] Furthermore, the operation of checking the checkbox 208 corresponding to the candidate image that corresponds to the object class specified in the class specification unit 204 (labeling operation) is performed for each object class. In other words, the user can switch the object class of the candidate image displayed in the representative image display unit by selecting a different object class from the drop-down list in the class specification unit 204, and by repeating the labeling operation for each object class, the labeling operation for all necessary object classes is completed.

[0127] Furthermore, the object recognition mode screen 201 is equipped with an "Object Recognition Learning" button 209. When the user operates the "Object Recognition Learning" button 209, the model generation server 1 executes a process (object recognition model learning process) to generate an object recognition model using training object images.

[0128] Next, we will explain the behavior recognition mode screen 301 (annotation screen) displayed on user terminal 2. Figure 11 is an explanatory diagram showing the behavior recognition mode screen 301. On the behavior recognition mode screen 301, the user performs operations related to annotation and training for generating the behavior recognition model.

[0129] The screen 301 in the behavior recognition mode is provided with a learning count display unit 302. The learning count display unit 302 displays the number of learning sessions, that is, the number of times learning (including additional learning and retraining) has been performed in the past for the creation and updating of the behavior recognition model.

[0130] Furthermore, the screen 301 in action recognition mode is equipped with an "Action Candidate Extraction" button 303. When the user operates the "Action Candidate Extraction" button 303, a process (action candidate extraction process) is executed to extract action images as candidate images for all action classes (screw tightening, touch, etc.) that are the target of the user's labeling operation. In the case of additional training and retraining, when the user operates the "Action Candidate Extraction" button 303, the action candidate extraction process is executed based on the updated class definition information and training videos.

[0131] Furthermore, the screen 301 in the action recognition mode is provided with a class selection unit 304. When the user operates the class selection unit 304, a drop-down list (not shown) is displayed. The user can select an action class (such as screw tightening or touching) from the drop-down list.

[0132] Furthermore, the screen 301 in the action recognition mode is provided with an image number specification unit 305. The image number specification unit 305 consists of a slider bar. By moving the slider, the user can increase or decrease the number of candidate images displayed in the representative image display unit. In the example shown in Figure 10, there are 10 candidate images, 10 clusters are generated, and 10 representative images for each cluster are displayed as candidate images. Note that if one representative image is selected as a candidate image for each cluster, the number of candidate images will match the number of clusters generated in the clustering process. Also, if multiple (e.g., two) representative images are extracted from one cluster, the number of clusters generated will be the number of candidate images divided by the number of representative images extracted from one cluster. In this case, if the division does not result in an equal distribution of clusters, a larger number of candidate images may be extracted from a particular cluster.

[0133] In the clustering process, the behavioral images may be sorted into optimal clusters based on the distribution of features in each behavioral image, and the optimal number of candidate images corresponding to the number of clusters may be presented to the user. Alternatively, the optimal number of candidate images (number of clusters) may be presented to the user as an initial value in the image number specification unit 305.

[0134] Furthermore, the screen 301 in the behavior recognition mode is provided with an extraction condition specification unit 306. When the user operates the extraction condition specification unit 206, a drop-down list (not shown) is displayed. In the drop-down list, the user can specify the extraction conditions for extracting representative images from the cluster. Specifically, the user can specify the centroid distance rank (a rank assigned in order of proximity to the centroid) as the extraction condition. In the example shown in Figure 11, since the centroid distance rank specified by the user is 1, the behavior image closest to the centroid is extracted as the representative image (candidate image).

[0135] Furthermore, the screen 301 in the behavior recognition mode is provided with a candidate image display unit 307. In the candidate image display unit 307, behavior images (thumbnails) that are assumed to belong to the behavior class specified in the class designation unit 304 are displayed in a row as candidate images. The candidate images are representative images for each cluster classified based on the features of the behavior images. In addition, the candidate images are extracted based on the object class provisionally set for the behavior images.

[0136] The candidate image display unit 307 is provided with a checkbox 308 (correct / incorrect input unit) for each candidate image. The user visually inspects each candidate image and determines whether each candidate image is appropriate as an action image corresponding to the action class specified in the class specification unit 304. The user then performs an operation (labeling operation) to check the checkbox 308 corresponding to the candidate image that is determined to be appropriate as an action image corresponding to the specified action class. In the example shown in Figure 11, the action class "touch" is specified in the class specification unit 304, and the user performs an operation to check the checkbox 308 corresponding to the candidate image that includes a touch action scene in the candidate image display unit 307.

[0137] At this time, the model generation server 1 targets the cluster to which the candidate images that the user has checked in checkbox 308 (in this case, three checkboxes are checked), that is, the candidate images (representative images) that correspond to the specified behavior class belong, and performs a process (labeling process) to assign the label of the specified behavior class to all behavior images included in the cluster.

[0138] In the candidate image display unit 307, based on the provisional action class set for each action image, action images that are assumed to belong to the action class specified in the class designation unit 304 are displayed as candidate images. However, since there may be errors in the provisional action class set for each action image, the candidate image display unit 307 also displays inappropriate candidate images that do not belong to the action class specified in the class designation unit 304. Therefore, by visually inspecting each candidate image and checking only the checkbox 308 corresponding to the candidate image that is judged to be appropriate as an action image belonging to the action class specified in the class designation unit 304, inappropriate action images that do not belong to the specified action class can be excluded from the action images for training.

[0139] Furthermore, the operation of checking the checkbox 308 corresponding to the candidate image that corresponds to the behavior class specified in the class specification unit 304 (labeling operation) is performed for each behavior class. In other words, the user can switch the behavior class of the candidate image displayed in the representative image display unit by selecting a different behavior class from the drop-down list in the class specification unit 304, and by repeating the labeling operation for each behavior class, the labeling operation for all necessary behavior classes is completed.

[0140] Furthermore, the screen 301 in the behavior recognition mode is equipped with a button 309 labeled "Behavior Recognition Learning". When the user operates the "Behavior Recognition Learning" button 309, the model generation server 1 executes a process (behavior recognition model learning process) to generate a behavior recognition model using training behavior images.

[0141] Next, we will explain the inference mode screen 401 displayed on user terminal 2. Figure 12 is an explanatory diagram showing the inference mode screen 401.

[0142] The inference mode screen 401 is equipped with a threshold setting unit 402 for object recognition and a threshold setting unit 403 for action recognition. The threshold setting units 402 and 403 consist of sliders. The thresholds can be adjusted by the user moving the sliders in the threshold setting units 402 and 403. The thresholds are compared with scores generated by the recognition models (object recognition model, action recognition model) to determine whether or not a predetermined event (object, action) can be recognized.

[0143] Furthermore, the inference mode screen 401 is equipped with a video input unit 404. The video input unit 404 allows the user to input verification videos of the worker's work. Specifically, verification videos can be input, for example, by moving the icon of a video file (a file containing the verification video) onto the video input unit 404 using a mouse operation (drag and drop operation), or by selecting a video file on a file selection screen (not shown) that appears when the mouse is right-clicked on the video input unit 404 using a mouse operation (click operation).

[0144] Here, it is preferable that a different video from the training video input to the video input unit 103 in the class definition mode screen 101 (see Figure 8) is input as the verification video. This allows the user to appropriately verify the recognition accuracy of the object recognition model and the action recognition model.

[0145] Furthermore, the inference mode screen 401 is equipped with a "Object Recognition Inference" button 405 and a "Behavior Recognition Inference" button 406. When the user operates the "Object Recognition Inference" button 405, inference is performed by the object recognition model on the verification video input to the video input unit 404. When the user operates the "Behavior Recognition Inference" button 406, inference is performed by the behavior recognition model on the verification video input to the video input unit 404.

[0146] Furthermore, the inference mode screen 401 is provided with a recognition result output unit 407. The recognition result output unit 407 displays a recognition result video. The recognition result video is a verification video on which images representing the recognition results are superimposed. Specifically, rectangular frames surrounding objects detected from each frame image of the verification video are superimposed on the verification video. In addition, characters representing the recognized objects and characters representing the recognized actions are superimposed on the verification video. Information regarding the processing status may also be superimposed on the verification video.

[0147] Here, the user can view the recognition result video displayed in the recognition result output unit 407 to check the accuracy of the recognition model (object recognition model, action recognition model), and if the accuracy of the recognition model is insufficient, they can perform additional training or retraining of the recognition model.

[0148] In other words, if the accuracy of the object recognition model is insufficient, the user can return to the object recognition mode screen 201 (see Figure 10) by operating the "Object Recognition" tab 102, restart the object candidate extraction process by operating the "Extract Object Candidates" button 203, restart the object recognition annotation process by performing a labeling operation on the presented candidate images, and restart the object recognition model training process by operating the "Object Recognition Training" button 209.

[0149] Furthermore, if the accuracy of the behavior recognition model is insufficient, the user can return to the behavior recognition mode screen 301 (see Figure 11) by operating the "Behavior Recognition" tab 102, restart the behavior candidate extraction process by operating the "Extract Behavior Candidates" button 303, restart the behavior recognition annotation process by performing a labeling operation on the presented candidate images, and restart the behavior recognition model training process by operating the "Behavior Recognition Training" button 309.

[0150] In this embodiment, the user can improve the accuracy of the object recognition model by redoing the object candidate extraction process, object recognition annotation process, and object recognition model training process. Furthermore, the user can improve the accuracy of the action recognition model by redoing the action candidate extraction process, action recognition annotation process, and action recognition model training process. In addition, since the user can specify various processing conditions on screen 201 in object recognition mode (see Figure 10) or screen 301 in action recognition mode (see Figure 11), the recognition model (object recognition model, action recognition model) can be customized to enable appropriate recognition processing according to the user's requests and the situation on site.

[0151] Next, we will explain the procedure for processing performed on the model generation server 1. Figure 13 is a flowchart showing the procedure for processing performed on the model generation server 1.

[0152] In the model generation server 1, the processor 13 first acquires the training video that the user inputs on the user terminal 2 and obtains the frame images that make up the training video (input video processing) (ST101). At this time, the user inputs a video file (a file containing the training video) in the video input unit 103 on the class definition mode screen 101 (see Figure 8) displayed on the user terminal 2.

[0153] Next, the processor 13 works in cooperation with the LLM server 3 to obtain class definition information (class definition processing) (ST102). The class definition information is information regarding the correspondence between object classes and action classes, that is, valid combinations of action classes and the object classes they target.

[0154] Next, processor 13 detects objects from the frame images of the training video (object detection process) (ST103). In this process, the first time, an object detection model tuned using zero-shot learning is used, and from the second time onward, an object detection model tuned using few-shot learning is used.

[0155] Next, when the user operates the "Extract Object Candidates" button 203 on the object recognition mode screen 201 (see Figure 10), the processor 13 extracts object images from the frame image based on the detection results of the object detection process, extracts feature quantities for each object image, then classifies the object images into multiple clusters based on the feature quantities for each object image, and extracts representative images for each cluster as candidate images (object candidate extraction process) (ST104).

[0156] Next, the processor 13 presents representative images for each cluster extracted in the object candidate extraction process to the user as candidate images, and in response to the user's operation to select a candidate image corresponding to a specified object class, it assigns the label of the specified object class to all object images included in the cluster to which the selected candidate image (representative image) belongs (object recognition annotation process) (ST105).

[0157] Next, when the user operates the "Object Recognition Learning" button 209 on the object recognition mode screen 201 (see Figure 10), the processor 13 performs machine learning using the labeled object images obtained in the object recognition annotation process to generate an object recognition model (object recognition model learning process) (ST106). The generated object recognition model is stored in the memory unit 12.

[0158] Next, the processor 13 acquires a verification video created by the user's input operations on the user terminal 2 and obtains frame images that make up the verification video (input video processing) (ST107). At this time, the user performs an operation to input a video file (a file containing the verification video) at the video input unit 404 on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2.

[0159] Next, on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2, when the user operates the "Object Recognition Inference" button 405, the processor 13 performs inference to derive object recognition results from the verification video using the object recognition model (object recognition inference process) (ST108).

[0160] Next, the processor 13 determines whether or not to restart the object recognition model based on the user's operation on the user terminal 2 (ST109). At this time, the user visually checks the recognition result video (inference result) on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2 and determines whether or not the recognition accuracy of the object recognition model is at a satisfactory level. If the recognition accuracy of the object recognition model is not at a satisfactory level, the user operates the "Object Recognition" tab 102, and it is determined that the object recognition model should be restarted.

[0161] If the object recognition model needs to be redone (Yes in ST109), the process returns to ST103, and the object detection process (ST103), object candidate extraction process (ST104), object recognition annotation process (ST105), and object recognition model training process (ST106) are performed again. At this time, the object candidate extraction process (ST110) is performed using the object recognition model generated in the previous object recognition model training process, so the object images, clusters, and representative images (candidate images) will be different from those generated in the previous object candidate extraction process.

[0162] On the other hand, if the object recognition model is not being redone, that is, if the user operates the "Action Recognition" tab 102 on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2 (No in ST109), the process proceeds to ST110.

[0163] Next, when the user operates the "Action Candidate Extraction" button 303 on the action recognition mode screen 301 (see Figure 11), the processor 13 extracts valid combinations of a person (subject of the action) and an object (object of the action) corresponding to a given action based on the class definition information acquired in the class definition relay processing (ST102). Next, it extracts action images that include the image regions of the person and object that make up the valid combination, extracts feature quantities for each action image, and then classifies the action images into multiple clusters based on the feature quantities for each object image, and extracts representative images for each cluster as candidate images (action candidate extraction processing) (ST110).

[0164] Next, the processor 13 presents representative images for each cluster extracted in the action candidate extraction process to the user as candidate images, and in response to the user's operation to select a candidate image corresponding to a specified action class, it assigns the label of the specified action class to all action images included in the cluster to which the selected candidate image (representative image) belongs (action recognition annotation process) (ST111).

[0165] Next, when the user operates the "Action Recognition Learning" button 309 on the action recognition mode screen 301 (see Figure 11), the processor 13 performs machine learning using the labeled action images obtained in the action recognition annotation process to generate an action recognition model (action recognition model learning process) (ST112). The generated action recognition model is stored in the memory unit 12.

[0166] Next, on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2, when the user operates the "Action Recognition Inference" button 406, the processor 13 performs inference to derive action recognition results from the verification video using the action recognition model (Action Recognition Inference Processing) (ST113).

[0167] Next, the processor 13 determines whether to restart the action recognition model or the object recognition model based on the user's operation on the user terminal 2 (ST114). At this time, the user visually inspects the recognition result video (inference result) on the inference mode screen 401 (see Figure 12) displayed on the user terminal 2 and determines whether the recognition accuracy of the action recognition model and the recognition accuracy of the object recognition model are at a satisfactory level. If the recognition accuracy of the action recognition model is not at a satisfactory level, the user can operate the "Action Recognition" tab 102 to determine whether to restart the action recognition model. Similarly, if the recognition accuracy of the object recognition model is not at a satisfactory level, the user can operate the "Object Recognition" tab 102 to determine whether to restart the object recognition model.

[0168] If the behavior recognition model needs to be redone, the process returns to ST110, and the behavior candidate extraction process (ST110), behavior recognition annotation process (ST111), and behavior recognition model training process (ST112) are performed again.

[0169] Furthermore, if the object recognition model needs to be redone, the process returns to ST103, and the object detection process (ST103), object candidate extraction process (ST104), object recognition annotation process (ST105), and object recognition model training process (ST106) are performed again.

[0170] On the other hand, if the action recognition model and object recognition model are not being redone, the process is terminated.

[0171] Thus, in this embodiment, if the accuracy of the recognition model (object recognition model, action recognition model) is insufficient, the recognition model can be repeatedly trained and retrained, making it easy to improve the accuracy of the recognition model.

[0172] Next, we will explain the input video processing performed on the model generation server 1 (ST101, ST107 in Figure 13). Figure 14 is a flowchart showing the procedure for input video processing.

[0173] In the model generation server 1, the communication unit 11 first receives the video file transmitted (uploaded) from the user terminal 2 (ST201). At this time, the user terminal 2 inputs the video file according to the user's operation, and that video file is transmitted from the user terminal 2 to the model generation server 1.

[0174] Next, processor 13 acquires the video (image) stored in the received video file (ST202). At this time, processor 13 acquires a training video in the case of the first input video processing (ST101 in Figure 13), and acquires a verification video in the case of the next input video processing (ST107 in Figure 13).

[0175] Next, processor 13 extracts frame images from the video (training video, verification video) (ST203). Then, processor 13 stores the frame images in storage unit 12 (ST204).

[0176] Next, we will explain the procedure for the class definition relay processing (ST102 in Figure 13) performed on the model generation server 1. Figure 15 is a flowchart showing the procedure for the class definition relay processing.

[0177] In the model generation server 1, the processor 13 first sends training videos (frame images) from the communication unit 11 to the LLM server 3 (ST301). The training videos are acquired from the user terminal 2 by input video processing (Figure 14) and stored in the storage unit 12.

[0178] Furthermore, when the communication unit 11 receives instruction text (prompt) entered by the user at the user terminal 2, the processor 13 transmits the received instruction text from the communication unit 11 to the LLM server 3 (ST302).

[0179] At this time, on the user terminal 2, on the class definition mode screen 101 (see Figure 8), when the user inputs instruction text containing recognition target information that describes the content of the action to be recognized, that instruction text is sent to the model generation server 1. Also, when the LLM server 3 receives the training video and instruction text from the model generation server 1, it performs a dialogue process that generates a response text in response to the instruction text, and a class definition process that generates class definition information based on the training video and instruction text.

[0180] Next, in the model generation server 1, when the communication unit 11 receives the response text and class definition information from the LLM server 3, the processor 13 transmits the received response text and class definition information from the communication unit 11 to the user terminal 2 (ST303).

[0181] At this time, when user terminal 2 receives the response text and class definition information from model generation server 1, the received response text and class definition information are displayed on screen 101 (see Figure 8) in class definition mode. The user checks the displayed class definition information and determines whether satisfactory class definition information has been obtained. The user also checks the displayed response text, and if the response text indicates that information necessary for the class definition is missing, the user enters instruction text to supplement the necessary information.

[0182] Next, in the model generation server 1, the processor 13 determines whether the user has instructed the completion of the class definition process based on the user's operation on the user terminal 2 (ST304). At this time, if the user is satisfied with the contents of the class definition information, they operate the "Complete Class Definition" button 106 on the class definition mode screen 101 (see Figure 8). On the other hand, if the user is not satisfied with the contents of the class definition information, they input new instruction text (prompt) in the dialogue section 104 of the class definition mode screen 101 (see Figure 8).

[0183] If the user then instructs the completion of the class definition process (Yes in ST304), the class definition information is stored in the storage unit 12.

[0184] On the other hand, if the user does not instruct the completion of the class definition process (No in ST304), the process returns to ST302. At this time, on the user terminal 2, the instruction text (prompt) newly entered by the user on the class definition mode screen 101 (see Figure 8) is sent to the model generation server 1.

[0185] Next, we will explain the object candidate extraction process (ST104 in Figure 13) and the object recognition annotation process (ST105 in Figure 13) performed on the model generation server 1. Figure 16(A) is a flowchart showing the procedure for the object candidate extraction process. Figure 16(B) is a flowchart showing the procedure for the object recognition annotation process.

[0186] As shown in Figure 16(A), in the object candidate extraction process, first, the processor 13 extracts an object image (a rectangular image region surrounding an object) from the frame image based on the detection results of the object detection process (ST103 in Figure 13) (object image extraction process) (ST401).

[0187] Next, processor 13 extracts features for each object image using a feature extraction model (object feature extraction process) (ST402). The features extracted in this object feature extraction process are referenced during the clustering process for each object class of the object image.

[0188] Next, processor 13 classifies each object image into a cluster (group of object images) based on the features of each object image (clustering process) (ST403). The clustering process is performed for each provisional object class.

[0189] Next, the processor 13 extracts a representative image for each cluster from the object images contained in each cluster (representative image extraction process) (ST404). At this time, for example, if the centroid distance rank specified by the user is 1, the object image closest to the centroid of the cluster is extracted as the representative image.

[0190] As shown in Figure 16(B), in the object recognition annotation process, the processor 13 first presents the representative images for each cluster extracted in the representative image extraction process (ST404 in Figure 16(A)) to the user as candidate images (ST501). Specifically, the representative images for each extracted cluster are displayed as a list of candidate images in the candidate image display unit 207 of the object recognition mode screen 201 (see Figure 10).

[0191] At this time, on the object recognition mode screen 201 (see Figure 10), the user specifies an object class, and a list of candidate images that are expected to belong to the specified object class is displayed. The user visually inspects the displayed candidate images, determines whether each candidate image belongs to the specified object class, and checks the checkbox 208 for the candidate images that belong to the specified object class.

[0192] Next, the processor 13, in response to user operations on the user terminal 2, obtains information on candidate images that the user has checked in checkbox 208, i.e., candidate images that belong to the specified object class, and identifies the cluster to which the candidate image belongs (ST502).

[0193] Next, the processor 13 targets the cluster to which the candidate images corresponding to the specified object class belong, and assigns the label of the specified object class to all object images included in the cluster (labeling process) (ST503). Note that in the labeling process, binary labels, that is, labels indicating whether or not the specified object is included, may be assigned.

[0194] This object recognition annotation process provides object recognition annotation information, where object class labels are assigned to object images extracted from frame images. Furthermore, object images included in the cluster to which candidate images (representative images) deemed appropriate by the user for a specified object class belong are labeled collectively. This allows users to efficiently label a large number of object images. Additionally, because appropriate training object images are obtained, the number of object images required to generate an object recognition model with sufficient accuracy can be significantly reduced.

[0195] Next, we will explain the action candidate extraction process (ST110 in Figure 13) and the action recognition annotation process (ST111 in Figure 13) performed on the model generation server 1. Figure 17(A) is a flowchart showing the procedure for the action candidate extraction process. Figure 17(B) is a flowchart showing the procedure for the action recognition annotation process.

[0196] As shown in Figure 17(A), in the action candidate extraction process, the processor 13 first extracts all combinations of people (subjects of the action) and objects (objects of the action) based on the object recognition annotation information obtained in the object recognition annotation process (see Figure 16(B)), that is, information on which object class labels (person, driver, etc.) are assigned to object images extracted from frame images (total combination extraction process) (ST601).

[0197] Next, the processor 13 extracts combinations that match the class definition from all combinations of person (subject of the action) and object (object of the action), based on the class definition information obtained in the class definition relay processing (see Figure 15), that is, information on valid combinations of action classes and the object classes that are the targets of the action (valid combination extraction processing) (ST602). As a result, impossible combinations of action and object (combinations without the "○" symbol in the example in Figure 4) are excluded in subsequent processing.

[0198] Next, the processor 13 extracts from the frame image an action image that includes a person (subject of the action) and an object (object of the action) whose combination is valid in response to a predetermined action, that is, a rectangular image region surrounding each image region of the person and the object (action image extraction process) (ST603).

[0199] Next, processor 13 extracts features from each behavior image using a feature extraction model (behavioral feature extraction process) (ST604). The features extracted in this behavioral feature extraction process are referenced during the clustering process of behavior images by behavior class.

[0200] Next, processor 13 classifies each action image into a cluster (group of action images) based on the features of each action image (clustering process) (ST605). Clustering is performed for each provisional object class.

[0201] Next, the processor 13 extracts a representative image for each cluster from the action images contained in each cluster (representative image extraction process) (ST606). At this time, for example, if the centroid distance rank specified by the user is 1, the action image closest to the centroid of the cluster is extracted as the representative image.

[0202] As shown in Figure 17(B), in the behavior recognition annotation process, the processor 13 first presents the representative images for each cluster extracted in the representative image extraction process (ST606 in Figure 17(A)) to the user as candidate images (ST701). Specifically, the extracted representative images for each cluster are displayed as candidate images on the candidate image display unit 307 of the behavior recognition mode screen 301 (see Figure 11).

[0203] At this time, on the action recognition mode screen 301 (see Figure 11), the user specifies an object class, and a list of candidate images that are expected to belong to the specified object class is displayed. The user visually inspects the displayed candidate images, determines whether each candidate image belongs to the specified action class, and checks the checkbox 308 for the candidate images that belong to the specified action class.

[0204] Next, the processor 13, in response to user operations on the user terminal 2, obtains information on candidate images that the user has checked in checkbox 308, i.e., candidate images that correspond to the specified behavior class, and identifies the cluster to which the candidate image belongs (ST702).

[0205] Next, the processor 13 targets the cluster to which the candidate images corresponding to the specified behavior class belong, and assigns the label of the specified behavior class to all behavior images included in the cluster (labeling process) (ST703). Note that in the labeling process, binary labels, that is, labels indicating whether or not the specified behavior is included, may be assigned.

[0206] This behavior recognition annotation process yields behavior recognition annotation information, in which behavior images extracted from frame images are labeled with behavior classes. Furthermore, behavior images included in the cluster to which a candidate image (representative image) deemed appropriate by the user for a specified behavior class belongs are labeled collectively. This allows users to efficiently label a large number of behavior images. Additionally, because appropriate training behavior images are obtained, the number of behavior images required to generate a sufficiently accurate behavior recognition model can be significantly reduced.

[0207] As described above, embodiments have been explained as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these embodiments and can be applied to embodiments that have been modified, replaced, added, or omitted. Furthermore, it is possible to create new embodiments by combining the components described in the above embodiments.

[0208] For example, in this embodiment, the model generation server 1 generates an object recognition model and also generates an action recognition model based on that object recognition model, but it is also possible to generate only the object recognition model. [Industrial applicability]

[0209] The class definition support system, class definition support device, and class definition support method according to the present invention have the effect of enabling even inexperienced users to appropriately perform the task of creating class definition information necessary for annotation and learning to generate a recognition model, and are useful as a class definition support system, class definition support device, and class definition support method that support user operations for obtaining class definition information related to the events to be recognized. [Explanation of Symbols]

[0210] 1: Model generation server (server device, class definition support device, object recognition annotation device, action recognition annotation device) 2: User terminal (terminal device) 3: LLM Server (Language Modeling Device) 11: Communications Department 12: Storage section 13: Processor 101: Class definition mode screen 102: Tab 103: Video Input Section 104: Dialogue Section 105: Definition result output section 106: Button 107: Model Selection Section 111,112: Message box 113: Input Box 114: Button 121: Dropdown list 201: Object Recognition Mode Screen 202: Learning Count Display Unit 203: Button 204: Class Designation Section 205: Image count specification section 206: Extraction condition specification part 207: Candidate image display section 208: Checkbox (True / False input section) 209: Button 301: Action Recognition Mode Screen 302: Learning Count Display Unit 303: Button 304: Class Designation Section 305: Image count specification section 306: Extraction condition specification part 307: Candidate image display section 308: Checkbox (True / False input section) 309: Button 401: Inference mode screen 402, 403: Threshold setting section 404: Video Input Section 405, 406: Buttons 407: Recognition result output unit

Claims

1. A class definition support system that assists user operations for obtaining class definition information related to events to be recognized, A terminal device in which the user performs an operation to input information to be recognized that describes the content of the event to be recognized, This terminal device is connected via a network to a server device that acquires the class definition information based on the recognized target information, The server device is A class definition support system characterized by having a language model perform a process to collect the recognition target information necessary for obtaining the class definition information while engaging in natural language dialogue with a user, and obtaining the class definition information generated by the language model.

2. The server device is The class definition support system according to claim 1, characterized in that it acquires information defining an object class corresponding to an action class as the class definition information relating to the action class.

3. The server device is The class definition support system according to claim 2, characterized in that it acquires information regarding effective combinations of an action class and the object class to which it is applied as the class definition information.

4. The server device is The class definition support system according to claim 3, characterized in that it displays a screen on the terminal device that visualizes the class definition information representing the correspondence between object classes and behavior classes in a tabular format.

5. The server device is A language model device equipped with the aforementioned language model is connected via a network, The class definition support system according to claim 1, characterized in that it acquires the class definition information in cooperation with the language model device.

6. The server device is The terminal device acquires the natural language instruction text entered by the user from the terminal device. The instruction text is sent to the language model device to instruct it to execute the class definition process. The class definition support system according to claim 5, characterized in that it obtains the class definition information based on the instruction text from the language model device.

7. The server device is The language model device is instructed to execute the class definition process by transmitting the captured image along with the instruction text. The class definition support system according to claim 6, characterized in that it obtains the class definition information based on the instruction text and captured images from the language model device.

8. A class definition support device equipped with a processor that performs processing to assist user operations for obtaining class definition information related to an event to be recognized, A terminal device is connected via a network to which the user performs an operation to input recognition target information that describes the content of the event to be recognized. The aforementioned processor, In obtaining the class definition information based on the aforementioned recognition target information, A class definition support device characterized by having a language model perform a process to collect the recognition target information necessary for obtaining the class definition information while engaging in natural language dialogue with a user, and thereby obtaining the class definition information generated by the language model.

9. A class definition support method that causes a processor to execute a process to assist user operations in obtaining class definition information related to an event to be recognized, In obtaining the aforementioned class definition information based on the recognition target information that describes the content of the event to be recognized, A class definition support method characterized by having a language model perform a process to collect the recognition target information necessary for obtaining the class definition information while engaging in natural language dialogue with a user, and obtaining the class definition information generated by the language model.

Images