Information processing system, information processing method, method for generating learning model, and computer program

Abstract

To provide an information processing system, an information processing method, a method for generating a learning model, and a computer program which allow for distinguishing between a work scene and a non-work scene to recognize them from a visual field image of a worker.SOLUTION: In an information processing system 1 comprising an analysis server 20 which analyzes a visual field image obtained by a wearable device 10, the analysis server comprises at least one processing unit, and the processing unit acquires the visual field image obtained by imaging a visual field of a captured worker by the wearable device 10 worn by the worker, and distinguishes between one or a plurality of types of work scenes and non-work scenes to recognize them by machine learning on the basis of the acquired visual field image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing method, a learning model generation method, and a computer program. [Background technology]

[0002] Patent Document 1 discloses a gas meter replacement work support device that acquires image information obtained by capturing images of a worker performing a gas meter replacement work, and determines whether the replacement work conforms to predetermined work procedures based on the acquired image information. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-124355 Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, it is possible to identify multiple types of work required to replace a gas meter, such as pre-airtightness testing, meter replacement, air purging, and post-airtightness testing, based on image information, but it is not possible to distinguish and recognize work scenes where these tasks are being performed from non-work scenes where no work is being performed.

[0005] An object of the present disclosure is to provide an information processing device, an information processing method, a learning model generation method, and a computer program that can distinguish and recognize work scenes and non-work scenes from a worker's field of view image. [Means for solving the problem]

[0006] An information processing system according to a first aspect of the present disclosure includes at least one processing unit, which acquires a field of view image obtained by capturing an image of a worker's field of view, and recognizes one or more types of work scenes and non-work scenes based on the acquired field of view image.

[0007] An information processing system according to a second aspect of the present disclosure is the information processing system according to the first aspect, wherein the processing unit distinguishes and recognizes multiple types of work scenes as the work scenes, including a first scene indicating work in progress, a second scene indicating work preparation, and a third scene indicating ancillary work.

[0008] An information processing system according to a third aspect of the present disclosure is the information processing system according to the first or second aspect, wherein the processing unit performs analysis processing on the field of view image of the recognized work scene according to the type of the work scene.

[0009] An information processing system according to a fourth aspect of the present disclosure is an information processing system according to any one of the first to third aspects, wherein the processing unit recognizes a movement scene between a work area and an area other than the work area as the non-work scene, and omits analysis processing for the recognized movement scene.

[0010] An information processing system according to a fifth aspect of the present disclosure is an information processing system according to any one of the first to fourth aspects, wherein the processing unit recognizes a movement scene between an indoor work area and an outdoor work area as the non-work scene, and omits analysis processing for the recognized movement scene.

[0011] An information processing system according to a sixth aspect of the present disclosure is the information processing system according to the fifth aspect, wherein the work target to be performed by the worker includes an indoor unit and an outdoor unit of an air conditioning device, and the processing unit recognizes a movement scene between the indoor unit and the outdoor unit as the non-work scene.

[0012] An information processing system according to a seventh aspect of the present disclosure is an information processing system according to any one of the first to sixth aspects, wherein the processing unit stores the recognition results for each scene together with the field of view image in a memory unit.

[0013] An information processing system according to an eighth aspect of the present disclosure is the information processing system according to the third aspect, wherein when the processing unit recognizes the third scene from the field of view image, the processing unit detects personal information or confidential information contained in the third scene by the analysis process, and stores the field of view image in which the detected personal information or confidential information has been anonymized in a memory unit.

[0014] An information processing method according to a ninth aspect of the present disclosure includes acquiring a field of view image obtained by capturing an image of a worker's field of view, and performing a process using a computer system to distinguish and recognize one or more types of work scenes from non-work scenes based on the acquired field of view image.

[0015] A method for generating a learning model according to a tenth aspect of the present disclosure involves acquiring a dataset including a visual field image obtained by capturing an image of a worker's visual field and correct answer data indicating whether the scene included in the visual field image is one or more types of work scene or a non-work scene, and learning using the acquired dataset as training data, so that when a visual field image is input, a computer executes a process to generate a learning model that outputs information indicating whether the scene included in the visual field image is one or more types of work scene or a non-work scene.

[0016] A computer program according to an eleventh aspect of the present disclosure acquires a field of view image obtained by capturing an image of a worker's field of view, and causes a computer to execute a process of distinguishing and recognizing one or more types of work scenes from non-work scenes based on the acquired field of view image. [Effects of the Invention]

[0017] According to the present disclosure, it is possible to distinguish and recognize work scenes and non-work scenes from the worker's field of view image. [Brief explanation of the drawings]

[0018] [Figure 1] FIG. 2 is an explanatory diagram illustrating an outline of processing executed by the information processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the internal configuration of the wearable device. [Figure 3] FIG. 2 is a block diagram showing the internal configuration of an analysis server. [Figure 4] FIG. 10 is an explanatory diagram illustrating a method for generating a task classification model. [Figure 5A] FIG. 10 is an explanatory diagram illustrating an annotation method for a visual field image. [Figure 5B] FIG. 10 is an explanatory diagram illustrating an annotation method for a visual field image. [Figure 5C] FIG. 10 is an explanatory diagram illustrating an annotation method for a visual field image. [Figure 6] FIG. 10 is an explanatory diagram illustrating a scene recognition method using a task classification model. [Figure 7] 10 is a flowchart illustrating a procedure for generating a task classification model. [Figure 8] 10 is a flowchart illustrating a scene recognition procedure. DETAILED DESCRIPTION OF THE INVENTION

[0019] Hereinafter, an information processing system according to an embodiment will be specifically described with reference to the drawings. (Embodiment 1) FIG. 1 is an explanatory diagram outlining the processing executed by an information processing system 1 according to a first embodiment. The information processing system 1 is a system for capturing an image of a worker's field of view, acquiring a visual field image, and analyzing the acquired visual field image. The worker in this embodiment is, for example, a worker who performs installation work, inspection work, repair work, etc. related to air conditioning equipment. Such a worker prepares for work indoors or outdoors, and also performs work related to indoor units indoors and work related to outdoor units outdoors. In addition, the work performed by the worker may include incidental work such as explaining the work to customers. Furthermore, the worker moves between indoor and outdoor work areas and takes breaks between tasks.

[0020] In the embodiment, a wearable device 10 with an imaging function is worn by a worker to capture an image of the worker's field of vision at a work site. In one example, the wearable device 10 is worn around the worker's neck. Alternatively, the wearable device 10 may be worn on the worker's head or on another body part such as the shoulder or arm. Furthermore, the wearable device 10 may be a goggle-type camera device, and any device with an imaging function such as a smartphone or an action camera may be used instead of the wearable device 10.

[0021] Wearable device 10 captures an image of the worker's field of view at a work site and generates a field of view image. In this embodiment, the field of view image generated by wearable device 10 is a moving image. Alternatively, the field of view image generated by wearable device 10 may be a plurality of still images captured at predetermined time intervals.

[0022] In this embodiment, the visual field image is a video from the worker's viewpoint (first-person viewpoint video), but does not mean a video that completely reproduces the worker's field of view. In other words, the range (field of view) captured by wearable device 10 does not need to strictly match the range that the worker sees, and the capturing direction of wearable device 10 does not need to strictly match the line of sight of the worker. Wearable device 10 only needs to be worn by the worker so that it faces generally forward of the worker, so that at least a portion of the worker's field of view can be captured.

[0023] The information processing system 1 includes an analysis server 20 that analyzes a visual field image obtained by the wearable device 10. The analysis server 20 is communicatively connected to the wearable device 10 via a communication network NW such as the Internet. The analysis server 20 acquires the visual field image from the wearable device 10 by communicating with the wearable device 10 via the communication network NW. The analysis server 20 may acquire the visual field image in real time while the wearable device 10 is performing a task, or may acquire the visual field image after the task is completed.

[0024] Furthermore, the route by which the analysis server 20 acquires the visual field image is not limited to a route by which the analysis server 20 acquires the visual field image through direct communication with the wearable device 10. For example, the visual field image captured by the wearable device 10 may be transmitted to the analysis server 20 via another terminal device (for example, the worker's smartphone). Furthermore, the visual field image captured by the wearable device 10 may be recorded on a portable recording medium such as an SD (Secure Digital) card. In this case, the analysis server 20 can acquire the visual field image by reading it from the portable recording medium.

[0025] Based on the acquired visual field image, the analysis server 20 executes a process of distinguishing and recognizing one or more types of work scenes from non-work scenes. The analysis server 20 classifies the worker's work using a work classification model MD1 (see FIG. 3) that classifies the work of the photographer of the visual field image (the worker himself / herself in this embodiment), and based on the classification results, distinguishes and recognizes one or more types of work scenes from non-work scenes. Because the visual field image in this embodiment is a moving image, the analysis server 20 can classify the visual field image into work scenes or non-work scenes at regular intervals (for example, every 6-second interval).

[0026] The analysis server 20 may further perform analysis processing according to the type of work scene on the view image of the section that has been recognized as a work scene.

[0027] For example, the analysis server 20 may perform an analysis process to identify the type of work in chronological order for the field of view image of a section in which a work scene (first scene) indicating that work is in progress is recognized, and to determine whether the work is being performed according to the correct work procedure.

[0028] In addition, if it is stipulated that protective equipment or auxiliary equipment must be worn to ensure safety during on-site work, the analysis server 20 may perform an analysis process to determine whether the worker has worn protective equipment or auxiliary equipment for the field of view image of the section in which a work scene (second scene) indicating that work preparation is in progress is recognized.

[0029] In addition, since there is a possibility that the customer's face or personal information may be reflected in the field of view image when explaining to the customer, the analysis server 20 may detect personal information or confidential information contained in the field of view image of the section in which a work scene (third scene) indicating ancillary work is recognized, and perform a process of concealing the detected personal information or confidential information.

[0030] On the other hand, the analysis server 20 can omit further analysis processing for the visual field image of the section that is recognized as a non-work scene.

[0031] As described above, when analysis server 20 according to the present embodiment acquires a visual field image captured by wearable device 10 worn by a worker, it can classify the visual field image into a work scene or a non-work scene for each section. Analysis server 20 performs analysis processing according to the work type for visual field images in sections classified as work scenes, and can omit analysis processing for visual field images in sections classified as non-work scenes, thereby reducing the processing load within the server.

[0032] 2 is a block diagram showing the internal configuration of wearable device 10. Wearable device 10 includes a processing unit 11, a storage unit 12, a communication unit 13, an imaging unit 14, a sound input unit 15, a sound output unit 16, a sensor unit 17, and the like.

[0033] The processing unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The ROM included in the processing unit 11 stores control programs and the like that control the operation of each hardware unit included in the wearable device 10. The CPU in the processing unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of each hardware unit, thereby causing the entire device to function as the wearable device 10 of the present disclosure. The RAM included in the processing unit 11 temporarily stores data used during the execution of various processes.

[0034] The storage unit 12 includes an auxiliary storage device and stores the field of view image generated by the imaging unit 14. An application program executed by the processing unit 11 may also be installed in the storage unit 12. The application program may be installed in advance or may be installed after use has begun.

[0035] The communication unit 13 includes a communication module for wireless communication with an external device such as the analysis server 20. The communication module may be a communication module for wireless communication using a known mobile communication standard such as 3G, 4G, or 5G, or a wireless LAN system such as WiFi (registered trademark). The communication unit 13 communicates with an external device such as the analysis server 20 via a communication network NW, transmitting necessary data such as a visual field image (video data) and receiving appropriate data transmitted from the external device. The communication unit 13 may also include a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark) for communication with a terminal such as a smartphone carried by the worker.

[0036] The imaging unit 14 includes an optical lens, an imaging element, a driver circuit, etc. A wide-angle lens is preferably used as the optical lens. The imaging element is a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge-Coupled Device), etc., and generates an electrical signal according to the intensity of light imaged through the optical lens. The driver circuit includes a timing generator (TG), etc., and sequentially reads out the electrical signals from the imaging element in synchronization with a clock signal output from the TG to generate video data. The video data generated by the imaging unit 14 is sent to the processing unit 11 and stored in the memory unit 12. Alternatively, the video data generated by the imaging unit 14 is transmitted to the analysis server 20 via the communication unit 13.

[0037] The sound input unit 15 includes a microphone for collecting sound, a processing circuit for converting the collected sound into a digital signal (sound data), etc. The sound data generated by the sound input unit 15 is sent to the processing unit 11, where appropriate processing such as noise removal is performed. The sound data generated by the sound input unit 15 is stored in the memory unit 12, or transmitted to the analysis server 20 via the communication unit 13.

[0038] The sound output unit 16 includes a speaker that outputs sound. The sound output unit 16 outputs sound based on the acoustic data provided by the processing unit 11.

[0039] The sensor unit 17 includes a non-contact sensor for detecting the worker's fingers and the like. The sensors included in the sensor unit 17 include a proximity sensor, a gesture sensor, and the like. The proximity sensor detects, for example, when the worker's fingers have approached within a predetermined range. The gesture sensor detects, for example, the movement of the worker's fingers. The detection result by the sensor unit 17 is notified to the processing unit 11. The processing unit 11 may issue an instruction to start capturing or stop capturing to the imaging unit 14 based on the detection result of the sensor unit 17.

[0040] 3 is a block diagram showing the internal configuration of the analysis server 20. The analysis server 20 is a dedicated or general-purpose server device, and includes a processing unit 21, a storage unit 22, a communication unit 23, an operation unit 24, and a display unit 25, for example.

[0041] The processing unit 21 includes a CPU, a ROM, a RAM, etc. The ROM included in the processing unit 21 stores a control program and the like that controls the operation of each hardware unit included in the analysis server 20. The CPU in the processing unit 21 reads and executes the control program stored in the ROM and a computer program (described below) stored in the storage unit 22, and executes processing to control the operation of each hardware unit, thereby causing the entire device to function as the analysis server 20 of the present disclosure. The RAM included in the processing unit 21 temporarily stores data used during the execution of various processes.

[0042] In the embodiment, the processing unit 21 is configured to include a CPU, a ROM, and a RAM, but the configuration of the processing unit 21 is not limited to the above. The processing unit 21 may be one or more processing circuits including, for example, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, volatile or non-volatile memory, etc. Furthermore, the processing unit 21 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until when a measurement end instruction is given, and a counter that counts numbers.

[0043] The storage unit 22 includes a storage device such as a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 22 stores various computer programs executed by the processing unit 21 and various data acquired via the communication unit 23.

[0044] The computer programs (program products) stored in the storage unit 22 include an analysis processing program PG1 for analyzing a visual field image. The analysis processing program PG1 is a computer program for causing a computer to execute a process of acquiring a visual field image obtained by capturing an image of the visual field of a worker, and distinguishing and recognizing one or more types of work scenes from non-work scenes based on the acquired visual field image.

[0045] The analysis processing program PG1 may further cause the computer to execute a process of identifying the type of work in chronological order based on the field of view image of the work screen divided during the work, and determining whether the work is being performed according to the correct work procedure.

[0046] The analysis processing program PG1 may further cause the computer to execute a process to determine whether or not the worker is wearing protective equipment or assistive devices based on the field of view image of the work scene divided during work preparation, and to check safety.

[0047] The analysis processing program PG1 may further cause the computer to execute processing to detect personal information or confidential information based on the field of view image of the work screen divided during the auxiliary work and to conceal the information.

[0048] The analysis processing program PG1 may be a single computer program or a group of programs made up of multiple computer programs. The analysis processing program PG1 may be executed by a single computer or may be executed by multiple computers (for example, the wearable device 10 and the analysis server 20) working together.

[0049] A computer program including the analysis processing program PG1 is provided by a non-transitory recording medium RM on which the computer program is readably recorded. The recording medium RM is a portable memory such as a CD-ROM, a USB memory, an SD card, a micro SD card, or a CompactFlash (registered trademark). The processing unit 21 reads various computer programs from the recording medium RM using a reading device (not shown) and stores the read various computer programs in the storage unit 22. The computer programs stored in the storage unit 22 may also be provided by communication. In this case, the processing unit 21 acquires the computer programs by communication via the communication unit 23 and stores the acquired computer programs in the storage unit 22.

[0050] The storage unit 22 also stores a task classification model MD1 used in the above-mentioned analysis processing program PG1. The task classification model MD1 is a learning model that has been trained to output information about the actions of the photographer (i.e., the worker himself / herself) in response to the input of a visual field image. The task classification model MD1 is described by its definition information. The definition information of the task classification model MD1 includes information about the layers that make up the model, information about the nodes that make up each layer, and parameters such as weighting and bias between nodes. These parameters are obtained by training using, as training data, a dataset that includes visual field images and annotation data that indicates the actions of the photographer shown in the visual field images. In this embodiment, it is assumed that the trained task classification model MD1 is stored in the storage unit 22.

[0051] The communication unit 23 includes a communication module for wireless communication with an external device such as the wearable device 10. The communication module may be a communication module for wireless communication using a known mobile communication standard such as 3G, 4G, or 5G, or a wireless LAN system such as WiFi (registered trademark). The communication unit 23 communicates with an external device such as the wearable device 10 via the communication network NW, receives data such as a visual field image (video data), and transmits appropriate data to be transmitted to the external device. The communication unit 23 may further include a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark).

[0052] The operation unit 24 is equipped with operation devices such as a touch panel, a keyboard, and switches, and receives various inputs and operations from a work manager, etc. The processing unit 21 acquires information input through the operation unit 24 and performs appropriate control based on various operation information provided by the operation unit 24.

[0053] The display unit 25 includes a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information to be notified to a work manager or the like in response to an instruction from the processing unit 21.

[0054] The analysis server 20 may be a single computer, or may be a computer system configured with multiple computers and peripheral devices, etc. Furthermore, the analysis server 20 may be a virtual machine whose entity is virtualized, or may be a cloud.

[0055] The task classification model MD1 used by the analysis server 20 will be described below. FIG. 4 is an explanatory diagram illustrating a method for generating a task classification model MD1. In this embodiment, a tuned model of EgoVLPv2 based on Transformer is used as the task classification model MD1. In EgoVLP (Egocentric Video-Language pre-training), a VLP model MD is constructed by pre-training using a dataset containing first-person perspective video clips and annotations (text narration) for the video clips as training data. EgoClip created by Ego4D is used as the training data for pre-training. Ego4D records the movements (annotations) of the camera wearer in the videos and timestamps for approximately 10,000 videos captured from a first-person perspective. By segmenting the videos based on the timestamps, a dataset (EgoClip) DS consisting of a large number of pairs of video clips and text is obtained.

[0056] The VLP model MD comprises a video encoder En1 that extracts features from input video clips and a text encoder En2 that extracts features from input text. In EgoVLP, the video encoder En1 and the text encoder En2 are trained by contrastive learning using EgoNCE (NCE: Noise Constastive Estimation) as the loss function. In contrastive learning, a self-supervised learning mechanism that compares data is used to learn features so that similar data are close and dissimilar data are far apart.

[0057] The second-generation EgoVLPv2 has a gating mechanism to turn on / off the cross-attention fusion between the video encoder En1 and the text encoder En2, and is characterized by the ability to flexibly switch between dual encoders and fusion encoders. Furthermore, compared to stacking transform layers and shared encoders specialized for fusion, it requires fewer fusion parameters, less GPU memory, less computational resources, and less training time.

[0058] In this embodiment, a task classification model MD1 is generated by tuning EgoVLPv2, an existing model, using a dataset containing field of view images captured at a work site using a wearable device 10 and annotations added to the field of view images as training data.

[0059] 5A to 5C are explanatory diagrams illustrating an annotation method for a visual field image. FIG. 5A shows an example in which a label is assigned every two seconds. In this embodiment, the accuracy of labels less than two seconds is not an issue, and the minimum unit for assigning labels as annotations is two seconds. In the example of FIG. 5A, if the first two-second section is a preparatory work scene, the visual field image for that section is labeled "preparatory work." If the next two-second section includes a preparatory work scene and a work scene, it is determined which scene accounts for the majority. If it is determined that the work scene accounts for the majority, the label "work" is assigned. Similarly, if the next two-second section is a work scene, the label "work" is assigned; if it is an incidental work scene, the label "incidental work" is assigned; and if it is a non-work scene, the label "non-work" is assigned. In this way, labels are assigned seamlessly to the video in the example of FIG. 5A.

[0060] In this embodiment, a "non-work" label can be assigned to a scene that does not fall under any of the following categories: work, preparatory work, and ancillary work. A "non-work" label can also be assigned to a scene in which a worker moves. For example, work related to air conditioning equipment may include work on an indoor unit installed indoors and work on an outdoor unit installed outdoors. In this case, it is expected that movement between the indoor work area and the outdoor work area will occur, so a "non-work" label can be assigned to a scene in which a worker moves between the indoor work area and the outdoor work area. Furthermore, work related to an air conditioning system may include work on a heat source device installed in an indoor machine room and work on an indoor unit installed in each room, and may involve movement between multiple indoor work areas. Furthermore, work related to an air conditioning system may not be limited to arriving at a customer's property by car and performing work at the customer's property, and may involve movement between the car and the work area at the customer's property. Furthermore, work at chemical plants, power supply facilities, etc. may involve movement between multiple outdoor work areas. Therefore, a scene of movement between a certain work area and an area other than the work area may be labeled as "non-work."

[0061] In this embodiment, four types of labels, "preparatory work," "work," "ancillary work," and "non-work," are assigned to the field of view images captured every two seconds, but more detailed labels may be assigned depending on the work object or work content. For example, labels such as "indoor unit," "outdoor unit," and "other" may be assigned depending on the work object, and labels such as "visual inspection," "check with measuring instruments," "dismantling," "repair / replacement," and "restore to original condition" may be assigned depending on the work content.

[0062] FIG. 5B shows an example of labeling based on the start timing of a scene. In the example of FIG. 5B, the label "preparation work" is assigned to the field of view image for two seconds after the start of the preparation work scene. If the next two seconds of field of view image contain both preparation work scenes and work scenes, no label is assigned, and the label "work" is assigned to the field of view image every two seconds after the start of the next scene (work scene). In this way, in the example of FIG. 5B, labels are assigned based on the start timing of the scenes.

[0063] FIG. 5C shows an example in which a label is assigned after excluding the video immediately after the start of a scene. Because there is a possibility that no substantial work is being performed immediately after a scene starts (or immediately after a scene changes), a predetermined number of frames are excluded from the start of each scene and then a label is assigned. In the example of FIG. 5C, a predetermined number of frames are excluded from the start of a preparatory work scene and then a label "preparatory work" is assigned, and a predetermined number of frames are excluded from the start of a work scene and then a label "work" is assigned to the field of view image every 2 seconds. In this way, in the example of FIG. 5C, a label is assigned after excluding the frame immediately after the start of each scene.

[0064] In this embodiment, in order to tune EgoVLPv2, training data was prepared by assigning labels to visual field images every two seconds using the method illustrated in FIG. 5A. Alternatively, training data may be prepared using the method illustrated in FIG. 5B or FIG. 5C. Due to the specifications of the base model, only six-second videos (visual field images) can be input as training data. Depending on how the video is cut, two or more types of labels may be assigned to a six-second video. In this embodiment, in order to assign one label to a six-second video, if two or more types of labels are included, a process of rounding them down to the majority label was performed.

[0065] The above process results in a dataset containing numerous videos (videos every 6 seconds) extracted from the visual field image and labels (text) assigned to each video. The task classification model MD1 according to this embodiment is generated by tuning the trained EgoVLPv2 using the obtained dataset as training data. The task classification model MD1 can be used for various tasks requiring a dual encoder and a fusion encoder due to the switching capability of EgoVLPv2. The analysis server 20 according to this embodiment uses the task classification model MD1 to perform a task of recognizing the actions (i.e., tasks) of the worker who is shooting the video. Based on the results of task classification using the task classification model MD1, the analysis server 20 distinguishes and recognizes whether the input video (visual field image) represents one or more types of work scenes or a non-work scene.

[0066] FIG. 6 is an explanatory diagram illustrating a scene recognition method using task classification model MD1. In the operation phase after task classification model MD1 is generated, a worker wears wearable device 10 and captures an image of the worker's own field of view using wearable device 10. Analysis server 20 acquires the worker's field of view image captured by wearable device 10 and inputs the field of view image in 6-second intervals (180-frame intervals for a 30-fps video) into task classification model MD1, thereby executing a task classification task using task classification model MD1. Analysis server 20 acquires a classification result including one of "preparatory work," "work," "incidental work," and "non-work" as the execution result of the task classification task.

[0067] Based on the execution results of the task classification task, the analysis server 20 distinguishes between work scenes and non-work scenes as scenes shown in each field of view image. That is, if the worker's task classified by the task classification task is either "preparatory work," "work," or "incidental work," the analysis server 20 recognizes the scene in that field of view image as a work scene. Furthermore, if the worker's task classified by the task classification task is "non-work," the analysis server 20 recognizes the scene in that field of view image as a non-work scene.

[0068] The analysis server 20 may distinguish between a first scene indicating that the worker is working, a second scene indicating that the worker is preparing for work, and a third scene indicating that the worker is performing an additional task as a work scene. If the worker's work classified by the work classification task is "work," the analysis server 20 recognizes it as the first scene, if it is "preparation work," the analysis server 20 recognizes it as the second scene, and if it is "additional work," the analysis server 20 recognizes it as the third scene.

[0069] The processing executed by the analysis server 20 will be described below. FIG. 7 is a flowchart illustrating the procedure for generating the task classification model MD1. It is assumed that a base model such as EgoVLPv2 is stored in the storage unit 22 before generating the task classification model MD1. To generate training data for tuning the base model, the processing unit 21 of the analysis server 20 acquires visual field images obtained by capturing images of the work site using the wearable device 10 (step S101) and accepts annotations for the acquired visual field images (step S102). The annotations are performed manually. For example, for each visual field image in a predetermined unit (e.g., every 2 seconds), the labels "preparatory work," "work," "auxiliary work," and "non-work" are accepted via the operation unit 24.

[0070] When a dataset containing a sufficient number of visual field images and labels for the visual field images is obtained through annotation, the processing unit 21 uses the obtained dataset as training data to tune the base model, thereby generating a task classification model MD1 (step S103). If the base model is EgoVLPv2, the base model can be tuned by performing contrastive learning using EgoNCE as a loss function. The generated task classification model MD1 is stored in the storage unit 22.

[0071] In this embodiment, a method for generating a task classification model MD1 using EgoVLPv2 has been described. However, other learning models, such as VideoMaev2 and CAST, may be used instead of EgoVLPv2, and learning models such as LSTM (Long-Term Memory) and 3D-CNN (Convolutional Neural Network) may also be used as the base model. The training data used to generate the learning model is not limited to the above examples and may be designed appropriately depending on the type of learning model used. Furthermore, instead of tuning the base model, the task classification model MD1 may be generated by training a model with initial parameters set from scratch.

[0072] In addition, although the present embodiment is configured to generate a task classification model MD1 for classifying tasks, a learning model for classifying scenes shown in a visual field image may be generated. That is, when a visual field image is input, the learning model may be trained to output information about the scene shown in the visual field image.

[0073] Furthermore, in this embodiment, a task classification model MD1 is shown that is configured to output task classification results in response to the input of a field of view image, but the task classification model MD1 may also be configured to input audio (acoustic data) recorded together with the field of view image, or text generated by voice recognition, and output task classification results in response to the input of the field of view image and audio (or the field of view image and text).

[0074] Furthermore, in this embodiment, the task classification model MD1 is generated in the analysis server 20, but it may also be generated in an external server. The generated task classification model MD1 may be stored in the storage unit 22 of the analysis server 20, or in a storage unit of an external server accessible from the analysis server 20.

[0075] FIG. 8 is a flowchart illustrating the scene recognition procedure. A worker who starts work at a work site wears the wearable device 10 on his or her body. The wearable device 10 automatically starts capturing images at an appropriate timing after the worker puts on the wearable device 10. Alternatively, the wearable device 10 starts capturing images when instructed by the worker or the analysis server 20. The wearable device 10 transmits the worker's field of view image captured by the imaging unit 14 to the analysis server 20 via the communication unit 13. The wearable device 10 may transmit the field of view image to the analysis server 20 in real time while the worker is working, or may transmit the captured field of view image to the analysis server 20 after the work is completed.

[0076] The processing unit 21 of the analysis server 20 reads out and executes the analysis processing program PG1 from the storage unit 22, and performs scene recognition processing according to the following procedure.

[0077] Processing unit 21 acquires a visual field image transmitted from wearable device 10 via communication unit 23 (step S121). Processing unit 21 inputs the acquired visual field image to task classification model MD1 at predetermined intervals (for example, every 6-second intervals) and executes a calculation (task classification task) using task classification model MD1 (step S122). Processing unit 21 acquires a task classification result as a result of executing the calculation using task classification model MD1 (step S123).

[0078] The processing unit 21 determines whether the classification result by the task classification model MD1 is "task" (step S124). If the processing unit 21 determines that the task is "task" (S124: YES), it recognizes that the field of view image of that section is the first scene (a scene showing work in progress) (step S125).

[0079] If the processing unit 21 determines that the classification result is not "work" (S124: NO), it determines whether the classification result according to the work classification model MD1 is "preparation work" (step S126). If the processing unit 21 determines that the classification result is "preparation work" (S126: YES), it recognizes that the field of view image of that section is the second scene (a scene showing work preparation) (step S127).

[0080] If the processing unit 21 determines that the classification result is not "preparatory work" (S126: NO), it determines whether the classification result according to the work classification model MD1 is "incidental work" (step S128). If the processing unit 21 determines that the classification result is "incidental work" (S128: YES), it recognizes that the field of view image of that section is a third scene (a scene showing incidental work) (step S129).

[0081] If the processing unit 21 determines that the classification result is not "incidental work" (S128: NO), it recognizes that the field of view image of that section is a non-work scene (step S130). The processing unit 21 may recognize a non-work scene after determining whether the classification result by the work classification model MD1 is "non-work".

[0082] After performing scene recognition, the processing unit 21 determines whether the field of view image has ended (step S131), and if it determines that the field of view image has not ended (S131: NO), it returns the processing to step S122 and executes the processing of steps S122 to S130 for the field of view image of the next section, thereby continuing the scene recognition processing.

[0083] When the processing unit 21 determines that the field of view image has ended (S131: YES), it executes an analysis process according to the recognized scene (step S132). For example, the processing unit 21 may execute an analysis process to identify the task type in chronological order for the field of view image recognized as the first scene (a scene in which work is being performed) and determine whether the work is being performed according to the correct work procedure. To determine whether the work procedure is correct or not, a learning model may be prepared that is trained to output information regarding the correctness of the work procedure when the field of view image of the first scene is input. When the processing unit 21 determines that the work procedure is incorrect, it may notify the wearable device 10 worn by the worker of this fact and guide the worker to the correct procedure via the sound output unit 16. Alternatively, when the processing unit 21 determines that the work procedure is incorrect, it may notify the work manager of this information.

[0084] Furthermore, processing unit 21 may perform an analysis process for determining whether the worker is wearing protective equipment or an assistive device for the field of view image recognized as the second scene (a scene in which the worker is preparing to work). When the field of view image of the second scene is input to detect whether the worker is wearing protective equipment or an assistive device, a learning model may be prepared that is trained to detect the worker's hands (where the protective equipment or assistive device is worn) included in the field of view image and the protective equipment or assistive device. Based on the output of such a learning model, processing unit 21 can determine whether the worker is wearing protective equipment or an assistive device by grasping the positional relationship between the worker's hands and the protective equipment or assistive device. If processing unit 21 determines that the worker is not wearing the necessary protective equipment or an assistive device, it may notify wearable device 10 worn by the worker of this fact and prompt the worker to wear the protective equipment or an assistive device via sound output unit 16.

[0085] Furthermore, the processing unit 21 may detect personal information or confidential information contained in the field of view image for a recognized image recognized as a third scene (a scene showing ancillary work) and perform a process of concealing the detected personal information or confidential information. When a worker explains the work to a customer or records the work on a form during or after the work is completed, personal information or confidential information, such as the customer's face or name, may appear in the field of view image. Therefore, the processing unit 21 detects personal information or confidential information contained in the field of view image and performs a concealment process, such as a mosaic process or a blackout process, on the detected personal information or confidential information. To detect personal information and confidential information, such as the customer's face or name, when the field of view image of the third screen is input, a learning model may be prepared that is trained to output information related to personal information and confidential information. When storing the field of view image of the third scene as evidence of on-site work, the processing unit 21 may store the concealed field of view image in the memory unit 22 instead of the raw field of view image.

[0086] The processing unit 21 can omit analysis processing for a field of view image recognized as a non-work scene. A scene recognized as a non-work scene is, for example, a scene of movement between an indoor work area and an outdoor work area. Alternatively, a scene recognized as a non-work scene may be a scene of movement between a specific work area and an area other than the specific work area.

[0087] As described above, when analysis server 20 according to the present embodiment acquires a visual field image captured by wearable device 10 worn by a worker, it can classify the visual field image into a work scene or a non-work scene for each section. Analysis server 20 performs analysis processing according to the work type for visual field images in sections classified as work scenes, and can omit analysis processing for visual field images in sections classified as non-work scenes, thereby reducing the processing load within the server.

[0088] In this embodiment, the analysis server 20 performs analysis processing on the field of view image recognized as the scene, but the configuration may also include a work check server that checks the work procedures for the field of view image recognized as the first scene, a safety check server that detects the wearing of protective equipment, etc. for the field of view image recognized as the second scene, and a recording server that records personal information and confidential information while concealing it, separate from the analysis server 20.

[0089] In this embodiment, the analysis server 20 is configured to perform the scene recognition process, but the wearable device 10 may also perform the scene recognition process, and depending on the results of the scene recognition process, the analysis server 20 may perform processes such as checking work procedures, safety checks to detect the wearing of protective equipment, and recording while concealing personal and confidential information.

[0090] Furthermore, as an application example of the information processing system according to this embodiment, a work scene involving air conditioning equipment has been envisaged, but it goes without saying that the system can be applied to a variety of work scenes, not limited to work scenes involving air conditioning equipment, such as work scenes relating to elevators, and work scenes relating to maintenance and inspection at chemical plants and power supply facilities.

[0091] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0092] 10. Wearable Devices 20 Analysis Server 21 Processing section 22 Memory section 23 Communications Department 24 Control section 25 Display section PG1 Analysis and Processing Program MD1 Task Classification Model

Claims

1. At least one processing unit; The processing unit A visual field image is acquired by capturing an image of the worker's visual field, Based on the acquired visual field image, one or more types of work scenes and non-work scenes are distinguished and recognized. Information processing system.

2. The processing unit The work scenes are recognized by distinguishing between a first scene showing work in progress, a second scene showing work preparation, and a third scene showing incidental work. The information processing system according to claim 1 .

3. The processing unit An analysis process is performed on the recognized visual field image of the work scene according to the type of the work scene. The information processing system according to claim 2 .

4. The processing unit Recognizing a movement scene between a work area and an area other than the work area as the non-work scene; Omit analysis processing for recognized moving scenes The information processing system according to claim 2 .

5. The processing unit Recognizing a moving scene between an indoor work area and an outdoor work area as the non-work scene; Omit analysis processing for recognized moving scenes The information processing system according to claim 1 .

6. The work target to be performed by the worker includes an indoor unit and an outdoor unit of an air conditioning device, The processing unit As the non-work scene, a scene of movement between the indoor unit and the outdoor unit is recognized. The information processing system according to claim 5 .

7. The processing unit The recognition result for each scene is stored in a storage unit together with the field of view image. The information processing system according to claim 1 .

8. The processing unit When the third scene is recognized from the field of view image, personal information or confidential information included in the third scene is detected by the analysis process; The visual field image in which the detected personal information or confidential information has been concealed is stored in a storage unit. The information processing system according to claim 3 .

9. A visual field image is acquired by capturing an image of the worker's visual field, Based on the acquired visual field image, one or more types of work scenes and non-work scenes are distinguished and recognized. An information processing method in which processing is performed by a computer system.

10. A data set is obtained that includes a visual field image obtained by capturing an image of the visual field of a worker and correct answer data that indicates whether the scene included in the visual field image is one or more types of work scene or a non-work scene; By learning using the acquired dataset as training data, a learning model is generated that, when a visual field image is input, outputs information indicating whether the scene included in the visual field image is one or more types of work scene or a non-work scene. A method for generating a learning model that executes processing by a computer.

11. A visual field image is acquired by capturing an image of the worker's visual field, Based on the acquired visual field image, one or more types of work scenes and non-work scenes are distinguished and recognized. A computer program that causes a computer to execute a process.

Images