Procedure manual creation device and procedure manual creation method

Abstract

A procedure manual creation device that can appropriately create a procedure manual from a moving image is provided. [Solution] The procedure manual creation device 100 comprises a procedure candidate frame extraction unit 12 that extracts procedure candidate frames, which are frames that separate moving images; a scene extraction unit 13 that extracts scenes that separate moving images; a scene content output unit 15 that assigns procedure candidate frames in the extracted scenes from the timestamp of each frame, inputs the procedure candidate frames to an image analysis tool, extracts information labels output by the image analysis tool, inputs the extracted information labels, prompt sentences and the procedure candidate frames to a language model and outputs the work content and work procedures of each scene from the language model; and a procedure manual creation unit 16 that extracts representative work images from the procedure candidate frames assigned to the scenes, and creates an overall work procedure manual that summarizes the scenes using the work content and work procedures of the output scenes and the extracted work images.

Description

[Technical Field]

[0001] The present invention relates to a procedure manual creation device and a procedure manual creation method. [Background technology]

[0002] Various methods have been proposed as support devices for creating manuals. Patent Document 1 discloses a manual creation support device including: a display information acquisition unit that acquires display information related to the display of a web page; an operation information recording unit that records operation information related to operations via a pointing device; a generation unit that generates a manual that explains operation procedures related to the web page based on the display information and the operation information; and a correction unit that corrects the manual generated by the generation unit based on an instruction from a user, wherein the operation information includes at least a position specified via the pointing device, designated position information related to movement of the position, and instruction information that instructs the execution of an operation at the position; the generation unit generates the manual based on the operation information and the display information; the correction unit corrects the manual by removing from the manual any description corresponding to a designated operation specified by the user among operations corresponding to the operation information; and the device further includes a first machine learning unit that machine learns to select operation information to be used in generating the manual based on the operation information and the correction; and the generation unit generates the manual based on the operation information selected using the machine learning of the first machine learning unit.

[0003] Patent Document 2 discloses a document generation program that causes a computer to execute the following process: acquire a moving image of a display screen of a device to be operated; acquire sensor information generated by an input operation to an input device associated with the device to be operated; generate a still image relating to an operation procedure corresponding to the input operation based on the moving image and the sensor information; and output a document that associates the operation procedure with the still image.

[0004] Non-Patent Document 1 discloses a method for automatically creating an image-attached manual using only operation videos. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 7475734 [Patent Document 2] Japanese Patent Application Publication No. 2020-123116 [Non-patent literature]

[0006] [Non-Patent Document 1] How to automatically create an image-based manual using only operation videos, [online], October 1, 2025, Internet<URL: https: / / zenn.dev / yuru_log / articles / video-to-manual-with-ai> Summary of the Invention [Problem to be solved by the invention]

[0007] The manual creation support device of Patent Document 1 requires operation information related to operations performed via a pointing device.

[0008] The program in Patent Document 2 requires the acquisition of sensor information generated by input operations to an input device associated with the device to be operated, but it is not always possible to acquire the sensor information. For this reason, it has been desired to create a procedure manual from a video of the operation of the device to be operated.

[0009] Non-Patent Document 1 claims that it is possible to automatically create manuals from operation videos using generation AI, but when it was put into practice, it was found that when trying to summarize using only prompts, the generated procedure manual, which is the output result, was inconsistent. Also, there were problems such as many missing steps, and it was an issue that it was not possible to create appropriate operation manuals.

[0010] The present invention has been made to solve the above-mentioned problems, and aims to provide a procedure manual creation device and a procedure manual creation method that can appropriately create a procedure manual from a moving image. [Means for solving the problem]

[0011] In order to achieve the above object, the procedure manual creation device of the present invention is a procedure manual creation device that creates a procedure manual from a video, and includes a procedure candidate frame extraction unit that analyzes image data cut out from the video at predetermined time intervals using an amount of image change and extracts procedure candidate frames that are frames that separate the video, a scene extraction unit that analyzes image data cut out from the video using an amount of image change and extracts scenes that separate the video, and a procedure candidate frame in the extracted scene is assigned based on the timestamp of each frame, the procedure candidate frames are input to an image analysis tool, information labels output by the image analysis tool are extracted, and the extracted information labels and Extract operation steps by comparing the information labels of two consecutive candidate frames. The present invention is characterized by comprising: a scene content output unit that inputs a prompt sentence and the procedure candidate frames into a language model and outputs the work content and work procedure of each scene from the language model; and a procedure manual creation unit that extracts a representative work image from the procedure candidate frames assigned to the scene and creates an overall work procedure manual summarizing the scenes using the work content and work procedure of the output scene and the extracted work image. Other aspects of the present invention will be described in the embodiments below. [Effects of the Invention]

[0012] According to the present invention, a procedure manual can be appropriately created from a video image. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a block diagram showing the configuration and functions of a procedure manual creation device according to an embodiment; [Figure 2] FIG. 10 is a diagram illustrating an example of a procedure manual creation process according to the embodiment. [Figure 3]FIG. 1 is a diagram illustrating an outline of processing from input to image processing. [Figure 4] 10 is a flowchart showing data processing using a generation AI of a generation AI processing unit. [Figure 5] 10 is a flowchart showing a representative image extraction process performed by the procedure manual creation unit. [Figure 6] FIG. 10 is a diagram showing the relationship between frame files. [Figure 7] FIG. 10 is a diagram illustrating a process of eliminating duplication of procedure candidates. [Figure 8] FIG. 10 is a diagram showing a specific example of a representative image extraction process. [Figure 9] FIG. 10 is a diagram illustrating an example of a work procedure manual. [Figure 10] FIG. 10 is an explanatory diagram showing an example of a procedure manual of a comparative example. [Figure 11] FIG. 2 is an explanatory diagram showing an example of a procedure manual according to the present embodiment. [Figure 12] 1 is a diagram illustrating a hardware configuration of a procedure manual creation device according to an embodiment. [Figure 13] FIG. 2 is a diagram illustrating an example of a functional configuration of a procedure manual creation device according to an embodiment. [Figure 14] FIG. 10 is a diagram showing another example of a work procedure manual. DETAILED DESCRIPTION OF THE INVENTION

[0014] Before describing an embodiment for carrying out the present invention, the features of this embodiment will be described with reference to FIGS. Fig. 10 is an explanatory diagram showing an example of a procedure manual of a comparative example. Fig. 11 is an explanatory diagram showing an example of a procedure manual of this embodiment. Fig. 10 is an example of a procedure manual created from video images of a display screen of a device to be operated. Fig. 10 shows procedure numbers, procedure explanations, image numbers, images, pixel values, etc. so that the relationship between procedure numbers and image numbers, etc. can be understood. For example, procedure number PR01 is a procedure for performing operation (1) on the ◎◎ screen, and procedure number PR02 is a procedure for performing operation (2) on the ◎◎ screen.

[0015] In contrast, in this embodiment, the concept of a scene is introduced, and operations on each screen are summarized, allowing the user to create a procedure manual that explains the overall operations in an easy-to-understand manner. A scene, for example, refers to a section of an action at a certain location. FIG. 11 shows scene numbers, scene descriptions, image numbers, images, pixel values, etc., so that the relationship between scene numbers and image numbers, etc., can be understood. According to this, the scene number SC01 clearly shows that step numbers PR01, PR02, and PR03 in FIG. 10 are operations on the same screen. Similarly, the scene number SC02 clearly shows that step numbers PR04 and PR05 in FIG. 10 are operations on the same screen. Furthermore, the scene number SC03 clearly shows that step numbers PR06, PR07, and PR08 in FIG. 10 are operations on the same screen.

[0016] In this embodiment, since showing images corresponding to all steps often makes the procedure manual difficult to understand, one of the features is that representative images corresponding to scenes are extracted. For this reason, images with diagonal lines have been removed from Figure 11. The representative image for each scene is selected by the representative image extraction process (see Figures 5 and 8), which will be described later.

[0017] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings as appropriate. <Overall structure> 1 is a block diagram showing the configuration and functions of a procedure manual creation device 100 according to an embodiment. The procedure manual creation device 100 includes a processing unit 10, a storage 20, an input unit 30, an output unit 40, etc. The procedure manual creation device 100 is a procedure manual creation device that creates and outputs a procedure manual from a video 50. The video 50 may be a video captured by a video camera, a mobile terminal, or the like, and is not particularly limited.

[0018] The processing unit 10 is composed of an image processing unit 11, a generation AI processing unit 14, a procedure manual creation unit 16, etc. The image processing unit 11 includes a procedure candidate frame extraction unit 12 that analyzes image data cut out from a video at predetermined time intervals based on the amount of image change and extracts procedure candidate frames, which are frames that mark divisions of the video, and a scene extraction unit 13 that analyzes image data cut out from the video based on the amount of image change and extracts scenes that mark divisions of the video.

[0019] The generation AI processing unit 14 includes a scene content output unit 15 that assigns candidate procedure frames in a scene extracted from the timestamp of each frame, inputs the candidate procedure frames into an image analysis tool (OCR), extracts information labels output by the image analysis tool, inputs the extracted information labels, prompt sentences, and candidate procedure frames into a language model (LLM), and outputs the work content and work procedures for each scene from the language model.

[0020] The procedure manual creation unit 16 extracts a representative work image from the procedure candidate frames assigned to the scene, and creates an overall work procedure manual that summarizes the scenes using the work content and work procedure of the output scene and the extracted work image.

[0021] The storage 20 stores a frame file group 21 processed by the processing unit 10, a manual template 23 for outputting a procedure manual, etc. The manual template 23 may be input via the input unit 30 when creating a procedure manual and stored in the storage 20.

[0022] 2 is a diagram showing an example of a procedure manual creation process S60 according to the embodiment. The processing unit 10 inputs a video file, which is a moving image 50, via the input unit 30 and extracts video metadata (process S61). The processing unit 10 divides the video file into frames, deletes overlapping images, and extracts text from the frames (process S62). Then, the processing unit 10 refers to the manual template 23, assigns representative images to the remaining "operations," and creates a procedure manual (process S63). Here, the "operations" are procedures extracted from the moving images in the video file.

[0023] 3 is a diagram showing an overview of process S70 from input to image processing. The processing unit 10 extracts time frames from a moving image 50 (e.g., an operation moving image) (process S71). The time frames are frames obtained by dividing an image at intervals of a specified time (e.g., a predetermined time). The processing unit 10 extracts an aggregated time frame that aggregates similar frames (process S72). Then, the processing unit 10 creates (extracts) procedure candidate frames, which are frames that separate the moving image, from the aggregated time frame (process S73).

[0024] On the other hand, the processing unit 10 extracts scene frames from the video 50 (process S75). As described above, a scene refers to a period of action at a certain location. Then, the processing unit 10 extracts aggregated scene frames that aggregate similar frames (process S76).

[0025] Then, the processing unit 10 assigns candidate procedure frames in the scene extracted from the timestamp of each frame, groups the images (process S77), inputs the candidate procedure frames into an image analysis tool (OCR), extracts the information labels output by the image analysis tool (process S78), and stores them in the storage 20 as an edited image file 24.

[0026] In process S73, the procedure candidate frame extraction unit 12 (see FIG. 1) creates (extracts) procedure candidate frames by selecting only frames in which the ROI (region of interest) or cursor movement amount or the image change amount is equal to or greater than a predetermined threshold from among frames extracted at predetermined time intervals. Specifically, the procedure candidate frame extraction unit 12 combines the image hash value and timestamp, and determines that images similar to multiple images within a recent predetermined time period are duplicates, while always leaving a representative image at regular intervals, thereby efficiently and strictly eliminating duplicates.

[0027] In process S75, the scene extraction unit 13 (see FIG. 1) uses a threshold value specified by an environmental variable to extract only frames with large scene changes using a filter. (1) Upload the video to FFmpeg, (2) Compare the statistical values ​​of pixel differences between previous and next frames to determine whether the scene score exceeds the value set in the environment variable. (3) Based on the judgment results, only frames with large scene changes are imaged. This completes the extraction of the scene frame. FFmpeg is free software for recording, converting, and playing video and audio. In this embodiment, scene changes are detected within a video and the frame at that timing is extracted. The frame is saved if it satisfies the ROI difference, cursor movement, and interval conditions from the most recent save. One of the features is that the time frame is divided into chunks using saved scene frames as boundaries.

[0028] Therefore, when extracting a scene frame, it is not necessary to compare the previous and next frames at the specified time interval of the time frame, but they may be the same.

[0029] 4 is a flowchart showing data processing S100 using the generation AI of the generation AI processing unit 14. Note that FIG. 4 includes part of the processing of the procedure manual creation unit 16. The generation AI processing unit 14 repeats the process of step S102 for each procedure candidate (process S101). In process S102, the generation AI processing unit 14 compares the information labels and UI element information of two consecutive procedure candidate frames, extracts an operation procedure (step), and stores it as a work procedure file (not shown). Details will be described later with reference to FIG. 6. Here, an example prompt is "Compare the information labels and UI element information of two consecutive procedure candidate frames, and extract an operation procedure."

[0030] The generation AI processing unit 14 repeats the process of step S104 for each scene (step S103). In step S104, the generation AI processing unit 14 generates a summary of each scene using the operation procedure extracted in the loop.

[0031] The generation AI processing unit 14 repeats the process of step S106 for each procedure candidate (step S105). In step S106, the generation AI processing unit 14 vectorizes the extracted work procedures, measures the similarity by comparing the previous and next procedures, and eliminates work procedures with high similarity as duplicates using a function. Details will be described later with reference to FIG. 7.

[0032] The generation AI processing unit 14 repeats the process of step S111 for each procedure candidate (step S 110) In step S111, the generation AI processing unit 14 determines the final work procedure to be merged by deleting the work procedures determined to be unnecessary by the generation AI.

[0033] The procedure manual creation unit 16 vectorizes the information labels of the work procedures and procedure candidate frames in the scene, selects the frame with the highest similarity as a representative, and generates a work image from the representative frame (process S112).The procedure manual creation unit 16 then aggregates the summaries of each scene and generates a project description of several sentences in a language model using the aggregated summaries as input (process S113).

[0034] For convenience of explanation, FIG. 5 will be postponed and the process of step S102 will be explained with reference to FIG. FIG. 6 is a diagram showing the relationship between frame files. FIG. 6 shows the scene frames, the times of the candidate procedure frames, and the state after unnecessary candidate procedure frames have been removed. Scene 1, Scene 2, Scene 3, etc. are extracted from the scene frames. Within Scene 1, candidate procedure 1 at time 1 and candidate procedure 2 at time 2 are extracted. Within Scene 2, candidate procedure 3 at time 4 is extracted. Within Scene 3, candidate procedure 4 at time 6 and candidate procedure 5 at time 8 are extracted.

[0035] In the above relationship, for the work procedure of scene 1, the operation contents of procedure candidate 1 and procedure candidate 2 are analyzed, and procedure candidate information is created that is organized as a series of operation procedures, Step 1, Step 2, Step 3, and Step 4.

[0036] Similarly, for the work procedure of scene 2, the operation contents of procedure candidate 2 and procedure candidate 3 are analyzed, and procedure candidate information is created that is organized as a series of operation steps, Step 5, Step 6, and Step 7.

[0037] Similarly, for the work procedures in scene 3, the operation details of candidate procedure 3 and candidate procedure 4 are analyzed, and candidate procedure information is created that is organized as a continuous operation procedure of Step 8, Step 9, Step 10, and Step 11. In scene 3, the operation details of candidate procedure 4 and candidate procedure 5 are further analyzed, and candidate procedure information is created that is organized as a continuous operation procedure of Step 12 and Step 13. That is, the previous and next procedure candidates are compared to generate a Step.

[0038] The scene content output unit 15 of this embodiment analyzes the operation contents included in two temporally adjacent procedure candidate frames to extract an operation procedure, thereby making it possible to extract an appropriate operation procedure. To explain the specific process, (1) The Python file retrieves the OCR text and UI element information for the target image pair from the database. (2) The acquired information is passed to the LLM (AI) along with a prompt. (3) LLM follows the prompts to compare the information in image 1 and image 2 and generate a steps array. (4) The Python file receives the step sequence (JSON format) returned from the LLM and saves it in the frame file group 21. Python is an open source programming language.

[0039] The process of step S106 will be described with reference to FIG. 7 is a diagram showing the process of eliminating duplication of candidate procedures. The scene content output unit 15 of the generation AI processing unit 14 outputs vectors by embedding the work procedures, compares the numerical values ​​before and after the work procedures, and determines and deletes procedures whose numerical values ​​are equal to or greater than a predetermined threshold as duplicate procedures, thereby reducing the number of work procedures.

[0040] The duplicate removal process example 120 in Figure 7 shows the relationship between text generated by LLM, vectorization using Embedding, comparison result values ​​(similarity scores), duplicate removal, and the remaining text. Vectorization of Steps 1, 2, 3, 4, and 5 using Embedding results in Vector-A, Vector-B, Vector-C, Vector-D, and Vector-E. The similarity score for Vector-A and Vector-B, which are before and after the work procedure, is calculated to be 0.85; the similarity score for Vector-B and Vector-C, which are before and after the work procedure, is 0.79; the similarity score for Vector-C and Vector-D, which are before and after the work procedure, is 0.90; and the similarity score for Vector-D and Vector-E, which are before and after the work procedure, is 0.61. In this case, duplicate removal occurs when the similarity score is equal to or greater than a predetermined threshold (e.g., 0.85). In other words, Step 2 and Step 4 are deleted, and the remaining sentences are Step 1, Step 3, and Step 5.

[0041] FIG. 5 is a flowchart showing the representative image extraction process S200 of the procedure manual creation unit 16. The procedure manual creation unit 16 repeats the processes of steps S202 to S204 for all scenes (step S201). The procedure manual creation unit 16 repeats the process of step S203 for all procedure candidates (step S202). In step S203, the procedure manual creation unit 16 vectorizes the image of each procedure candidate and compares it with the work procedure of the corresponding scene to determine the similarity. Then, the procedure manual creation unit 16 selects the image with the highest similarity as the representative image.

[0042] FIG. 8 is a diagram showing a specific example of the representative image extraction process. (1) The procedure manual creation unit 16 outputs a vector of the work procedure of the scene, (2) The information labels of the candidate procedure frames in the scene are vectorized using embedding and output. (3) Compare the information labels of the candidate procedure frames with the numerical values ​​of the vectors of the work procedures. (4) The frame with the highest numerical value obtained as a result of the comparison is selected as the representative frame of the procedure for the scene, and a working image is created from the representative frame of the procedure.

[0043] In the example of FIG. 8, the comparison result between vector D of procedure candidate 4 and the vector of the work procedure of the scene is 0.9, which is the highest numerical value of the comparison result and is therefore determined to be the representative image.

[0044] For each scene, the procedure manual creation unit 16 calculates the vector similarity (cosine similarity) between the scene text and the information labels of each frame image belonging to the same scene, and can select the image with the highest similarity as the representative image of that scene.

[0045] Figure 9 is a diagram showing an example of a work procedure manual. Figure 9 shows an example of the procedure for managing IAM (Identity and Access Management) user accounts within the AWS console. The work procedure manual in Figure 9 is composed of item numbers, work content, work procedures, work images (representative images), etc. Item numbers 1, 2, and 3 correspond to scene numbers.

[0046] Specifically, item 1 is the scene where "Open the URL in your browser, enter your authentication information, and log in." The steps for this scene are "1.1 Open your browser and access the URL," "1.2 Wait until the AWS Management Console login page appears," "1.3 Enter... in the "Username" field," "1.4 Enter your password in the "Password" field," and "1.5 Press the "Login" button." The representative image shows the login page of 1.2.

[0047] Item 2 is the scene where "Use the search bar to navigate to the IAM service page." The procedure for this scene is listed as "2.1 Open the AWS Management Console homepage, enter "IAM" in the search bar, and execute a search. Wait until the search results are displayed, and then check the results." The representative image shows the IAM input screen from 2.1.

[0048] Item 3 is the scene "Check the user's policies." The steps for this scene are: "3.1 Click on Users on the Dashboard.", "3.2 Enter "143" in the search bar on the Users tab and identify the user "admin143" from the list.", "3.3 On the user list page, wait until the user list is displayed. The list includes attributes such as "Username," "Pass," "Group," "Last Activity," "MFA," and "Not Password.", "3.4 Click on the name of user "admin143" to open the user details page.", and "3.5 Scroll down to the "Authorization Policies" section and check the policies associated with user "admin143." The representative image shows the input screen for the search bar in 3.2.

[0049] In this embodiment, the concept of a scene is introduced, and by summarizing the operations on each screen, it is possible to create a procedure manual that makes the overall operations easy for the user to understand. A scene refers to, for example, a section of an action in a certain location.

[0050] FIG. 12 is a diagram illustrating a hardware configuration of the procedure manual creation device according to the embodiment. A procedure manual creation device 1200 shown in Fig. 12 is, for example, one of the implementations shown in Fig. 1. The procedure manual creation device 100 includes a memory 1201, a processor 1202 (processing unit), a storage device 1203 (database) of a storage 20 such as a hard disk (HD), a communication unit 1204 such as a network interface card (NIC), and a user interface unit 1205. Note that while a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) may be considered as an example of the processor, other semiconductor devices may also be used as long as they are capable of executing predetermined processing.

[0051] Then, the program stored in the storage device 1203 is loaded into the memory 1201, and the loaded program is executed by the processor 1202. The procedure manual creation device 1200 may have a user interface unit 1205 including a display, a touch panel, a mouse, and a keyboard.

[0052] 13 is a diagram showing an example of the functional configuration of a procedure manual creation device according to an embodiment. As shown in FIG. 13, the procedure manual creation device 100 includes an image processing unit 11 (procedure candidate frame extraction unit 12, scene extraction unit 13), a generation AI processing unit 14 (scene content output unit 15), a procedure manual creation unit 16, etc. The procedure manual creation unit 16 aggregates the summaries of each scene and uses them as input to have the LLM generate a project description of several sentences.

[0053] Fig. 14 is a diagram showing another example of the work procedure manual. In Fig. 9, the example of the work procedure manual was explained based on the moving image on the display screen of the operation target device, but it is not limited to this. Fig. 14 explains the memory replacement work.

[0054] The work procedure manual in Fig. 14, like the work procedure manual in Fig. 9, is composed of item numbers, work content, work procedures, work images (representative images), etc. Item number 1, item number 2, item number 3, etc. correspond to scene numbers.

[0055] Specifically, item 1 is the scene of "Check that the 'IO' button is present on the back of the PC case," and the representative image shows an image of the back of the PC case. The on symbol for the 'IO' button is an 'I' (straight line), indicating that the circuit is closed and power is flowing, while the off symbol is an 'O' (circle), indicating that the circuit is disconnected. Item 2 is the scene of "Press the 'IO' button on the back panel of the PC case and unplug the cord," and the representative image shows an image of the cord unplugged.

[0056] Item 3 is a scene in which "the PC case is removed, the "FROZN" card is checked, the lock is released, and the image shown is the unlocking process." Item 4 is a scene in which "the "crukl Door Pro" card is inserted and the PC case is put on," and the image shown is the insertion of the card. Item No. 5 is a scene in which "insert the cord into the rear panel of the PC case and press the 'IO' button." The representative image shows the 'IO' button being pressed.

[0057] The procedure manual creation device 100 and procedure manual creation method of the present embodiment described above have the following features. (1) A procedure manual creation device for creating a procedure manual from a video, comprising: a procedure candidate frame extraction unit 12 that analyzes image data cut out from the video at predetermined time intervals based on image change amounts and extracts procedure candidate frames, which are frames that separate the video; a scene extraction unit 13 that analyzes image data cut out from the video based on image change amounts and extracts scenes that separate the video; a scene content output unit 15 that assigns procedure candidate frames in the extracted scenes based on the timestamps of each frame, inputs the procedure candidate frames into an image analysis tool (e.g., OCR), extracts information labels output by the image analysis tool, inputs the extracted information labels, prompt sentences, and procedure candidate frames into a language model (e.g., LLM), and outputs the work content and work procedures of each scene from the language model; and a procedure manual creation unit 16 that extracts representative work images from the procedure candidate frames assigned to the scenes and creates an overall work procedure manual that summarizes the scenes using the work content and work procedures of the output scenes and the extracted work images. This allows for appropriate creation of procedure manuals from video.

[0058] (2) In the procedure manual creation device of (1), the procedure manual creation unit 16 outputs vectors of work procedures for a scene, vectorizes information labels of procedure candidate frames in the scene by embedding, and outputs the vectors. The information labels of the procedure candidate frames are compared with the values ​​of the vectors of the work procedures, and the one with the higher value is used as a procedure representative frame for the scene. A work image is created from the procedure representative frame (see FIGS. 8 and 11). This makes it possible to create a procedure manual that is easy for the user to understand.

[0059] (3) In the procedure manual creation device of (1), the scene content output unit 15 vectorizes the work procedures by embedding and outputs them, compares the values ​​before and after the work procedures, and determines and deletes those whose values ​​are equal to or greater than a predetermined threshold as redundant procedures, thereby reducing the number of work procedures (see FIG. 7). This allows the user to create a concise procedure manual.

[0060] (4) In the procedure manual creation device of (1), the procedure manual creation unit 16 aggregates summaries of each scene and inputs the aggregated summaries to a language model to generate a project description of several sentences.

[0061] (5) In the procedure manual creation device of (1), the procedure manual creation unit 16 calculates the vector similarity between the scene text and the information labels of each frame image belonging to the scene for each scene, and selects the image with the highest similarity as the representative image of the scene (see FIGS. 8 and 11). This makes it possible to create a procedure manual that is easy for the user to understand.

[0062] (6) In the procedure manual creation device of (1), the procedure candidate frame extraction unit 12 can select frames extracted at predetermined time intervals, and leave only those frames in which the amount of movement of the region of interest or cursor, or the amount of change in the image, is equal to or exceeds a predetermined threshold value. This allows appropriate procedure candidate frames to be extracted.

[0063] (7) In the procedure manual creation device of (1), the scene extraction unit 13 can extract only frames with large scene changes using a filter, using a threshold value specified by an environmental variable. This allows appropriate scenes to be extracted.

[0064] (8) In the procedure manual creation device of (1), the scene content output unit 15 compares the information labels and UI element information of two consecutive procedure candidate frames to extract an operation procedure (see FIG. 6). This makes it possible to extract an appropriate operation procedure.

[0065] (9) A procedure manual creation method of a procedure manual creation device 100 that creates a procedure manual from a video, wherein the processing unit 10 of the procedure manual creation device 100 includes a procedure candidate frame extraction step of analyzing image data cut out from the video at predetermined time intervals based on image change amounts and extracting procedure candidate frames that are frames that separate the video, a scene extraction step of analyzing the image data cut out from the video based on image change amounts and extracting scenes that separate the video, a scene content output step of assigning procedure candidate frames in the extracted scenes based on the timestamps of each frame, inputting the procedure candidate frames to an image analysis tool, extracting information labels output by the image analysis tool, inputting the extracted information labels, prompt sentences, and procedure candidate frames to a language model, and outputting the work content and work procedures of each scene from the language model, and a procedure manual creation step of extracting representative work images from the procedure candidate frames assigned to the scenes and creating an overall work procedure manual that summarizes the scenes using the work content and work procedures of the output scenes and the extracted work images. This allows a procedure manual to be created appropriately from the video. [Explanation of symbols]

[0066] 10 Processing section 11 Image processing section 12 Procedure candidate frame extraction unit 13 Scene Extraction Unit 14 Generative AI processing section 15 Scene content output section 16 Procedure Manual Creation Department 20. Storage 21 Frame files 23 Manual Templates 30 Input section 40 Output section 50 Videos 100 Procedure manual creation device 1200 Procedure manual creation device 1201 memory 1202 Processor (processing unit) 1203 Storage 1204 Communications Department 1205 User Interface Section S60 Procedure creation process S70 Processing from input to image processing S100 Data Processing S200 Representative image extraction processing

Claims

1. A procedure manual creation device that creates a procedure manual from a video, a procedure candidate frame extraction unit that analyzes image data cut out from the moving image at predetermined time intervals based on an amount of image change and extracts procedure candidate frames that are frames that separate the moving image; a scene extraction unit that analyzes image data cut out from the moving image based on an amount of image change and extracts scenes that separate the moving image; a scene content output unit that assigns procedure candidate frames in the extracted scenes based on the timestamp of each frame, inputs the procedure candidate frames into an image analysis tool, extracts information labels output by the image analysis tool, compares the extracted information labels with information labels of two consecutive procedure candidate frames, inputs a prompt sentence for extracting an operation procedure and the procedure candidate frames into a language model, and outputs the work content and work procedures of each scene from the language model; and a procedure manual creation unit that extracts a representative work image from the procedure candidate frames assigned to the scene, and creates an overall work procedure manual that summarizes the scene using the work content and work procedure of the output scene and the extracted work image.

2. The procedure manual creation device according to claim 1, The procedure manual creation unit outputs a vector of a work procedure for the scene, vectorizes information labels of procedure candidate frames in the scene by embedding and outputs the vectorized information labels, compares the numerical values ​​of the information labels of the procedure candidate frames with the numerical values ​​of the vectors of the work procedures, and sets the one with the higher numerical value as a procedure representative frame for the scene, and creates the work image from the procedure representative frame. A procedure manual creation device characterized by:

3. The procedure manual creation device according to claim 1, The scene content output unit vectorizes the work procedures by embedding and outputs them, compares the values ​​before and after the work procedures, and determines and deletes those whose values ​​are equal to or greater than a predetermined threshold as duplicated steps, thereby reducing the number of work procedures. A procedure manual creation device characterized by:

4. The procedure manual creation device according to claim 1, The procedure manual creation unit aggregates summaries of each scene and uses the aggregated summaries as input to a language model to generate a project description of several sentences. A procedure manual creation device characterized by:

5. The procedure manual creation device according to claim 1, The procedure manual creation unit calculates the vector similarity between the scene text and the information labels of each frame image belonging to the scene, and selects the image with the highest similarity as the representative image of the scene. A procedure manual creation device characterized by:

6. The procedure manual creation device according to claim 1, The procedure candidate frame extraction unit selects frames extracted at each predetermined time interval and selects only frames in which the amount of movement of a region of interest or a cursor, or the amount of change in an image, is equal to or greater than a predetermined threshold. A procedure manual creation device characterized by:

7. The procedure manual creation device according to claim 1, The scene extraction unit uses a threshold value specified by an environment variable to extract only frames with large scene changes using a filter. A procedure manual creation device characterized by:

8. The procedure manual creation device according to claim 1, The scene content output unit compares information labels and UI element information of two consecutive procedure candidate frames to extract an operation procedure. A procedure manual creation device characterized by:

9. A procedure manual creation method for a procedure manual creation device that creates a procedure manual from a video, comprising: The processing unit of the procedure manual creation device includes: a procedure candidate frame extraction step of analyzing image data cut out from the moving image at predetermined time intervals based on an amount of image change, and extracting procedure candidate frames that are frames that separate the moving image; a scene extraction step of analyzing image data cut out from the moving image based on an amount of image change and extracting scenes that separate the moving image; a scene content output step of assigning a procedure candidate frame in the extracted scene from the timestamp of each frame, inputting the procedure candidate frame into an image analysis tool, extracting an information label output by the image analysis tool, comparing the extracted information label with the information labels of two consecutive procedure candidate frames, inputting a prompt sentence for extracting an operation procedure and the procedure candidate frame into a language model, and outputting the work content and work procedure of each scene from the language model; A procedure manual creation method comprising: a procedure manual creation step of extracting a representative work image from the procedure candidate frames assigned to the scene, and creating an overall work procedure manual that summarizes the scene using the work content and work procedure of the output scene and the extracted work image.

Images