MANUAL ANALYSIS DEVICE, MANUAL PROVIDING DEVICE, MANUAL READING SYSTEM, METHOD, AND PROGRAM

Abstract

To provide a manual analysis apparatus which allows a user to understand a manual without images of the manual, a manual providing apparatus, a manual comprehension system, a method and a program.SOLUTION: A manual comprehension support system 1 includes a manual analysis apparatus 10, a manual providing apparatus 20, and a user terminal 30. The manual analysis apparatus 10 analyzes a digitized manual so that the manual including text data and image data can be understood only with the text data. The manual providing apparatus 20 presents work procedure information compensated so that the digitized manual including the text data and the image data can be understood only with the text data, in response to a request from the user terminal 30.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a manual analysis device, a manual provision device, a manual reading system, a method, and a program. [Background technology]

[0002] Conventionally, various manuals (also called instruction manuals or user guides) are used in various situations (Patent Document 1).

[0003] For example, one of the maintenance tasks of customer engineers in the manufacturing industry includes repairing products at customer sites. One particularly common task is replacing product parts. If a customer engineer does not know the replacement procedure, he or she will replace the part in question while referring to the manual. This type of work is often performed in a small space and the customer engineer's hands often get dirty. For this reason, there is a need for voice confirmation that allows customer engineers to refer to the manual without having to change their posture significantly or touch it with dirty hands. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in addition to the text explaining the procedure, the manual contains images with superimposed marks indicating the positional relationship between the target part and the surrounding parts and the direction in which the target part should be moved or removed. Also, the surrounding parts shown in the images may not be mentioned in the text explaining the procedure. Therefore, simply reading out the text of the manual lacks information regarding the positional relationship between the target part and the surrounding parts and the direction in which the target part should be moved or removed, and it is not possible to present sufficient information for performing the work.

[0005] Therefore, an object of one embodiment of the present invention is to enable a user to understand a manual even without an image of the manual. [Means for solving the problem]

[0006] An apparatus according to one embodiment of the present invention includes a manual analysis unit that acquires training data, performs machine learning using the training data, and generates a machine learning model that outputs work procedure information when image data in a manual and text data in the manual are input as input data and work procedure information supplemented by adding text data generated based on the image data to the text data in the manual as output data. Effect of the Invention

[0007] In one embodiment of the present invention, a manual can be understood without an image of the manual. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram for explaining an overview of the present invention. [Diagram 2] 1 is an overall configuration diagram according to an embodiment of the present invention; [Diagram 3] 1 is a hardware configuration diagram of a manual analysis device, a manual providing device, and a user terminal according to an embodiment of the present invention. [Figure 4] 1 is a functional block diagram of a manual analysis device according to an embodiment of the present invention. [Diagram 5] 1 is a functional block diagram of a manual providing device according to an embodiment of the present invention; [Figure 6] 1 is a flowchart of a manual analysis process according to an embodiment of the present invention. [Figure 7] 11 is a flowchart of a manual providing process according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0010] <Terminology explanation> In this specification, a "manual (also called an instruction manual or user guide)" refers to an electronic manual that includes text data and image data. For example, the manual is a manual for repairing precision equipment such as a multifunction printer, but is not limited to this and may be any manual. For example, a customer engineer or the like can use the manual to replace parts of a precision equipment such as a multifunction printer in order to repair the precision equipment.

[0011] <Summary> Fig. 1 is a diagram for explaining an overview of the present invention. In the present invention, it is possible to present work procedure information that is supplemented by adding text data based on image data in a manual to the text data in the manual.

[0012] For example, suppose that the manual describes the procedure for a task as shown in Figure 1. In other words, the manual includes text data such as "1. Open the front cover [A]. 2. While pressing down the stopper, slide the front cover [A] to the right to remove it," and image data. The image data is superimposed with a mark indicating "[A]," a mark indicating the "main body," a mark indicating the "stopper," a mark indicating the direction in which to move the front cover, a mark indicating the direction in which to remove the front cover, and a mark indicating the direction in which to move the stopper.

[0013] In the manual, the target part (here, A) is indicated on the image data, and is described in the text data as "Front Cover [A]". Note that there may be cases where there is no description in the text data (i.e., only instructions by marks on the image data). The marks on the image data have multiple meanings and contain a lot of information such as the direction in which the part should be moved, the direction in which the part should be removed, instructions for the target part described in the text data, and the order of each procedure. In the present invention, machine learning is performed using training data in which image data (image data in the manual) containing such various information and text data in the manual are input data, and text data (hereinafter also referred to as work procedure information) that completely expresses the contents of the work is output data. Then, the work procedure information can be presented using a machine learning model generated by machine learning.

[0014] The following is an example of work instruction information.

[0015] Before (i.e., text data in the manual) 1. Open the front cover [A]. 2. While pressing down the stopper, slide the front cover [A] to the right to remove it.

[0016] After (i.e., work procedure information supplemented by adding text data based on images in the manual to the text data in the manual) 1. Open the front cover of the device towards you. 2. While pressing down on the stopper located slightly to the left of the center where the main body and front cover join, slide the front cover to the right to remove it.

[0017] As described above, it can be seen that the work procedure information is presented in a state where the positions of the target parts and peripheral parts, the operation direction, etc. are supplemented to the text data of "Before." Note that the wording [A] used when checking simultaneously with the image as in this example may be deleted from the work procedure information, or may be replaced with the wording indicated by [A].

[0018] <Overall configuration diagram> Fig. 2 is an overall configuration diagram according to an embodiment of the present invention. As shown in Fig. 2, the manual reading support system 1 can include a manual analysis device 10, a manual provision device 20, and a user terminal (customer engineer terminal) 30. The user terminal 30 can transmit and receive data to and from the manual provision device 20 via any network. Note that in this specification, the manual analysis device 10 and the manual provision device 20 are described as two devices, but the manual analysis device 10 and the manual provision device 20 may be implemented in one device.

[0019] The manual analysis device 10 analyzes a manual that has been digitized and contains text data and image data so that the manual can be understood only from the text data. Specifically, the manual analysis device 10 acquires training data in which the image data and text data in the manual are input data, and work procedure information supplemented by adding text data generated based on the image data in the manual to the text data in the manual is output data. The manual analysis device 10 also performs machine learning using the training data to generate a machine learning model that outputs work procedure information when the image data and text data in the manual are input.

[0020] In response to a request from a user terminal 30, the manual providing device 20 presents work procedure information that has been supplemented so that an electronic manual including text data and image data can be understood from the text data alone.

[0021] The manual providing device 20 may present work procedure information that has been generated in advance (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

[0022] When presenting work procedure information generated in advance, the manual providing device 20 acquires information (e.g., image data) for identifying an object (e.g., a part of a precision device such as a multifunction printer) from the user terminal 30. The manual providing device 20 also identifies at least one of image data and text data in the manual that corresponds to the information for identifying the object. The manual providing device 20 also identifies work procedure information generated in advance that is linked to at least one of image data and text data in the manual. The manual providing device 20 also outputs the work procedure information to the user terminal 30.

[0023] When generating and presenting work procedure information in response to a request from the user terminal 30, the manual providing device 20 acquires information (e.g., image data) for identifying an object (e.g., a part of a precision device such as a multifunction printer) from the user terminal 30. The manual providing device 20 also identifies image data and text data in the manual that correspond to the information for identifying the object. The manual providing device 20 also inputs the image data and text data in the manual into a machine learning model to output work procedure information. The manual providing device 20 also outputs the work procedure information to the user terminal 30.

[0024] The user terminal (customer engineer terminal) 30 is a terminal used by a person who uses a manual according to one embodiment of the present invention (for example, a customer engineer who repairs precision equipment such as a multifunction printer). Specifically, the user terminal 30 transmits information (for example, image data) for identifying an object to the manual providing device 20. The user terminal 30 also receives work procedure information for the object from the manual providing device 20. For example, the user terminal 30 is a smart device, smart glasses, etc.

[0025] The user terminal (customer engineer terminal) 30 may use the work procedure information stored in the user terminal (customer engineer terminal) 30.

[0026] <Hardware configuration> FIG. 3 is a diagram showing the hardware configuration of the manual analysis device 10, manual providing device 20, and user terminal 30 according to an embodiment of the present invention.

[0027] As shown in FIG. 3, the manual analysis device 10, manual provision device 20, and user terminal 30 are constructed by computers, and as shown in FIG. 3, are equipped with a CPU 1001, a ROM 1002, a RAM 1003, a HD 1004, a HDD (Hard Disk Drive) controller 1005, a display 1006, an external device connection I / F (Interface) 1007, a network I / F 1008, a data bus 1009, a keyboard 1010, a pointing device 1011, a DVD-RW (Digital Versatile Disk Rewritable) drive 1013, and a media I / F 1015.

[0028] Among these, the CPU 1001 controls the overall operation of the manual analysis device 10, the manual provision device 20, and the user terminal 30. The ROM 1002 stores a program such as an IPL used to drive the CPU 1001. The RAM 1003 is used as a work area for the CPU 1001. The HD 1004 stores various data such as programs. The HDD controller 1005 controls the reading or writing of various data from the HD 1004 under the control of the CPU 1001. The display 1006 displays various information such as a cursor, a menu, a window, a character, or an image. The external device connection I / F 1007 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I / F 1008 is an interface for data communication using a communication network. The bus line 1009 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 1001 shown in FIG. 3.

[0029] The keyboard 1010 is a type of input means having a plurality of keys for inputting characters, numbers, various instructions, etc. The pointing device 1011 is a type of input means for selecting and executing various instructions, selecting a processing target, moving a cursor, etc. The DVD-RW drive 1013 controls reading and writing of various data from a DVD-RW 1012 as an example of a removable recording medium. Note that the medium is not limited to a DVD-RW, and may be a DVD-R, etc. The media I / F 1015 controls reading and writing (storing) of data from a recording medium 1014 such as a flash memory.

[0030] <Function block> Hereinafter, the functional blocks of the manual analysis device 10 will be described with reference to FIG. 4, and the functional blocks of the manual providing device 20 will be described with reference to FIG.

[0031] Fig. 4 is a functional block diagram of a manual analysis device 10 according to an embodiment of the present invention. As shown in Fig. 4, the manual analysis device 10 includes an in-manual image data storage unit 101, an in-manual text data storage unit 102, and a manual analysis unit 103. Moreover, the manual analysis device 10 functions as the manual analysis unit 103 by executing a program. Each of these will be described below.

[0032] The manual image data storage unit 101 stores image data included in the manual. For example, the image data is electronic data such as photographs and illustrations showing the procedure of the work. On the image data, a mark indicating the target part (a part to be worked on, for example, a part to be replaced by a customer engineer), a mark indicating a peripheral part, a mark indicating the direction in which the part should be moved, a mark indicating the direction in which the part should be removed, etc. may be superimposed.

[0033] The manual text data storage unit 102 stores the text data included in the manual.

[0034] The manual analysis unit 103 executes a manual analysis process. Specifically, the manual analysis unit 103 acquires image data stored in the manual image data storage unit 101 and text data stored in the manual text data storage unit 102. The manual analysis unit 103 also acquires complemented text data (text data complemented by text data generated based on image data in the manual). The manual analysis unit 103 also associates "image data in the manual and text data in the manual" with "complemented text." The manual analysis process will be described in detail with reference to FIG. 6.

[0035] Fig. 5 is a functional block diagram of a manual providing device according to one embodiment of the present invention. As shown in Fig. 5, manual providing device 20 includes object data input section 201, work procedure information selection section 202, work procedure information output section 203, manual image data storage section 101, and manual text data storage section 102. Moreover, manual providing device 20 executes a program to function as object data input section 201, work procedure information selection section 202, and work procedure information output section 203. Each of these will be described below.

[0036] The manual providing device 20 may use the in-manual image data storage unit 101 and the in-manual text data storage unit 102 of the manual analysis device 10.

[0037] The object data input unit 201 acquires, from the user terminal 30, information (eg, image data, voice, text, etc.) for identifying an object (eg, a part of a precision device such as a multifunction peripheral).

[0038] The work procedure information selection unit 202 may present work procedure information that has been generated in advance (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

[0039] When presenting work procedure information generated in advance, the work procedure information selection unit 202 identifies at least one of image data and text data in the manual that corresponds to information for identifying the object acquired by the object data input unit 201. In addition, the work procedure information selection unit 202 identifies work procedure information generated in advance that is linked to at least one of image data and text data in the manual.

[0040] When generating and presenting work procedure information in response to a request from the user terminal 30, the work procedure information selection unit 202 identifies image data and text data in the manual that correspond to information for identifying the object acquired by the object data input unit 201. In addition, the work procedure information selection unit 202 inputs the image data and text data in the manual to a machine learning model to output work procedure information.

[0041] The work procedure information output unit 203 outputs the work procedure information to the user terminal 30. For example, the work procedure information output unit 203 transmits audio data of the work procedure information to the user terminal 30.

[0042] <Processing method> Hereinafter, the manual analysis process will be described with reference to FIG. 6, and the manual provision process will be described with reference to FIG.

[0043] FIG. 6 is a flowchart of a manual analysis process according to an embodiment of the present invention.

[0044] In step 11 (S11), the manual analysis unit 103 acquires image data stored in the in-manual image data storage unit 101 and text data stored in the in-manual text data storage unit 102. This image data and text data become training data.

[0045] In step 12 (S12), the manual analysis unit 103 acquires the complemented text data (text data complemented by text data generated based on image data in the manual). This complemented text data becomes training data. For example, a system administrator or the like creates the complemented text data.

[0046] In step 13 (S13), the manual analysis unit 103 associates "image data in the manual and text data in the manual" with "complemented text."

[0047] Specifically, the manual analysis unit 103 acquires training data in which the image data and text data in the manual acquired in S11 are input data, and the complemented text data acquired in S12 (i.e., work procedure information complemented by adding text data generated based on image data to the text data in the manual) is output data, and performs machine learning using the training data. Then, the manual analysis unit 103 generates a machine learning model that outputs work procedure information when the image data and text data in the manual are input. For example, the manual analysis unit 103 can use VQA (visual question answering), which is an example of a multimodal (image and text) method, as a machine learning method.

[0048] The following is an example of text data generated based on image data in a manual. For example, the text data generated based on image data in a manual relates to marks (symbolic information) included in the image data in the manual, such as the positional relationship between the target part (the part that is the target of the work, e.g., the part that the customer engineer replaces) and surrounding parts, the direction in which the target part should be moved, and the direction in which the target part should be removed. -What aspect does the target part fit from the customer engineer's perspective? -Which side of that is it? What colour? Where do you move it from (what do you do)? In which direction do you want to move it? others What shape is it? How big is it? How many are there?

[0049] FIG. 7 is a flowchart of a manual providing process according to one embodiment of the present invention.

[0050] In step 21 (S21), the object data input unit 201 acquires information (e.g., image data) for identifying the object. Specifically, the object data input unit 201 acquires information (e.g., image data) for identifying the object from the user terminal 30. For example, the object is an object on which a customer engineer works (e.g., a part that the customer engineer replaces). Smart glasses worn by the customer engineer or the like may capture an image and transmit it to the manual providing device 20, or a smart device or the like may capture an image and transmit it to the manual providing device 20 via an interface such as a chatbot or an input box in a web browser.

[0051] As described above, the work procedure information selection unit 202 may present work procedure information that has been generated in advance (which may be stored in any storage device), or may generate and present work procedure information in response to a request from the user terminal 30.

[0052] When presenting pre-generated work procedure information, in step 22 (S22), the work procedure information selection unit 202 acquires at least one of image data and text data in the manual. Specifically, the work procedure information selection unit 202 identifies at least one of image data and text data in the manual that corresponds to information for identifying the object in S201. For example, the work procedure information selection unit 202 identifies the object included in the image data in S201, and acquires at least one of image data and text data in the manual that describes the object.

[0053] When generating and presenting work procedure information in response to a request from the user terminal 30, in step 22 (S22), the work procedure information selection unit 202 acquires image data and text data in the manual. Specifically, the work procedure information selection unit 202 identifies image data and text data in the manual that correspond to information for identifying the object in S201. For example, the work procedure information selection unit 202 identifies the object included in the image data in S201, and acquires image data and text data in the manual that explain the object.

[0054] In step 23 (S23), the work procedure information selection unit 202 complements the text data.

[0055] When presenting work procedure information that has been generated in advance, in S23, the work procedure information selection unit 202 identifies work procedure information that has been generated in advance and that is linked to at least one of the image data in the manual identified in S22 and the text data in the manual.

[0056] When generating and presenting work procedure information in response to a request from the user terminal 30, in S23, the work procedure information selection unit 202 inputs the image data and text data in the manual identified in S22 into a machine learning model (the machine learning model generated by the manual analysis process in Figure 6) and outputs the work procedure information.

[0057] In one embodiment of the present invention, the work procedure information (i.e., the text data after completion) contains part names obtained from the text data in the manual (i.e., the text data before completion), so by comparing "that part name" with "the part name obtained from information for identifying the object in S21 (by captioning, voice recognition, etc.)", it is possible to select work procedure information without going through the text data in the manual.

[0058] In step 24 (S24), the work procedure information output unit 203 outputs the work procedure information. Specifically, the work procedure information output unit 203 outputs the work procedure information to the user terminal 30. For example, the work procedure information output unit 203 outputs the work procedure information by voice. Note that the work procedure information output unit 203 may display the work procedure information on a smart device, display the work procedure information in AR (Augmented Reality) on smart glasses, or the like.

[0059] <Effects> Thus, in one embodiment of the present invention, a customer engineer or the like can hear audio of work procedure information in which text based on images in the manual is added to the text in the manual. Therefore, the customer engineer or the like can understand the manual only from the text information even without images. As a result, the customer engineer or the like can work while understanding the repair procedures for precision equipment such as a multifunction printer without changing his / her position to look at the manual or holding the manual with dirty hands.

[0060] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above.

[0061] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to the specific embodiments described above, and various modifications and variations are possible within the scope of the gist of the present invention as defined in the claims. [Explanation of symbols]

[0062] 1. Manual Reading Support System 10 Manual Analysis Device 20 Manual provision device 30 User terminal (customer engineer terminal) 101 Image data storage section in manual 102 Text data storage section in manual 103 Manual Analysis Section 201 Object data input section 202 Work procedure information selection section 203 Work procedure information output section [Prior art documents] [Patent documents]

[0063] [Patent Document 1] JP 2002-269147 A

Claims

1. acquiring training data, the input data being image data in a manual and text data in the manual, and the output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual; Perform machine learning using the training data; a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are inputted; An apparatus comprising:

2. The apparatus according to claim 1 , wherein the text data generated based on the image data indicates a positional relationship between a target part and surrounding parts, a direction in which the target part is to be moved, and a direction in which the target part is to be removed.

3. 3. The apparatus according to claim 1, wherein the manual is a manual for a multifunction peripheral.

4. A manual reading support system including a manual analysis device, a manual provision device, and a user terminal, The manual analysis device includes: acquiring training data, the input data being image data in a manual and text data in the manual, and the output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual; Perform machine learning using the training data; a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are input; The manual providing device includes: an object data input unit that acquires information for identifying an object from the user terminal; a work procedure information selection unit that identifies image data and text data in a manual that correspond to information for identifying the object, and inputs the image data and the text data in the manual into the machine learning model to output work procedure information; a work procedure information output unit that outputs the work procedure information to the user terminal.

5. 1. A computer-implemented method comprising: acquiring training data, the input data being image data in a manual and text data in the manual, and the output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual; A step of performing machine learning using the training data; generating a machine learning model that outputs work procedure information when image data and text data in the manual are input; The method includes:

6. Computer acquiring training data, the input data being image data in a manual and text data in the manual, and the output data being work procedure information supplemented by adding text data generated based on the image data to the text data in the manual; Perform machine learning using the training data; a manual analysis unit that generates a machine learning model that outputs work procedure information when image data and text data in the manual are inputted; A program to function as a

Images