Data processor, method for processing data, and data processing program

Abstract

To provide an AI which can properly machine-learn the content of the instruction manual of a product.SOLUTION: A data processor 12 includes: a recognition unit 310 for reading image information on the instruction manual of a product and recognizing plural pieces of letter information; a provision unit 312 for numbering the plural pieces of letter information so that the letter information can be understood as a sentence; a re-configuration unit 316 for understanding the meaning of the letter information represented as collections of words of the plural pieces of letter information and re-configuring the letter information as a consistent sentence; a generation unit 318 for generating an instruction manual DB 70 including the numbered plural pieces of letter information and the reconfiguration information obtained by re-configuration; and a learning unit 320 for causing a generation AI to learn the content of the instruction manual DB 70.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The technology disclosed herein relates to a data processing device, a data processing method, and a data processing program. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] By the way, the instruction manuals for various household products are different from general books in that the letters and sentences are arranged in a way that is easy for humans to understand, making it difficult for AI to machine-learn them. For this reason, even if AI is trained on the contents of the instruction manual as is, there is a possibility that it will learn incorrect interpretations.

[0005] The present disclosure aims to provide a data processing device, a data processing method, and a data processing program that enable AI to appropriately machine-learn the contents of product instruction manuals. [Means for solving the problem]

[0006] A first aspect of the technology disclosed herein is a data processing device comprising: a recognition unit that reads image information described in a product instruction manual and recognizes multiple pieces of character information; an assignment unit that assigns an order to the multiple pieces of character information so that the meaning of the multiple pieces of character information can be understood as a sentence; a reconstruction unit that understands the meaning of character information expressed as a list of words among the multiple pieces of character information and reconstructs it into a coherent sentence; a generation unit that generates an instruction manual database including the multiple pieces of character information with the order assigned and the reconstructed information obtained by the reconstruction; and a learning unit that trains a generation AI to learn the contents of the instruction manual database.

[0007] A second aspect of the technology disclosed herein is the first aspect, further comprising a grouping unit that groups the multiple pieces of character information into predetermined units, and the instruction manual database further includes grouping information obtained by the grouping.

[0008] A third aspect of the technology disclosed herein is a device according to the first aspect, wherein the product is an electrical appliance, and the instruction manual includes an instruction manual section regarding an operating device for operating the electrical appliance, the instruction manual section including a display name of a button on the operating device and an explanatory section for the button, and further includes a matching section that matches first character information representing the display name of the button recognized by the recognition section with second character information representing the explanatory section of the button, and a creation section that understands meanings of the matched first character information and second character information and creates an explanatory sentence regarding the button, wherein the generation section generates an operation database including the first character information, the second character information, and the explanatory sentence, and the learning section causes the generation AI to learn the contents of the operation database.

[0009] A fourth aspect of the technique disclosed herein is the third aspect, wherein the operation database further includes information representing an operation signal corresponding to the button provided on the operation device.

[0010] A fifth aspect of the technology disclosed herein is the fourth aspect, wherein the operating device is a remote control device for remotely operating the electrical appliance, and the operating signal is an infrared signal or a wireless signal of the remote control device.

[0011] A sixth aspect of the technology disclosed herein is that in the first aspect, the generation AI determines the user's literacy level for the product based on the user's usage history of the product, and outputs an explanation tailored to the literacy level based on the contents of the instruction manual database.

[0012] A seventh aspect of the technology disclosed herein is that in the first aspect, the generation AI identifies the type, manufacturer, and model of the product from an image obtained by photographing the product or from a dialogue with a user of the product, and outputs a description corresponding to the identified type, manufacturer, and model of the product based on the contents of the instruction manual database.

[0013] An eighth aspect of the technology disclosed herein is that in the first aspect, the generation AI determines the user's work status from a captured image obtained by photographing the user's work status on the product, and outputs audio guidance on the next task that the user should perform in accordance with the determined work status of the user based on the contents of the instruction manual database.

[0014] A ninth aspect of the technology disclosed herein is that in the first aspect, when the generation AI detects dangerous wiring or usage of the product from a captured image obtained by photographing the installation environment in which the product is installed, it turns off the power of the product and outputs a message to a user of the product.

[0015] A tenth aspect of the technology disclosed herein is that in the first aspect, the generation AI learns the correspondence between the living environment of a user of the product and the maintenance conditions of the product, and outputs the maintenance conditions of the product according to the living environment of the user of the product.

[0016] An eleventh aspect of the technology disclosed herein is that in the first aspect, the generation unit further generates a comfort database including operation information of a user of the product, living environment information of the user, health care information of the user, and outdoor environment information of the user, and the learning unit causes the generation AI to learn the contents of the comfort database, and the generation AI outputs an answer for improving the comfort of the user based on the contents of the comfort database.

[0017] A twelfth aspect of the technology disclosed herein is, in the eleventh aspect, that when the user's comfort level improves as a result of the answer, the learning unit updates the comfort level database of similar users who are similar to the user.

[0018] A thirteenth aspect of the technology of the present disclosure is a data processing method in which a data processing device reads image information described in a product instruction manual, recognizes a plurality of pieces of character information, assigns an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence, understands the meaning of character information expressed as a list of words among the plurality of pieces of character information, and reconstructs it into a coherent sentence, generates an instruction manual database including the plurality of pieces of character information with the order assigned and reconstructed information obtained by the reconstruction, and has a generation AI learn the contents of the instruction manual database.

[0019] A fourteenth aspect of the technology of the present disclosure is a data processing program that causes a computer to execute the following processes: read image information described in a product instruction manual, recognize multiple pieces of character information, assign an order to the multiple pieces of character information so that the meaning of the multiple pieces of character information can be understood as a sentence, understand the meaning of character information expressed as a list of words among the multiple pieces of character information, reconstruct the information into a coherent sentence, generate an instruction manual database including the multiple pieces of character information with the order assigned and reconstructed information obtained by the reconstruction, and have a generation AI learn the contents of the instruction manual database. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a conceptual diagram illustrating an example of a configuration of a data processing system. [Diagram 2] FIG. 2 is a conceptual diagram showing an example of main functions of a data processing device and a smart device. [Diagram 3] 2 is a diagram illustrating an outline of a functional configuration of a specific processing unit of the data processing device. FIG. [Figure 4] FIG. 10 is a diagram illustrating an example of an operational flow of a specific process by a data processing device. [Diagram 5] 1 is a block diagram showing an example of a functional configuration of a data processing device according to an embodiment. [Figure 6] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 7] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 8] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 9] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 10] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 11] FIG. 13 is a diagram for explaining a process for generating an instruction manual DB. [Figure 12] FIG. 13 is a diagram illustrating a process for generating an operation DB. [Figure 13] FIG. 13 is a diagram illustrating a process for generating an operation DB. [Figure 14] 13A and 13B are diagrams illustrating a process for generating an operation DB. [Figure 15] 13 is a diagram for explaining a form in which image data of a main body of an electrical appliance or a remote control device is combined. FIG. [Figure 16] FIG. 13 is a diagram for explaining a setup support function for an electrical product using a data generation model. [Figure 17] 13 is a flowchart showing an example of the flow of an electrical product identification process using a data generation model. [Figure 18]13(A) and 13(B) are diagrams for explaining a form in which explanatory portions corresponding to buttons on a remote control device are displayed. [Figure 19] 13 is a diagram for explaining a form in which an instruction manual notifies a user of an updated function. FIG. [Figure 20] FIG. 1 is a diagram illustrating a configuration in which the AR function of a smart device is combined with audio guidance. [Figure 21] FIG. 1 is a diagram illustrating a configuration in which the AR function of a smart device is combined with audio guidance. [Figure 22] FIG. 1 is a diagram illustrating a configuration in which the AR function of a smart device is combined with audio guidance. [Diagram 23] 10 is a diagram illustrating a configuration in which the power supply of an electrical appliance is automatically turned off when the wiring or the like of the electrical appliance is in a dangerous state. FIG. [Figure 24] 13 is a diagram for explaining a form for instructing a maintenance method for an electrical appliance. FIG. [Diagram 25] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Figure 26] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Figure 27] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Figure 28] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Figure 29] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Diagram 30] 10 is a diagram for explaining a form in which a user comfort DB is constructed and the comfort DB is utilized. FIG. [Diagram 31] FIG. 13 is a diagram for explaining a form in which a data generation model is used to detect power consumption in a home and change settings of electrical appliances. [Diagram 32] FIG. 13 is a diagram for explaining a form in which a solution to an error code or an error message of an electrical appliance is automatically updated. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, an example of an embodiment of a data processing device, a data processing method, and a program according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (Tensor Processing Unit).

[0024] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0025] In the following embodiments, the storage with the code is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., Examples include hard disks, or magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the embodiment.

[0029] As shown in Fig. 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server. An example of the smart device 14 is a smartphone. In this embodiment, the data processing device 12 is an example of a "data processing device" according to the technology of the present disclosure.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by the touch of the pointer. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the person 20 by outputting the data in a form of expression that the person 20 can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a Complementary Metal-Oxide-Semiconductor (CMOS) image sensor or a Charge Coupled Device (CCD) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0036] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58. The data generation model 58 is used by the specific processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 62. The reception output program 62 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 62 from the storage 50, and executes the read reception output program 62 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 62 executed on the RAM 48.

[0039] Next, we will explain the processing of the specific processing unit 290 when the data processing device 12 performs specific processing that enables the operation of various products used in the home using an AI that has learned the instruction manual (also called the "product manual" or "operation manual").

[0040] As shown in FIG. 3, the specific processing unit 290 includes an input unit 292, a processing unit 294, and an output unit 296.

[0041] The input unit 292 acquires user input received by the smart device 14. Specifically, the input unit 292 acquires at least one of text, voice, and image data of the user received by the smart device 14.

[0042] The processing unit 294 performs a specific process using the data generation model 58. Specifically, character, voice, and image data input by the user is input to the data generation model 58, and a generation result is obtained.

[0043] The output unit 296 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0044] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0045] Next, the operation of the data processing system 10 will be described.

[0046] An example of the flow of the specific processing will be described with reference to Fig. 4. Note that the flow of the specific processing shown in Fig. 4 is an example of a "data processing method" according to the technology of the present disclosure.

[0047] In step S300, the processing unit 294 determines whether a predetermined trigger condition is met.

[0048] If the trigger condition is met in step S300 (step S300; Yes), the data processing system 10 proceeds to step S301. On the other hand, if the trigger condition is not met in step S300 (step S300; No), the data processing system 10 ends the identification process.

[0049] In step S301, the processing unit 294 generates a prompt by adding an instruction for obtaining a result of a specific process to the text representing the input.

[0050] In step S303, the processing unit 294 inputs the generated prompt to the data generation model 58, and obtains the result of the specific process based on the output of the data generation model 58.

[0051] In step S304, the output unit 296 outputs the result of the identification process to the user terminal, and the identification process ends.

[0052] The following provides additional information about the identification process of this embodiment. The identification process of this embodiment has the following functions in order to achieve the following objectives. In this embodiment, an electrical product is used as an example of a product, but the identification process of this embodiment may be applied to products other than electrical products.

[0053] In other words, purchasing new IoT devices places a heavy burden on household finances. One possible solution is to make the remote controls for existing electrical appliances IoT-enabled, but conventional learning and smart remote controls are too complicated and multifunctional, making it difficult to operate electrical appliances easily.

[0054] In addition, the instruction manual of an electrical appliance describes various methods for making the most of the electrical appliance. However, not all users necessarily read the instruction manual thoroughly. Therefore, in this embodiment, the performance of the electrical appliance is enhanced by having the data generation model 58, which is a generation AI, learn the contents of the instruction manual, and furthermore, a more comfortable usage environment is provided by combining multiple electrical appliances.

[0055] In addition, for general books, although there are minor differences in vertical or horizontal writing, character size, number of characters per line, etc., the format is standardized to a certain extent. In contrast, instruction manuals for electrical appliances are different from general books in that they are heavily influenced by corporate concepts and product designs, and characters and sentences are arranged with diagrams and illustrations, and vertical and horizontal writing are mixed on each page, and there is no fixed format, making the layout difficult for the data generation model 58 to machine learn. For this reason, even if the contents of the instruction manual are simply processed by OCR (Optical Character Recognition) and the data generation model 58 is machine-learned as it is, it may not be able to correctly recognize vocabulary and context, and may learn with an incorrect interpretation.

[0056] In response to this, the data processing device 12 according to the present embodiment allows the data generation model 58 to appropriately perform machine learning on the contents of the instruction manual for the electrical product. Specifically, the processing unit 294 functions as each unit shown in FIG.

[0057] FIG. 5 is a block diagram showing an example of the functional configuration of the data processing device 12 according to the present embodiment.

[0058] As shown in FIG. 5, the processing unit 294 of the data processing device 12 according to this embodiment functions as a recognition unit 310, an assignment unit 312, a grouping unit 314, a reconstruction unit 316, a generation unit 318, a learning unit 320, an association unit 322, and a creation unit 324.

[0059] The recognition unit 310 reads image information described in the instruction manual for the electrical product and recognizes multiple pieces of character information. Specifically, for example, the recognition unit 310 performs OCR processing on the instruction manual and recognizes the character information.

[0060] For example, the assigning unit 312 assigns an order to the plurality of pieces of character information recognized by the recognition unit 310 so that the meaning of the plurality of pieces of character information can be understood as a sentence. Specifically, for example, the assigning unit 312 may accept an input of the order from a user, or may assign the order using a generation AI.

[0061] The grouping unit 314 groups a plurality of pieces of character information recognized by the recognition unit 310 into predetermined units. The "unit" here may be, for example, a meaningful word or a certain coherent explanatory part, or a larger unit such as a chapter or a section.

[0062] The reconstruction unit 316 understands the meaning of character information represented as a list of words among the multiple pieces of character information recognized by the recognition unit 310, and reconstructs the information into a coherent sentence. Specifically, for example, the reconstruction unit 316 narrows the range of character information by using prompt engineering, takes the OCR recognition result as an input into the generation AI, and reconstructs the content output from the generation AI into a coherent sentence.

[0063] The generating unit 318 generates an instruction manual database (instruction manual DB) 70 including the plurality of pieces of character information to which the order has been assigned and the reconstructed information obtained by the reconstruction. The instruction manual DB 70 generated by the generating unit 318 is stored in the storage 32.

[0064] The learning unit 320 causes the generation AI to learn the contents of the instruction manual DB 70, and generates the data generation model 58. The data generation model 58 generated by the learning unit 320 is stored in the storage 32.

[0065] Next, the process of generating the instruction manual DB 70 will be specifically described with reference to FIGS.

[0066] First, when the instruction manual shown in Figure 6 is subjected to OCR processing, it is recognized as multiple pieces of character information, as shown in Figure 7. The part surrounded by a rectangle in Figure 7 indicates what has been recognized as character information.

[0067] As shown in Figures 6 and 7, instruction manuals are different from general books in that they do not simply provide text information, but are arranged in a specific way to make them easy for people to understand. Therefore, when reading with OCR, multiple pieces of scattered text information on the page are read irregularly, which creates the risk of misunderstanding the meaning of the text. To prevent this, it is necessary to accurately specify the reading order of the scattered text information.

[0068] Next, as shown in Figure 8, the OCR-processed instruction manual is given an appropriate order for the text information arranged to be easy for people to understand. In the example in Figure 8, the order is represented by numbers in circles. If the OCR reading units are insufficient, it is desirable to clarify the order by clearly indicating the start and end points. Through this procedure, when the instruction manual is converted into text information, it is organized so that it makes sense as a sentence.

[0069] Next, as shown in FIG. 9, since each page does not form a single sentence, grouping is performed, for example, by explanation unit. In the example of FIG. 9, character parts numbered 5 to 8 are grouped as a group G1. At this time, it is desirable to consider cases where grouping is required in larger units such as chapters or sections. This allows further grouping to be performed and misinterpretation of the sentences to be prevented. Note that, when the amount of training data increases, the generation AI can also learn these processes, making it possible to perform grouping automatically.

[0070] Here, there are some parts of the instruction manual that are expressed as a list of words rather than complete sentences. When such content is read as text using OCR, the readability of the text may be low. For this reason, the text information is reconstructed using a generation AI, as shown in Figure 10. The example in Figure 10 shows the output from the generation AI in response to the input to the generation AI. Specifically, as described above, prompt engineering is used to narrow down the range of text information, and the OCR recognition results are taken into the generation AI as input. The content output by the generation AI is then reconstructed as coherent text.

[0071] Next, as shown in FIG. 11, an instruction manual DB70 is generated, which includes a plurality of pieces of character information to which orders have been assigned, reconstructed information obtained by reconstruction, and grouped information obtained by grouping. In the example of FIG. 11, a plurality of pieces of character information to which orders have been assigned corresponds to "OCR reading", reconstructed information obtained by reconstruction corresponds to "semantic understanding by the generation AI", and grouped information obtained by grouping corresponds to "grouping". In the example of FIG. 11, character parts to which orders have been assigned, 5 and 6, are grouped together as a group (group number 5), and character parts to which orders have been assigned, 7 and 8, are grouped together as a group (group number 6). Furthermore, character parts to which group numbers 5 and 6 have been assigned are grouped together as a group (group number 7).

[0072] The generation AI performs machine learning on the contents of the instruction manual DB 70 generated as described above, thereby generating a data generation model 58.

[0073] Next, the process of generating the operation DB 72 will be specifically described with reference to Fig. 12, Fig. 13, Fig. 14(A), and Fig. 14(B). The instruction manual includes an instruction section regarding an operation device for operating an electrical appliance, and the instruction section includes display names of buttons provided on the operation device and explanations of the buttons. The operation device is, for example, a remote control device. In the following, a remote control device will be described as an example of the operation device, but the operation device is not limited to a remote control device and may be an operation panel of a main body, etc.

[0074] As an example, as shown in Fig. 12, the association unit 322 (see Fig. 5) associates the first character information representing the display name of the button with the second character information representing the explanatory portion of the button, both of which are recognized by the recognition unit 310. For example, a generation AI is used for this association.

[0075] As shown in Fig. 12, when associating the display names of the buttons on the remote control device written in the instruction manual with the explanatory parts of the buttons on the remote control device written in the instruction manual, the function explanation text shown for each button on the remote control device is associated with the button by adding a corresponding line (or arrow). The display names of the buttons are also read at the same time, but since the display names are not necessarily unique, it is desirable to assign a unique ID (button ID) separately.

[0076] The creation unit 324 (see FIG. 5) understands the meaning of the first character information and the second character information associated by the association unit 322, and creates an explanatory text about the button. For example, a generation AI is used to create the explanatory text.

[0077] 13, the generating unit 318 generates an operation DB 72 including first character information, second character information, explanatory text, and information representing an operation signal corresponding to a button provided on a remote control device, and stores the operation DB 72 in the storage 32. The operation signal is, for example, an infrared signal or a wireless signal of the remote control device. In the example of FIG. 13, the first character information corresponds to "button display name", the second character information corresponds to "button explanation / OCR", the explanatory text corresponds to "meaning understanding by generation AI", and the information representing the operation signal corresponding to the button provided on the remote control device corresponds to "infrared communication".

[0078] The learning unit 320 makes the generation AI learn the contents of the operation DB 72, and generates the data generation model 58. After associating the button ID of each button on the remote control device with the text read by OCR, the generation AI learns the explanatory content and understands the meaning of the function. In addition, the generation AI also learns the infrared signal (or wireless signal) corresponding to each button on the remote control device. This allows the generation AI to properly understand operation instructions from the user, and enables the generation AI to operate electrical appliances.

[0079] Here, the information on the infrared signal may be collected using a known method. Specifically, for example, it may be provided by the manufacturer of the electrical appliance via a network or the like. Alternatively, a generation AI may generate a sentence instructing the user to operate a specific button, and the data processing device 12 may receive an infrared signal transmitted in response to the user's operation of a remote control, and record the received infrared signal in the data processing device 12 as an infrared signal corresponding to the designated button.

[0080] Fig. 14(A) shows an example of a case where the generation AI is made to learn the explanation of the buttons on a remote control device. The example in Fig. 14(A) shows an output 1 from the generation AI in response to an input 1 to the generation AI. Also, Fig. 14(B) shows an example of a case where an explanation of the buttons on a remote control device is requested from the generation AI. The example in Fig. 14(B) shows an output 2 from the generation AI in response to an input 2 to the generation AI.

[0081] Next, a form for combining image data of the main body of an electric appliance or a remote control device will be described with reference to FIG.

[0082] As shown in FIG. 15, in order to deepen understanding of the instruction manual, a specific function explanation may be provided on the screen of the smart device 14 by combining image data of the main body of the electrical appliance or a remote control device. The function explanation is read from the instruction manual using OCR, and the read content is trained by a generation AI to provide an easy-to-understand explanation tailored to each user through voice or dialogue chat. In addition, for example, by using an AR (Augmented Reality) function to hold the smart device 14 over a remote control device, which is an example of an object, and displaying the captured image data of the remote control device on the screen of the smart device 14 while providing a function explanation corresponding to each button, it becomes possible to obtain necessary information more intuitively and instantly.

[0083] Next, with reference to FIG. 16, a setup support function for electrical products using a data generation model 58, which is a generation AI, will be described.

[0084] The data generation model 58 determines the literacy level of the user with respect to the electrical appliance from the user's usage record of the electrical appliance, and outputs an explanation according to the literacy level based on the contents of the instruction manual DB 70. The literacy level is the degree of understanding that indicates the degree to which the electrical appliance is understood. In other words, if the user has a relatively large usage record, the literacy level is considered to be high, and if the user has a relatively small usage record, the literacy level is considered to be low.

[0085] As shown in FIG. 16, the data generation model 58 may support the setup method of an electrical appliance in a chat format. The data generation model 58 determines the literacy level of the user regarding electrical appliances based on the user's usage history, and if the literacy level is low, converts the explanation into simple sentences and provides an explanation. In other words, an explanation can be provided that matches the user's level of understanding. Furthermore, even if the user asks a question about something they are unsure about (e.g., what does "pairing" mean?), the data generation model 58 outputs an appropriate answer to the question (e.g., "pairing" means connecting a TV and a remote control wirelessly).

[0086] Next, with reference to Figs. 17 to 19, a description will be given of a data generation model 58, which is a generation AI, and an electrical product guide function using the AR function of the smart device 14.

[0087] The data generation model 58 identifies the type, manufacturer, and model of the electrical appliance from an image obtained by photographing the electrical appliance or from a dialogue with the user of the electrical appliance, and outputs an explanation corresponding to the identified type, manufacturer, and model of the electrical appliance based on the contents of the instruction manual DB 70.

[0088] FIG. 17 is a flowchart showing an example of the flow of an electrical product identification process using the data generation model 58.

[0089] In step S401 of FIG. 17, the processing unit 294 acquires a captured image of a target electrical product.

[0090] In step S402, the processing unit 294 uses the data generation model 58 to determine whether or not the type of electrical appliance (e.g., television, refrigerator, air conditioner, etc.) has been identified from the captured image acquired in step S401. If it is determined that the type of electrical appliance has been identified (positive determination), the process proceeds to step S403, and if it is determined that the type of electrical appliance cannot be identified (negative determination), the process proceeds to step S406. In this case, the data generation model 58 has previously learned image patterns such as televisions, refrigerators, air conditioners, etc., and is capable of outputting the type of electrical appliance for the input captured image.

[0091] In step S403, the processing unit 294 judges whether or not the manufacturer has been identified from the captured image using the data generation model 58. If it is judged that the manufacturer has been identified (if the judgment is positive), the process proceeds to step S404, and if it is judged that the manufacturer cannot be identified (if the judgment is negative), the process proceeds to step S406. In this case, the data generation model 58 has previously learned the image patterns for each manufacturer, and is capable of outputting the manufacturer name for the input captured image.

[0092] In step S404, the processing unit 294 uses the data generation model 58 to determine whether or not the model of the electrical appliance (e.g., model number, etc.) can be identified from the captured image. If it is determined that the model of the electrical appliance can be identified (positive determination), the process proceeds to step S405, and if it is determined that the model of the electrical appliance cannot be identified (negative determination), the process proceeds to step S406. In this case, the data generation model 58 has previously learned the models of electrical appliances for each manufacturer, and is capable of outputting the model of the electrical appliance for the input captured image.

[0093] In step S405, the processing unit 294 associates the model of the electrical product identified in step S404 with the instruction manual, and ends the electrical product identification process.

[0094] On the other hand, in step S406, if the processing unit 294 cannot identify the type of electrical appliance in step S402, it notifies the user that it is not an electrical appliance or that the electrical appliance has not been properly photographed, and ends the electrical appliance identification process. If the electrical appliance has not been properly photographed, the processing unit 294 may notify the user to adjust the camera orientation. If the processing unit 294 cannot identify the manufacturer in step S403, it notifies the user that the amount of learning of image patterns for each manufacturer is insufficient. If the processing unit 294 cannot identify the model of the electrical appliance in step S404, it notifies the user to photograph the model number, etc., since the model cannot be identified.

[0095] Also, if it is difficult to identify the model from the captured image, the model may be identified through a dialogue with the user. The processing unit 294 uses a chat function to inquire of the user about the model number of the manufacturer. If the user does not know, the processing unit 294 inquires about the type of electrical appliance (e.g., television, refrigerator, air conditioner, etc.) and further inquires about the manufacturer. The processing unit 294 may also acquire an operation sound of the electrical appliance via the smart device 14 and identify the model from the operation sound of the electrical appliance. The processing unit 294 checks the presence or absence of a function from information in the instruction manual, associates the identified model of the electrical appliance with the instruction manual, and ends the electrical appliance identification process.

[0096] Next, a mode for displaying an explanatory portion corresponding to a button on a remote control device will be described with reference to Figs. 18(A) and 18(B).

[0097] As shown in Fig. 18(A), the screen of smart device 14 displays text information of the explanatory portion corresponding to each button of the remote control device together with the image of the remote control device. In this case, if the area on the screen for displaying the image of the remote control device is equal to or larger than a certain size, that is, if the area for displaying the text information of the explanatory portion is small compared to the area for displaying the image of the remote control device, the text information of the explanatory portion may be displayed only for frequently used buttons. In the example of Fig. 18(A), the "Drive" button of the remote control device is a frequently used button, so the text information of the explanatory portion is displayed only for the "Drive" button.

[0098] The example of Figure 18(A) shows a form in which text information of the explanatory portion is displayed only for frequently used buttons, but if you want to display text information of the explanatory portion for all displayed buttons, you can switch to full display by shaking the smart device 14, as shown in Figure 18(B), for example.

[0099] Next, a form in which the user is notified of an updated function in the instruction manual will be described with reference to FIG.

[0100] As shown in Fig. 19, the contents of the updated functions in the latest instruction manual may be displayed on the screen of the smart device 14 to notify the user. At this time, the notification may be made taking into consideration whether the updated functions are necessary for the user. For example, the frequency of past actions or the frequency of use of the functions for the user may be stored, and if the frequency of past actions or the frequency of use of the functions is equal to or greater than a certain level, the user may be notified of the updated functions.

[0101] Next, a combination of the AR function of the smart device 14 and a voice guide will be described with reference to Figs. 20 to 22.

[0102] As shown in Fig. 20, the data generation model 58 may display an image of an electrical appliance using an AR function according to the screen size of the smart device 14, while displaying a message on the screen of the smart device 14 to guide the user as to the order in which maintenance work should be performed on the electrical appliance, or may provide guidance using a voice function of the smart device 14. The example of Fig. 20 shows a state in which the maintenance work is guided in order while displaying an image of a recorder as an example of an electrical appliance. Note that since displaying a large amount of information on the screen at once may confuse the user, it is desirable to display a small amount of information on the screen at an appropriate timing.

[0103] In addition, as shown in FIG. 21, the data generation model 58 may determine the user's work status from a captured image obtained by photographing the user's work status on an electrical appliance, and output an audio guide on the next task that the user should perform according to the determined work status of the user based on the contents of the instruction manual DB 70.

[0104] It is difficult for a user to perform maintenance work while holding the smart device 14. Therefore, as shown in Fig. 21, with the smart device 14 fixed in a predetermined position, the smart device 14 captures an image of the user's work status and outputs audio guidance on the work that the user should do next.

[0105] 22, the data generation model 58 may classify the functions of the electrical appliance from the instruction manual DB 70 and output a function explanation taking into consideration the frequency of use of the user and the history of button operations. For example, the button operations are simply displayed on the screen, and the function introduction is output by audio guidance. The data generation model 58 may change the function explanation text that it has generated to a simpler and easier-to-understand text, or may change the text to suit the age or level of understanding of the user.

[0106] Next, with reference to FIG. 23, a description will be given of a mode in which the power supply of an electric appliance is automatically turned off when the wiring or the like of the electric appliance is in a dangerous state.

[0107] In this case, if data generation model 58 detects dangerous wiring or usage of the electrical appliance from a captured image obtained by photographing the environment in which the electrical appliance is installed, it may turn off the power to the electrical appliance and output a message to the user of the electrical appliance.

[0108] In (S1) of FIG. 23, the processing unit 294 acquires images captured by the smart device 14 during maintenance of the electrical appliance and during consideration of the installation location of the electrical appliance.

[0109] In (S2), the processing unit 294 uses the data generation model 58 to extract electrical appliances with relatively high power consumption from the captured image acquired in (S1), and further extracts portions of the captured image in which multiple codes can be identified.

[0110] In (S3), the processing unit 294 uses the app to guide the user to take an image of the outlet of the electrical appliance. Specifically, for example, the processing unit 294 guides the user by displaying a message such as "Please take an image of the outlet of the electrical appliance" on the screen of the smart device 14.

[0111] In (S4), the processing unit 294 uses the data generation model 58 to determine the risk of the outlet part of the electrical appliance from the photographed image of the outlet part acquired in (S3), and further determines the risk of deterioration over time after the electrical appliance is installed.

[0112] In (S5), if the processing unit 294 determines that the data generation model 58 is highly dangerous, it takes action such as turning off the power to the relevant electrical appliance to stop it, suggesting a wiring change, or suggesting a replacement of the electrical appliance, and then ends the series of processes.

[0113] Next, a form for instructing how to perform maintenance on an electrical appliance will be described with reference to FIG.

[0114] In this case, the data generation model 58 may learn the correspondence between the living environment of the user of the electrical appliance and the maintenance conditions of the electrical appliance, and output the maintenance conditions of the electrical appliance according to the living environment of the user of the electrical appliance.

[0115] In (S11) of FIG. 24, the processing unit 294 acquires images captured by the smart device 14 during maintenance of the electrical appliance and during consideration of the installation location of the electrical appliance.

[0116] In (S12), the processing unit 294 measures the cleanliness of the electrical appliances from the captured image acquired in (S11), measures the number of items on the shelves and in the vicinity, and automatically extracts the user's maintenance frequency. The processing unit 294 then inputs the cleanliness of these electrical appliances, the number of items on the shelves and in the vicinity, and the user's maintenance frequency to the data generation model 58, which then comprehensively determines the degree of maintenance.

[0117] In (S13), if the degree of maintenance is less than the threshold, that is, the degree of maintenance is relatively low, based on the determination result of the data generation model 58, the processing unit 294 proposes a simple maintenance method to the user after a certain period of time. Also, if the degree of maintenance is equal to or greater than the threshold, that is, the degree of maintenance is relatively high, based on the determination result of the data generation model 58, the processing unit 294 proposes a better maintenance method to the user.

[0118] In (S14), the processing unit 294 uses the data generation model 58 to determine the deterioration time of the electrical appliance, and based on the determination result, suggests to the user when to replace the electrical appliance, and ends the series of processes.

[0119] Next, with reference to Figs. 25 to 30, a form in which the user comfort level DB 74 is constructed and utilized will be described.

[0120] In this case, the generating unit 318 further generates a comfort level DB 74 including operation information of the user of the electrical appliance, living environment information of the user, health care information of the user, and outdoor environment information of the user, as shown in Fig. 25. The operation information includes the usage status of the remote control device operated by the user, and the power start time of each electrical appliance (e.g., air conditioner, television, air purifier, recorder, lighting fixture, vacuum cleaner, etc.). The living environment information includes, for example, illuminance, amount of indoor pollutants, etc. The health care information includes, for example, pulse, body temperature, number of steps, calories burned, sleep time, time spent operating the smart device 14, etc. The outdoor environment information includes, for example, temperature, humidity, weather, etc.

[0121] The learning unit 320 trains the generation AI to learn the contents of the comfort level DB 74 and generates the data generation model 58.

[0122] The data generation model 58 outputs an answer for improving the user's comfort level based on the contents of the comfort level DB 74.

[0123] Fig. 26 is a diagram showing an example of a flow of answer output for increasing the comfort level. As shown in Fig. 26, instruction manual data is collected, and the collected instruction manual data is stored in a vector DB and a large-scale DB for advance learning.

[0124] Next, the extraction engine is used to extract comfort level data of similar users from the vector DB, and the extracted comfort level data is input to the prompt as reference information. Comfort level data of similar users is data that indicates cases in which similar users have increased their comfort level. The "reference information" of the prompt includes not only the comfort level data of similar users, but also additional information from the manufacturer's personnel. The "instructions" of the prompt indicate instructions to the user on button operations or panel operations. The "conditions" of the prompt indicate conditions for selecting a language model specific to the application, either for operation methods, maintenance methods, or repair methods, in order to lead to an appropriate answer.

[0125] On the other hand, the trained model uses data obtained from a large-scale pre-training database as training data to learn basic operations such as how to operate, maintain, repair, and what not to do.

[0126] A use-specific model is prepared for each of operation methods, maintenance methods, and repair methods, and each use-specific model outputs answers in descending order of comfort level score.

[0127] In addition, when the comfort level of the user is improved by the answer of the data generation model 58, the learning unit 320 may update the comfort level DB 74A of similar users who are similar to the user. For example, as shown in FIG. 27, the comfort level DB 74A of similar users by the manufacturer may be updated. The example of FIG. 27 shows the case of a passive user. In this case, the data generation model 58 automatically feeds back the change in the comfort level to the manufacturer. The person in charge of the manufacturer supports the data generation model 58 so that the data generation model 58 can make a proposal to the user to increase the comfort level while checking the user's comfort level. When the user's comfort level is increased, the learning unit 320 updates the comfort level DB 74A of similar users who are similar to the user. This automatically improves the accuracy of the comfort level.

[0128] Also, as shown in FIG. 28, the comfort level DB74A of similar users may be updated by UI operations of the smart device 14. The example of FIG. 28 shows a case for an active user. In this case, the data generation model 58 automatically feeds back changes in the comfort level to the comfort level DB74. That is, the data generation model 58 acquires feedback from the user, via UI operations of the smart device 14, on what the user actually wants. The learning unit 320 updates the comfort level DB74A of similar users based on the acquired feedback. This makes it possible to automatically improve the accuracy of the comfort level.

[0129] Also, as shown in FIG. 29, the similar user comfort level DB74A may be updated by the dialogue mode of the smart device 14. The example of FIG. 29 also shows the case for an active user. In this case, the data generation model 58 automatically feeds back the change in comfort level to the comfort level DB74. That is, the data generation model 58 acquires feedback through dialogue with the user via the dialogue mode of the smart device 14. The learning unit 320 updates the similar user comfort level DB74A based on the acquired feedback. This makes it possible to automatically improve the accuracy of the comfort level.

[0130] Also, as shown in FIG. 30, the comfort level DB74A of similar users may be updated in cooperation with the indoor camera 80. In this case, the data generation model 58 collects the user's room situation in cooperation with the indoor camera 80, and determines the user's comfort level based on the collected room situation. Then, the data generation model 58 automatically feeds back the change in the comfort level to the comfort level DB74. The learning unit 320 updates the similar user comfort level DB74A based on the acquired feedback. This makes it possible to automatically improve the accuracy of the comfort level. Note that the indoor camera 80 may be any other device as long as it can collect the room situation. For example, an object detection sensor using radio waves or ultrasonic waves, an infrared camera, or the like may be used.

[0131] Next, with reference to FIG. 31, a description will be given of an embodiment in which the amount of power consumption in a home is detected using the data generation model 58 and the settings of electrical appliances are changed.

[0132] As shown in Fig. 31, the data generation model 58 may be used to detect the amount of power usage in the home and change the settings of electrical appliances. In this case, the data generation model 58 can detect the amount of power usage in the home because it can grasp which electrical appliances are turned on. Therefore, the data generation model 58 can change the settings of electrical appliances and control them so that the breaker does not trip, for example, "the breaker trips when the microwave oven is turned on, so change the settings of the air conditioner."

[0133] Next, a mode of automatically updating a solution to an error code or an error message of an electrical appliance will be described with reference to FIG.

[0134] 32, the countermeasures for the error code or error message of the electrical appliance may be automatically updated. In this case, a countermeasure DB 75 in which countermeasures for the error code or error message of the electrical appliance are registered is stored in the storage 32 of the data processing device 12. The countermeasure DB 75 is configured as a part of the instruction manual DB 70, but may be configured separately from the instruction manual DB 70.

[0135] The data generation model 58 learns in advance the contents of the countermeasure DB 75. The data generation model 58 acquires an error code or an error message signal from the electrical appliance. If the error code or error message can be dealt with by a countermeasure obtained from the countermeasure DB 75, the data generation model 58 teaches the user the countermeasure for the error obtained from the countermeasure DB 75, and automatically feeds back the time required for the solution and the problem to the countermeasure DB 75. The learning unit 320 updates the countermeasure DB 75A of similar electrical appliances based on this feedback.

[0136] On the other hand, when the error code or error message cannot be dealt with using the countermeasure DB 75 alone, the data generation model 58 automatically cooperates with the manufacturer to create a new countermeasure DB and update the countermeasure DB 75. Specifically, the person in charge at the manufacturer instructs the data generation model 58 on the countermeasure in an interactive mode, and the data generation model 58 informs the user of the instruction and automatically feeds back the time required to solve the problem and the problem to the countermeasure DB 75. The learning unit 320 updates the countermeasure DB 75A for similar home appliances based on this feedback.

[0137] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone, etc. The method according to the present disclosure may be implemented in a SaaS (Software as a Service) format. may be provided to the user.

[0138] In the above embodiment, an example was given in which a specific process is performed by one computer 22, but the technology disclosed herein is not limited to this, and distributed processing of the specific process may be performed by multiple computers, including computer 22.

[0139] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0140] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0141] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0142] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0143] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0144] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0145] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0146] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0147] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, and standard was specifically and individually indicated to be incorporated by reference.

[0148] The following supplementary notes are further disclosed regarding the above embodiment.

[0149] (Appendix 1) A recognition unit that reads image information described in an instruction manual for the product and recognizes a plurality of pieces of character information; an assignment unit that assigns an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; a reconstruction unit that understands the meaning of character information expressed as a list of words among the plurality of character information and reconstructs the character information into a coherent sentence; a generating unit that generates an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; A learning unit that causes a generation AI to learn the contents of the instruction manual database; A data processing device comprising: (Appendix 2) A grouping unit that groups the plurality of pieces of character information into predetermined units, The instruction manual database further includes grouping information obtained by the grouping. 2. A data processing device according to claim 1. (Appendix 3) The product is an electrical product, The instruction manual includes an instruction section for operating an operating device for operating the electrical appliance, the instruction manual portion includes display names of buttons provided on the operation device and explanations of the buttons, a correspondence unit that associates first character information representing a display name of the button recognized by the recognition unit with second character information representing an explanatory portion of the button; a creation unit that understands the meaning of the associated first character information and the associated second character information and creates an explanatory text about the button; Further comprising: The generation unit generates an operation database including the first character information, the second character information, and the explanatory text; The learning unit causes the generation AI to learn the contents of the operation database. 3. A data processing device according to claim 1 or 2. (Appendix 4) The operation database further includes information representing an operation signal corresponding to the button provided on the operation device. 4. A data processing device according to claim 3. (Appendix 5) the operation device is a remote control device for remotely operating the electrical appliance, The operation signal is an infrared signal or a radio signal of the remote control device. 5. A data processing device according to claim 4. (Appendix 6) The generating AI determines the literacy level of the user for the product based on the user's usage record of the product, and outputs an explanation according to the literacy level based on the contents of the instruction manual database. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 7) The generation AI identifies the type, manufacturer, and model of the product from a photographed image of the product or a dialogue with a user of the product, and outputs a description corresponding to the identified type, manufacturer, and model of the product based on the contents of the instruction manual database. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 8) The generation AI determines the user's work status from a captured image obtained by capturing an image of the user's work status on the product, and outputs a voice guide on the next task that the user should perform according to the determined work status of the user based on the contents of the instruction manual database. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 9) When the generation AI detects dangerous wiring or usage of the product from a captured image of the installation environment in which the product is installed, the generation AI turns off the power of the product and outputs a message to a user of the product. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 10) The generation AI learns the correspondence between the living environment of the user of the product and the maintenance conditions of the product, and outputs the maintenance conditions of the product according to the living environment of the user of the product. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 11) The generating unit further generates a comfort level database including operation information of a user of the product, living environment information of the user, health care information of the user, and outdoor environment information of the user; The learning unit causes the generation AI to learn the contents of the comfort level database, The generating AI outputs an answer for improving the comfort level of the user based on the contents of the comfort level database. A data processing device according to any one of Supplementary Note 1 to Supplementary Note 5. (Appendix 12) when the comfort level of the user is improved by the answer, the learning unit updates the comfort level database of similar users who are similar to the user. 12. A data processing device according to claim 11. (Appendix 13) A data processing device comprising: It reads the image information in the product's instruction manual and recognizes multiple pieces of text information. giving an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; understanding the meaning of character information expressed as a string of words among the plurality of character information, and reconstructing the information into a coherent sentence; generating an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; The contents of the instruction manual database are trained on a generation AI. Data processing methods. (Appendix 14) It reads the image information in the product's instruction manual and recognizes multiple pieces of text information. giving an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; understanding the meaning of character information expressed as a string of words among the plurality of character information, and reconstructing the information into a coherent sentence; generating an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; A process of having a generation AI learn the contents of the instruction manual database, A data processing program for execution by a computer. [Explanation of symbols]

[0150] 10 Data Processing System 12 Data Processing Device 14 Smart Devices 58 Data Generation Model 70 User Manual DB 72 Operation DB 74, 74A Comfort DB 75, 75A Solution DB 290 Specific Processing Department 292 Input section 294 Processing section 296 Output section 310 Recognition part 312 Granting Department 314 Grouping Division 316 Reconstruction part 318 Generation part 320 Learning Department 322 Mapping section 324 Creation Department

Claims

1. A recognition unit that reads image information described in an instruction manual for the product and recognizes a plurality of pieces of character information; an assignment unit that assigns an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; a reconstruction unit that understands the meaning of character information expressed as a list of words among the plurality of character information and reconstructs the character information into a coherent sentence; a generating unit that generates an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; A learning unit that causes a generation AI to learn the contents of the instruction manual database; A data processing device comprising:

2. A grouping unit that groups the plurality of pieces of character information into predetermined units, The instruction manual database further includes grouping information obtained by the grouping.

2. A data processing apparatus according to claim 1.

3. The product is an electrical product, The instruction manual includes an instruction section for operating an operating device for operating the electrical appliance, the instruction manual portion includes display names of buttons provided on the operation device and explanations of the buttons, a correspondence unit that associates first character information representing a display name of the button recognized by the recognition unit with second character information representing an explanatory portion of the button; a creation unit that understands the meaning of the associated first character information and the associated second character information and creates an explanatory text about the button; Further comprising: The generation unit generates an operation database including the first character information, the second character information, and the explanatory text; The learning unit causes the generation AI to learn the contents of the operation database.

2. A data processing apparatus according to claim 1.

4. The operation database further includes information representing an operation signal corresponding to the button provided on the operation device.

4. A data processing device according to claim 3.

5. the operation device is a remote control device for remotely operating the electrical appliance, The operation signal is an infrared signal or a radio signal of the remote control device.

5. A data processing apparatus according to claim 4.

6. The generating AI determines the literacy level of the user for the product based on the user's usage record of the product, and outputs an explanation according to the literacy level based on the contents of the instruction manual database.

2. A data processing apparatus according to claim 1.

7. The generation AI identifies the type, manufacturer, and model of the product from a photographed image of the product or a dialogue with a user of the product, and outputs a description corresponding to the identified type, manufacturer, and model of the product based on the contents of the instruction manual database.

2. A data processing apparatus according to claim 1.

8. The generation AI determines the user's work status from a captured image obtained by photographing the user's work status for the product, and outputs a voice guide on the work that the user should do next according to the determined work status of the user based on the contents of the instruction manual database.

2. A data processing apparatus according to claim 1.

9. When the generating AI detects dangerous wiring or usage of the product from a photographed image of the installation environment in which the product is installed, the generating AI turns off the power of the product and outputs a message to a user of the product.

2. A data processing apparatus according to claim 1.

10. The generation AI learns the living environment of the user of the product and the maintenance conditions of the product in association with each other, and outputs the maintenance conditions of the product according to the living environment of the user of the product.

2. A data processing apparatus according to claim 1.

11. The generating unit further generates a comfort level database including operation information of a user of the product, living environment information of the user, health care information of the user, and outdoor environment information of the user; The learning unit causes the generation AI to learn the contents of the comfort level database, The generating AI outputs an answer for improving the comfort level of the user based on the contents of the comfort level database.

2. A data processing apparatus according to claim 1.

12. when the comfort level of the user is improved by the answer, the learning unit updates the comfort level database of similar users who are similar to the user. A data processing apparatus according to claim 11.

13. A data processing device comprising: It reads the image information in the product's instruction manual and recognizes multiple pieces of text information. giving an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; understanding the meaning of character information expressed as a string of words among the plurality of character information, and reconstructing the information into a coherent sentence; generating an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; The contents of the instruction manual database are learned by a generation AI. Data processing methods.

14. It reads the image information in the product's instruction manual and recognizes multiple pieces of text information. giving an order to the plurality of pieces of character information so that the meaning of the plurality of pieces of character information can be understood as a sentence; understanding the meaning of character information expressed as a string of words among the plurality of character information, and reconstructing the information into a coherent sentence; generating an instruction manual database including the plurality of pieces of character information to which the order has been assigned and reconstructed information obtained by the reconstruction; A process of having a generating AI learn the contents of the instruction manual database, A data processing program for execution by a computer.

Images