Home electric appliance operation support system and home electric appliance operation support system

Abstract

To enable a user to understand more operation or a state of an apparatus such as a defect by notifying the user of information relating to the operation, etc., of a home electric appliance in a more concrete form.SOLUTION: A home electric appliance operation support system supports operation of a home electric appliance 2. The home electric appliance operation support system comprises: an imaging section 11 which images the home electric appliance and acquires image data; a state estimation section 13 which estimates a state of the home electric appliance using the image data; a content identification section 33 which identifies a content relating to the operation regarding the home electric appliance corresponding to the estimated state; a display information creation section 15 which creates display information by superposing the image data and the content; and a display section 18 which displays the display information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for supporting the operation of various devices, such as so-called home appliances. [Background technology]

[0002] Home appliances and other devices have multiple functions, and users may get confused about how to operate them. In addition, as the functions and configurations become more complex, a wide variety of malfunctions may occur in the devices. In such cases, users may not be able to understand the nature of the malfunction.

[0003] In order to address these issues, Patent Documents 1 and 2 have been proposed. Patent Document 1 describes the operation as follows: It is stated that "a usage method according to the usage status of the device is provided to the home appliance." More specifically, it is disclosed that "The server 10 and the home appliance 20 are connected to each other so that they can communicate with each other via a communication network 1. The home appliance 20 transmits usage status information indicating the operating status of the home appliance to the server 10. The server 10 updates the user information database 16 based on the received usage status information. The server 10 also refers to the design information database 15 and the user information database 16 to obtain consulting information indicating how to use the home appliance 20, and transmits it to the home appliance 20. The consulting information is information on operations to resolve problems in using the home appliance 20 and information on how to use the home appliance in an optimal state. The home appliance 20 operates according to the received consulting information."

[0004] Patent Document 2 also describes a technology for supporting measures against malfunctions. More specifically, it discloses an error removal support device 2 that includes an imaging device 22 that captures an error screen displayed on a display device 473 of a device 1 that is a support target for removing an error, a display device 22 that displays a guidance screen showing a method for removing an error in the device 1, a communication device 23 that communicates with a maintenance system 4 of the device 1, and a control unit 26 that reads an error code 474 in image data captured by the imaging device 21 to identify an error occurring in the device 1, notifies the maintenance system 4 of a repair request via the communication device 23 if the error is a system error that requires a repair request, and displays a guidance screen on the display device 22 showing a method for removing the error if the error is other than a system error. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2012-109995 A [Patent Document 2] JP 2015-118519 A Summary of the Invention [Problem to be solved by the invention]

[0006] When providing support for user operations, such as dealing with malfunctions, it is desirable to provide support in a format that is easy for users to understand. In contrast, in Patent Documents 1 and 2, notifications are provided only in text or schematic displays, and do not indicate the specific state of the device, making it difficult for users to intuitively understand the operations to be performed or the contents of the malfunction.

[0007] Therefore, an object of the present invention is to provide a method for notifying a user of information relating to the operation of a home appliance in a more specific format, thereby making it easier for the user to understand the state of the appliance, such as its operation or a malfunction. [Means for solving the problem]

[0008] In order to solve the above problems, in the present invention, the appearance (including the display contents) of a device such as a home appliance is photographed, the state of the device is estimated, and display information about the state of the device is displayed by merging (including superimposing) the photographed image data and content related to operations according to the estimation result.

[0009] A more specific aspect of the present invention is a home appliance operation assistance system that assists in the operation of a home appliance, the home appliance operation assistance system having an imaging unit that photographs the home appliance and acquires image data, a state estimation unit that uses the image data to estimate a state of the home appliance, a content identification unit that identifies content related to the operation of the home appliance in accordance with the estimated state, a display information creation unit that creates display information by superimposing the image data and the content, and a display unit that displays the display information.

[0010] Furthermore, the present invention also includes a home appliance operation support method using each device or subsystem constituting the above-mentioned home appliance operation support system. Furthermore, the present invention also includes a program for causing each device to function as a computer and a storage medium for storing this program. Effect of the Invention

[0011] According to the present invention, the state of a device, such as its operation or malfunction, can be more easily understood. [Brief description of the drawings]

[0012] [Figure 1] 1 is a system configuration diagram of a home appliance operation assistance system according to an embodiment of the present invention; [Diagram 2] 1 is a hardware configuration diagram of a user terminal 1 in one embodiment of the present invention. [Diagram 3] 2 is a functional block diagram of a home appliance 2 according to an embodiment of the present invention. FIG. [Figure 4] 2 is a hardware configuration diagram of a home appliance operation assistance device 3 according to an embodiment of the present invention. FIG. [Diagram 5]FIG. 11 is a diagram showing home appliance management information 191 used in one embodiment of the present invention. [Figure 6] 1 is a flowchart showing a process flow in one embodiment of the present invention. [Figure 7] 5 is a flowchart showing a specific example of a process for creating and displaying display information in an embodiment of the present invention. [Figure 8A] 1 is a flowchart (part 1) showing a processing flow in the first embodiment. [Figure 8B] 1 is a flowchart (part 2) showing the processing flow in the first embodiment. [Figure 9A] 1 is a diagram showing display contents (part 1) of display information on a display unit 18 in the first embodiment. FIG. [Figure 9B] 13 is a diagram showing the display contents (part 2) of the display information on the display unit 18 in the first embodiment. FIG. [Figure 9C] 13 is a diagram showing the display contents (part 3) of the display information on the display unit 18 in the first embodiment. FIG. [Figure 9D] FIG. 11 is a diagram showing the display contents (part 4) of the display information on the display unit 18 in the first embodiment. [Figure 10] 11 is a flowchart showing a processing flow in the second embodiment. [Figure 11A] 13 is a flowchart (part 1) showing a processing flow in the third embodiment. [Figure 11B] 13 is a flowchart (part 2) showing the process flow in the third embodiment. [Figure 12A] FIG. 13 is a diagram showing display contents (part 1) of display information on the display unit 18 in the third embodiment. [Figure 12B] FIG. 13 is a diagram showing a second example of the display content of the display information on the display unit 18 in the third embodiment. [Figure 12C] FIG. 13 is a diagram showing the display contents (part 3) of the display information on the display unit 18 in the third embodiment. [Figure 12D] FIG. 13 is a diagram showing the display contents (part 4) of the display information on the display unit 18 in the third embodiment. [Figure 12E]FIG. 13 is a diagram showing a fifth example of the display content of the display information on the display unit 18 in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An embodiment of the present invention will be described below. In this embodiment, the device performs operation support for a plurality of home appliances as a representative of devices. This operation support includes supporting the operation of the home appliances and understanding the state of the home appliances. In addition, the operation includes controlling the operation of the home appliances and dealing with malfunctions.

[0014] 1 is a system configuration diagram of a home appliance operation assistance system according to the present embodiment. The home appliance operation assistance system has a user terminal 1 and a home appliance operation assistance device 3, which manage home appliances 2. Note that the home appliances 2 may be multiple or singular, and hereinafter, when the home appliance 2 is referred to, it also means an individual home appliance.

[0015] The user terminal 1, the home appliance 2, and the home appliance operation assistance device 3 are connected to each other. To achieve this, they may be connected via a router 4 or a network 5, or may be connected directly using a short-range communication function. The user terminal 1 and the home appliance operation assistance device 3 work together to assist in the operation of the home appliance 2. Each of these devices, including the home appliance 2, will be described below.

[0016] First, the user terminal 1 is an information processing device used by a user of the home appliance 2. Therefore, the user terminal 1 can be realized by various computers such as a smartphone, a tablet, a mobile phone, or a PC. As shown in Fig. 1, the user terminal 1 has a communication unit 11, an image capturing unit 12, a state estimation unit 13, a content identification unit 14, a display information creation unit 15, a control command unit 16, an input unit 17, a display unit 18, and a storage unit 19.

[0017] First, the communication unit 11 has a function of communication via the router 4 and the network 5 and a function of short-distance wireless communication. The photographing unit 12 is realized by a camera or the like, and acquires image data of the surroundings. The state estimation unit 13 estimates the state of the home appliance using the image data acquired by the photographing unit 12. This state includes not only the state and situation currently occurring in the home appliance 2, but also functions requested by the user, operations for executing the functions, and operation areas for the operations (e.g., buttons).

[0018] The content identification unit 14 also identifies content related to the operation of the home appliance 2 according to the estimated state. This content includes the details of the malfunction indicated by the error code and / or a method for dealing with the malfunction. The content also includes the estimated operation area itself and operation areas for the operation area. Furthermore, the content also includes functions operated by the home appliance 2 depending on the operation area and their details.

[0019] Moreover, the display information creation unit 15 creates display information using the acquired image data and the identified content. In creating this display information, the image data and the content are merged, more preferably the content is superimposed on the image data. For this purpose, so-called AR (Augmented Reality) technology can be used. Furthermore, in creating the display information, the state estimated by the state estimation unit 13 may be included in the display information. In this case, the estimated state is used as the content, that is, the content identification unit 14 can identify the state itself as the content. Furthermore, the display information creation unit 15 can create display information that enables operation of the home appliance 2. That is, the display information creation unit 15 superimposes an operation area on image data showing the display operation panel of the home appliance 2, and creates display information that enables a user's instruction to be received for the operation area. At this time, the contents of the function by the operation area may also be included in the display information.

[0020] Furthermore, the control command unit 16 creates a control command for the home appliance 2. The creation of this control command includes creating a control command according to an instruction from a user for display information including an operation area. Here, at least one of the state estimation unit 13 to the display information creation unit 15 can be omitted by providing a corresponding function in the home appliance 2 or the home appliance operation assistance device 3.

[0021] Furthermore, the input unit 17 accepts various operations from the user. Furthermore, the display unit 18 displays display information, etc. The input unit 17 and the display unit 18 may be integrated into one unit, such as a touch panel. Furthermore, the storage unit 19 stores home appliance management information 191. This home appliance management information 191 will be described later.

[0022] Next, an implementation example of the user terminal 1 will be described. As described above, the user terminal 1 can be realized by various computers such as a smartphone, a tablet, a mobile phone, or a PC. Fig. 2 is a hardware configuration diagram of the user terminal 1 in this embodiment. In Fig. 2, the user terminal 1 has a camera 101, a touch panel 102, a processing device 103, a communication device 104, and a storage device 105, which are connected to each other via a communication path.

[0023] First, camera 101 is an implementation example of image capturing unit 12, and has an image capturing function. Touch panel 102 is configured to serve both input unit 17 and display unit 18 in Fig. 1, and accepts user operations and displays various information such as display information. Touch panel 102 may be configured separately as an input device and an output device.

[0024] The processing device 103 can be realized by a processor such as a CPU (Central Processing Unit), and executes calculations according to a home appliance operation assistance program 106 stored in a storage device 105, which will be described later.

[0025] The home appliance operation assistance program 106 is composed of a state estimation module 107, a content identification module 108, a display information creation module 109, and a control command module 110 for each function. Each of these modules may be realized as an individual program or a partial combination. Furthermore, the home appliance operation assistance program 106 may be realized as one function of an integrated management application that manages the home appliances 2 and has functions other than the above-mentioned modules.

[0026] Here, the configuration shown in the functional blocks of the user terminal 1 in FIG. 1, which execute the same functions as the modules, is as follows. State estimation module 107: state estimation unit 13 Content Identification Module 108: Content Identification Unit 14 Display information creation module 109: Display information creation unit 15 Control command module 110: control command unit 16 Therefore, the processing device 103 executes the processes of the state estimation unit 13, the content identification unit 14, the display information creation unit 15, and the control command unit 16 in accordance with the home appliance operation assistance program 106. The home appliance operation assistance program 106 can be stored in the storage device 105 described later or other storage media.

[0027] 1, and has a function of communication via the router 4 and the network 5 and a short-distance wireless communication function. The storage device 105 corresponds to the storage device 19 in FIG. 1, and stores the above-mentioned home appliance operation assistance program 106 and home appliance management information 191. For this reason, the storage device 105 may be realized by a main storage device such as a memory and a secondary storage device (storage medium) that is a so-called storage. The secondary storage device may be realized by an external HDD (Hard Disk Drive), SSD (Solid State Drive), memory card, etc. This concludes the description of the configuration of the user terminal 1 of this embodiment.

[0028] Next, the home appliances 2 of this embodiment will be described. The home appliances 2 are the objects to be operated in this embodiment, and are used by users in houses and the like. In FIG. 1, an IH cooker 21, a washing machine 22, and a refrigerator 23 are shown as the home appliances 2, but the home appliances 2 are not limited to these. Here, the information processing configuration of each of these home appliances 2 will be described. FIG. 3 is a functional block diagram of the home appliances 2 in this embodiment. Since each home appliance 2 executes common information processing, the functions thereof will be described collectively.

[0029] The home appliance 2 includes a control unit 201, a communication unit 206, a display operation unit 207, an operating unit 208, and a sensor 209. These are connected to each other via a communication path. Each component will be described below.

[0030] First, the control unit 201 creates control signals for implementing the functions of each home appliance 2, such as cooking and laundry, to control the operation of the home appliance 2, and executes other information processing. For this purpose, the control unit 201 has a control signal creation unit 202, a state estimation unit 203, a content identification unit 204, and a display information creation unit 205.

[0031] The control signal creation unit 202 creates a control signal for executing a function such as cooking according to an operation from the user to the display operation unit 207 or a control command from the user terminal 1. The state estimation unit 203 to the display information creation unit 205 have the same functions as the state estimation unit 13 to the display information creation unit 15, respectively. Therefore, a description of these functions will be omitted. In this way, these can be provided in the user terminal 1 or the home appliance operation assistance device 3, and at least one of them can be omitted. The control unit 201 can be realized by an MPU (Micro-Processing Unit). Therefore, each of the control unit 201, the control signal creation unit 202 to the display information creation unit 205 can be realized by dedicated hardware or software such as a program.

[0032] The communication unit 206 also communicates with the user terminal 1 and the home appliance operation assistance device 3 by communication via the router 4 or a network or by a short-range communication function. The communication unit 206 receives control commands from the user terminal 1 and the home appliance operation assistance device 3 and notifies them of various information. The communication unit 206 also realizes pairing with the user terminal 1 in the short-range communication function. Note that the pairing function or the communication unit 206 itself may be omitted. In other words, this embodiment can also be applied to a home appliance 2 that does not have a communication function.

[0033] Furthermore, the display operation unit 207 receives operations from the user on the home appliance 2, and outputs an error message regarding a malfunction (e.g., an error) that occurs in the home appliance 2. This error message includes an error code that identifies the malfunction. Furthermore, the display operation unit 207 includes an operation area (e.g., an operation button) for receiving operations from the user. Note that the operation area may be realized as an area displayed on the screen, such as a software keyboard, or may be realized as physical buttons. Note that the display operation unit 207 can be configured as a fixed display operation panel that displays and receives operations.

[0034] Moreover, the operating unit 208 operates according to the control signal generated by the control signal generating unit 202. Furthermore, the sensor 209 measures the operating status of the operating unit 208. And, as the sensor 209, an imaging unit (e.g., a camera) that captures an image of the home appliance 2 itself including the display operation unit 207 may be used. The state estimation unit 203 may estimate the state of the home appliance 2 using image data captured by this imaging unit, or the state estimation unit 13 of the user terminal 1 may estimate the state. In the latter case, the image data acquired by the imaging unit is notified to the user terminal 1 via the communication unit 206. This concludes the description of the home appliance 2, and next, the home appliance operation assistance device 3 will be described with reference to FIG. 1.

[0035] 1, the home appliance operation assistance device 3 includes a communication unit 31, a state estimation unit 32, a content identification unit 33, a display information creation unit , a control instruction unit 35, and a storage unit .

[0036] First, the communication unit 31 is connected to the home appliances 2 and the user terminal 1 via the network 5. The state estimation unit 32 to the control command unit 35 have the same functions as the state estimation unit 13 to the control command unit 16 of the user terminal 1, respectively. Therefore, a description of these functions will also be omitted.

[0037] The storage unit 36 ​​also stores home appliance management information 361. This will be described later together with home appliance management information 191.

[0038] Furthermore, the home appliance operation assistance device 3 may be connected to an operation assistance terminal used by an operator or a service engineer. The operation assistance terminal receives operation instructions from the operator or the service engineer and displays information corresponding to the instructions. For this reason, the operation assistance terminal may be realized by an information processing device (computer) such as a smartphone, tablet, or PC. The operation assistance terminal displays the estimated state, contents, display information, etc. of the home appliance 2. As a result, the operator or the service engineer can give the user appropriate advice according to the state of the home appliance 2. Furthermore, the operation assistance device may be configured to notify the user terminal 1 of the advice.

[0039] Next, a hardware configuration as an implementation example of the home appliance operation assistance device 3 will be described. The home appliance operation assistance device 3 can be realized by a computer such as a server or a cloud system. Fig. 4 is a hardware configuration diagram of the home appliance operation assistance device 3 in this embodiment. In Fig. 4, the home appliance operation assistance device 3 has a processing device 301, a communication device 302, a memory 303, and a secondary storage device 304, which are connected to each other via a communication path.

[0040] First, the processing device 301 can be realized by a processor such as a CPU, and executes calculations according to a home appliance operation assistance program 305 stored in a sub-storage device 304 described later. Also, the communication device 302 corresponds to the communication unit 31 in Fig. 1, connects to the network 5, and communicates with other devices such as the user terminal 1.

[0041] 1. The memory 303 and the secondary storage device 304 expand information used for processing by the home appliance operation assistance program 305 and the processing device 301 stored in the secondary storage device 304. The secondary storage device 304 can be realized by a so-called storage. The secondary storage device 304 stores the home appliance operation assistance program 305 and the home appliance management information 361. The secondary storage device 304 may be realized by various storage media such as an external hard disk drive (HDD), a solid state drive (SSD), or a memory card, or may be realized by a device separate from the home appliance operation assistance device 3, such as a file server.

[0042] Here, the home appliance operation assistance program 305 is composed of a state estimation module 306, a content identification module 307, a display information creation module 308, and a control command module 309 for each function. Each of these modules may be realized as an individual program or a partial combination. Furthermore, the home appliance operation assistance program 305 may be realized as one function of an integrated management program that manages the home appliance 2 having other functions as well.

[0043] Here, the configuration shown in FIG. 1 which performs the same functions as each module is as follows. State Estimation Module 306: State Estimation Unit 32 Content Identification Module 307: Content Identification Unit 33 Display information creation module 308: Display information creation unit 34 Control command module 309: control command unit 35 Therefore, the processing device 301 executes the processes of the state estimation unit 32 to the control command unit 35 in accordance with the home appliance operation assistance program 305.

[0044] Here, the home appliance operation assistance program 305 executes the same process as the home appliance operation assistance program 106 of the user terminal 1. The home appliance operation assistance program 305 can be stored in the secondary storage device 304 or other storage media.

[0045] The home appliance operation assistance device 3 is used by manufacturers and distributors of the home appliances 2, and these distributors manage users as members and also manage each home appliance 2. For this reason, the home appliance operation assistance device 3 may be realized as a home appliance management device that comprehensively manages home appliances. This concludes the explanation of the home appliance operation assistance device 3.

[0046] The router 4 is a device that mediates communication between the user terminal 1 and the home appliances 2, but may be omitted. The network 5 connects the above-mentioned devices and can be realized by a wide area network such as the Internet or a combination of the wide area network and a local network. This concludes the explanation of the configuration of each device in this embodiment.

[0047] Next, the home appliance management information 191 and home appliance management information 361 used in this embodiment will be described. Fig. 5 is a diagram showing the home appliance management information 191 used in this embodiment. The home appliance management information 191 is information for managing each home appliance 2, and more specifically includes information about its display operation unit 207. For this reason, the home appliance management information 191 has the following items for each home appliance: model, display operation unit reference data, error management information, and function management information.

[0048] First, the model is an item for identifying the corresponding home appliance, and a model number or the like can be used. The display operation unit reference data is data that serves as a reference for identifying the display operation unit of the corresponding home appliance on the image data. In this embodiment, an image of the corresponding display operation unit is used.

[0049] Moreover, the error code management information is an example of the content in this embodiment, and is information on an error that is an example of a malfunction of the home appliance. For this reason, the error management information has each item of an error code, a display position, and an error content / handling method. The error code is a code that identifies the error. The display position indicates the display position of the error code on the home appliance, particularly on the display operation unit. The error content / handling method indicates the content of the error indicated by the error code and the handling method for that error. Note that in this embodiment, multiple error contents and their handling methods are associated with the error code. For this reason, even if an error code is displayed, it is very difficult for a user to understand the content of the error and the handling method. For this reason, this embodiment makes it possible to identify the error content and the handling method and present them to the user. Note that there may be one error content / handling method for each error code.

[0050] Moreover, the function management information is an item related to the function of the display contents on the corresponding display operation unit. Therefore, the function management information has each item of operation button, display position, and function. The operation button is a type of operation area, and executes a function according to a user's specification. That is, the control signal creation unit 202 creates a control signal for executing the corresponding function, that is, the operation of the operation unit 208. As a result, the home appliance realizes a function according to the operation button.

[0051] The display position indicates the display position of the corresponding operation button on the home appliance, particularly on the display operation unit. The function indicates the content of the function executed by the corresponding operation button. The function includes execution of pairing between the user terminal 1 and the corresponding home appliance.

[0052] In addition, the error code of the error management information and the operation button of the function management information are examples of the state of the home appliance, and the error content / countermeasure method of the error management information and the function of the function management information are examples of the content. Note that the error content / countermeasure method may be at least one of the error content and the countermeasure method.

[0053] The error management information and the function management information may be treated as separate information. Furthermore, the home appliance management information 361 of the home appliance operation assistance device 3 is the same information as the home appliance management information 191, and has the same items. However, the home appliance management information 191 is information limited to the home appliance 2, that is, the home appliance 2 of the relevant user. In contrast, the home appliance management information 361 is not limited to a user, and is information on home appliances of multiple users managed using the home appliance operation assistance device 3. In this way, the home appliance management information 191 and the home appliance management information 361 are the same type of information, and either one may be omitted, or both of them may be used.

[0054] The above is the explanation of the information used in this embodiment, and next, the processing flow in this embodiment will be explained. Fig. 6 is a flowchart showing the processing flow in this embodiment. The contents will be explained below, with the main processing entity being explained using the functional blocks in Figs. 1 and 3.

[0055] First, as a premise of this processing flow, a state is assumed in which a user desires operational assistance based on display information, such as an error code being displayed on the display operation unit 207 of the home appliance 2. Other examples include a case in which the user desires to know the functions of the home appliance 2, or a case in which the user desires pairing between the home appliance 2 and the user terminal 1.

[0056] The flowchart will be described below. First, in step S1, a relevant function of the user terminal 1 is started in response to an operation by the user. Specifically, at least one of the state estimation unit 13, the content identification unit 14, and the display information creation unit 15 is started.

[0057] In step S2, the photographing unit 12 starts photographing and executes photographing processing. This step may be started by a photographing instruction from the user, or may be started by control of the state estimation unit 13 started in step S1. In this step and the following steps, the image data photographed by the photographing unit 12 is continuously displayed on the display screen of the display unit 18. At this time, the displayed image data also changes as the user terminal 1 moves and changes in direction.

[0058] In step S3, the state estimation unit 13 determines whether the image data includes operation-related information such as the display operation unit 207 or an error code. If the image data includes the related information, the state estimation unit 13 extracts the related information and uses it to estimate the state of the home appliance 2.

[0059] For this purpose, for example, in step S1, an input for identifying related information is accepted, and in this step, it is possible to determine whether related information is included based on the content of this input. This related information also includes operation areas in the display operation unit 207. The input for identifying related information also includes user requests such as checking an error code and checking each operation area, and function specifications for this purpose.

[0060] Moreover, the state of the home appliance 2 estimated by the state estimation unit 13 includes an error code and an operation area in the display operation unit 207. Furthermore, a pairing operation area may be estimated as the operation area. Such an estimation process including clipping can be performed using an image processing technique. For example, the display operation unit reference data of the home appliance management information 191 is used to identify the display operation unit. Furthermore, the error management information and the function management information of the home appliance management information 191 are used to clip the error code and the operation button. More specifically, the display position corresponding to the error code and the operation button of the home appliance management information 191 is used. At this time, it is desirable to use the clipped display operation unit 207 together.

[0061] In addition, in step S4, the content identification unit 14 identifies content corresponding to the estimated state of the home appliance 2 from the home appliance management information 191. That is, the error content / countermeasure or function corresponding to the estimated error code or operation button is identified. Note that in this step, the estimated function itself may be identified as the content. For example, the estimated operation button is identified as the content. Also, in this step, multiple contents may be identified. In this way, when multiple contents are identified, multiple states may be estimated in step S3, and there may be multiple contents corresponding to the estimated states.

[0062] In step S5, the display information creation unit 15 creates display information using the image data captured by the image capture unit 12 and the identified content. At this time, the display information may be created by superimposing the content on the image data using AR technology. In addition, when multiple contents are identified in step S5, it is desirable that the display information includes guidance information for guiding the display from any content to other content. In addition, the display information may be created so that multiple contents, for example, error contents and their countermeasures, are displayed on the same screen, or each content is displayed by changing the display according to the guidance. In addition, in step S6, the display information creation unit 15 displays the created display information on the display unit 18. In this display, as described above, the content may be guided according to the guidance information. In addition, the creation of the display information in step S5 and the display of the display information in step S6 may be executed so that the error contents and their countermeasures are displayed on the same screen.

[0063] Here, a specific example of the creation process and display process of the display information executed in steps S5 and S6 will be described. Fig. 7 is a flow chart showing a specific example of the creation process and display process of the display information in this embodiment. In Fig. 7, steps S51 to S54 correspond to step SS5, and steps S61 to S65 correspond to step S6.

[0064] First, in step S51, the display information creation unit 15 determines whether multiple contents are identified in step S4. As a result, if multiple contents are identified (YES), the process proceeds to step S52. If multiple contents are not identified (NO), the process proceeds to step S53.

[0065] Furthermore, in step S52, the display information creation unit 15 creates guidance information and embeds it in the identified content. The guidance information indicates guidance from each of a plurality of contents to other content (destination content), indicates a predetermined display order, and can be indicated, for example, in the form of an arrow to the other content. This display order includes an operation order for the home appliance 2. This operation order includes not only normal operations of the home appliance 2, but also an operation order for dealing with errors. The operation order for normal operations is used when a function is specified as the content, and the operation order for dealing with errors is used when error content / dealing method is specified as the content. Furthermore, in order to specify this display order, it is desirable to include information indicating the display order in the functions of the function management information of the home appliance management information 191. Furthermore, in step S53, if the identified content is an operation area such as an operation button, the display information creation unit 15 creates control information and embeds it in the content. This control information has contents according to the functions of the home appliance management information 191. Therefore, the display information creation unit 15 creates the control information based on the functions of the home appliance management information 191. Furthermore, the home appliance management information 191 may include the control information.

[0066] Furthermore, in step S54, the display information creation unit 15 creates display information by superimposing the content on the image data. Then, in step S61, the display information creation unit 15 displays the display information on the display unit 18. At this time, as described above, the error content and the method for dealing with the error included in the display information may be displayed on the same screen.

[0067] Furthermore, in step S62, the display information creation unit 15 determines whether it is necessary to display guidance information. To this end, the display information creation unit 15 determines whether guidance information is embedded in the display information. Then, the display information creation unit 15 determines whether a predetermined operation has been received from the user via the input unit 17 for the content in which the guidance information is embedded, particularly the operation area. For example, the predetermined operation may be a tap on the operation area, or a request for specification of a function content or its details. If the result of the determination in this step is that it is necessary to display guidance information (YES), the process proceeds to step S63. If it is not necessary (NO), the process proceeds to step S7.

[0068] Furthermore, in step S63, the display information creation unit 15 uses the display unit 18 to display guidance information to the destination content. In response to this, in step S64, the display information creation unit 15 determines whether the display has been moved (changed) to the destination content. This determination can be made based on whether the destination content is displayed. Alternatively, it may be determined based on whether the corresponding operation area in the image data is displayed. Note that, in order for the display to be moved in this manner, the user moves the user terminal 1 itself, changes its orientation, or receives an instruction to display the destination content. If the determination result in this step is that the content has been moved (YES), the process proceeds to step S65. If the content has not been moved (NO), the process returns to step S63.

[0069] Furthermore, in step S65, the display information creation unit 15 displays the display information in which the destination content is superimposed on the image data on the display unit 18. This completes the explanation of Fig. 7, and the processing flow of this embodiment will now be explained with reference back to Fig. 6.

[0070] More preferably, in step S7, the home appliance 2 is controlled in response to an operation from the user. To this end, the control command unit 16 creates a control command in response to the operation and notifies the home appliance 2 via the communication unit 11. As a result, the control signal creation unit 202 of the home appliance 2 creates a control signal in response to the control command and controls the operation of the operation unit 208.

[0071] Here, step S7 may be executed on the condition that the display information in which the control information is embedded has been created in step S6.

[0072] In the above description, each step in the flowchart of Fig. 6 has been described as being executed by the user terminal 1. However, at least a part of the steps may be executed by corresponding components (units) of the home appliance 2 or the home appliance operation assistance device 3. For example, step S3 and / or step S4 may be executed by the state estimation unit 32 or the content identification unit 33 of the home appliance operation assistance device 3. In this case, the image data captured by the imaging unit 12 or the state estimated by the state estimation unit 13 is notified to the home appliance operation assistance device 3 from the user terminal 1.

[0073] Also, at least one of steps S2 to S6 may be executed by the home appliance 2. In this case, image data captured by the home appliance 2 and an estimated state are notified from the home appliance 2 to the user terminal 1 or the home appliance operation assistance device 3. Display information created by the user terminal 1 or the home appliance operation assistance device 3 is notified to the home appliance 2, and the display operation unit 207 displays this. Furthermore, each step may be executed by the home appliance 2 or the home appliance operation assistance device 3 alone, or the user terminal 1, the home appliance 2, and the home appliance operation assistance device 3 may cooperate to execute the processing flow shown in the flowchart of FIG. 6. This concludes the description of this embodiment, and below, each example, which is a specific example of this embodiment, will be described. Note that in each example, the configuration and information of FIGS. 1 to 5 are used, so the processing flow and display contents will be described. EXAMPLES

[0074] The first embodiment is an example in which, as content, an error content, which is a type of malfunction, and a method for dealing with the error content are presented to a user. Details of the first embodiment will be described below with reference to the drawings. FIGS. 8A and 8B are flowcharts showing a processing flow in the first embodiment. In these flowcharts, the user terminal 1 and the home appliance operation assistance device 3 cooperate to execute the processing, but this is only an example, and there are various variations in the processing subject.

[0075] 8A, a malfunction, for example, an error, occurs in one of the home appliances 2. In addition, in step S102, the display operation unit 207 of the home appliance 2 displays an error code indicating the occurred error.

[0076] In response to this, the start-up process of step S1 is executed. Then, in step S103, the photographing unit 12 starts photographing. Then, in step S104, the state estimation unit 13 judges whether the display operation unit 207 on which the error code is displayed is photographed or displayed. This judgment may be made by judging whether the error code is photographed or displayed. As a result, if the display operation unit 207 is photographed (YES), the process transitions to step S106. If the display operation unit 207 is not photographed (NO), the process transitions to step S105.

[0077] Furthermore, in step S105, the state estimation unit 13 displays a change instruction on the display unit 18 so that the display operation unit 207 or the error code can be photographed. Furthermore, in step S106, the state estimation unit 13 fixes the photographed image data and displays it on the display unit 18. At this time, it is desirable for the state estimation unit 13 to extract the display operation unit 207 or the error code from the image data.

[0078] In step S107, the state estimation unit 13 performs image processing on the captured image data of the display operation unit 207 or the error code to estimate the model of the target home appliance 2. The model may be estimated when an error occurs or in advance using a two-dimensional code containing model and color information. The model may also be identified by reading the model name on the panel or estimated from the appearance of the product. In step S108, the state estimation unit 13 determines whether the user has accepted an instruction to confirm the displayed error code via the input unit 17. Depending on the user or the displayed error code, it may not be necessary to confirm the contents of the error code. For this reason, the user's wish to confirm is confirmed.

[0079] As a result, if a confirmation instruction has been received (YES), the process proceeds to step S110. If a confirmation instruction has not been received (NO), the process proceeds to step S109. Then, in step S109, the process ends, that is, the image capturing and display.

[0080] 8B, the state estimation unit 13 estimates the captured error code, which is an example of the state. That is, the code is identified by image recognition. In response to this, in step S111, the state estimation unit 13 notifies the home appliance operation assistance device 3 of the estimated error code via the communication unit 11.

[0081] In step S112, the communication unit 31 of the home appliance operation assistance device 3 receives the notified error code. In step S113, the content identification unit 33 uses the home appliance management information 361 to identify an error content / countermeasure, which is an example of a content corresponding to the received error code. At this time, it is preferable to identify a display position on the display operation unit 207 in the image data of the identified error content / countermeasure, from the error management information. In response to this, in step S114, the content identification unit 33 notifies the user terminal 1 of the identified error content / countermeasure via the communication unit 31. At this time, it is preferable to notify the identified display position as well.

[0082] In step S115, the communication unit 11 of the user terminal 1 accepts the notified error content / measures. In response to this, in step S116, if the display position is specified in step S113, the display information creation unit 15 uses this to specify the display position for the image data of the accepted content.

[0083] Then, in step S117, the display information creation unit 15 creates display information using the notified error content / measures and image data acquired by the imaging unit 12, and displays this on the display unit 18. Here, the error content and the measures included in the display information may be displayed on the same screen. Here, the same screen may be such that the error content and the measures are displayed by scrolling the display unit 18 (screen), or may be displayed on the same screen without scrolling or the like. This step is executed, for example, according to the flowchart shown in FIG. 7. Here, the display content in the first embodiment will be described.

[0084] 9A to 9D are diagrams showing the display contents of the display information on the display unit 18 in Example 1. The display contents transition in the order of Figs. 9A to 9D. First, in Fig. 9A, the display unit 18 has an image data display area 181 that displays image data captured by the imaging unit 12. In the image data display area 181, an IH cooker 21 (part), which is an example of the home appliance 2, is displayed.

[0085] Furthermore, boundary information 182-1 and error content / measures 182-2, which are examples of a content display area, are displayed on the display unit 18. Here, the boundary information 182-1 is related to the displayed error code and is information for distinguishing the corresponding operation area from others.

[0086] Moreover, the error content / solution method 182-2 indicates the content of the error, that the power is not turned on. Furthermore, guidance information 182-21 indicated by a "triangle" is displayed, and when specified by the user, the display content is changed to that shown in Fig. 9B, including the destination content.

[0087] In Fig. 9B, the details of the error content shown in Fig. 9A are displayed as the destination content. That is, as the error content / solution method 182-3, an image 182-4 indicating that the power plug needs to be inserted into the outlet and indicating the insertion is displayed. That is, a solution is displayed. Then, guidance information 182-31 is displayed, and when specified by the user, the display content is changed to that shown in Fig. 9C including the destination content. Note that this change in the display content may be performed when the display information creation unit 15 determines that the operation indicated in the error content / solution method 182-3 is performed. This determination can be performed by image processing. This determination can also be performed in the following processes in the same manner.

[0088] In Fig. 9C, the operation to be performed following the solution method shown in Fig. 9B is displayed as the destination content. That is, the error content / solution method 182-5 displays that it is necessary to open and close the door of the IH cooking appliance and turn on the power. Note that, as in Fig. 9B, the error content / solution method 182-5 in Fig. 9C may also include an image showing the solution method. Then, guidance information 182-51 is displayed, and when specified by the user, the display content is changed to that shown in Fig. 9D including the destination content.

[0089] In Fig. 9D, confirmation content regarding the countermeasure shown in Fig. 9C is displayed as the destination content. That is, as error content / countermeasure 182-6, whether to confirm the countermeasure again or to end the confirmation is displayed. As a result, if the user specifies to confirm again, the display transitions to the display content of Fig. 9A again. Also, if the user specifies to end the operation confirmation, the display of the corresponding error code is ended. This concludes the explanation of the display content of step S117, and the explanation of the processing flow continues by returning to Fig. 8B.

[0090] In step S118 of Fig. 8B, the display information creation unit 15 checks whether there are any other error codes that have not been dealt with or checked, that is, whether each error code has been dealt with. As a result, if each error code has been dealt with (YES), the process proceeds to step S119 and ends this processing flow. On the other hand, if each error code has not been dealt with (NO), the process proceeds to step S120. Then, in step S120, the display information creation unit 15 updates the display information to deal with the undealt with error codes, and performs the processing of step S117.

[0091] This concludes the description of the first embodiment, which can help users intuitively understand the content of an error corresponding to an error code and how to deal with the error. EXAMPLES

[0092] The second embodiment is an example of a content that supports a user in an operation for pairing a user terminal 1 with a home appliance 2. Details of the second embodiment will be described below with reference to the drawings. Fig. 10 is a flowchart showing a process flow in the second embodiment.

[0093] First, in step S201, the input unit 17 accepts a pairing instruction from a user. Then, steps S103 to S106 are executed in the same manner as in the first embodiment. Here, in step S106 in the second embodiment, the state estimation unit 13 cuts out the display operation unit 207 from the fixed image data.

[0094] In step S201, the state estimation unit 13 performs image recognition on the display operation unit 207 of the cut-out image data, and estimates a pairing button for executing pairing processing. In step S203, the content identification unit 14 identifies operation guidance to the pairing button. This operation guidance is the same type as the boundary information 182-1 in the first embodiment, and is information for distinguishing the pairing button from others. In addition, it may be in a format different from the boundary information 182-1. For example, the pairing button can be colored or displayed larger than other buttons. Then, the content identification unit 14 displays the image data on the display unit 18 by superimposing the operation guidance on the image data.

[0095] Then, in step S204, similar to step S117 in the first embodiment, the display information creation unit 15 creates display information and displays it on the display unit 18. Here, in step S204, similar to Fig. 9A, the display information creation unit 15 superimposes operation guidance on the pairing button of the display operation unit 207 in the image data. Also, the display information creation unit 15 superimposes, as content, on the image data, the fact that it is a pairing button.

[0096] As a result, the user can refer to this display and operate the home appliance 2 (display operation unit 207) to execute pairing processing. That is, in step S205, the display operation unit 207 of the home appliance 2 accepts a pairing operation from the user. Then, in step S206, the communication unit 206 notifies the user terminal 1 of a pairing request. In response to this, in step S207, the communication unit 11 of the user terminal 1 executes pairing processing between the user terminal 1 and the home appliance 2 in response to the pairing request. Note that steps S206 and S207 are not limited to the steps of FIG. 10 as long as the communication unit 206 and the communication unit 11 cooperate to execute the pairing processing. Then, when the user terminal 1 and the home appliance 2 are paired in step S207, the display processing ends in step S208.

[0097] This concludes the description of the second embodiment. According to the second embodiment, it is possible to appropriately support the user in the pairing operation in an intuitive and easy-to-understand manner. EXAMPLES

[0098] In the second embodiment, the pairing operation is supported, whereas in the third embodiment, operation support for executing various functions including functions other than the pairing function is performed. Details of the third embodiment will be described below with reference to the drawings. Fig. 11A and Fig. 11B are flowcharts showing a process flow in the third embodiment.

[0099] When the user desires operational assistance, the corresponding function of the user terminal 1 is started in the same manner as in step S1 of Fig. 6. Furthermore, steps S103 to S106 are executed in the same manner as in the first embodiment. Here, in step S106 of the second embodiment, the state estimation unit 13 cuts out the display operation unit 207 from the fixed image data.

[0100] In step S302, the state estimation unit 13 performs image recognition on the display operation unit 207 of the cut-out image data, and extracts various operation buttons on the display operation unit 207. In step S303, the content identification unit 14 identifies operation guidance to the estimated operation buttons. Then, the content identification unit 14 superimposes the operation guidance on the image data and displays it on the display unit 18. This operation guidance is the same as that in the first embodiment.

[0101] In step S304, the input unit 17 receives a designation of a button for which the user desires to have its function explained. At this time, the input unit 17 can receive designation of a plurality of operation buttons. Then, the content identification unit 14 identifies the display position of the designated operation button in the cut-out display operation portion 207.

[0102] Then, in step S305, the content identification unit 14 notifies the home appliance operation assistance device 3 via the communication unit 11 of the display position of the identified operation button.

[0103] In step S306, the communication unit 31 of the home appliance operation assistance device 3 receives the notified display position. In step S307, the content identification unit 33 identifies the function of the button specified in step S304. To this end, the content identification unit 33 uses the function management information of the home appliance management information 361. That is, the content identification unit 33 identifies the function corresponding to the display position received in step S306 from the function management information. Note that, if multiple operation buttons are specified in step S304, the functions corresponding to the respective operation buttons, that is, multiple functions, are identified. In step S308, the content identification unit 33 notifies the communication unit 31 of the identified functions for the buttons.

[0104] Furthermore, in step S309, the communication unit 11 of the user terminal 1 accepts the notified functions. Then, in step S310, the display information creation unit 15 creates display information using the notified functions and image data acquired by the photographing unit 12, and displays this on the display unit 18. This is the same process as step S117 in the first embodiment, but the display contents are different. Here, the display contents in the third embodiment will be described. Here, in addition to the display in step S310, the display contents of the operation guidance in steps S106 and S303 will also be described.

[0105] 12A to 12E are diagrams showing the display contents of the display information on the display unit 18 in Example 3. Here, FIG. 12A is displayed in step S106, and FIG. 12B is displayed in step S303. Then, in step S310, the display transitions to the order of FIG. 12C to FIG. 12ED.

[0106] First, in Fig. 12A, the display unit 18 has an image data display area 181 that displays image data captured by the imaging unit 12. Then, an IH cooker 21, which is an example of a home appliance 2, is displayed in the image data display area 181. Furthermore, boundary information 184-1 is superimposed on the display operation portion 207 of the IH cooker 21 cut out on the image data. Then, when an operation button is extracted in step S302, the process proceeds to step S303, and the button is displayed as shown in Fig. 12B.

[0107] 12B, a cut-out explanation area 184-2 is displayed indicating that the extracted item is the display operation section. Also, the cut-out display operation section 207 is displayed in the image data display area 181 of the display section 18. Then, on the display operation section 207, boundary information 184-3 is superimposed on each of the cut-out operation buttons. As a result, the user can easily specify the operation button (step S304).

[0108] Then, in step S310, the display shown in FIG. 12C to FIG. 12E is performed. First, when the user terminal 1 accepts a function from the home appliance operation assistance device 3, the display shown in FIG. 12C is performed. In FIG. 12C, since the leftmost operation button is specified in step S304, an operation guide 184-5 is superimposed on the corresponding location. Furthermore, a function display area 184-4 indicating the function accepted in step S309 is displayed. In the function display area 184-4 of FIG. 12C, it is displayed that the specified operation button is a "manual button" and the contents of the function. Furthermore, in the function display area 184-4, guidance information 184-41 is displayed as in the first embodiment. When this guidance information 184-41 is specified, the display contents are changed to FIG. 12D.

[0109] In FIG. 12D, guidance information 184-61 for displaying the functions of the specified "manual button" and details of these functions is displayed in the function display area 184-6. Then, when any of the guidance information 184-61 is selected, the display content is changed to FIG. 12E. In FIG. 12E, details of the function "grill heating" corresponding to the selected guidance information 184-61 are displayed in the function display area 184-7. In this way, the user can understand the functions of the operation area exemplified by various operation buttons of the home appliance 2 such as the IH cooker 21. This concludes the explanation of the display content of the third embodiment, and the explanation of the processing flow is continued by returning to FIG. 11B.

[0110] In step S311 of Fig. 11B, the control instruction unit 16 determines whether or not the user has specified an operation mode via the input unit 17. Here, the operation mode refers to a mode for operating the home appliance 2 using the display content of the display unit 18 of the user terminal 1. If the result of the determination in this step is that the specification of the operation mode has been accepted (YES), the process proceeds to step S312. On the other hand, if the specification has not been accepted (NO), the process continues to step S311.

[0111] Furthermore, in step S312, input unit 17 accepts a designation for an operation button of a function that the user wishes to operate. In this step, designation for the display content of display unit 18 is accepted. Here, the display content can be any of those shown in Figs. 12B to 12E.

[0112] In step S313, the content identification unit 14 notifies the home appliance operation assistance device 3 of the display position of the specified operation button via the communication unit 11. In step S314, the communication unit 31 of the home appliance operation assistance device 3 receives the notified display position. In step S315, the content identification unit 33 identifies the function of the button specified in step S312. Here, steps S313 to S315 can be realized in the same way as steps S305 to S307. In step S316, the control command unit 35 creates a control command for executing the function identified in step S315. Then, in step S317, the control command unit 35 notifies the home appliance 2 via the communication unit 31 of the created control command.

[0113] Furthermore, in step S318, the communication unit 206 of the home appliance 2 receives a control command. In response to this, in step S319, the control signal creation unit 202 creates a control signal according to the control command. Then, in step S320, the control signal creation unit 202 executes control by outputting the created control signal to the corresponding operation unit 208. As a result, the operation unit 208 executes the corresponding function.

[0114] This concludes the description of the process flow of the third embodiment, but the third embodiment is not limited to the above. For example, the notification of the control command in step S317 may be sent via the user terminal 1. Also, the control signal itself may be generated by the home appliance operation assistance device 3 or the user terminal 1. According to the third embodiment, it is possible to appropriately assist the user in operating the home appliance 2 in an intuitively easy-to-understand manner.

[0115] This concludes the description of the embodiments of the present invention, but the present invention is not limited to the above-mentioned embodiments. For example, the processing entity of the processing flow is not limited to the above-mentioned examples, and at least a part of the processing of the user terminal 1 can be executed by the home appliance 2 or the home appliance operation assistance device 3. Similarly, at least a part of the processing of the home appliance 2 or the home appliance operation assistance device 3 may be executed by another device such as the user terminal 1. [Explanation of symbols]

[0116] 1...user terminal, 11...communication unit, 12...photography unit, 13...state estimation unit, 14...content identification unit, 15...display information creation unit, 16...control command unit, 17...input unit, 18...display unit, 19...storage unit, 191...home appliance management information, 2...home appliance, 21...induction cooker, 22...washing machine, 23...refrigerator, 201...control unit, 202...control signal creation unit, 203...state estimation unit, 204...content identification unit, 205...display information creation unit, 206...communication unit, 207...display operation unit, 208...operation unit, 209...sensor, 3...home appliance operation support device, 31...communication unit, 32...state estimation unit, 33...content identification unit, 34...display information creation unit, 35...control command unit, 36...storage unit, home appliance management information 361, 4...router, 5...network

Claims

1. In a home appliance operation support system for supporting the operation of home appliances, An imaging unit that captures an image of the home appliance and acquires image data; A state estimation unit that estimates a state of the home appliance by using the image data; a content identification unit that identifies content related to an operation of the home appliance according to the estimated state; a display information generating unit that generates display information by superimposing the image data and the content; A home appliance operation support system having a display unit that displays the display information.

2. The home appliance operation support system according to claim 1, The home appliance has a display panel that displays an error code indicating a malfunction occurring in the home appliance, The state estimation unit estimates the error code using the image data, The content identification unit is a home appliance operation support system that identifies the content of the malfunction indicated by the error code and a method of dealing with the malfunction.

3. The home appliance operation support system according to claim 2, The display unit is a home appliance operation support system that displays the content of the malfunction and the solution to the malfunction on the same screen.

4. The home appliance operation support system according to claim 2, The state estimation unit fixes the image data when the image data includes the display panel, and estimates the state using the fixed image data.

5. The home appliance operation support system according to claim 1, The home appliance has a display panel that displays a pairing operation area for pairing with a user terminal used by a user, The state estimation unit estimates a pairing operation area on the display panel; The content identification unit identifies an operation guidance to the pairing operation area, The display information creation unit creates the display information by superimposing the pairing operation area and the operation guidance on a display panel of the image data.

6. The home appliance operation support system according to claim 5, The state estimation unit fixes the image data when the image data includes the display panel, and estimates the state using the fixed image data.

7. The home appliance operation support system according to claim 1, The home appliance has a display operation panel including a plurality of operation areas for operating the home appliance, The state estimation unit estimates the operation area by using the image data; The content identification unit is a home appliance operation support system that identifies a function to be operated based on the operation area.

8. The home appliance operation support system according to claim 7, The display unit displays the image data, The state estimation unit extracts an operation region included in the image data, The display unit displays the extracted operation area in a manner that distinguishes it from other areas, The content identification unit is configured to, when receiving a designation for the differentiated operation area, identify a function of the operation area.

9. Computers to assist in the operation of home appliances A state estimation unit that estimates a state of the home appliance using image data acquired by photographing the home appliance by the photographing unit; a content identification unit that identifies content related to an operation of the home appliance according to the estimated state; a display information generating unit that generates display information by superimposing the image data and the content; A home appliance operation assistance program that causes the display unit to function as a display unit that displays the display information.

10. 10. The home appliance operation assistance program according to claim 9, The home appliance has a display panel that displays an error code indicating a malfunction occurring in the home appliance, The state estimation unit estimates the error code using the image data, The content identification unit is a home appliance operation assistance program that identifies the content of the malfunction indicated by the error code and a method of dealing with the malfunction.

11. The home appliance operation assistance program according to claim 10, The display unit is a home appliance operation assistance program that displays the content of the malfunction and the solution to the malfunction on the same screen.

12. The home appliance operation assistance program according to claim 10, The state estimation unit fixes the image data when the image data includes the display panel, and estimates the state using the fixed image data.

13. 10. The home appliance operation assistance program according to claim 9, The home appliance has a display panel that displays a pairing operation area for pairing with a user terminal used by a user, The state estimation unit estimates a pairing operation area on the display panel; The content identification unit identifies an operation guidance to the pairing operation area, The display information creation unit is a home appliance operation assistance program that creates the display information by superimposing the pairing operation area and the operation guidance on a display panel of the image data.

14. The home appliance operation assistance program according to claim 13, The state estimation unit fixes the image data when the image data includes the display panel, and estimates the state using the fixed image data.

15. 10. The home appliance operation assistance program according to claim 9, The home appliance has a display operation panel including a plurality of operation areas for operating the home appliance, The state estimation unit estimates the operation area by using the image data; The content identification unit is a home appliance operation assistance program that identifies a function to be operated based on the operation area.

16. The home appliance operation assistance program according to claim 15, The display unit displays the image data, The state estimation unit extracts an operation region included in the image data, The display unit displays the extracted operation area in a manner that distinguishes it from other areas, The home appliance operation assistance program, when the content identification unit receives a designation for the differentiated operation area, identifies a function of the operation area.

Images