Household electrical appliance management system, application program, and moving image creation method

Abstract

To obtain a household electrical appliance management system with a new configuration that can easily create moving images in which a plurality of operation screens are transitioned.SOLUTION: In a household electrical appliance management system, when an application program is executed by a computer, the application program causes the computer to execute a step of recording the history of operation of household electrical appliances, and a step of, when there is input of instructing creation of moving images indicating operating procedures of the household electrical appliances, transmitting operation history information including the history of operation of the household electrical appliances to a server device. The server device has an acquisition unit that acquires the operation history information from the application program, and a creation unit that creates, on the basis of the operation history information, moving images in which a plurality of operation screens are transitioned in the order of operations of the household electrical appliances.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Embodiments of the present invention relate to a home appliance management system, an application program, and a video generation method.

Background Art

[0002] In a home appliance management system that manages a plurality of home appliances connected to a network, there is known a technique of remotely operating a home appliance with an application executed on a terminal device such as a smartphone. When a problem occurs in such an application and an inquiry is made to a customer center or the like, the operation screen displayed by the application records how the screen transitions by the user's operation and sends it to the customer center, making it easier to quickly identify the cause of the problem.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in order to record the operation screen, for example, the user has to record the operation screen while adjusting the operation timing of the terminal device, which imposes complicated work on the user.

[0005] An example of the problem to be solved by the present invention is to obtain a home appliance management system, an application program, and a video generation method with a novel configuration that can easily generate a video in which a plurality of operation screens transition.

Means for Solving the Problems

[0006] The home appliance management system according to one embodiment of the present invention includes a server device for managing home appliances, and an application program to be executed on a computer of a terminal device that is connected to the server device via a network and includes a display device. When the application program is executed by the computer, the application program causes the computer to record an operation history of the home appliance, and when there is an input instructing generation of a video showing an operation procedure of the home appliance, transmit operation history information including the operation history of the home appliance to the server device. The server device has an acquisition unit that acquires the operation history information from the application program, and a generation unit that generates a video in which a plurality of operation screens transition in the operation order of the home appliance based on the operation history information.

[0007] In the home appliance management system, for example, the video includes the plurality of operation screens from an operation screen that is an operation starting point of the home appliance to a final operation screen in the operation history information.

[0008] In the home appliance management system, for example, the video includes the plurality of operation screens from an operation screen that is an operation starting point of the home appliance to a final operation screen in the operation history information with the shortest operation procedure.

[0009] In the home appliance management system, for example, the application program causes the computer to receive an input for selecting an operation screen that is a start screen of the video, and transmit input information of the operation screen that is the start screen of the video to the server device.

[0010] In the home appliance management system, for example, the application program causes the computer to receive an input for selecting an operation screen that is an end screen of the video, and transmit input information of the operation screen that is the end screen of the video to the server device.

[0011] In the home appliance management system, for example, for each of the plurality of operation screens, a display time in the video is preset.

[0012] In the home appliance management system, for example, the application program causes the computer to execute a step of receiving an input for designating a display time in the video of at least one of the plurality of operation screens, and a step of transmitting input information of the display time to the server device.

[0013] In the home appliance management system, for example, when there is an input for instructing generation of the video, the application program causes the computer to execute a step of transmitting the operation history information including event information indicating an operation event corresponding to at least one of the plurality of operation screens to the server device.

[0014] In the home appliance management system, for example, when there is an input for instructing generation of the video, the application program causes the computer to execute a step of transmitting the operation history information including event information corresponding to at least one of the plurality of operation screens and indicating a communication event with the server device to the server device.

[0015] In the home appliance management system, for example, the generation unit generates the video by superimposing information indicating an operation location in the operation screen corresponding to the event information based on the event information.

[0016] In the home appliance management system, for example, the generation unit generates the video by superimposing the event information on the operation screen corresponding to the event information.

[0017] In the home appliance management system, for example, the server device causes the computer to execute a step of remotely operating the application program and transmitting the operation history information to the server device.

[0018] An application program according to an embodiment of the present invention is an application program for causing a computer of a terminal device including a display device to be executed by being connected to a server device that manages home electric appliances via a network, and when the application program is executed by the computer, receiving an input for instructing generation of a video showing an operation procedure of the home electric appliance; recording an operation history of the home electric appliance when there is an input for instructing generation of the video; and generating a video in which a plurality of operation screens transition in the operation order of the home electric appliance at respective predetermined display times based on operation history information including the operation history of the home electric appliance, and causing the computer to execute the above steps.

[0019] A video generation method according to an embodiment of the present invention is a video generation method executed by a home electric appliance management system including a server device that manages home electric appliances and an application program for causing a computer of a terminal device including a display device and connected to the server device via a network to be executed, and when the application program is executed by the computer, receiving an input for instructing generation of a video showing an operation procedure of the home electric appliance; recording an operation history of the home electric appliance when there is an input for instructing generation of the video; and generating a video in which a plurality of operation screens transition in the operation order of the home electric appliance at respective predetermined display times based on operation history information including the operation history of the home electric appliance.

[0020] According to the above home electric appliance management system, application program, and video generation method, a video in which a plurality of operation screens transition can be easily generated.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

[0022] Hereinafter, embodiments of the air conditioner according to the present disclosure will be described with reference to the drawings. In this specification, the components according to the embodiments and the descriptions of the components may be described in a plurality of expressions. The components and their descriptions are examples and are not limited by the expressions in this specification. The components may be specified by different names from those in this specification. Also, the components may be described by expressions different from those in this specification.

[0023] In addition, the drawings are schematic, and the dimensional relationships of the respective elements, the ratios of the respective elements, etc. may be different from the actual ones. Also, there may be parts where the dimensional relationships and ratios of the drawings are different from each other. In this specification, ordinal numbers are used only for distinguishing parts, members, sites, positions, directions, etc., and do not indicate order or priority.

[0024] (Configuration example of home appliance system) FIG. 1 is a system configuration diagram showing an example of a home appliance device management system 1 according to an embodiment. As shown in FIG. 1, the home appliance device management system 1 of the embodiment includes a server device 10 and a terminal device 20 of a user U, and controls and manages a plurality of home appliance devices 30 (30A, 30B, 30C, 30D ···). Note that the configuration including the server device 10 and the terminal device 20 of the user U and a plurality of home appliance devices 30 may be hereinafter referred to as a home appliance system.

[0025] The plurality of home appliance devices 30 may include home appliance devices having various functions. The refrigerator 30A and the oven range 30B installed in the kitchen, the washing machine 30C installed in the washroom, and the air conditioners 30D (30Da, 30Db, 30Dc) installed in the living room, study, and bedroom, respectively, shown in FIG. 1 are examples of such home appliance devices 30.

[0026] The server device 10 is connected to a network NT such as the Internet, and can communicate with a plurality of home appliance devices 30 via a modem MD and a router RT also connected to the network NT. Further, the server device 10 can communicate with the terminal device 20 connected directly to the network NT or via the modem MD and the router RT or the like.

[0027] With the above configuration, the server device 10 manages a plurality of home appliance devices 30, relays communication between the terminal device 20 and the home appliance devices 30, and enables the terminal device 20 to operate the home appliance devices 30. Note that the modem MD and the router RT may be included in the above-described home appliance device management system 1.

[0028] The terminal device 20 is a smartphone, a tablet terminal, a notebook-type or desktop-type personal computer, etc. An application program for remotely operating the home electric appliance 30 is installed in the terminal device 20. The application program may be hereinafter referred to as an "application" in some cases.

[0029] When the user U starts the application on the terminal device 20 and performs various operations, the application communicates with the home electric appliance 30 via the server device 10 via the network NT. As a result, the user U can remotely operate the home electric appliance 30 from the terminal device 20.

[0030] Here, in the home electric appliance management system 1 capable of remotely operating the home electric appliance 30 from the application of the terminal device 20 by relaying through the server device 10, there is a need to capture the state of operation of the home electric appliance 30 by the application as a video. Such a need may occur, for example, when a problem occurs in the application or the like and correspondence by the customer center is required, or when the user U uploads an introduction video of the home electric appliance 30 to an SNS (Social Networking Service) or the like.

[0031] In the home electric appliance management system 1 of the embodiment, when the application is started on the terminal device 20 and various operations are performed, the user U can instruct the generation of a video indicating the operation content for the home electric appliance 30.

[0032] When a video generation instruction is input from the terminal device 20, the application transmits a snapshot of the operation screen sequentially displayed on the terminal device 20 to the server device 10. The server device 10 converts the snapshot received from the application into a video format and generates a video showing the state in which a plurality of operation screens are transitioning.

[0033] (Configuration example of the terminal device) FIG. 2 is a block diagram showing an example of the functional configuration of the terminal device 20 according to the embodiment. As shown in FIG. 2, the terminal device 20 of the embodiment includes an application (application program) 201, a communication unit 202, a display unit 203, an input unit 204, and a storage unit 205. The application 201 includes an acquisition unit 211, an input / output control unit 212, and a control unit 213.

[0034] The terminal device 20 is configured as a computer including a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc., not shown. In the ROM of the terminal device 20, for example, a control program for realizing the functions of the terminal device 20, an application 201 for remotely operating the home electric appliance 30, etc. are stored.

[0035] By the CPU expanding and executing the control program stored in the ROM in the RAM, the communication unit 202, the display unit 203, the input unit 204, and the storage unit 205 described above are realized as functional units of the terminal device 20. However, at least a part of these functional units of the terminal device 20 may be realized by an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like.

[0036] Also, by the CPU expanding and executing the application 201 stored in the ROM in the RAM, the acquisition unit 211, the input / output control unit 212, and the control unit 213 described above are realized as functional units of the terminal device 20.

[0037] Note that the application 201 can be provided by recording it on a recording medium readable by a computer device such as a CD (Compact Disc)-ROM (Read Only Memory), a flexible disk (FD), a CD-R (Recordable), or a DVD (Digital Versatile Disk) in an installable or executable file format.

[0038] Alternatively, the application 201 may be configured to be stored on a computer such as a server connected to a network such as the Internet and downloaded via the network. Further, the application 201 may be configured to be provided or distributed via a network such as the Internet.

[0039] The application 201 has a module configuration including the above-described functional units. As actual hardware, the CPU of the terminal device 20 reads a program from the ROM and executes it, so that the above functional units are loaded onto a main storage device such as a RAM and generated on the main storage device.

[0040] The acquisition unit 211 acquires various information regarding the home appliances 30, such as the states of the plurality of home appliances 30, via the communication unit 202 of the server device 10 and the terminal device 20.

[0041] The input / output control unit 212 causes the display unit 203 to display various information regarding the home appliance 30 acquired from the home appliance 30. Further, the input / output control unit 212 receives an operation performed by the user U on the input unit 204.

[0042] The control unit 213 transmits various instruction contents from the user U received by the input / output control unit 212 to the server device 10 via the communication unit 202.

[0043] Further, when the input / output control unit 212 receives a video generation instruction from the user U, thereafter, the control unit 213 records in the storage unit 205 a snapshot 251 of the operation screen displayed on the display unit 203 according to the operation of the user U, and event information 252 such as operation events and communication events performed on those operation screens.

[0044] The event information 252 such as operation events and communication events is, for example, an event log or the like. Operation events performed on a predetermined operation screen include, for example, pressing a button displayed on the operation screen. Communication events performed on a predetermined operation screen include, for example, transmitting an operation signal for operating the home appliance 30 to the server device 10.

[0045] Thereafter, when the input / output control unit 212 receives an end instruction for video generation from the user U, the control unit 213 generates operation history information 253 of the home appliance 30 including the respective snapshots 251 of the plurality of operation screens recorded between the start instruction and the end instruction of video generation, and event information 252 such as operation events and communication events. Further, the control unit 213 transmits the generated operation history information 253 to the server device 10 via the communication unit 202.

[0046] The communication unit 202 is a communication circuit or the like that realizes a communication function for communicating with an external device. In the terminal device 20 of the embodiment, the communication unit 202 establishes communication with the server device 10.

[0047] The display unit 203 is configured by a display device such as a liquid crystal panel.

[0048] The input unit 204 is composed of a touch panel switch formed on the display screen of the display unit 203, a mechanical switch provided around the display unit 203, and the like.

[0049] The storage unit 205 is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. By the above-described control unit 213, in the storage unit 205, for example, snapshots 251 of respective operation screens, event information 252 such as operation events and communication events, and operation history information 253 including these are stored.

[0050] (Configuration example of server device) FIG. 3 is a block diagram showing an example of the configuration of the server device 10 according to the embodiment. As shown in FIG. 3, the server device 10 of the embodiment is configured as a computer having a CPU 11, a RAM 12, and a ROM 13. The server device 10 also includes a communication interface (I / F) 14, a storage device 15, and an encoder 16.

[0051] The CPU 11 is a hardware processor that controls the entire server device 10.

[0052] The RAM 12 functions as a primary storage device and serves as a working area for the CPU 11. The RAM 12 may be, for example, a volatile semiconductor memory or the like.

[0053] The ROM 13 functions as a storage area in the server device 10, and stores, for example, a control program executed by the CPU 11 to realize the functions of the server device 10. The information stored in the ROM 13 is retained even when the power of the server device 10 is turned off.

[0054] Note that the control program executed by the server device 10 can be provided by recording it in a recordable medium readable by a computer device such as a CD-ROM, FD, CD-R, DVD, etc. in an installable format or an executable format file. Also, the control program may be provided or distributed via a network NT such as the Internet.

[0055] The communication I / F 14 is a communication device or the like having a predetermined communication protocol, and is configured to be connectable to, for example, the network NT. Through the communication I / F 14, the server device 10 can communicate with the terminal device 20 and the plurality of home appliances 30.

[0056] The storage device 15 is an HDD, an SSD, or the like, and functions as an auxiliary storage device for the CPU 11.

[0057] The encoder 16 generates a video in which a plurality of operation screens transition based on the operation history information 253 of the home appliance 30 transmitted from the application 201 in response to a video generation instruction by the user U.

[0058] FIG. 4 is a block diagram showing an example of the functional configuration of the server device 10 according to the embodiment. As shown in FIG. 4, the server device 10 of the embodiment includes, as functional units, an acquisition unit 101, a processing unit 102, a generation unit 103, an operation unit 104, a transmission unit 105, and a storage unit 106.

[0059] These functional units are realized, for example, when the above-described CPU 11 expands and executes a control program stored in the ROM 13 in the RAM 12. However, at least a part of the above functional units of the server device 10 may be realized by an FPGA or an ASIC or the like.

[0060] The acquisition unit 101 communicates with the application 201 of the terminal device 20 to acquire operation history information 253 such as the operation content for the home appliance 30 and the snapshot 251 when there is a video generation instruction. Further, the acquisition unit 101 communicates with the plurality of home appliances 30 to acquire various information such as the states of these home appliances 30.

[0061] The processing unit 102 performs preliminary processing for video generation on the snapshot 251 acquired from the application 201 of the terminal device 20.

[0062] As a preprocessing for the snapshot 251, for example, there is a process of deleting unnecessary parts included in the snapshot 251, that is, a notification area displayed at the corner of the operation screen by the OS (Operating System) of the terminal device 20, an icon indicating a processing waiting state, etc., and parts unrelated to the operation of the home appliance 30.

[0063] In addition, the operations of the user U after the video generation instruction may include some useless operations such as going back and forth between several operation screens several times or entering an incorrect operation screen. As a preprocessing for the snapshot 251, the processing unit 102 deletes the snapshot 251 of the operation screen displayed by these useless operations, and extracts the snapshot 251 of the operation screen that is displayed when the shortest operation procedure is taken from the operation screen at the start instruction of video generation to the operation screen at the end instruction of video generation, and arranges them in order of operation. The operation order at this time may be, for example, the ideal operation order when the shortest operation from the start screen to the end screen of the video is performed.

[0064] In addition, when arranging the snapshots 251 of the operation screen, the rearrangement of the snapshots 251 may be performed so that the video starts from the operation screen that is the basis of the operation of the home appliance 30, rather than the screen that was displayed when the video generation instruction was given. The operation screen that is the basis of the operation of the home appliance 30 is, for example, the home screen or the top screen of each home appliance 30. When arranging the snapshots 251 of the plurality of operation screens, for example, the rearrangement may be performed so that the snapshots 251 of these home screens or top screens are at the beginning.

[0065] In addition, the processing unit 102 may superimpose a display that emphasizes the pressed button on the snapshot 251 so that the operation content such as pressing the button on each operation screen can be known, or may superimpose an icon of a finger or the like on or near the pressed button. In addition, the processing unit 102 may superimpose operation events, communication events, etc. on the snapshot 251.

[0066] Note that the processing unit 102 performs the above various preprocesses by referring to event information 252 such as operation events and communication events.

[0067] The generation unit 103 combines the snapshots 521 preprocessed by the processing unit 102 and converts them into a video format to generate a video 161 in which a plurality of operation screens transition. At this time, the generation unit 103 adjusts the display time of each operation screen in the video 161.

[0068] The display time of each operation screen is set in advance in, for example, the storage unit 106 so as to be a time suitable for viewing the video. The display time of each operation screen may be different for each operation screen, or may be the same for all operation screens. When the display time is made different for each operation screen, for example, when there are many display items, or when operation events or communication events are superimposed, a longer display time can be set for an operation screen with a large amount of information than for other operation screens.

[0069] The generation unit 103 stores the video 161 generated as described above in, for example, the storage unit 106. When the video 161 is generated for the customer center's response to a defect in the application 201 or the like, the customer center can download the video 161 from the storage unit 106 of the server device 10. Also, when the video 161 is required by the user U, for example, for uploading to an SNS, the application 201 of the terminal device 20 can download the video 161 from the storage unit 106 of the server device 10 according to the instruction of the user U.

[0070] The operation unit 104 transmits the operation signal from the application 201 of the terminal device 20 acquired by the acquisition unit 101 to the corresponding home appliance 30 via the transmission unit 105. As a result, the target home appliance 30 performs various operations according to the operation signal.

[0071] The transmission unit 105 is a communication circuit or the like that realizes a communication function for communicating with an external device. In the server device 10 of the embodiment, the transmission unit 105 establishes communication with the applications 201 of the plurality of home appliances 30 and the terminal device 20.

[0072] The storage unit 106 is composed of a storage device such as an HDD. The storage unit 106 stores various information and data such as the video 161 generated by the generation unit 103 and the operation history information 253 acquired from the application 201.

[0073] (Example of video generation) Next, with reference to FIGS. 5 to 7, an example of the video generation operation by the home appliance management system 1 of the embodiment will be described. FIGS. 5 to 7 are schematic diagrams showing how the home appliance management system 1 according to the embodiment generates a video. In the example of FIGS. 5 to 7, a case where a video is generated for an operation of changing the settings of the refrigerator 30A among the plurality of home appliances 30 is shown.

[0074] The user U starts the application 201 on the terminal device 20 and selects an item "Start recording operation", for example, from the menu screen.

[0075] Thereafter, the control unit 213 of the application 201 associates and records a snapshot of the operation screen displayed on the display unit of the terminal device 20 in response to the operation of the user U, as well as snapshots corresponding to operation events and communication events generated in response to the operation of the user U. While the user U performs various operations from the application 201, the recording of these snapshots, operation events, and communication events is performed in the background.

[0076] When the user U finishes a series of operations to be made into a video, the user U selects an item "End recording operation", for example, from the menu screen. Thereby, the recording of the snapshot of the operation screen, operation events, and communication events by the control unit 213 is ended.

[0077] As shown in FIG. 5, by the above operation, snapshots of each of a plurality of operation screens displayed by a series of operations by the user U are recorded. Also, operation events related to the operations performed on the operation screen, communication events related to the communication performed between the server device 10, etc. are recorded in association with these snapshots. In the example of FIG. 5, a plurality of snapshots shown in FIGS. 5(a) to 5(g), as well as operation events and communication events (not shown) associated with these are obtained.

[0078] Here, in the example of FIG. 5, it is assumed that the user U has traveled back and forth several times between the operation screens shown in FIGS. 5(b) and 5(c) (FIGS. 5(b)(c)(b)'(c)'). Also, after reaching the operation screen shown in FIG. 5(c)', it is assumed that several unnecessary operations are performed on that operation screen (FIGS. 5(c)'(c)''(c)''').

[0079] The control unit 213 reads out from the storage unit 106 the snapshots from FIGS. 5(a) to 5(g) recorded between "start recording operations" and "end recording operations", as well as the operation events and communication events associated with these snapshots, and generates operation history information including these. Also, the control unit 213 transmits the generated operation history information to the server device 10 via the communication unit 202.

[0080] In this way, the control unit 213 may automatically transmit the operation history information to the server device 10 after generating it, or may transmit the operation history information to the server device 10 when a transmission instruction from the user U is additionally given.

[0081] When receiving the operation history information from the application 201 of the terminal device 20, the processing unit 102 of the server device 10 performs preliminary processing for video generation while referring to the operation events, communication events, etc. associated with each of the snapshots for the snapshots included in the operation history information.

[0082] As shown in FIG. 6, the processing unit 102 deletes overlapping and unnecessary snapshots from among a plurality of snapshots, such as those shown in FIGS. 5(b), (c), (c)”, and (c)”’ described above. Further, the processing unit 102 rearranges, in the order of operations, the snapshots of a plurality of screens leading to the setting completion screen, with, for example, the home screen serving as the operation base point of the home appliance 30, at the beginning, as necessary. Also, the processing unit 102 deletes from each snapshot unnecessary portions DP such as the notification area displayed at the corner of the operation screen and icons indicating the processing wait state, etc., by the OS of the terminal device 20.

[0083] Furthermore, so that the operation locations on each operation screen can be understood, for example, the processing unit 102 may highlight an operation button or the like that has been operated, or superimpose an icon of a finger or the like near the button, or may also superimpose operation events, communication events, etc. In the example of FIG. 6, the processing unit 102 superimposes an icon MK of a finger near each button.

[0084] Note that such an icon MK of a finger or the like is preferably superimposed in a manner such that the size, color, position, etc. do not interfere with video viewing. The same applies when superimposing an emphasis display on the operation screen.

[0085] The generation unit 103 joins together these snapshots and converts them into a video format so that the plurality of snapshots processed as described above are displayed at predetermined time intervals. Thereby, for example, the videos shown in FIGS. 7(a) to 7(g) are generated.

[0086] FIG. 7(a) is the home screen displayed at the beginning of the video. On the home screen, icons of a plurality of home appliances 30 that can be remotely operated by the application 201 of the user U's terminal device 20 are displayed, and the user U can select the home appliance 30 to be operated from among them. In the example of FIG. 7(a), an icon MK of a finger is superimposed near the icon indicating the refrigerator 30A among the plurality of home appliances 30, and it can be seen that in this video, the user U is operating the refrigerator 30A as the operation target.

[0087] Figure 7(b) is the top screen related to the operation of the refrigerator 30A after transitioning from the home screen when the icon of the refrigerator 30A is selected. On the top screen of the refrigerator 30A, the current status of the refrigerator 30A and various setting contents are displayed, and the user U can, for example, change a predetermined setting of the refrigerator 30A. In the example of Figure 7(b), among a plurality of statuses and setting items, an icon MK of a finger is superimposed near the setting item of the chill room. From this, it can be understood that the operation purpose of the user U is to change the setting of the chill room or the like.

[0088] Figure 7(c) is the setting screen of the chill room that has transitioned from the top screen related to the refrigerator 30A when the setting item of the chill room is selected. On the setting screen of the chill room, it is shown that the current setting of the chill room is "Normal Chill", and setting items that can be set for the chill room such as "Normal Chill", "Quick Fresh Chill", and "Defrost Mode" are shown. In the example of Figure 7(c), among a plurality of setting items, an icon MK of a finger is superimposed near the setting item of "Quick Fresh Chill", and it can be understood that the user U is trying to change the setting of the chill room from "Normal Chill" to "Quick Fresh Chill".

[0089] Figure 7(d) is the setting change screen that has transitioned from the setting screen of the chill room when a setting item different from the current one is selected. On the setting change screen, a "Determine" button and a "Cancel" button are displayed, and the user U can confirm whether there is an error in the setting item selected on the setting screen of the chill room. In the example of Figure 7(d), an icon MK of a finger is superimposed near the "Determine" button, and it can be understood that the user U has determined to select "Quick Fresh Chill" as the selected item.

[0090] FIG. 7(e) is a confirmation screen that transitions from the chill room setting change screen when "Quick Fresh Chilled" is determined as a selection item. On the confirmation screen, a button for confirming the selection item and a "Cancel" button are displayed, and user U can select whether to confirm the change of the chill room setting. In the example of FIG. 7(e), a finger icon MK is superimposed near the "Set to Quick Fresh Chilled" button, indicating that user U has confirmed the setting change to "Quick Fresh Chilled".

[0091] FIG. 7(f) is a guidance screen that transitions from the confirmation screen when the setting change to "Quick Fresh Chilled" is confirmed. On the guidance screen, confirmation items such as the usage method and precautions of the chill room under the setting of "Quick Fresh Chilled" are displayed, and user U can confirm these contents. In the example of FIG. 7(f), a finger icon MK is superimposed near the "OK" button at the end of the confirmation items, indicating that user U has acknowledged the confirmation items.

[0092] FIG. 7(g) is the top screen related to the refrigerator 30A that transitions from the guidance screen when the operation regarding the chill room setting is completed. In the example shown in FIG. 7, the video ends when returning to the top screen.

[0093] (Example of video generation process) Next, with reference to FIG. 8, a processing example of video generation in the home appliance management system 1 of the embodiment will be described. FIG. 8 is a sequence diagram showing an example of the procedure of the video generation process executed in the home appliance management system 1 according to the embodiment.

[0094] As shown in FIG. 8, when user U displays the menu screen from the home screen of the application 201 and selects "Start recording operation" (step S101), snapshots of the operation screens displayed by the subsequent operations of user U, as well as the recording of operation events and communication events associated with these snapshots, are started (steps S102 to S109). Note that the processes in steps S102 to S109 are repeated the required number of times.

[0095] That is, when user U performs a screen operation on application 201 (step S102), the control unit 213 of application 201 records a snapshot of the operation screen and an operation event related to the screen operation (step S103).

[0096] Also, when the screen operation by user U is an operation that requires communication with server device 10 or home appliance 30 via server device 10, communication with server device 10 or home appliance 30 is performed (steps S104 to S109). In the example of FIG. 8, it is assumed that communication with home appliance 30 via server device 10 is performed in the processes of steps S104 to S109.

[0097] That is, the control unit 213 records a communication event related to the content to be transmitted to home appliance 30, which is generated in conjunction with the screen operation by user U (step S104). Also, the control unit 213 requests server device 10 to control home appliance 30 (step S105). In response to this, server device 10 controls home appliance 30 (step S106), and home appliance 30 returns the control result to server device 10 (step S107).

[0098] Furthermore, server device 10 returns the implementation result of the control request to application 201 (step S108). The control unit 213 records a snapshot of the operation screen at this time and a communication event related to the received content from server device 10 (step S109).

[0099] As described above, after repeating the processes of steps S102 to S109 the necessary number of times, when a series of operations on application 201 is completed, user U can display the menu screen from the home screen of application 201 and select "End operation recording" (step S110).

[0100] In response to this, the control unit 213 generates operation history information including the snapshots recorded in the processes of steps S102 to S109, as well as the operation events and communication events associated with the snapshots, and transmits it to the server device 10 (step S111).

[0101] The server device 10 generates a video from the snapshots included in the operation history information (step S112).

[0102] After that, for example, when the user U instructs to download a video from the application 201 (step S113), the control unit 213 acquires the video from the server device 10 via the communication unit 202 (step S114).

[0103] Thus, the video generation process in the home appliance management system 1 of the embodiment ends.

[0104] (Summary) When there is a problem with the application for remotely operating home appliances, providing a video recording the operations at the time of the problem occurrence to the customer center may facilitate identifying the cause of the problem. Also, there is a user need to upload a video of the operation state to make explanations easier when introducing home appliances on SNS, etc. In such cases, it is conceivable for the user himself / herself to shoot a video using the recording function of a terminal device or the like. However, in order to obtain an easy-to-watch video, video shooting must be performed while adjusting the display time of each screen and the timing of button operations, etc., which makes the work complicated.

[0105] Also, when reporting to the customer center, it is preferable to provide not only the operation screen but also information such as operation events and communication events. However, when providing operation events and communication events separately, it is difficult to grasp the association between the operation screen and these event logs.

[0106] According to the above-described embodiment, a video in which a plurality of operation screens transition can be easily generated. It is also possible to superimpose operation events and communication events associated with these operation screens.

[0107] In the above-described embodiment, the home screen or the top screen of the home appliance 30 or the like is set in advance as the operation screen for starting the video. However, the user U may be able to set from which operation screen to start the video and at which operation screen to end the video. At this time, the user U may be able to select an arbitrary snapshot from, for example, the recorded snapshots of the operation screens as the start screen and the end screen of the video.

[0108] In the above-described embodiment, the display time in the video of each operation screen is set in advance. However, the user U may be able to arbitrarily set the display time of each operation screen.

[0109] In the above-described embodiment, when generating a video, it is assumed that the user U inputs a video generation instruction before performing a series of operations. However, it may be possible to give an instruction to generate a video for the operated content after the user U performs a predetermined operation. In this case, when the user U operates on the terminal device 20, the application 201 can temporarily pool snapshots, operation events, and communication events for a predetermined period or a predetermined capacity. Thereby, it becomes possible to generate a video for a predetermined operation retroactively.

[0110] (Modification Example 1) Next, with reference to FIG. 9, a home appliance management system according to Modification Example 1 of the embodiment will be described. The home appliance management system according to Modification Example 1 is different from the above-described embodiment in that the customer center CS causes the home appliance management system according to Modification Example 1 to generate a video by remote operation.

[0111] FIG. 9 is a sequence diagram showing an example of the procedure of video generation processing executed in the home appliance management system according to Modification Example 1 of the embodiment. As shown in FIG. 9, the home appliance management system of Modification Example 1 can also generate a video by remote operation from the customer center CS. The customer center CS shown in FIG. 9 represents, for example, a computer operable by an operator.

[0112] That is, for example, when the customer center CS that has received a report of a problem with the application 201a from the user U requests the server device 10a to start operation recording (step S201), the server device 10a accesses the application 201a and starts recording the operation (step S202).

[0113] The subsequent processing up to steps S102 to S109 is the same as the processing up to steps S102 to S109 shown in FIG. 8 of the above-described embodiment, and is performed a plurality of times as necessary. The user U reproduces the operation procedure in which the problem occurs, for example, according to the instructions of the operator.

[0114] When a snapshot of the operation screen, operation events, and communication events necessary for the analysis of the problem are obtained, the customer center CS requests the server device 10a to end the operation recording (step S203). The server device 10a accesses the application 201a and ends the operation recording (step S204). The application 201a generates operation history information and uploads it to the server device 10a (step S205).

[0115] When the server device 10a acquires the operation history information from the application 201a, it generates a video from the snapshot included in the operation history information and stores it in the storage unit 106 (step S206). Thereafter, the customer center CS downloads the video from the server device 10a (step S207).

[0116] Thus, the video generation processing in the home appliance management system of Modification Example 1 ends.

[0117] In addition, in the configuration of Modification Example 1, the display times of the start screen and the end screen of the video, as well as each operation screen, may be arbitrarily set from the customer center CS.

[0118] (Modification Example 2) Next, the server device 10b according to Modification Example 2 of the embodiment will be described with reference to FIG. 10. The server device 10b according to Modification Example 2 is different from the above-described embodiment in that video is generated based on the operation events and communication events acquired from the application 201 of the terminal device 20.

[0119] FIG. 10 is a block diagram showing an example of the configuration of the server device 10b according to Modification Example 2 of the embodiment. As shown in FIG. 10, the server device 10b according to Modification Example 2 includes an emulator 17 in addition to each part of the server device 10 of the above-described embodiment.

[0120] The emulator 17 can reproduce an operation screen or the like displayed on the terminal device 20 by the application 201 by emulating the execution of the application 201 of the terminal device 20 in the operating environment of the server device 10b.

[0121] In the home appliance management system according to Modification Example 2, when generating a video related to the operation of the home appliance 30, the application 201 of the terminal device 20 records the operation events and communication events generated in response to the operation of the user U. Further, when the recording of the operation events and communication events is completed, the application 201 generates operation history information including these operation events and communication events, and transmits it to the server device 10b.

[0122] In the server device 10b, the processing unit emulates the execution of the application 201 of the terminal device 20 by using the function of the emulator 17 described above, and reproduces the operation of the user U based on the acquired operation events and communication events. Thereby, the operation screen displayed on the terminal device 20 by the operation of the user U can be reproduced and recorded.

[0123] The generation unit of the server device 10b converts the operation screen acquired as described above into a video format, and generates a video in which a plurality of operation screens transition in the operation order of the home appliance 30.

[0124] When the customer center or the like analyzes the defect of the application 201, it is also useful to refer to the records of the operation event and the communication event for identifying the defect. However, the operation event and the communication event etc. may take time to interpret as they are. By reproducing the operation screen from the operation event and the communication event and converting it into a video, for example, analysis by an operator at the customer center becomes easier.

[0125] In the above-described embodiment and Modification Example 2 etc., the functions of the encoder 16 and the emulator 17 are assumed to be realized by hardware. However, the functions of the encoder 16 and the emulator 17 may be realized by software.

[0126] (Modification Example 3) Next, with reference to FIGS. 11 and 12, a home appliance management system according to Modification Example 3 of the embodiment will be described. The home appliance management system according to Modification Example 3 is different from the above-described embodiment in that the application 201c of the terminal device 20c has a video generation function.

[0127] FIG. 11 is a block diagram showing an example of the functional configuration of the terminal device 20c according to Modification Example 3 of the embodiment. As shown in FIG. 11, the terminal device 20c according to Modification Example 3 includes an application 201c and a storage unit 205c instead of the application 201 and the storage unit 205 of the above-described embodiment.

[0128] The application (application program) 201c of Modification Example 3 incorporates a plugin application 206. As a result, in addition to having functional units similar to those of the application 201 in the above-described embodiment, when the plugin application 206 is activated in response to a video generation instruction from the user U, the application 201c will include a processing unit 261 and a generation unit 262 as functional units of the plugin application 206.

[0129] Similar to the processing unit 102 of the server device 10 in the above-described embodiment, the processing unit 261 performs preliminary processing for generating a video on the snapshot 251.

[0130] Similar to the generation unit 103 of the server device 10 in the above-described embodiment, the generation unit 262 uses the snapshot 251 that has undergone preliminary processing to generate a video 254 in which a plurality of operation screens transition in the order of operations. The function of such a generation unit 262 is, for example, a function of the encoder 16 provided in the server device 10 in the above-described embodiment realized by software.

[0131] The storage unit 205c of Modification Example 3 stores the snapshot 251, event information 252, and operation history information 253 recorded or generated by the control unit 213. Further, the storage unit 205c stores the video 254 generated by the generation unit 262.

[0132] FIG. 12 is a sequence diagram showing an example of the procedure of video generation processing executed in the home appliance management system according to Modification Example 3 of the embodiment.

[0133] The processing from steps S101 to S110 shown in FIG. 12 is the same as the processing from steps S101 to S110 shown in FIG. 8 of the above-described embodiment. That is, when a video generation instruction is given by the user U (step S101), the application 201c appropriately records snapshots of the operation screen, operation events, communication events, etc. (steps S102 to S109).

[0134] After that, when the user U gives an instruction to end video generation (step S110), the plugin application 206 incorporated in the application 201c is launched, and a video is generated from the snapshots included in the operation history information (step S312).

[0135] Thus, the video generation process in the home appliance management system of Modification 3 ends.

[0136] In the above-described Modification 3, when generating a video, the control unit 213 records a snapshot of the operation screen, and the processing unit 261 and the generation unit 262 generate a video from the snapshot. However, similar to Modification 2 above, the application 201c of Modification 3 may generate a video based on operation events and communication events without recording a snapshot of the operation screen.

[0137] In this case, the application 201c can reproduce the operation screen based on operation events and communication events, for example, within the memory space, and generate a video using the reproduced operation screen.

[0138] Also, in the above-described embodiments and Modifications 1 to 3, the server devices 10, 10a, 10b are configured as a single device, but it is not limited thereto. For example, the acquisition unit 101, the processing unit 102, the generation unit 103, the control unit 104, the transmission unit 105, the storage unit 106, etc., which are functional parts of the server devices 10, 10a, 10b, or the functions of each part may be distributed and configured by a plurality of devices. Even in this case, the users who use the applications 201, 201c of the terminal devices 20, 20c can enjoy all the operational effects described in the above-described embodiments and Modifications 1 to 3.

[0139] Further, when the server devices 10, 10a, and 10b in the above-described embodiments and Modifications 1 to 3 are configured to be provided overseas outside Japan, or when the server devices 10, 10a, and 10b are configured to be distributed among a plurality of devices, and a part of the functional units such as the acquisition unit 101, the processing unit 102, the generation unit 103, the control unit 104, the transmission unit 105, and the storage unit 106, or a part of the functions of each unit are configured to be provided overseas, if it is substantially considered to be executed within Japan, users who use the applications 201 and 201c of the terminal devices 20 and 20c can enjoy all of the operational effects described in the above-described embodiments and Modifications 1 to 3.

[0140] (Summary) As described above, the home appliance management system 1 according to the embodiment includes a server device 10 (10a, 10b) that manages home appliances 30, and an application program 201 (201a) for causing a computer of a terminal device 20 connected to the server device 10 (10a, 10b) via a network NT and including a display device (203) to execute. When the application program 201 (201a) is executed by a computer, the application program 201 (201a) causes the computer to execute steps of recording an operation history of the home appliance 30, and when there is an input instructing generation of a video showing an operation procedure of the home appliance 30, transmitting operation history information including the operation history of the home appliance 30 to the server device 10. The server device 10 includes an acquisition unit 101 that acquires operation history information from the application program 201 (201a), and a generation unit 103 that generates a video in which a plurality of operation screens transition in the operation order of the home appliance 30 based on the operation history information.

[0141] According to such a configuration, a video in which a plurality of operation screens transition can be easily generated.

[0142] In the home appliance management system 1, for example, the video includes a plurality of operation screens from an operation screen that is an operation base point of the home appliance 30 to the final operation screen in the operation history information.

[0143] According to such a configuration, for example, a video can be generated starting from a home screen or a top screen related to the operation of each home appliance 30, and the operation content in the video becomes easier to understand.

[0144] In addition, in the home appliance management system 1, for example, the video includes a plurality of operation screens from the operation screen that is the operation starting point of the home appliance 30 to the final operation screen in the operation history information with the shortest operation procedure.

[0145] According to such a configuration, for example, even when the user U makes an operation error, the video is generated with the shortest operation procedure without including unnecessary operation screens, so the operation content in the video becomes easier to understand.

[0146] In addition, in the home appliance management system 1, for example, the application program 201(201a) causes a computer to execute a step of receiving an input for selecting an operation screen that is the start screen of the video, and a step of transmitting the input information of the operation screen that is the start screen of the video to the server device 10(10a, 10b).

[0147] According to such a configuration, for example, the user U can arbitrarily select the start screen of the video, and can more accurately meet the needs of the user U for video generation.

[0148] In addition, in the home appliance management system 1, for example, the application program 201(201a) causes a computer to execute a step of receiving an input for selecting an operation screen that is the end screen of the video, and a step of transmitting the input information of the operation screen that is the end screen of the video to the server device 10(10a, 10b).

[0149] According to such a configuration, for example, the user U can arbitrarily select the end screen of the video, and can more accurately meet the needs of the user U for video generation.

[0150] In addition, in the home appliance management system 1, for example, for each of a plurality of operation screens, a display time in the video is preset.

[0151] According to such a configuration, for example, each operation screen can be displayed for a display time suitable for viewing without disturbing the user.

[0152] In addition, in the home appliance management system 1, for example, the application program 201(201a) causes a computer to execute a step of receiving an input for designating a display time in the video of at least one operation screen of a plurality of operation screens, and a step of transmitting input information of the display time to the server device 10(10a, 10b).

[0153] According to such a configuration, for example, the user U can arbitrarily set the display time of each operation screen in the video, and can more accurately meet the needs of the user U for video generation.

[0154] In addition, in the home appliance management system 1, for example, when there is an input for instructing generation of a video, the application program 201(201a) causes a computer to execute a step of transmitting operation history information including event information indicating an operation event corresponding to at least one operation screen of a plurality of operation screens to the server device 10(10a, 10b).

[0155] According to such a configuration, the server devices 10, 10a can generate a video more appropriately by referring to the operation event. Also, like the server device 10b in the second modification example, a video can be generated using an operation screen reproduced based on event information such as an operation event. In this way, by generating and visualizing a video based on event information, it becomes easy to identify defects in the applications 201, 201a by, for example, a customer center or the like.

[0156] In the home appliance management system 1, for example, when there is an input instructing the generation of a video, the application program 201(201a) causes the computer to execute a step of transmitting operation history information including event information indicating a communication event with the server device 10(10a, 10b) corresponding to at least one of a plurality of operation screens to the server device 10(10a, 10b).

[0157] According to such a configuration, the server devices 10 and 10a can generate a video more appropriately by referring to the communication event. Also, like the server device 10b in Modification 2, it is also possible to generate a video using an operation screen reproduced based on event information such as a communication event. Thus, by generating and visualizing a video based on event information, it becomes easy to identify defects in the applications 201 and 201a by, for example, a customer center or the like.

[0158] In the home appliance management system 1, for example, the generation unit 103 generates a video by superimposing information indicating an operation location in the operation screen corresponding to the event information based on the event information.

[0159] According to such a configuration, when viewing the video, it is possible to easily grasp what operations were performed on each operation screen. Also, by clearly indicating the operation locations in the operation screen, it becomes even easier to grasp the relationship between the operation timing and the events that occurred as a result of the operation.

[0160] In the home appliance management system 1, for example, the generation unit 103 generates a video by superimposing the event information on the operation screen corresponding to the event information.

[0161] According to such a configuration, when watching a video, it is possible to easily grasp what operations or communications have been performed on each operation screen. Further, for example, it becomes easier to identify malfunctions or the like that have occurred in the applications 201, 201a by, for example, a customer center or the like. Further, by displaying event information in the operation screen, it becomes even easier to grasp the relationship between the operation timing, the communication timing, and the events in which the results of the operation or communication occurred.

[0162] Further, in the home appliance management system, for example, the server device 10a causes a computer to execute a step of remotely operating the application program 201a and transmitting operation history information to the server device 10a.

[0163] According to such a configuration, for example, a customer center CS or the like can cause a video of an operation on the application 201a to be generated by remote operation via the server device 10a. Therefore, when identifying a malfunction of the application 201a or the like, it is possible to obtain necessary information without bothering the user U. Further, it becomes even easier to identify a malfunction of the application 201a by a customer center CS or the like.

[0164] The application program 201c according to the embodiment is an application program 201c for causing a computer of a terminal device 20c including a display device (203) to be executed, which is connected to a server device 10 that manages a home appliance 30 via a network NT. When the application program 201c is executed by a computer, the method includes receiving an input for instructing generation of a video showing an operation procedure of the home appliance 30, recording an operation history of the home appliance 30 when there is an input for instructing generation of the video, and generating, based on operation history information including the operation history of the home appliance 30, a video in which a plurality of operation screens transition in the operation order of the home appliance 30 at a predetermined display time, and causing the computer to execute the steps.

[0165] According to such a configuration, it is possible to easily generate a video in which a plurality of operation screens transition.

[0166] The video generation method according to the embodiment is a video generation method executed by a home appliance management system (1) including a server device 10 (10a, 10b) that manages a home appliance 30 and an application program 201 (201a, 201c) that is executed on a computer of a terminal device 20 (20c) connected to the server device 10 (10a, 10b) via a network NT and includes a display device (203). When the application program 201 (201a, 201c) is executed by a computer, the method includes: receiving an input for instructing generation of a video showing an operation procedure of the home appliance 30; recording an operation history of the home appliance 30 when there is an input for instructing generation of the video; and generating a video in which a plurality of operation screens transition in the operation order of the home appliance 30 at respective predetermined display times based on operation history information including the operation history of the home appliance 30.

[0167] According to such a configuration, a video in which a plurality of operation screens transition can be easily generated.

[0168] The embodiments of the present invention have been described above. However, the above embodiments are merely examples and are not intended to limit the scope of the invention. The above embodiments can be implemented in various forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The above embodiments are included in the scope and gist of the invention and are included in the invention described in the claims and its equivalent scope.

Description of Reference Numerals

[0169] 1... Home appliance management system, 10, 10a, 10b... Server device, 20, 20c... Terminal device, 30... Home appliance, 101... Acquisition unit, 102, 261... Processing unit, 103, 262... Generation unit, 201, 201c... Application program, 206... Plugin application, 213... Control unit, U... User.

Claims

1. A server device for managing home appliances, and an application program for causing a computer of a terminal device connected to the server device via a network and provided with a display device to execute. The application program When the application program is executed by the computer, records the operation history of the home appliances, When there is an input instructing generation of a video showing the operation procedure of the home appliances, causes the computer to transmit operation history information including the operation history of the home appliances to the server device, The server device An acquisition unit that acquires the operation history information from the application program, A generation unit that generates a video in which a plurality of operation screens transition in the operation order of the home appliances based on the operation history information. A home appliance management system.

2. The video Includes the plurality of operation screens from the operation screen that is the operation starting point of the home appliances to the final operation screen in the operation history information. The home appliance management system according to claim 1.

3. The video Includes the plurality of operation screens from the operation screen that is the operation starting point of the home appliances to the final operation screen in the operation history information with the shortest operation procedure. The home appliance management system according to claim 1.

4. The application program Receives an input for selecting an operation screen to be the start screen of the video, Causes the computer to transmit input information of the operation screen to be the start screen of the video to the server device. The home appliance management system according to claim 1.

5. The application program receives an input for selecting an operation screen that becomes the end screen of the video, and causes the computer to execute a step of transmitting input information of the operation screen that becomes the end screen of the video to the server device. The home appliance management system according to claim 1.

6. For each of the plurality of operation screens, a display time in the video is preset. The home appliance management system according to claim 1.

7. The application program receives an input for designating a display time in the video of at least one of the plurality of operation screens, and causes the computer to execute a step of transmitting input information of the display time to the server device. The home appliance management system according to claim 1.

8. The application program when there is an input for instructing generation of the video, causes the computer to execute a step of transmitting the operation history information including event information indicating an operation event corresponding to at least one of the plurality of operation screens to the server device. The home appliance management system according to claim 1.

9. The application program when there is an input for instructing generation of the video, causes the computer to execute a step of transmitting the operation history information including event information indicating a communication event with the server device and corresponding to at least one of the plurality of operation screens to the server device. The home appliance management system according to claim 1.

10. The generation unit Generating the video by superimposing information indicating an operation location on an operation screen corresponding to the event information based on the event information. The home appliance management system according to claim 8.

11. The generation unit Generating the video by superimposing the event information on an operation screen corresponding to the event information. The home appliance management system according to claim 8 or claim 9.

12. The server device Causing the computer to execute a step of remotely operating the application program and transmitting the operation history information to the server device. The home appliance management system according to claim 1.

13. An application program for causing a computer of a terminal device including a display device to be executed, the terminal device being connected to a server device for managing home appliances via a network, When the application program is executed by the computer, receiving an input for instructing generation of a video showing an operation procedure of the home appliance; When there is an input for instructing generation of the video, recording an operation history of the home appliance; Causing the computer to execute a step of generating a video in which a plurality of operation screens transition in the operation order of the home appliance at respective predetermined display times based on operation history information including the operation history of the home appliance. Application program.

14. A server device for managing home appliances, A video generation method executed by a home appliance management system including the server device and an application program for causing a computer of a terminal device including a display device and connected to the server device via a network to be executed, When the application program is executed by the computer, receiving an input for instructing generation of a video showing an operation procedure of the home electric appliance; When there is an input for instructing generation of the video, recording an operation history of the home electric appliance; Generating a video in which a plurality of operation screens transition in the operation order of the home electric appliance at respective predetermined display times based on operation history information including the operation history of the home electric appliance. Video generation method.

Images