Operation assistance method, operation assistance device, operation assistance system, and storage medium storing operation assistance program

Abstract

The present invention provides an operation assistance method, an operation assistance device, an operation assistance system, and an operation assistance program that acquire an object image obtained by imaging an operation button provided to controlled equipment, acquire an icon associated with the operation button as identified by image recognition applied to the object image, and display the icon via a display unit.

Description

Operation support method, operation support device, operation support system, and storage medium storing operation support program

[0001] The present disclosure relates to an operation assistance method, an operation assistance device, an operation assistance system, and a storage medium storing an operation assistance program.

[0002] A technology has been proposed that improves user convenience when operating facility equipment from a mobile terminal (see Patent Document 1). In this technology, the mobile terminal acquires facility equipment identification information for identifying the facility equipment from the facility equipment, sends the facility equipment identification information to a server, acquires operation permission information indicating permission to operate the facility equipment from the server, and sends the operation permission information and a command indicating the operation content of the facility equipment to the facility equipment. The facility equipment acquires the operation permission information and the command, and controls the device itself in accordance with the operation content indicated in the command.

[0003] International Publication No. 2019 / 012599

[0004] According to the technology described in Patent Document 1, when there are a large number of control devices to be operated remotely, the task of registering the control devices becomes complicated, which reduces user convenience.

[0005] The present disclosure has been made in view of the above-mentioned problems, and aims to provide an operation assistance method, an operation assistance device, an operation assistance system, and an operation assistance program that can simplify the task of registering control devices and improve user convenience when there are a large number of control devices to be operated remotely.

[0006] In order to solve the above-mentioned problems, an operation assistance method, an operation assistance device, an operation assistance system, and an operation assistance program according to one aspect of the present disclosure acquire a target image of an operation button provided on a control device, acquire an icon associated with the operation button identified by image recognition of the target image, and display the icon via a display unit.

[0007] According to the present disclosure, when there are a large number of control devices to be operated remotely, the task of registering the control devices can be simplified, improving user convenience.

[0008] Fig. 1 is a block diagram showing a configuration of an operation assistance system according to a first embodiment of the present disclosure. Fig. 2 is a sequence diagram showing processing of the operation assistance system according to the first embodiment of the present disclosure. Fig. 3 is a block diagram showing a configuration of an operation assistance system according to a second embodiment of the present disclosure. Fig. 4 is a sequence diagram showing processing of the operation assistance system according to the second embodiment of the present disclosure. Fig. 5 is a block diagram showing a configuration of an operation assistance system according to a third embodiment of the present disclosure. Fig. 6 is a sequence diagram showing processing of the operation assistance system according to the third embodiment of the present disclosure.

[0009] Next, embodiments of the present disclosure will be described in detail with reference to the drawings. In the description, the same components are designated by the same reference numerals and redundant description will be omitted.

[0010] [First embodiment] An example configuration of an operation assistance system will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an operation assistance system according to a first embodiment of the present disclosure. As shown in Fig. 1, the operation assistance system includes an operation assistance device 1 and a control device 400.

[0011] For example, the control device 400 may be any of various information devices equipped with operation buttons. For example, the control device 400 may be a navigation device mounted on a vehicle, or a terminal that provides various information services to a user. The control device 400 may be the vehicle itself or a part of the vehicle. Furthermore, the control device 400 may be any of various electrical machinery and appliances used by a user. The control device 400 is not limited to the examples given here.

[0012] The operation assistance device 1 includes a communication unit 110, a display unit 120, and a controller 130. In addition, the operation assistance device 1 may include an operation unit 150, a database 160, and an imaging device 170.

[0013] Communication unit 110 is communicably connected to control device 400. Communication between communication unit 110 and control device 400 may be via the Internet, for example. Furthermore, communication between communication unit 110 and control device 400 may be via a mobile communication function such as 4G / LTE or 5G.

[0014] Alternatively, a short-range wireless system may be used for communication between the communication unit 110 and the control device 400. Examples of short-range wireless systems include near field communication (NFC), Wi-Fi, Bluetooth, and various other systems.

[0015] The communication section 110 may transmit information for identifying the communication method used in communication with the control device 400 to the controller 130, which will be described later.

[0016] The display unit 120 displays various icons (icons, supplementary icons) based on commands from the controller 130. Here, the "icons" are linked to operation buttons provided on the control device 400. Additionally, the display unit 120 may display received messages based on commands from the controller 130.

[0017] For example, the display unit 120 is a display device such as a computer screen or a television LCD screen. Alternatively, the display unit 120 may be a display device used in a mobile phone, a portable game console, a calculator, a clock, or the like. Examples of the display device include a display (monitor) and a projector. Examples of the display include a liquid crystal display (LCD), an organic light-emitting diode display (OLED), a plasma display (PDP), a micro LED display (mLED), a video projector, a cathode ray tube (CRT), and the like. The display unit 120 is not limited to the examples given here.

[0018] The operation unit 150 accepts user operations corresponding to various commands from the user to the operation assistance device. For example, the operation unit 150 may be an input interface having a plurality of buttons or a touch panel having a touch interface.

[0019] More specifically, the display unit 120 may display icons represented by pictures or symbols so that the user can operate them, and the operation unit 150 may accept user operations when the user touches or moves the displayed icons. The "icon" may be an image or text. For example, if the "icon" is associated with the execution of a playback function for music, video, or the like, the "icon" may be an image of an arrow or triangle indicating the direction of playback, or the word "play."

[0020] In particular, operation unit 150 accepts operations on icons displayed by display unit 120. As will be described later, controller 130 associates "operations on icons" with the execution of functions of control device 400.

[0021] For example, "operations on icons" include "tap," "double tap," "long press," "scroll," "drag," "swipe from the edge of the screen," "swipe an item," "pinch," and "rotate."

[0022] A "tap" is the action of lightly touching an icon. A "double tap" is the action of quickly touching an object twice. A "long press" is the action of pressing and holding an object for a predetermined period of time and then releasing it. A "scroll" is the action of sliding a finger horizontally or vertically across the screen. A "drag" is the action of moving a finger in either direction while still touching an object. A "swipe from the edge of the screen" is the action of moving a finger in either direction while still touching an object at the edge of the screen. A "swipe on an item" is the action of briefly sliding an item in a direction that does not scroll the page. A "pinch" is the action of touching the screen or an item with two or more fingers and moving the fingers closer together or apart. A "rotate" is the action of placing two or more fingers on an item and rotating them. An "operation on an icon" is not limited to the examples given here. For example, if an "icon" is associated with the execution of a playback function such as music or video playback, an "operation on the icon" may be a tap on an arrow or triangle image indicating the playback direction, or the word "play."

[0023] Database 160 stores operation buttons provided on control device 400 in association with icons. More specifically, database 160 may store information specifying the external characteristics of the operation buttons, identifiers for identifying the operation buttons, and other information in association with the icons. Database 160 may also store information such as control device 400 related to the operation buttons or identifiers for identifying control device 400 in association with the icons.

[0024] The database 160 may store predetermined images as icons. The database 160 may also store information related to the layout of the icons, linked to the icons. Examples of the information related to the layout of the icons include the display position of the icons when displayed on the display unit 120 and the relative positional relationship between the display positions of the icons. The information related to the layout of the icons may include information about the orientation of the icons when displayed. Additionally, the database 160 may store identifiers for identifying the icons. The database 160 may also store manuals for the functions of the operation buttons related to the icons, linked to the icons.

[0025] The database 160 may store the control signal for the control device 400 in association with operation information that identifies an operation on an icon.

[0026] Alternatively, the database 160 may store an icon in association with a status signal of the control device 400. For example, the database 160 may store an icon for each status that the control device 400 can be in.

[0027] If the operation button associated with the icon is a button for switching a function of the control device 400 on and off, the database 160 may store both an icon corresponding to when the function is on and an icon corresponding to when the function is off. The icons stored for each possible state of the control device 400 are not limited to the examples given here. If there are N possible states of the control device 400, where N is an integer of 2 or greater, the database 160 may store N corresponding icons.

[0028] The database 160 may be configured to register at least one of the control devices 400 that have been connected to the communication unit 110 and the control devices 400 that the user has specified. The database 160 may store operation buttons and icons related to the registered control devices 400 in association with each other.

[0029] The possibility of registering information in database 160 may be managed by controller 130. For example, when controller 130 determines that the distance between control device 400 and communication unit 110 is equal to or less than a predetermined distance, registration of control device 400 in database 160 may be permitted. Furthermore, when controller 130 determines that control device 400 is connected to communication unit 110 by a predetermined short-range wireless communication method, registration of control device 400 in database 160 may be permitted.

[0030] The imaging device 170 captures a target image of an operation button provided on the control device 400. More specifically, the user sets the imaging direction of the control device 400 so that the control device 400, which is the target of the user's operation, fits within the angle of view of the imaging device 170, and the control device 400 captures the target image based on an instruction from the user. The imaging device 170 may capture the target image in the form of either a still image or a video. The target image captured by the imaging device 170 is transmitted to the controller 130.

[0031] For example, the imaging device 170 is a digital camera equipped with a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 170 captures an image of the surface of the control device 400 and acquires a target image including a part of or the entire array of operation buttons provided on the control device 400. The imaging device 170 acquires the target image by setting the focal length, the angle of view of the lens, the vertical and horizontal angles of the camera, etc. The imaging device 170 is not limited to the examples given here.

[0032] Alternatively, the target image acquired by the imaging device 170 may be transmitted to a storage device (not shown) connected to the controller 130 and stored in the storage device. The storage device may be included in the operation assistance device 1. Alternatively, the storage device may be provided externally to the operation assistance device 1 (for example, a cloud service). The storage device is not limited to the examples given here.

[0033] The controller 130 (an example of a control unit or processing unit) is a general-purpose computer equipped with a CPU (Central Processing Unit), memory, and input / output units. A computer program (operation assistance program) for functioning as part of the operation assistance device is installed in the controller 130. By executing the computer program, the controller 130 functions as multiple information processing circuits (131, 133, 135, 137, 139, 141, 143) equipped in the operation assistance device.

[0034] Here, an example is shown in which the multiple information processing circuits (131, 133, 135, 137, 139, 141, 143) provided in the operation assistance device are realized by software. However, it is also possible to prepare dedicated hardware for executing each of the information processes described below and configure the information processing circuits (131, 133, 135, 137, 139, 141, 143). Also, the multiple information processing circuits (131, 133, 135, 137, 139, 141, 143) may be configured by individual hardware.

[0035] The controller 130 includes an image acquisition unit 131, an image recognition unit 133, an icon acquisition unit 135, a display setting unit 137, an operation acquisition unit 141, and a control output unit 143 as a plurality of information processing circuits (131, 133, 135, 137, 141, 143).

[0036] As will be described in a second embodiment, the controller 130 may include an inquiry unit 139 instead of the icon acquisition unit 135. As will be described in a third embodiment, the controller 130 may include an inquiry unit 139 instead of the image recognition unit 133 and the icon acquisition unit 135.

[0037] The operation assistance program may be stored in a computer-readable storage medium or may be provided via a telecommunications line.

[0038] The image acquisition unit 131 acquires a target image obtained by capturing an image of an operation button provided on the control device 400. The image acquisition unit 131 may acquire the target image via the imaging device 170, or may acquire the target image stored in a storage device (not shown).

[0039] The image recognition unit 133 identifies an operation button reflected in the target image by performing image recognition on the target image acquired by the image acquisition unit 131. When identifying the operation button, the image recognition unit 133 may also identify an identifier that identifies the operation button.

[0040] For example, the image recognition unit 133 may read information specifying the external characteristics of the operation button from the database 160, and specify the operation button or an identifier that identifies the operation button based on the read characteristics.

[0041] The image recognition unit 133 may identify an identifier for identifying an operation button by using the features of other operation buttons located around the operation button to be identified and the positional relationship between the operation button to be identified and the other operation buttons. Alternatively, the image recognition unit 133 may determine an identifier for identifying an operation button based on an acquired target image by using a learning model generated based on training data that pairs the appearance of the operation button with an identifier for identifying the operation button.

[0042] Alternatively, the image recognition unit 133 may identify the control device 400 associated with an operation button reflected in the target image. For example, the image recognition unit 133 may read information that identifies the appearance characteristics of the operation button from the database 160, and identify the control device 400 or an identifier that identifies the control device 400 based on the read characteristics. After identifying the control device 400 or the identifier that identifies the control device 400, the image recognition unit 133 may refer to a list of operation buttons stored in association with the control device 400, and identify operation buttons (supplementary operation buttons) that are not reflected in the target image among the operation buttons provided on the control device 400.

[0043] Alternatively, the image recognition unit 133 may determine whether or not multiple operation button candidates have been extracted as a result of image recognition for one operation button reflected in the target image. If multiple operation button candidates have been extracted, the image recognition unit 133 may determine one operation button candidate selected based on a selection instruction from the user as the identified operation button. In other words, if multiple operation button candidates have been found for one operation button reflected in the target image and the button cannot be identified, the image recognition unit 133 may further accept a selection instruction from the user and identify the operation button.

[0044] The icon acquiring unit 135 acquires an icon linked to an operation button identified by image recognition of the target image. More specifically, the icon acquiring unit 135 acquires the icon linked to the identified operation button by referring to the database 160. The icon acquiring unit 135 may also acquire information related to the arrangement of the icon by referring to the database 160.

[0045] The icon acquiring unit 135 may be provided in the identified control device 400 and may acquire a supplementary icon linked to a supplementary operation button that is not reflected in the target image. More specifically, the icon acquiring unit 135 may refer to the database 160 and acquire an icon linked to a supplementary operation button as the supplementary icon.

[0046] The display setting unit 137 displays the acquired icon via the display unit 120. More specifically, the display setting unit 137 refers to the acquired icon and information related to the icon arrangement, and displays the icon at a predetermined position on the display unit 120. Additionally, when displaying an icon, the display setting unit 137 may also display a manual for the function of the operation button related to the icon.

[0047] The orientation of the displayed icon may not match the orientation of the operation button included in the target image, but may match the orientation in which the user operates the operation button provided on the control device. For example, suppose that an image of the operation button is captured in a rotated orientation, resulting in the vertical and horizontal orientation of the icon being reversed. In this case, the display unit 120 may display the icon in a reversed state, or may display the icon in a normal operation orientation with the vertical and horizontal orientation reversed. The normal operation orientation may be determined based on information about the orientation when the icon is displayed, which is included in the information related to the icon placement.

[0048] Furthermore, the display setting unit 137 may receive, from the control device 400, a status signal indicating the result of executing a function of the control device 400 via the communication unit 110. The display setting unit 137 may change the form of an icon based on the status signal received from the control device 400. For example, the display setting unit 137 may switch the icon displayed corresponding to an operation button to an icon corresponding to the state of the control device 400, depending on the state of the control device 400. Note that the display setting unit 137 may display only the acquired icon via the display unit 120, or may display supplementary icons in addition to the acquired icon via the display unit 120.

[0049] Alternatively, the display setting unit 137 may determine whether the communication method between the control device 400 and the communication unit 110 is a short-range wireless communication method based on information identifying the communication method used in communication between the control device 400 and the communication unit 110. Note that "when the communication method between the control device 400 and the communication unit 110 is not a short-range wireless communication method" includes a case where short-range wireless communication is not established (when communication is not taking place) between the control device 400 and the communication unit 110. Furthermore, when the communication method between the control device 400 and the communication unit 110 is not a short-range wireless communication method, the display setting unit 137 may display, via the display unit 120, a message requesting a change of communication method or a message requesting the establishment (connection) of short-range wireless communication before displaying the icon.

[0050] The operation acquisition unit 141 acquires a user operation received via the operation unit 150. Then, the operation acquisition unit 141 generates operation information that identifies the operation on the icon. Furthermore, the operation acquisition unit 141 may determine whether or not the icon has been operated based on the user operation. Then, when it is determined that the icon has been operated, the operation acquisition unit 141 may generate operation information that identifies the operation on the icon.

[0051] The control output unit 143 acquires a control signal corresponding to an operation on an icon. Here, the control signal corresponding to an operation on an icon is a control signal for executing a function related to the operation button associated with the operated icon. For example, the control output unit 143 may refer to the database 160 and acquire a control signal for the control device 400 that is linked to the operation information generated by the operation acquisition unit 141. Additionally, the control output unit 143 may set parameters included in the control signal depending on the strength of the operation on the icon (e.g., the distance, speed, and duration of movement of the icon during the operation).

[0052] Thereafter, the control output unit 143 transmits the acquired control signal to the control device 400 via the communication unit 110 .

[0053] Next, a processing procedure of the operation assistance system according to the present embodiment will be described. Fig. 2 is a sequence diagram showing processing of the operation assistance system according to the first embodiment of the present disclosure.

[0054] In step S101 , the image acquisition unit 131 acquires a target image of an operation button provided on the control device 400 .

[0055] In step S103, the image recognition unit 133 performs image recognition on the target image to identify operation buttons that appear in the target image.

[0056] In step S105, the icon obtaining unit 135 obtains an icon associated with the identified operation button.

[0057] In step S107 , the display setting unit 137 displays the acquired icon via the display unit 120 .

[0058] In step S109 , the operation acquisition unit 141 acquires the user's operation accepted via the operation unit 150 .

[0059] In step S111, the operation acquisition unit 141 determines whether or not the icon has been operated. If it is determined that the icon has not been operated (NO in step S111), the process returns to step S109.

[0060] On the other hand, if it is determined that the icon has been operated (YES in step S111), in step S113, the control output unit 143 acquires a control signal corresponding to the operation on the icon and transmits the acquired control signal to the control device 400.

[0061] In step S115, the control device 400 executes the function related to the control signal.

[0062] In step S117, the display setting section 137 receives from the control device 400 a status signal indicating the result of executing the function of the control device 400.

[0063] In step S119 , the display setting unit 137 changes the form of the icon based on the status signal received from the control device 400 .

[0064] [Second embodiment] An example configuration of an operation assistance system will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the configuration of an operation assistance system according to a second embodiment of the present disclosure. As shown in Fig. 3, the operation assistance system includes an operation assistance device 1, a server 2, and a control device 400.

[0065] Unlike the controller 130 shown in FIG. 1, the controller 130 included in the operation assistance device 1 in FIG. 3 includes an inquiry unit 139 instead of the icon acquisition unit 135 .

[0066] The query unit 139 acquires an identifier for identifying an operation button identified by image recognition of the target image from the image recognition unit 133. Then, the query unit 139 transmits the identifier for identifying the operation button to the server 2 via the communication unit 110. After that, the query unit 139 acquires an icon linked to the operation button related to the identifier from the server 2 via the communication unit 110.

[0067] Next, the server 2 includes a communication unit 210 , a controller 230 , and a database 260 .

[0068] The communication unit 210 is communicably connected to the communication unit 110 included in the operation assistance device 1. The communication between the communication unit 110 and the communication unit 210 may be performed using, for example, the Internet. Furthermore, the communication between the communication unit 110 and the communication unit 210 may be performed using a mobile communication function such as 4G / LTE or 5G.

[0069] The database 260 stores operation buttons provided on the control device 400 in association with icons. The information stored in the database 260 is similar to the information stored in the database 160.

[0070] The controller 230 (an example of a control unit or a processing unit) is a general-purpose computer including a CPU (Central Processing Unit), a memory, and an input / output unit. The controller 230 includes an icon acquisition unit 235.

[0071] The icon acquiring unit 235 acquires an icon linked to the operation button related to the acquired identifier. More specifically, the icon acquiring unit 235 acquires the icon linked to the specified operation button by referring to the database 260. The icon acquiring unit 235 may also acquire information related to the arrangement of the icon by referring to the database 260.

[0072] The icon acquiring unit 235 may be provided in the identified control device 400 and may acquire a supplementary icon linked to a supplementary operation button that is not reflected in the target image. More specifically, the icon acquiring unit 235 may refer to the database 260 and acquire an icon linked to a supplementary operation button as the supplementary icon.

[0073] The icon acquiring unit 235 transmits the icon linked to the operation button related to the acquired identifier to the operation assistance device 1 via the communication unit 210. In addition, the icon acquiring unit 235 may transmit information related to the arrangement of the icons and supplementary icons to the operation assistance device 1.

[0074] Next, a processing procedure of the operation assistance system according to the present embodiment will be described. Fig. 4 is a sequence diagram showing the processing of the operation assistance system according to the second embodiment of the present disclosure. Differences from the sequence diagram of Fig. 2 will be described.

[0075] In step S201 , the inquiry unit 139 acquires an identifier for identifying the specified operation button, and transmits the identifier for identifying the operation button to the server 2 .

[0076] In step S203, the icon obtaining unit 235 obtains an icon linked to the operation button associated with the obtained identifier.

[0077] In step S205 , the icon acquisition unit 235 transmits the icon associated with the operation button corresponding to the acquired identifier to the operation assistance device 1 via the communication unit 210 .

[0078] [Third embodiment] An example configuration of an operation assistance system will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of an operation assistance system according to a third embodiment of the present disclosure. As shown in Fig. 5, the operation assistance system includes an operation assistance device 1, a server 2, and a control device 400.

[0079] Unlike the controller 130 shown in FIG. 1, the controller 130 included in the operation assistance device 1 in FIG. 5 includes an inquiry unit 139 instead of the image recognition unit 133 and the icon acquisition unit 135 .

[0080] The query unit 139 transmits the target image acquired by the image acquisition unit 131 and an identifier for identifying the operation button to the server 2 via the communication unit 110. Thereafter, the query unit 139 acquires, from the server 2 via the communication unit 110, an icon linked to the operation button identified by image recognition of the target image.

[0081] Unlike the controller 230 shown in FIG. 3, the controller 230 in FIG. 5 includes an image recognition unit 233 and an icon acquisition unit 235 .

[0082] The image recognition unit 233 performs image recognition on the acquired target image to identify operation buttons reflected in the target image. Furthermore, when identifying the operation buttons, the image recognition unit 233 may also identify identifiers that identify the operation buttons. The image recognition process by the image recognition unit 233 is similar to the image recognition process by the image recognition unit 133.

[0083] The icon acquisition unit 235 acquires an icon associated with the operation button identified by the image recognition unit 233 .

[0084] The icon acquiring unit 235 transmits the icon associated with the identified operation button to the operation assistance device 1 via the communication unit 210. In addition, the icon acquiring unit 235 may transmit information related to the arrangement of the icons and supplementary icons to the operation assistance device 1.

[0085] Next, a processing procedure of the operation assistance system according to the present embodiment will be described. Fig. 6 is a sequence diagram showing the processing of the operation assistance system according to the third embodiment of the present disclosure. Differences from the sequence diagram of Fig. 2 will be described.

[0086] In step S301, the inquiry unit 139 transmits the target image acquired by the image acquisition unit 131 and an identifier for identifying the operation button to the server 2 via the communication unit 110.

[0087] In step S303, the image recognition unit 233 performs image recognition on the target image to identify operation buttons that appear in the target image.

[0088] In step S305, the icon obtaining unit 235 obtains an icon associated with the identified operation button.

[0089] In step S307 , the icon acquisition unit 235 transmits the icon associated with the identified operation button to the operation assistance device 1 via the communication unit 210 .

[0090] [Effects of the embodiment] As described in detail above, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment acquire a target image of an operation button provided on a control device, acquire an icon linked to the operation button identified by image recognition of the target image, and display the icon via a display unit.

[0091] This simplifies the task of registering control devices and improves user convenience when there are many control devices to be operated remotely. In particular, image recognition of target images is used to identify control devices and operation buttons, so the task of the user inputting information to identify control devices when registering control devices is reduced. As a result, user convenience is improved.

[0092] Furthermore, because icons are displayed on the display unit based on the target image, the user can easily recognize the correspondence between the operation buttons reflected in the target image and the icons. As a result, a user who is accustomed to directly operating operation buttons can relatively easily operate the icons displayed on the display unit. Conversely, a user who is accustomed to operating icons displayed on the display unit can relatively easily directly operate the operation buttons. In other words, when comparing direct operation and remote operation, the user's operability of the control device is prevented from significantly differing. As a result, user convenience is improved.

[0093] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may transmit a control signal to a control device to execute a function associated with an operation button when an icon is operated via an operation unit. This simplifies the process of registering the control device and makes it easy to remotely control the control device. In particular, all that is required is to call up a target image capturing an operation button associated with the function the user wants to execute and operate the icon associated with the operation button. Therefore, even if there are multiple control devices or operation buttons, the user can easily execute the function they want to execute on the control device.

[0094] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may receive a status signal indicating the result of executing a function from the control device. This makes it possible to identify the status of the control device immediately after executing the function. As a result, when continuously transmitting control signals to the control device, it becomes possible to transmit control signals that take into account the status of the control device. As a result, it becomes possible to flexibly control the control device.

[0095] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may change the shape of an icon based on a status signal. This allows the user to easily recognize the result of executing a function on the control device through the shape of the displayed icon. As a result, user convenience is improved.

[0096] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may perform image recognition to identify operation buttons. This simplifies the process of registering control devices and allows operation buttons to be identified with high accuracy. As a result, user convenience is improved.

[0097] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may refer to a database that stores operation buttons and icons in association with each other, and acquire an icon associated with the identified operation button. This allows an icon to be assigned to each operation button, making it easier for the user to recognize the association between the operation button and the icon. A developer of the operation assistance system can design an icon in association with the function of the operation button and set the association between the operation button and the icon. For example, the same icon can be assigned to different operation buttons having the same function. As a result, differences in the appearance of operation buttons between control devices are absorbed, improving operability for the user when remotely operating the control device.

[0098] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the controller may be connected via a communication unit to a server including a database that stores operation buttons and icons in association with each other. The controller may then transmit an identifier for identifying the identified operation button to the server via the communication unit, the server may refer to the database to obtain an icon associated with the operation button associated with the identifier, and transmit the obtained icon to the controller, which may then obtain the icon from the server via the communication unit. This allows the operation buttons to be associated with the icons through processing on the server side. As a result, the operation assistance service provided by the operation assistance system can be improved at any time, even after the service has been provided to the user.

[0099] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the controller may be connected via a communication unit to a server including a database that stores operation buttons and icons in association with each other. The controller may then transmit a target image to the server via the communication unit, the server may perform image recognition to identify the operation buttons, refer to the database to obtain icons associated with the identified operation buttons, and transmit the obtained icons to the controller, which may then obtain the icons from the server via the communication unit. This allows the operation buttons reflected in the target image to be associated with the icons through processing on the server side. As a result, the operation assistance service provided by the operation assistance system can be improved at any time, even after the service has been provided to the user.

[0100] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the database may register at least one of control devices that have been connected to the communication unit and control devices specified by the user, and may store operation buttons associated with icons associated with the registered control devices. This limits the control devices targeted by the operation assistance system, improving security. Furthermore, using the database in which at least one of control devices that have been connected to the communication unit and control devices specified by the user is registered, it is possible to limit the control devices and operation buttons associated with the control devices that are specific targets in image recognition of a target image. As a result, the accuracy of image recognition can be improved compared to when image recognition is performed on an unspecified number of control devices and operation buttons associated with the control devices.

[0101] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may permit registration of a control device in the database when the distance between the control device and the communication unit is equal to or less than a predetermined distance, or when the control device is connected to the communication unit via a short-range wireless communication system. This limits the control devices that are targets of the operation assistance system, improving security.

[0102] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may display, via a display unit, icons associated with operation buttons reflected in the target image. This allows only icons associated with operation buttons reflected in the target image, out of all operation buttons provided on the control device, to be displayed, improving operability when the user operates the control device. In particular, even if the control device is multifunctional and has multiple operation buttons, the user can proceed with the operation by focusing on the operation button associated with the function executed by the control device.

[0103] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may identify a control device associated with an operation button reflected in a target image, obtain a supplemental icon associated with a supplemental operation button provided on the identified control device and not reflected in the target image, and display the supplemental icon in addition to the current icon via a display unit. This allows a user to operate icons associated with other operation buttons provided on the control device based on a target image capturing some of the operation buttons on the control device, thereby improving user convenience.

[0104] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the controller may be connected to an imaging device and acquire a target image via the imaging device. This allows the user to instruct the execution of a function related to the control device captured by the imaging device. As a result, user convenience is improved. In particular, compared to directly operating the operation button, displaying an icon and manual associated with the operation button reflected in the target image and operating the icon improves user operability.

[0105] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the controller may be connected to a storage device and acquire target images stored in the storage device. This allows the user to instruct the execution of a function related to the control device based on past target images stored in the storage device. As a result, the user does not need to be near the control device when operating the control device. This improves user operability.

[0106] Furthermore, in the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment, the control device may be connected to the communication unit via a short-range wireless communication method, thereby limiting the control devices that are targets of the operation assistance system and improving security.

[0107] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiments may be configured to, if communication is not established between the control device and the communication unit, display a message on the display unit requesting establishment of communication between the control device and the communication unit before displaying the icon. This limits the control devices that are targeted by the operation assistance system, improving security.

[0108] Furthermore, the operation assistance method, operation assistance device, operation assistance system, and operation assistance program according to the present embodiment may be configured to, when multiple operation button candidates are extracted as a result of image recognition for a single operation button reflected in a target image, select one operation button candidate based on a selection instruction from a user as the identified operation button. As a result, even if a single operation button reflected in a target image cannot be identified because multiple operation button candidates are found, the operation button can be identified based on an instruction from the user. As a result, even if the accuracy of image recognition is poor, it is possible to register a control device in the operation assistance system and further control the control device.

[0109] Each of the functions described in the above embodiments may be implemented by one or more processing circuits, including programmed processors, electrical circuits, and even devices such as application specific integrated circuits (ASICs), or circuit components arranged to perform the described functions.

[0110] Although the contents of the present disclosure have been described above based on the embodiments, the present disclosure is not limited to these descriptions, and various modifications and improvements are possible, which will be apparent to those skilled in the art. The descriptions and drawings that form part of this disclosure should not be understood as limiting the present disclosure. Various alternative embodiments, examples, and operating techniques will be apparent to those skilled in the art from this disclosure.

[0111] Of course, the present disclosure includes various embodiments not described herein. Therefore, the technical scope of the present disclosure is defined only by the invention-specifying matters according to the scope of the claims that are appropriate from the above description.

[0112] REFERENCE SIGNS LIST 1 Operation support device 2 Server 110 Communication unit 120 Display unit 130 Controller 131 Image acquisition unit 133 Image recognition unit 135 Icon acquisition unit 137 Display setting unit 139 Inquiry unit 141 Operation acquisition unit 143 Control output unit 150 Operation unit 160 Database 170 Imaging device 210 Communication unit 230 Controller 233 Image recognition unit 235 Icon acquisition unit 260 Database 400 Control device

Claims

1. An operation support method relating to an operation support device comprising a communication unit, a display unit, and a controller, The aforementioned communication unit is connected to the control equipment in a manner that enables communication. The aforementioned controller, A target image is obtained of the operation buttons provided on the control device. The icon associated with the operation button identified by image recognition on the target image is obtained. The icon is displayed via the display unit, Identify the control device related to the operation button that is visible in the aforementioned target image, The system acquires the replenishment icon associated with the replenishment operation button that is located on the identified control device and is not visible in the target image. The display unit displays the supplementary icons in addition to the icons. An operation support method characterized by the following.

2. The aforementioned controller, It is connected to an operation unit that accepts operations on the aforementioned icon, When the icon is operated via the operation unit, a control signal to execute the function related to the operation button is transmitted to the control device via the communication unit. The operation support method according to claim 1, characterized by the above.

3. The aforementioned controller, The communication unit receives a status signal from the control device indicating the result of performing the function. The operation support method according to claim 2, characterized by the above.

4. The aforementioned controller, The shape of the icon is changed based on the status signal. The operation support method according to claim 3, characterized by the above.

5. The aforementioned controller, Perform the aforementioned image recognition to identify the operation button. The operation support method according to claim 1, characterized by the above.

6. The aforementioned controller, It is connected to a database that stores the association between the aforementioned operation buttons and the aforementioned icons, Referencing the aforementioned database, obtain the icon associated with the identified operation button. The operation support method according to claim 5, characterized by the above.

7. The communication unit is connected to a server that has a database that stores the operation buttons and icons in association with each other. The aforementioned controller, The communication unit transmits an identifier that identifies the specified operation button to the server. The aforementioned server, By referring to the database, obtain the icon associated with the operation button corresponding to the identifier, The acquired icon is sent to the controller. The aforementioned controller, The icon is obtained from the server via the aforementioned communication unit. The operation support method according to claim 5, characterized by the above.

8. The communication unit is connected to a server that has a database that stores the operation buttons and icons in association with each other. The aforementioned controller, The target image is transmitted to the server via the communication unit. The aforementioned server, Perform the aforementioned image recognition to identify the operation button, By referring to the aforementioned database, the icon associated with the identified operation button is obtained, The acquired icon is sent to the controller. The aforementioned controller, The icon is obtained from the server via the aforementioned communication unit. The operation support method according to claim 1, characterized by the above.

9. The aforementioned database contains: At least one of the control devices that have been connected to the aforementioned communication unit, and the control devices identified by the user, is registered. The operation buttons and icons related to the registered control device are stored in association with each other. The operation support method according to claim 6, characterized by the above.

10. The aforementioned controller, If the distance between the control device and the communication unit is less than or equal to a predetermined distance, or if the control device is connected to the communication unit by short-range wireless communication, then the registration of the control device in the database is permitted. The operation support method according to claim 9, characterized by the above.

11. The aforementioned controller, The icon associated with the operation button that is visible in the target image is displayed via the display unit. The operation support method according to claim 1, characterized by the above.

12. The aforementioned controller, Connected to the imaging device, Acquiring the target image via the aforementioned imaging device. The operation support method according to claim 1, characterized by the above.

13. The aforementioned controller, Connected to a memory device, To acquire the target image stored in the aforementioned storage device. The operation support method according to claim 1, characterized by the above.

14. The control device is connected to the communication unit by a short-range wireless communication method. The operation support method according to claim 1, characterized by the above.

15. The aforementioned controller, If communication has not been established between the control device and the communication unit, a message requesting the establishment of communication between the control device and the communication unit will be displayed via the display unit before the icon is displayed. The operation support method according to claim 1, characterized by the above.

16. The aforementioned controller, If, as a result of the image recognition, multiple candidate operation buttons are extracted for one of the operation buttons that appears in the target image, the one candidate operation button selected based on the user's selection instruction will be designated as the identified operation button. An operation support method according to any one of claims 1 to 15, characterized by the above.

17. An operation support device comprising a communication unit, a display unit, and a controller, The aforementioned communication unit is connected to the control equipment in a manner that enables communication. The aforementioned controller, A target image is obtained of the operation buttons provided on the control device. The icon associated with the operation button identified by image recognition on the target image is obtained. The icon is displayed via the display unit, Identify the control device related to the operation button that is visible in the aforementioned target image, The system acquires the replenishment icon associated with the replenishment operation button that is located on the identified control device and is not visible in the target image. The display unit displays the supplementary icons in addition to the icons. An operation support device characterized by the following.

18. Control equipment and An operation support device comprising a communication unit, a display unit, and a controller, An operation support system comprising, The communication unit is connected to the control device in a manner that enables communication. The aforementioned controller, A target image is obtained of the operation buttons provided on the control device. The icon associated with the operation button identified by image recognition on the target image is obtained. The icon is displayed via the display unit, Identify the control device related to the operation button that is visible in the aforementioned target image, The system acquires the replenishment icon associated with the replenishment operation button that is located on the identified control device and is not visible in the target image. The display unit displays the supplementary icons in addition to the icons. An operation support system characterized by the following.

19. A communication unit and a display unit that are connected to the control equipment in a communication manner. A storage medium that stores an operation support program to be executed on a controller connected to it, The aforementioned operation support program, The steps include acquiring a target image of an operation button provided on the control device, The steps include obtaining an icon associated with the operation button identified by image recognition of the target image, The steps include: displaying the icon via the display unit; The steps include identifying the control device related to the operation button that is visible in the target image, The steps include: obtaining a replenishment icon associated with a replenishment operation button provided on the identified control device and not visible in the target image; The steps include displaying the supplementary icons in addition to the icons via the display unit, including A storage medium containing an operation support program, characterized by the following features.