Control device, processing method, and program

Abstract

The present invention displays a plurality of pieces of information associated with states of a control target. The present invention displays an image relating to the control target in a state associated with one piece of information selected from among the plurality of pieces of information.

Description

[Technical Field]

[0001] The present invention relates to a control device, a processing method, program Regarding. [Background technology]

[0002] Many technologies for remotely controlling a control target are being studied. Patent Document 1 discloses a technology related to a remote control system that remotely operates a control target using a display terminal located at another base via a communication network, and that controls a mobile object equipped with an imaging means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2020-5146 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-described control technology, when a plurality of control targets equipped with image capturing means are controlled, it is necessary to further improve operability.

[0005] Therefore, one of the objects of the present invention is to provide a control device, a processing method, and a storage medium that solve the above-mentioned problems. [Means for solving the problem]

[0006] According to a first aspect of the present invention, a control device includes a state display means for displaying a plurality of pieces of information linked to the state of a controlled object, and an image display means for displaying an image relating to the controlled object in that state based on the selection of one piece of information from the plurality of pieces of information.

[0007] According to a second aspect of the present invention, a processing method displays multiple pieces of information linked to the state of a controlled object, and based on the selection of one piece of information from the multiple pieces of information, displays an image related to the controlled object in that state. Processing method.

[0008] According to a third aspect of the present invention, the storage medium causes the computer of the control device to function as a status display means that displays multiple pieces of information linked to the status of the controlled object, and an image display means that displays an image related to the controlled object in that status based on the selection of one piece of information from the multiple pieces of information. [Effects of the Invention]

[0009] According to the present invention, it is possible to improve operability when controlling a plurality of control targets. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic configuration of a control system including a control device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a functional block diagram of a control device according to the present embodiment. [Figure 3] 10 is a diagram conceptually showing an example of an image (or a moving image) displayed in a first display area. FIG. [Figure 4] FIG. 10 is a diagram conceptually illustrating an example of display information. [Figure 5] FIG. 10 is a diagram illustrating an example of display information. [Figure 6] FIG. 10 is a diagram illustrating an example of display information. [Figure 7] 10 is a flowchart showing a processing flow of the control device. [Figure 8] 10 is a flowchart showing a processing flow for a first operation C among the processes of the control device. [Figure 9] 10 is a flowchart showing a processing flow for a second operation T among the processes of the control device. [Figure 10]10 is a flowchart showing a processing flow for a third operation A among the processes of the control device. [Figure 11] FIG. 2 is a diagram illustrating a configuration of a control device according to the present embodiment. [Figure 12] 4 is a flowchart showing a processing flow by the control device according to the present embodiment. [Figure 13] FIG. 2 is a hardware configuration diagram of a control device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] A control device according to an embodiment of the present invention will be described below with reference to the drawings. 1 is a diagram showing a schematic configuration of a control system 100 including a control device according to this embodiment. The control system 100 includes a control device 1, a control target 2, an imaging device 3, and a communication network 8. The control device 1 is connected to the control target 2 and the imaging device 3 via the communication network 8 so as to be able to communicate with each other.

[0012] In the example shown in FIG. 1, the control device 1 is communicatively connected to one control target 2 and one imaging device 3. The control device 1 may be communicatively connected to multiple control targets 2 and multiple imaging devices 3. A user operates the control device 1. The control device 1 controls one or more control targets 2 and imaging devices 3 based on the user's operation. The imaging device 3 is, for example, a camera, and captures a range of space in which the movement and state of the control target 2 are captured. If the control target 2 is a robot arm, the imaging device 3 captures, for example, a range of angle of view that includes the arm of the control target 2 and captures the movement state of the object held by the arm from the origin to the destination. One imaging device 3 may capture an image that includes multiple control targets 2 and the space in which those control targets 2 operate. Alternatively, multiple imaging devices 3 may capture an image that includes one control target 2 and the space in which that control target 2 operates. The imaging device 3 may be separate from or integrated with the control target 2. The image captured by the imaging device 3 may be a moving image or a still image.

[0013] The control device 1 controls the control target 2 via a communication network. The control device 1 acquires an image of the control target 2 captured by the imaging device 3. The control device 1 is, for example, a tablet terminal.

[0014] FIG. 2 is a functional block diagram of the control device according to this embodiment. The control device 1 has an information acquisition unit 11, a status display unit 12, an image display unit 13, and a control unit 14. It executes a control program, which causes the control device 1 to fulfill the functions of the information acquisition unit 11, the status display unit 12, the image display unit 13, and the control unit 14. The status display unit 12 displays information associated with the status of at least some of the control targets 2 among the multiple control targets 2. The status display unit 12 may display information corresponding to the status of at least some of the control targets 2 among the multiple control targets 2. Hereinafter, for convenience of explanation, the term "corresponding" will be used to explain processing, etc., but the term may also be "associated" or "linked," etc.

[0015] In the following description, it is assumed that the control target 2 has a plurality of states. The plurality of states may be, for example, an abnormal state, a confirmation state, an execution state, a manual recognition state, a manual control state, an on-site adjustment state, a work request state, etc., as will be described later with reference to Fig. 5 etc. The plurality of states may include states different from those described above, or may be some of the states described above.

[0016] The information acquisition unit 11 acquires the state of the control target 2 and images (or moving images) captured by the imaging device 3 from a computer (for example, a control device 1 or recognition device operated by a user; hereinafter referred to as an external device) that is communicatively connected to the control target 2, the imaging device 3, etc. In response to a selection of information corresponding to one state from among information corresponding to a plurality of states, the image display unit 13 displays, for example, information such as an image relating to the control target 2 in the state represented by the selected icon on a display (described later with reference to FIG. 5). The display may be, for example, a touch panel display, a mobile terminal, an operation terminal for operating the control target, or a display device such as a notebook computer. The image display unit 13 may receive instructions for the control target 2 and instructions for the control device 1 from an external device via an operation on the display. Alternatively, the image display unit 13 may receive instructions for the control target 2 and instructions for the control device 1 from an external device via an operation on an input device such as a mouse or an operating lever. The control unit 14 controls the control target 2 according to instructions input via the image display unit 13, for example. The control unit 14 may be a remote control unit that controls the control target 2 via a communication network 8.

[0017] In this embodiment, the image display unit 13 receives a first operation C for selecting an icon representing one state from among a plurality of states and corresponding information. Based on the received first operation C, the image display unit 13 performs display control to display a list of information such as images of the control target 2 in that state. The image display unit 13 may display the list by superimposing it on the area displaying the image of the control target 2. Alternatively, the image display unit 13 may display the list in an area different from the area displaying the image of the control target 2.

[0018] The status display unit 12 displays icons representing at least some of the multiple states. The status display unit 12 may divide the control targets 2 in one state into multiple groups and display icons representing each group. The status display unit 12 may display not only an icon representing one state, but also an icon representing information comprehensively representing at least some of the multiple states (hereinafter referred to as a "general icon"). The comprehensive icon is, for example, a single icon representing both the confirmation state and the manual recognition state. In the following description, it is assumed that the status display unit 12 displays an icon representing each of a plurality of statuses. The state display unit 12 displays the number of control targets 2 in each state by superimposing the number of control targets 2 in that state or information indicating the number of control targets 2 (the number of dots corresponding to the number of control targets 2, the number of rectangles corresponding to the number of control targets 2, etc.) on the icon representing each state. The state display unit 12 may display the number of control targets 2 in each state in association with the icon representing each state (for example, above the icon, below the icon, to the side of the icon, etc.).

[0019] The control unit 14 acquires a second operation T for selecting at least one of the images in the list and a third operation A for selecting one of the multiple icons. Display information including the images and the icons (illustrated in FIG. 5) is displayed on the display. The image displayed in the second display area A2 in the display information exemplified in FIG. 5 is information representing the control target 2 in a state represented by the icon selected by the first operation C. For example, the control unit 14 performs control so that the control target 2 represented by the image selected by the second operation T transitions to a state represented by the icon selected by the third operation A. In other words, through the first operation C, the second operation T, and the third operation A, the control target 2 selected by the second operation T is controlled so as to change from a state represented by the icon selected by the first operation C to a state represented by the icon selected by the third operation A.

[0020] FIG. 3 is a diagram conceptually showing an example of an image (or a moving image) displayed in the first display area A1. The first display area A1 displays an image (or a moving image) of the control target 2 being captured (or captured) by the imaging device 3. The moving image illustrated in Fig. 3 is an image showing the control target 2 grasping an object placed at a certain position and performing an action of moving the object to another position.

[0021] FIG. 4 is a diagram conceptually illustrating an example of display information. The display information includes a first display area A1, a second display area A2, and a third display area A3. In the first display area A1, an image of the control target 2 captured by the imaging device 3 is displayed. The second display area A2 displays a list of thumbnails for the control targets 2. For example, in response to the first operation C, the list includes thumbnails for each control target 2 that is in a state represented by the icon selected by the first operation C. The list does not need to include multiple thumbnails, and may include a thumbnail for a single control target 2. In the third display area A3, icons representing the respective states of the control target 2 are displayed.

[0022] The first display area A1, the second display area A2, and the third display area A3 are not limited to the form illustrated in Fig. 4. In display information, for example, the first display area A1, the second display area A2, and the third display area A3 may be arranged horizontally or vertically.

[0023] In the display information, the second display area A2 and the third display area A3 may be positioned adjacent to each other, which has the advantage of reducing the amount of operation required to move between the first operation C and the second operation T, and between the second operation T and the third operation A.

[0024] The display information may be arranged in the order of the second display area A2, the third display area A3, and the first display area A1. In this manner, the order of operations from the first operation C to the third operation A corresponds to one round trip of the display information, which has the effect of reducing the amount of operation required to move between each operation when the set of operations from the first operation C to the third operation A is repeated multiple times.

[0025] Alternatively, the display information may include a third display area A3 (for convenience, referred to as "third display area A3C") corresponding to the first operation C and a third display area A3 (for convenience, referred to as "third display area A3A") corresponding to the third operation A, and the areas are arranged in the order of third display area A3C, second display area A2, and third display area A3A (referred to as "display order"). This type of arrangement provides the advantage of ease of operation, since the order of operations from the first operation C to the third operation A corresponds to the above-mentioned display order.

[0026] Next, an example of the state of the control target 2 will be described. For convenience of explanation, the state of the control target 2 is assumed to be any one of an abnormal state, a confirmation state, an execution state, a manual recognition state, a manual control state, an on-site adjustment state, and a work request state.

[0027] The abnormal state is a state in which the controlled object 2 is abnormal. Alternatively, the abnormal state is a state in which it is recognized that an abnormality has occurred in the controlled object 2. The confirmation state is a state in which the state and operation of the control target 2 must be confirmed by a communicatively connected external device. The confirmation state indicates, for example, a state in which the control device 1 is waiting for confirmation of the control target 2 by a user operating the control device 1. The confirmation state indicates a state in which the user is confirming whether the operation of the control target 2 has been completed normally.

[0028] The execution state is a state in which the control target 2 is performing a predetermined series of operations. The manual recognition state is a state in which the control target 2 waits to acquire, from an external device, information required for the operation of the control target 2. This state is a state in which, for example, an object to be grasped by the control target 2, an object to be transported by the control target 2, an object on which the control target 2 performs an operation, the position of the object, the shape of the object, the size of the object, etc. are provided from an external device. The manual recognition state indicates, for example, a state in which the control target 2 waits for input of information required for the operation of grasping by operating the control device 1 by the user.

[0029] The manual control state is a state in which the control target 2 waits for an operation from the control device 1 regarding the operation of the control target 2. The manual control state indicates a state in which, for example, a predetermined part of the control target 2, such as a robot arm, moves by an input operation of the control device 1 for the amount of movement of the predetermined part. The manual control state is, for example, a state in which the control target 2 moves an object from one position to another in accordance with a series of operations of the control device 1 by the user.

[0030] The on-site adjustment state is a state in which the controlled object 2 is waiting for physical adjustment at the location where the controlled object 2 is operating. The on-site adjustment state is, for example, a state in which measures are being taken to restore the controlled object 2 to a normal state from an abnormal state caused by a physical cause (such as a machine failure or a dropped object to be grasped). The on-site adjustment state can also be said to indicate a state in which repairs, various adjustments, and other work are being carried out at the site where the controlled object 2 is located, or a state in which the controlled object 2 is waiting to be repaired.

[0031] The work request state indicates a state in which a user is requesting another user who operates another control device to perform an operation related to the control of a control target 2. The work request state icon is used, for example, when a user is performing operations related to a plurality of control targets 2 using the control device 1 and wants to delegate operations related to some of the control targets 2 to another user to reduce the user's own workload. The state of the controlled object 2 does not necessarily have to include all of the states described above, and may include states different from the examples described above.

[0032] Next, the icons representing the states of the control target 2 and the operations for those icons will be described. In the following, for the sake of convenience, it is assumed that one icon corresponds to each of the above-mentioned states. In the above example, the icon representing the state of the control target 2 displayed in the third display area A3 is any one of an abnormality icon 31, a confirmation icon 32, an execution icon 33, a manual recognition icon 34, a manual control icon 35, an on-site adjustment icon 36, and a work request icon 37. The icon displayed in the third display area A3 may be an icon representing a state different from the above-mentioned states.

[0033] The abnormality icon 31 is an icon that represents an abnormal state of the control target 2. The abnormality icon 31 has a function of displaying, in the second display area A2, a list of thumbnails representing the control target 2 that is in an abnormal state, by a first operation C.

[0034] The confirmation icon 32 is an icon that indicates the confirmation state of the control target 2. The confirmation icon 32 has a function of displaying, for example, in the second display area A2, a list of thumbnails that indicate the control targets 2 that are in the confirmation state, in response to a first operation C.

[0035] The execution icon 33 is an icon that represents the execution state of the control target 2. The execution icon 33 has a function of displaying, for example, in the second display area A2, a list of thumbnails representing the control target 2 in the execution state, in response to a first operation C. Furthermore, the execution icon 33 has a function of controlling, in response to a second operation T and a third operation A, the state of the control target 2 to be in the execution state.

[0036] The manual recognition icon 34 is an icon that represents the manual recognition state. The manual recognition icon 34 has a function of displaying, for example, in the second display area A2, a list of thumbnails that represent the control targets 2 that are in the manual recognition state, in response to a first operation C.

[0037] The manual control icon 35 is an icon that represents a manual control state. The manual control icon 35 has a function of displaying, for example, in the second display area A2, a list of thumbnails that represent control targets 2 that are in a manual control state, in response to a first operation C.

[0038] The on-site adjustment icon 36 is an icon that represents the on-site adjustment state. The on-site adjustment icon 36 has a function of displaying, for example, in the second display area A2, a list of thumbnails that represent the control target 2 in the on-site adjustment state, in response to a first operation C.

[0039] The work request icon 37 is an icon that represents a work request status. The work request icon 37 has a function of displaying, for example, in the second display area A2, a list of thumbnails that represent the control targets 2 that are in a work request status, in response to a first operation C.

[0040] In the example shown in FIG. 4, the state display unit 12 displays information indicating the number of control targets 2 in that state superimposed on the icon displayed in the third display area A3.

[0041] Furthermore, operations on icons will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of display information displayed on a display or the like. The operations that can be performed on the display information are a first operation C, a second operation T, and a third operation A.

[0042] The first operation C is, for example, an operation of selecting an icon in the third display area A3 and swiping it toward the first display area A1. As an example, it is assumed that the control device 1 is a tablet terminal. When performing the first operation C on an icon, the user touches the icon and, while still touching the icon, moves his / her finger from the third display area A3 to the first display area A1. For example, in the first operation C, the manual control icon 35 displayed in the third display area A3 is swiped toward the first display area A1. In this case, a list of thumbnails, each of which is a frame of a moving image including the space in which the control target 2 operates, which is the state represented by the manual control icon 35, is displayed in the second display area A2.

[0043] Then, in response to the selection of one thumbnail from the list by the second operation T, the image display unit 13 displays in the first display area A1 a moving image including the space in which the control target 2 represented by the selected thumbnail operates. Alternatively, when a moving image including the space in which the control target 2 operates is displayed in the first display area A1, in response to the selection of one thumbnail from the list by the second operation T, the image display unit 13 switches the moving image displayed in the first display area A1 to the moving image including the space in which the control target 2 represented by the selected thumbnail operates.

[0044] The third operation A will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of display information displayed on a display or the like. With a moving image including the space in which the control target 2 operates displayed in the first display area A1, the user performs the third operation A on a certain icon. The user touches the execution icon 33 through the third operation A. The image display unit 13 identifies the icon on which the third operation A has been performed. The control unit 14 controls the control object 2 displayed in the first display area A1 so that the control object 2 changes to the state represented by the identified icon. The control unit 14 transmits a control signal to the control object 2, instructing the control object 2 displayed in the first display area A1 to change to the state represented by the execution icon 33.

[0045] The processing in the control device 1 will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of processing in the control device 1. It is assumed that the control target 2 is photographed by the imaging device 3. That is, it is assumed that the imaging device 3 creates a moving image including the space in which the control target 2 operates. In the initial state, it is assumed that nothing is displayed in the first display area A1 and the second display area A2. It is assumed that the above-mentioned abnormality icon 31 to work request icon 37 are displayed in the third display area A3.

[0046] The control device 1 connects to the imaging device 3 that captures the control target 2 and acquires a moving image including the space in which the control target 2 operates (step S101). The moving image acquired by the control device 1 may be of one control target 2 or may be of multiple control targets 2.

[0047] For convenience of explanation, it is assumed that the state request signal is transmitted to each controlled object 2. The information acquisition unit 11 transmits a status request signal to each control target 2 (step S102). The control device 1 does not necessarily need to transmit a status request signal to all control targets 2, and may transmit a status request signal to some of the control targets 2. When the imaging device 3 recognizes the status of the control target 2, the control device 1 may transmit a status request signal to the imaging device 3. The control target 2 receives the status request signal and identifies its own status. The control target 2 transmits information representing an identifier (hereinafter referred to as a "status identifier") representing the identified status to the control device 1.

[0048] The information acquisition unit 11 receives the information (step S103) and creates state information in which the state identified by the information is associated with an identifier (hereinafter referred to as a "controlled object identifier") representing the controlled object 2 to which the state request signal was sent. Therefore, the state information can also be said to be information in which the controlled object identifier representing the controlled object 2 is associated with a state identifier representing the state of the controlled object 2. The information acquisition unit 11 may store the created state information in a storage unit (not shown) (step S104).

[0049] The state display unit 12 calculates the number of control targets 2 in each state based on the state information. For example, the state display unit 12 reads the control target identifiers stored in the state information in association with the state identifiers representing each state, and counts the number (step S105). The state display unit 12 superimposes on each icon displayed in the third display area A3 the number of identifiers of control targets 2 in the state indicated by that icon (step S106). That is, the state display unit 12 displays the number of control targets 2 in the state indicated by the icon in association with the icon.

[0050] Next, processing for the first operation C will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the processing flow for the first operation C among the processing of the control device 1. The first operation C is an example of processing for displaying a list of thumbnails related to the control object 2 in a certain state in the second display area A2.

[0051] Assume that the user selects one of the icons displayed in the third display area A3 in a first operation C. The information acquisition unit 11 identifies the icon selected by the user and identifies a state identifier representing the state indicated by the icon (step S201). The information acquisition unit 11 outputs information representing the identified state identifier to the image display unit 13.

[0052] The image display unit 13 acquires information representing the state identifier. The image display unit 13 acquires a control object identifier stored in association with the state identifier in the state information (step S202). If there are multiple control object identifiers, the image display unit 13 acquires those control object identifiers. If there is no control object identifier stored in association with the state identifier in the state information, the image display unit 13 does not acquire a control object identifier.

[0053] The image display unit 13 identifies an identifier (hereinafter referred to as an "imaging device identifier") representing the imaging device 3 capturing an image of the control target 2 (step S203). The image display unit 13, for example, refers to system information in which the control target identifier representing the control target 2 is associated with the imaging device identifier identifying the imaging device 3 capturing an image of the control target 2, and identifies the imaging device identifier associated with the control target identifier in the system information. Based on the acquired imaging device identifier, the image display unit 13 transmits a thumbnail request signal to the imaging device 3 identified by the imaging device identifier, requesting the imaging device 3 to send information representing a thumbnail of the control target 2 captured by the imaging device 3 (hereinafter referred to as "thumbnail information") to the control device 1 (step S204).

[0054] The imaging device 3 receives the thumbnail request signal. In response to the thumbnail request signal, the imaging device 3 creates a thumbnail of, for example, one frame in a moving image generated by photographing the control target 2, and transmits thumbnail information representing the created thumbnail to the control device 1. The image display unit 13 receives the thumbnail information (step S205). The image display unit 13 displays the thumbnail represented by the thumbnail information in the second display area A2 (step S206). When there are multiple control targets 2 in the state indicated by the icon selected by the first operation C, the image display unit 13 receives thumbnail information from the imaging devices 3 capturing images of each control target 2. The image display unit 13 displays the thumbnails included in each piece of received thumbnail information in the second display area A2. That is, the image display unit 13 displays a list of thumbnails in the second display area A2. As a result, a list of thumbnails related to all control targets 2 in the state indicated by the icon for which the first operation C was performed is displayed in the second display area A2. Therefore, according to the above process, it is possible to obtain thumbnail information for each control target 2 in the state indicated by the icon. The image display unit 13 may create thumbnail management information in which an identifier representing the thumbnail information is associated with the imaging device identifier of the imaging device 3 that transmitted the thumbnail information and stored, and store the created thumbnail management information in the storage unit.

[0055] Assume that in step S201 described above, the manual control icon 35 displayed in the third display area A3 is swiped toward the first display area A1 by the first operation C. In this case, the image display unit 13 displays in the second display area A2 a list of thumbnails, each of which is a frame of a moving image including a space in which the control target 2 operates, which is the state represented by the manual control icon 35. In this case, the image display unit 13 may further display, in a part of the display area such as the third display area A3, an image of a controller that allows the user to manually control the amount of movement of a moving part, such as a robot arm, of the control target 2 in predetermined increments, superimposed on the third display area A3. If the control target 2 has a robot arm, the controller may be, for example, a controller that indicates the movement direction of the robot arm.

[0056] Next, the processing for the second operation T will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the processing flow for the second operation T among the processing of the control device 1. When processing the second operation T, the image display unit 13 has already established a communication session with the imaging device 3 capturing images of the control target 2 for the control target 2 represented by at least some of the thumbnails in the thumbnail list. When the communication session with the imaging device 3 is established, the image display unit 13 may buffer the images captured by the imaging device 3 in the control device 1. Therefore, the image display unit 13 is on standby so that it can display moving images corresponding to the thumbnails displayed in the second display area A2 in response to the second operation T.

[0057] The image display unit 13 may establish a communication session in response to the process of displaying a list of thumbnails in the second display area A2. The image display unit 13 may hold the communication session established in response to the process of transmitting a status request signal to the control target (step S102 in FIG. 7). In other words, the timing of establishing a communication session is not limited to the above-described example. The image display unit 13 may establish a communication session in response to detecting that the second operation T is performed. In the second operation T, the user selects a desired thumbnail by touching a desired thumbnail from the list of thumbnails displayed in the second display area A2, for example.

[0058] The information acquisition unit 11 identifies the selected thumbnail and identifies an identifier representing the identified thumbnail (hereinafter, "thumbnail identifier T1") (step S301). The information acquisition unit 11 outputs information representing the thumbnail identifier T1 to the image display unit 13. The image display unit 13 identifies the imaging device 3 that transmitted the thumbnail represented by the thumbnail identifier T1 based on the thumbnail identifier T1 (step S302). For example, the control device 1 identifies the imaging device 3 that transmitted the thumbnail identifier T1 by reading the imaging device identifier that is stored in the above-mentioned thumbnail management information in association with the thumbnail identifier T1. The image display unit 13 acquires an image (or a moving image) captured by the identified imaging device 3, for example, via a pre-established communication session, and displays the acquired image in the first display area A1 (step S303).

[0059] In other words, the image display unit 13 transmits a request to acquire a moving image to the imaging device 3 via the communication session. Upon receiving the distribution request, the imaging device 3 transmits the captured moving image to the control device 1 using a technique such as streaming distribution. The image display unit 13 in the control device 1 acquires the moving image via the communication session and displays the acquired image in the first display area A1. As a result, the control device 1 can display a moving image corresponding to the thumbnail represented by the thumbnail identifier T1.

[0060] The processing performed in response to the third operation A will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the processing flow for the third operation A among the processing by the control device. It is assumed that first display area A1 displays a moving image of a certain control target 2. It is assumed that the user, in a third operation A, selects one of the icons displayed in third display area A3. The information acquisition unit 11 identifies the selected icon and identifies the state identifier of the state represented by the identified icon (step S401). The information acquisition unit 11 outputs the state identifier to the control unit .

[0061] The control unit 14 acquires a state identifier (step S402). The control unit 14 acquires a control object identifier representing the control object 2 in the image displayed in the first display area A1 (step S403). The control unit 14 may acquire the control object identifier stored in association with the imaging device identifier representing the imaging device capturing the image, based on the system information. The control unit 14 transmits a control signal to the control object 2 represented by the control object identifier, for controlling the state of the control object 2 to transition to the state represented by the acquired state identifier (step S404). The controlled object 2 receives a control signal. The controlled object 2 acquires the state identifier contained in the control signal and operates to enter the state represented by the state identifier. For example, when the state identifier contained in the control signal indicates "execute," the controlled object 2 operates according to a predetermined program representing the operation. When the state identifier contained in the control signal indicates "confirm," the controlled object 2 stops the operation it is executing based on the program.

[0062] The control target 2 may, for example, identify the state to which it will transition after performing the operation by receiving the above-mentioned control signal. The control target 2 may identify the state after performing the operation by analyzing the state of its own device. Alternatively, the imaging device 3 capturing the image of the control target 2 may identify the state after performing the operation by analyzing the state of the control target 2. The control target 2 creates information representing the state identifier of the identified state and transmits the created information to the control device 1. The control device 1 receives the information (step S405). The control device 1 creates state information that associates the control object identifier of the control object 2 with the state identifier of the state included in the information, and stores the created state information in the storage unit of the control device 1 (step S406).

[0063] The state display unit 12 counts the number of control object identifiers stored in association with the state identifier of the state indicated by each icon in the state information (step S407). The state display unit 12 superimposes on each icon displayed in the third display area A3 the number of control object identifiers of the control objects 2 in the state indicated by that icon (step S408). That is, the state display unit 12 displays the number of control objects in the state indicated by each icon in association with the icon. As a result, when the user performs the third operation A on an icon indicating a certain state, and the number of control objects 2 in that state changes, the latest number of control objects 2 after the change is superimposed on the icon indicating that state. This allows the user to grasp the latest number of control objects 2 in each state.

[0064] Assume that the user selects one of the icons displayed in the third display area A3 and then selects another icon displayed in the third display area A3. In this case, the control device 1 repeats the above-described process from step S401 based on the selection of the other icon. The control unit 14 detects the selection of the other icon based on the status identifier acquired in step S402. The control unit 14 then acquires the status identifier and acquires, from the system information, an imaging device identifier representing the imaging device capturing the image displayed in the first display area A1. The control unit 14 outputs the imaging device identifier to the image display unit 13. Based on the imaging device identifier, the image display unit 13 identifies the imaging device 3 capturing the image so as to include the operating space of the control target 2 that sent the control signal. The image display unit 13 determines whether an image (or video) captured by the identified imaging device 3 is displayed in the first display area A1. If the image (or video) captured by the identified imaging device 3 is not displayed in the first display area A1, the image display unit 13 starts display processing. In this display process, the image display unit 13, similar to the process described above, acquires an image (or a moving image) captured by the identified imaging device 3, for example, via a pre-established communication session, and displays the acquired image in the first display area A1.

[0065] The process described using Figure 10 is an example of a process that controls, based on a second operation T on one image and a third operation A on information corresponding to one state, the control object 2 related to the selected image so that it becomes in the state represented by the icon to which the third operation A has been performed.

[0066] The above-described control device 1 can provide a user interface for efficiently operating multiple control targets 2. Furthermore, the processing of the control device 1 described above can display a list of images of the control targets 2 in a state selected by the user using the first operation C, and transition to a state selected by the user using the third operation A for the control target 2 corresponding to the displayed video, all with a single icon. This reduces the number of display items, such as icons, displayed in the user interface, ensuring high visibility in the user interface for controlling the control targets 2. Furthermore, because the control target 2 can be controlled using only the first operation C and the third operation A, the user can operate intuitively, easily remember the operations, and feel comfortable operating the control target 2. Furthermore, the processing of the control device 1 described above can display a state in which each image of the control target 2 in a predetermined state can be confirmed using only the first operation C. Furthermore, the control target 2 can be controlled using the third operation A so that the control target 2 related to the video displayed in the first display area A1 reaches a desired state, thereby achieving the desired state with just one user operation. This improves user work efficiency and reduces operational errors.

[0067] The information acquisition unit 11 of the control device 1 may change the size of the video displayed in the first display area A1 and the thumbnail displayed in the second display area A2 based on an operation from another device connected for communication. This allows the operating state of the control target 2 to be grasped with good visibility based on the video image of the imaging device 3 corresponding to the control target 2 even when remotely operated, thereby reducing human error by the user.

[0068] In the above process, the status display unit 12 of the control device 1 may be provided with a function to restrict operation of an icon representing a different state so that a state different from a predetermined state transition cannot be selected when controlling a control target 2 in a certain state. For example, suppose that when a user performs a first operation C on an abnormality icon 31, a list of thumbnails of moving images corresponding to the control target 2 in the state represented by the abnormality icon 31 is displayed in the second display area A2. Furthermore, suppose that a transition from the abnormality state to the execution state is not permitted. In this case, the control device 1 may be controlled not to detect an operation on the execution icon 33 representing the execution state. Alternatively, the control device 1 may, for example, gray out the execution icon 33 to notify the user that a transition to the execution state is not permitted. This prevents user errors.

[0069] Furthermore, the information acquisition unit 11 of the control device 1 may frame or highlight a selected thumbnail or icon in the second display area A2, allowing the user to intuitively recognize the progress of the operation.

[0070] The control device 1 displays the first display area A1 and the second display area A2 adjacent to each other in the display information, thereby enabling the user to easily perform the first operation C on the icon.

[0071] In the second display area A2, the image display unit 13 of the control device 1 displays a list of thumbnails related to the control target 2 in a row that is in a state indicated by the icon on which the first operation C has been performed. The image display unit 13 displays a video corresponding to a thumbnail selected from the list in the first display area A1. However, if the video displayed in the first display area A1 cannot be displayed due to some kind of failure, the image display unit 13 may select another thumbnail from the list, switch to the video corresponding to the selected thumbnail, and display the video in the first display area A1. The image display unit 13 may rearrange the list so that the thumbnail corresponding to the video displayed in the first display area A1 is at the top of the list and display the list. If playback of the video stops due to some kind of failure of the video, the image display unit 13 may delete the thumbnail corresponding to the video from the list.

[0072] In the above example, the control device 1 controls the control target 2 so that the target grasps the object and moves it from one position to another. However, the control target 2 operated by the control device 1 may be a moving object such as a robot such as an articulated robot, an airplane, a drone, a car, or an automatic guided vehicle, or may be another device such as an imaging device 3 or heavy machinery such as a crane.

[0073] Furthermore, the state indicated by the icon displayed in the third display area A3 as display information may be an icon displaying a list of control objects in other states in the third display area A3. For example, if the control object is a drone, the icon may be an icon representing an ascending flight state, an icon representing a descending flight state, an icon representing a hovering state, an icon representing a horizontal flight state, an icon representing a landing state, etc. In other words, the control device 1 may display information corresponding to each state of a plurality of control objects, and, based on selection of information corresponding to one state from the information corresponding to the plurality of states, display an image of the control object corresponding to the selected state.

[0074] Furthermore, in the above example, an embodiment is shown in which the multiple icons displayed in the third display area A3 correspond to certain states among multiple states related to the control object 2. However, in other embodiments, the multiple icons displayed in the third display area A3 may correspond to control of the control object. For example, the icons may be icons corresponding to a first control, an icon corresponding to a second control, an icon corresponding to a third control, etc.

[0075] In this case, the control device 1 may display information representing the control of each of the multiple control objects 2, and based on the selection of information representing one control from the information representing the multiple controls, display an image related to the control object 2 corresponding to that control. In this case, based on a first operation C of information representing one of the plurality of controls, the control device 1 displays a list of thumbnails related to the control objects 2 corresponding to the control in the second display area A2. The control device 1 displays icons corresponding to a plurality of different control contents. The control device 1 displays, on each icon, the number of control objects 2 being controlled in accordance with the control content represented by the icon, superimposed on the icon. Based on a selection operation on one of the images related to the plurality of control objects 2 and a second operation T on the information representing one of the plurality of controls, the control device 1 controls the control object 2 related to the image selected, so that the control object 2 operates in accordance with the control content represented by the second operation T.

[0076] The above-mentioned icons are one form of information corresponding to the state of each control target 2, and information other than icons may be displayed in the third display area A3 as information operated by the user in the control device 1.

[0077] The above description shows an example in which a user can perform a first operation C and a third operation A on an icon. However, the control device 1 may also be capable of detecting various other operations, such as a fourth operation and a fifth operation, performed by the user on the icon and performing corresponding control or operation. In this case, the control device 1 performs display control to display a list of images related to a control object in a state represented by one of the multiple pieces of information in an area different from the display area of ​​the captured image, based on one of the multiple operations performed on one piece of information. Also in this case, the control device 1 controls the control object related to the selected image based on the selection of one image from the list of images and one of the multiple operations performed on one piece of information, so that the control object related to the selected image is in a state represented by the operated information.

[0078] FIG. 11 is a diagram showing the configuration of the control device according to this embodiment. FIG. 12 is a flowchart showing a processing flow by the control device according to this embodiment. The control device 1 includes a status display unit 12 and an image display unit 13. The state display unit 12 displays a plurality of pieces of information linked to the state of the control target 2 (step S501). Based on the selection of one piece of information from the plurality of pieces of information, the image display unit 13 displays an image relating to the control target 2 in that state (step S502). The status display unit 12 can be realized using functions similar to those of the status display unit 12 in Fig. 2. The image display unit 13 can be realized using functions similar to those of the image display unit 13 in Fig. 2. Therefore, the control device 1 can be realized using functions similar to those of the control device 1 in Fig. 2.

[0079] (Example of hardware configuration) FIG. 13 is a hardware configuration diagram of the control device. An example of the configuration of hardware resources for implementing the control device according to each embodiment of the present invention described above using one calculation processing device (information processing device, computer) will be described. However, such a control device may be implemented physically or functionally using at least two calculation processing devices. Furthermore, such a control device may be implemented as a dedicated device.

[0080] 13 is a block diagram showing an example of the hardware configuration of a calculation processing device capable of realizing a control device according to each embodiment of the present invention. The calculation processing device 20 has a central processing unit (Central Processing Unit, hereinafter referred to as "CPU") 21, a volatile storage device 22, a disk 23, a non-volatile recording medium 24, and a communication interface (hereinafter referred to as "communication IF") 27. The calculation processing device 20 may be connectable to an input device 25 and an output device 26. The calculation processing device 20 can send and receive information to and from other calculation processing devices and communication devices via the communication IF 27.

[0081] The nonvolatile recording medium 24 is a computer-readable medium, such as a compact disc or a digital versatile disc. The nonvolatile recording medium 24 may also be a universal serial bus memory (USB memory), a solid state drive, or the like. The nonvolatile recording medium 24 stores the program and enables portability without requiring a power supply. The nonvolatile recording medium 24 is not limited to the above-mentioned media. Instead of the nonvolatile recording medium 24, the program may be transported via the communication IF 27 and a communication network.

[0082] The volatile storage device 22 is computer-readable and can temporarily store data. The volatile storage device 22 is a memory such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0083] That is, when the CPU 21 executes a software program (computer program: hereinafter simply referred to as "program") stored on the disk 23, it copies the program to the volatile storage device 22 and executes the arithmetic processing. The CPU 21 reads data necessary for executing the program from the volatile storage device 22. When display is required, the CPU 21 displays the output result on the output device 26. When inputting a program from an external device such as another device connected for communication, the CPU 21 reads the program from the input device 25.

[0084] The CPU 21 interprets and executes the control programs (FIGS. 7, 8, 9, 10, or 12) stored in the volatile storage device 22 that correspond to the functions (processes) represented by the units shown in FIG. 2 or 11. The CPU 21 executes the processes described in the above-described embodiments of the present invention.

[0085] In other words, in such a case, each embodiment of the present invention can be realized by the control program.Furthermore, each embodiment of the present invention can be realized by a computer-readable non-volatile recording medium on which the control program is recorded.

[0086] The program may also be a program for realizing some of the functions described above, or may be a so-called differential file (differential program) that can realize the functions described above in combination with a program already recorded in the computer system. The present invention has been described above using the above-described embodiment as an exemplary example. However, the present invention is not limited to the above-described embodiment. In other words, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention. [Explanation of symbols]

[0087] 1. Control device 2. Control target 3. Imaging device 100 Control System 11...Information acquisition section 12 Status display section 13 Image display section 14. Control section

Claims

1. a state display means for displaying a plurality of state icons linked to a plurality of different states of the control object, the state icons being associated with respective states to which the control object transitions by controlling the control object; an image display means for displaying in a second display area a list of thumbnail images relating to all control objects in a state indicated by a selected one of the plurality of status icons, and for displaying in a first display area an image relating to any one of the control objects in a state indicated by the selected thumbnail image; A control device comprising:

2. a control means for controlling a control object related to the thumbnail image, which is an image selected from the list of images based on a user's operation, and a target status icon, which is one status icon selected from the plurality of status icons based on a user's operation, so that the control object related to the thumbnail image transitions to a status indicated by the target status icon; The control device of claim 1 further comprising:

3. The image is a moving image or a still image of the control target. The control device according to claim 1 or 2.

4. The status display means displays the number of control targets in the status indicated by the status icon in a manner superimposed on the status icon. The control device according to any one of claims 1 to 3.

5. The computer displays a plurality of state icons, each of which is associated with a plurality of different states of the control object and which transitions to a state by controlling the control object; A list of thumbnail images relating to all control objects in the state indicated by a selected one of the plurality of status icons is displayed in a second display area, and an image relating to any one of the control objects in the state indicated by the selected one of the status icons is displayed in a first display area based on a selection operation of a thumbnail image in the list. Processing method.

6. Displaying a plurality of state icons associated with a plurality of different states of the control object, the states being transitioned by controlling the control object; a list of thumbnail images relating to all control objects in the state indicated by a selected one of the plurality of status icons is displayed in a second display area based on the selection of the selected one of the status icons, and an image relating to any one of the control objects in the state indicated by the selected one of the status icons is displayed in a first display area based on a selection operation of a thumbnail image in the list; A program that enables a computer to perform a function.

Images