Display control system and program

Abstract

The aim is to simplify the operation of display devices installed within the facility. [Solution] A display control system for controlling the display on a display device capable of displaying multiple display screens in a facility where multiple devices are arranged, comprising: a gesture recognition unit that recognizes predetermined operation gestures made by workers included in a predetermined specific range within the shooting range of the shooting unit, based on images obtained by a shooting device that photographs the inside of the facility; and an operation reception unit that receives display operations, which are operations related to the display of the display device, corresponding to the operation gestures recognized by the gesture recognition unit.

Description

Technical Field

[0001] The present invention relates to a display control system and a program.

Background Art

[0002] In production factories such as food factories, large monitors are installed, and various production situations and the like are displayed. For example, the same screen is always displayed on the monitor, and there are cases where a plurality of screens are switched and displayed at regular intervals. When an operator or the like wants to switch the display content in such a production factory, they had to move to the location where a display control device for controlling the display on the display device was installed and operate a mouse, keyboard, or the like.

[0003] Patent Document 1 discloses a technique for observing a gesture by a user of industrial machinery as an instruction for the industrial machinery and operating the industrial machinery according to the instruction.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In a production factory, an operator had to move to the location where a display control device was installed to switch the display and touch an operation unit to perform an operation. For this reason, there was a problem that the operation was troublesome.

[0006] The present invention has been made in consideration of such points, and an object thereof is to simplify the operation on a display device arranged in a facility.

Means for Solving the Problems

[0007] The present invention relates to a display control system for controlling the display of a display device capable of displaying multiple display screens in a facility where multiple devices are arranged, comprising: a gesture recognition unit that recognizes predetermined operation gestures made by a worker included in a predetermined specific range within the shooting range of a shooting device, based on a captured image obtained by a shooting device that photographs the inside of the facility; and an operation reception unit that receives a display operation, which is an operation related to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit.

[0008] Another embodiment of the present invention is a display control system for controlling displays on multiple display devices capable of displaying multiple different display screens relating to products within a facility where multiple devices are arranged, comprising: a gesture recognition unit that recognizes predetermined operation gestures made by an operator based on captured images obtained by one or more shooting devices that photograph the facility; an operation reception unit that receives a display operation which is an operation relating to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit; and a display processing unit that displays a display screen corresponding to the display operation on a display device determined based on the position of the operator from which the operation gesture was recognized by the gesture recognition unit.

[0009] Another embodiment of the present invention is a program that causes a computer to control the display on a display device capable of displaying multiple display screens in a facility where multiple devices are arranged, to function as a gesture recognition unit that recognizes predetermined operation gestures made by workers included in a predetermined specific range within the shooting range of the shooting unit, based on images captured by a shooting unit that photographs the inside of the facility, and an operation reception unit that accepts display operations, which are operations related to the display of the display device, corresponding to the operation gestures recognized by the gesture recognition unit.

[0010] Another embodiment of the present invention is a program that causes a computer to function as a display processing unit that controls the display on multiple display devices capable of displaying multiple different display screens relating to products in a facility where multiple devices are arranged, based on captured images obtained by one or more cameras that photograph the facility, the computer to function as a display processing unit that displays a display screen corresponding to the display operation on a display device determined based on the position of the operator whose operation gesture was recognized by the gesture recognition unit. [Effects of the Invention]

[0011] According to the display control system and program of the present invention, operation of display devices installed within a facility can be simplified. [Brief explanation of the drawing]

[0012] [Figure 1] This is an overall diagram of the display system. [Figure 2] This figure shows an example of the arrangement of the display device and imaging device. [Figure 3] This figure shows an example of the data structure in the worker table. [Figure 4] This figure shows an example of the data structure of a gesture table. [Figure 5] This is a diagram showing the first display screen. [Figure 6] This is a diagram showing the second display screen. [Figure 7] This is an explanatory diagram for displaying a horizontally oriented graph as a slideshow. [Figure 8] This flowchart shows the display control process. [Figure 9] This is a diagram showing the display unit relating to the second modified example. [Figure 10] This figure shows an example of the arrangement of multiple display devices. [Figure 11]It is a diagram showing display examples of two screens. [Figure 12] It is a diagram showing display examples of two screens. [Figure 13] It is an explanatory diagram of specific examples of operation gestures and display operations. [Figure 14] It is a diagram showing a display example of alarm notification information.

Embodiments for Carrying Out the Invention

[0013] FIG. 1 is an overall configuration diagram of the display system 1. The display system 1 includes a display control device 10, a display device 20, and a photographing device 30. The display control device 10, the display device 20, and the photographing device 30 are communicably connected to each other via a communication network N such as the Internet.

[0014] FIG. 2 is a diagram showing an arrangement example of the display device 20 and the photographing device 30. The display device 20 and the photographing device 30 are arranged in the facility S. In the facility S, a plurality of devices such as production machines are arranged, and work is performed by a plurality of workers. Here, examples of the facility S include a packaging factory where a bag-making packaging machine is installed. The positional relationship between the display device 20 and the photographing device 30 is not particularly limited, but the photographing device 30 is preferably arranged around the display device 20.

[0015] Note that the facility where the photographing device 30 is arranged may be any area where a plurality of devices are installed and work is performed by a plurality of workers, and is not limited to the embodiment. Examples of the facility include production factories for foods, automobile parts, etc., and distribution facilities related to distribution.

[0016] In this embodiment, when the worker is located within the photographing range of the photographing device 30 and performs a predetermined operation gesture, a display screen corresponding to the operation gesture is displayed on the display device 20. In this way, the worker can switch the display screen on the display device 20 without using a user interface such as a mouse or a keyboard. The

[0017] The display control device 10 controls the display on the display device 20. The display device 20 displays multiple display screens. Examples of display screens include images taken inside the facility S and graphs showing the status of equipment inside the facility S. Examples of images taken include images showing the supply status of food products that are packaged, and images showing the transport status of packaged products. Furthermore, if the facility S is divided into multiple areas, such as spanning multiple floors, images taken in each area may be displayed on the display device 20. In this embodiment, these predetermined multiple display screens are displayed sequentially on the display device 20 by switching between them at regular intervals.

[0018] The camera 30 captures the operation gestures of an operator located within the shooting range. The captured images are transmitted to the display control device 10. Based on the video acquired from the camera 30, the display control device 10 recognizes the operation gestures and switches the display screen on the display device 20 according to the operation gestures.

[0019] The display control device 10 is an information processing device and comprises a control unit 100, a storage unit 110, and a communication unit 120.

[0020] The control unit 100 includes a processor such as a CPU (Central Processing Unit) and controls the operation of the display control device 10. The storage unit 110 includes, for example, an HDD (Hard Disk Drive), RAM (Random Access Memory), ROM (Read Only Memory), and SSD (Solid State Drive). Furthermore, the storage unit 110 is not limited to being built into the display control device 10, but may be a storage medium that can be detachably attached to the display control device 10 (for example, a USB memory). In this embodiment, the storage unit 110 stores various programs executed by the control unit 100 and various data.

[0021] The communication unit 120 includes a communication interface for communicating with external devices wirelessly or via radio. The control unit 100 transmits and receives data with the display device 20 and the imaging device 30 via the communication unit 120.

[0022] The display device 20 comprises a display unit 200 and a communication unit 210. The display unit 200 displays various information, and the communication unit 210 communicates with external devices wirelessly or via radio.

[0023] The imaging device 30 comprises an imaging unit 300 and a communication unit 310. The imaging unit 300 performs imaging within the imaging range. In this embodiment, the imaging unit 300 performs video recording. The communication unit 310 communicates with an external device wirelessly or via communication.

[0024] Next, the configuration of the control unit 100 of the display control device 10 will be described. The control unit 100 functions as an image acquisition unit 101, an authentication unit 102, an operation level identification unit 103, a gesture recognition unit 104, an operation reception unit 105, and a display processing unit 106 by executing a program stored in the storage unit 110. In the following, the processes described as being performed by the image acquisition unit 101, the authentication unit 102, the operation level identification unit 103, the gesture recognition unit 104, the operation reception unit 105, and the display processing unit 106 are processes performed by the control unit 100 by executing a program.

[0025] The image acquisition unit 101 acquires the captured image taken by the imaging unit 300 of the imaging device 30. The authentication unit 102 refers to the worker table 40 and authenticates the worker recognized in the captured image. The operation level identification unit 103 refers to the worker table 40 and identifies the operation level of the worker recognized in the captured image. Here, the operation level is, for example, a value of 5 levels from 1 to 5. An operation level is set for each worker according to their years of experience, etc., and the authority to perform display operations on the display device 20 is restricted based on the operation level. Workers with operation level 1 are only permitted to perform the minimum display operations, while workers with operation level 5 are permitted to perform all display operations.

[0026] Figure 3 shows an example of the data structure of the worker table 40. The worker table 40 stores the worker ID and operation level of each worker performing work within facility S, in an associated manner. The worker table 40 is stored, for example, in the storage unit 110.

[0027] Returning to Figure 1, the gesture recognition unit 104 refers to the gesture table and recognizes that a pre-registered gesture has been performed in the captured image acquired by the image acquisition unit 101. In this embodiment, the gesture is assumed to be a hand operation. The operation reception unit 105 receives a display operation corresponding to the gesture recognized by the gesture recognition unit 104. Here, the display operation is an operation related to the display of the display device 20, that is, an operation on the display screen displayed on the display device 20. Examples of display operations include operations to display a predetermined display screen on the display device 20, or operations to horizontally scroll the display screen that the display device 20 is currently displaying.

[0028] Figure 4 shows an example of the data structure of the gesture table 50. The gesture table 50 stores multiple operation gestures, each associated with a gesture ID, gesture, display operation, and operation level. The gesture table 50 is stored, for example, in the storage unit 110.

[0029] For example, gesture 1 is the action of raising the index finger to indicate "1," and gesture 1 is associated with the display on the first display screen. The first display screen is a list screen of multiple graphs, including the time-series change of production volume per hour, as shown in Figure 5. Gesture 2 is the action of raising two fingers to indicate "2," and gesture 2 is associated with the second display screen. The second display screen is a screen of a single graph showing the time-series change of the detection results of multiple sensors, as shown in Figure 6.

[0030] Gesture 3 is a gesture of holding up three fingers to indicate "3," and Gesture 3 is associated with a third display screen. The third display screen is, for example, a screen displaying images taken in the first area of ​​facility S.

[0031] Furthermore, gesture 10 is an operation of sliding the palm to the right, and gesture 10 is associated with rightward scrolling of the display screen. This allows, for example, a horizontally long graph that cannot be fully displayed on the display unit 200 to be displayed by sliding it, as shown in the graph in Figure 7. Thus, gestures 1 and onward correspond to operation gestures for displaying the display screen on the display device 20.

[0032] On the other hand, gesture 0 is a clenched fist, which is registered as the starting gesture. When the gesture recognition unit 104 recognizes this starting gesture 0, it can determine that an operation gesture will be performed next. In this way, by recognizing the starting gesture before the operation gesture, the accuracy of operation gesture recognition can be improved.

[0033] The gesture recognition unit 104 detects hand images in the captured image using pattern matching or the like, and extracts movement features such as motion vectors by tracking the hand images. Based on these detection results, the operation gesture and start gesture are recognized.

[0034] Furthermore, in this embodiment, the gesture recognition unit 104 performs gesture recognition when a worker is included in a specific range of the captured image. For example, the left side of the captured image is set as the specific range. Thus, the specific range is a part of the captured image. In facility S, a mark P may be placed on the floor at the position where the gesture is performed so that the worker is visible in this specific range. The worker stands at this mark P and performs the gesture towards the camera 30. This results in a captured image in which the worker performing the gesture is included in the specific range.

[0035] Within facility S, multiple workers move around, and it is conceivable that multiple workers may be included in the shooting range simultaneously. In contrast, by defining a specific range, the gesture recognition unit 104 can accurately identify the worker performing the gesture.

[0036] Returning to Figure 1, when the operation reception unit 105 recognizes an operation gesture, it refers to the gesture table 50 to check whether the operation level of the worker who performed the gesture meets the operation level of the recognized operation gesture. If the operation level of the worker meets the operation level of the operation gesture, the operation reception unit 105 accepts the display operation associated with the operation gesture in the gesture table 50.

[0037] The display processing unit 106 controls the display of various information on the display unit 200 of the display device 20. Specifically, the display processing unit 106 sends a display instruction to the display device 20 via the communication unit 120, causing the display unit 200 to display various display screens.

[0038] Figure 8 is a flowchart illustrating the display control process. As a prerequisite for the display control process, the display control device 10 is assumed to acquire captured images from the imaging device 30 in real time.

[0039] Then, the gesture recognition unit 104 checks whether or not a worker has been detected in a specific area of ​​the captured image (step S100). Step S100 is repeated until a worker is detected (N in step S100), and when a worker is detected (Y in step S100), the authentication unit 102 authenticates the worker (step S102).

[0040] In this embodiment, the worker possesses an RF tag (not shown) on which the worker ID is stored, and holds it over an RFID reader (not shown) installed near the camera 30. The RFID reader transmits the read user ID to the display control device 10 via the communication network N. The authentication unit 102 authenticates the worker by comparing the worker ID transmitted from the RFID reader with the worker ID stored in the worker table 40. Note that the authentication method is not limited to this embodiment.

[0041] If authentication fails (N in step S104), the control unit 100 displays an error on the display device 20 (step S106) and terminates the process. In this way, by not performing subsequent processing if the worker's authentication is unsuccessful, it is possible to restrict which workers can perform gesture-based operations.

[0042] If authentication is successful (Y in step S104), the operation level identification unit 103 identifies the operator's operation level (step S108). Specifically, the operation level identification unit 103 identifies the operation level associated with the operator ID in the operator table 40.

[0043] Next, the gesture recognition unit 104 waits until a start gesture is recognized (N in step S110), and once a start gesture is recognized (Y in step S110), it then waits until an operation gesture is recognized (N in step S112). Once an operation gesture is recognized (Y in step S112), the gesture recognition unit 104 refers to the gesture table 50 and identifies the display operation corresponding to the operation gesture (step S114).

[0044] Next, the operation reception unit 105 determines whether or not it is possible to accept a display operation based on the operator's operation level (step S116). Specifically, the operation reception unit 105 refers to the gesture table 50 and identifies the operation level associated with the display operation identified in step S114. Then, the operation reception unit 105 determines that it is possible to accept a display operation if the operator's operation level satisfies the identified operation level.

[0045] In this way, by defining the acceptable level of operation for each operation gesture, it is possible to restrict which operators can perform the operation, thereby preventing the display on the display unit 200, which is viewed by multiple operators, from being changed unnecessarily.

[0046] If the operation reception unit 105 determines that it is not possible to accept a display operation (N in step S116), the control unit 100 displays an error on the display device 20 (step S118) and terminates the process. If the operation reception unit 105 determines that it is possible to accept a display operation (Y in step S116), the display processing unit 106 performs display processing according to the display operation (step S120). This completes the display control process.

[0047] As described above, in the display system 1 according to this embodiment, the display control device 10 can accept display operations by an operator performing gestures within a predetermined range. In other words, it is possible to simplify operations on the display devices installed within the facility.

[0048] The embodiments described above are merely examples for carrying out the present invention, and various other embodiments can be adopted. For example, various modifications and changes are possible within the scope of the gist of the present invention as described in the claims, such as applying one modification to another. For example, some of the components of the above embodiments may be omitted, or the order of processing may be changed or omitted.

[0049] The first modified example will now be described. The display control device of this embodiment is an example of a display control system that controls the display on a display device installed in a facility, but the display control system may be implemented by multiple devices. That is, some functions of the display control device may be implemented by a first device, and other functions of the display control device may be implemented by a second device. For example, the first device may include a gesture recognition unit 104, and the second device may include an authentication unit 102 and an operation level identification unit 103.

[0050] As a second modification, as shown in Figure 9, a display unit 200 formed across the entire side surface of a cylinder may be placed within the facility. In this case, multiple imaging devices 30 may be arranged to photograph the area around the display unit 200, and as another example, an imaging device 30 may be placed on top of the display unit 200 to photograph its entire circumference.

[0051] In this case, the display processing unit 106 may display a display screen corresponding to the operation gesture in a display area of ​​the display unit 200 that corresponds to a specific area where the worker was detected. For example, as shown in Figure 9, a display screen corresponding to the operation gesture may be displayed in a display area 201 opposite to the area where the worker was detected.

[0052] Furthermore, as a third modification, as shown in Figure 10, multiple display devices 20 may be arranged within the facility. In this case, the control unit 100 identifies the position of the worker who performed the operation gesture from the captured image. The display processing unit 106 then displays a display screen corresponding to the display operation on the display device 20 determined based on the worker's position.

[0053] For example, suppose that there are multiple display devices 20, and that each display device 30 is located near each display device 20. In this case, the display device 20 corresponding to the display device that obtained the image of the worker will display a display screen related to the display operation.

[0054] A fourth variation is that when an operation gesture to change the display screen is received, the display of the screen may differ according to the priority of the display screen. In this case, each display screen is assumed to have a priority assigned to it.

[0055] For example, suppose a command to display pie chart screen B is input while bar graph screen A is displayed on the display unit 200. In this case, if the priority of bar graph screen A is lower than the priority of pie chart screen B, as shown in Figure 11, the display of bar graph screen A will not disappear, and pie chart screen B may be displayed superimposed on bar graph screen A in a part of the display unit 200. Another example is that, as shown in Figure 12, bar graph screen A and pie chart screen B may be displayed side by side on the display unit 200 at the same time.

[0056] Furthermore, after a certain period of time has elapsed since the operation gesture was received, the display on the display unit 200 may switch back to the display screen that was displayed at the time the operation gesture was received (for example, the home screen). For example, the priority of the display screen that was displayed when the operation gesture was received may be set to increase after a certain period of time has elapsed since the operation gesture was received, and the display may switch back to the original display screen accordingly.

[0057] On the other hand, if the priority of bar graph screen A is higher than the priority of pie chart screen B, the display of bar graph screen A may disappear, and the display on the display unit 200 may switch from bar graph screen A to pie chart screen B.

[0058] Thus, when a new display screen is shown via a display operation, a decision may be made on whether to close an already displayed display screen based on the priority of the display screen. This ensures that highly important display screens, such as those that many workers would want to check, remain displayed simultaneously even when a new display screen is shown via a display operation.

[0059] As a fifth modification, the operation gestures and corresponding display operations are not limited to the embodiment. Specific examples of operation gestures and display operations are described below.

[0060] For example, a display screen may be generated by an operation gesture. For instance, as shown in Figure 13(a), a screen frame is displayed on the left side of the display unit 200, and multiple display screens are displayed on the right side of the display unit 200 in a scrollable manner. In this state, the operator can perform an operation gesture of sliding their hand downwards to scroll down the multiple display screens. Furthermore, as shown in Figure 13(b), with a display screen to be added to the screen frame displayed, the operator can perform an operation gesture of pinching the display screen and moving it to the screen frame. This inserts the display screen into the screen frame. By repeating such operations, a display screen customized by the operator can be generated and displayed, as shown in Figure 13(c).

[0061] Another example is that, while an alarm status notification screen, which is a display screen indicating that an abnormality has occurred in a device such as a bag-making and packaging machine located in facility S, is displayed, a confirmed operation, such as a display operation, may be performed by an operation gesture. Specifically, when an abnormality occurs in the bag-making and packaging machine, an alarm sound is emitted from the alarm device of the bag-making and packaging machine, and information that an abnormality has occurred is notified from the bag-making and packaging machine to the display control device 10. The display processing unit 105 of the display control device 10 then superimposes the alarm notification information 300 on the display unit 200, as shown in Figure 14.

[0062] In this state, the operator makes an operation gesture to acknowledge and cancel the alarm notification information 300, and when this is recognized by the gesture recognition unit 104, the operation reception unit 105 accepts the operation to cancel the alarm notification information 300. The display processing unit 106 then cancels the alarm notification information 300, i.e., deletes it from the display unit 200, and the control unit 100 further stops the alarm sound from the alarm device of the bag-making and packaging machine. An example of an operation gesture to indicate that the alarm notification information 300 has been acknowledged is a gesture of clenching the palm and extending only the thumb. The operation to cancel the alarm notification information 300 is an example of a display operation, which is an operation related to the display. In addition, the alarm device of the bag-making and packaging machine may be an alarm lamp that lights up or flashes, as well as a speaker that emits an alarm sound. [Explanation of Symbols]

[0063] 1 Display System 10 Display control device 20 Display device 30 Imaging device 40 Worker Table 50 Gesture Table 100 Control Unit 101 Image acquisition unit 102 Certification Department 103 Operation Level Identification Unit 104 Gesture recognition unit 105 Operation Reception Section 106 Display Processing Unit 110 Storage section 120 Communications Department 200 Display section 210 Communications Department 300 Photography Department 310 Communications Department

Claims

1. A display control system for controlling the display on a display device capable of displaying multiple screens within a facility where multiple devices are arranged, A gesture recognition unit recognizes predetermined operation gestures performed by workers within a predetermined specific range of the shooting range of the aforementioned facility, based on images obtained by a shooting device that photographs the inside of the aforementioned facility. An operation receiving unit that receives a display operation, which is an operation related to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit, A display control system equipped with the following features.

2. The display control system according to claim 1, further comprising a display processing unit that displays a display screen corresponding to the display operation in a display area of ​​the display device corresponding to the specific range.

3. The display control system according to claim 1, wherein the gesture recognition unit starts recognizing the operation gesture when a start gesture is recognized.

4. The system further includes an authentication unit for authenticating the aforementioned worker, The display control system according to claim 1, wherein the operation reception unit accepts the display operation corresponding to the operation gesture when the operator has been authenticated.

5. For each type of display operation, a user skill level is set for the operator who is capable of performing the operation. The unit further comprises an operation level identification unit that identifies the operation level of the aforementioned worker, The display control system according to claim 1, wherein the operation reception unit displays the display operation corresponding to the operation gesture when the operator's operation level is the operation level associated with the operation gesture.

6. Priorities are set in advance for each of the multiple display screens shown on the aforementioned display device. The display control system according to claim 1, further comprising a display processing unit that displays the first display screen while continuing to display the second display screen when the display operation is a display operation of the first display screen and the priority of the first display screen is lower than the priority of the second display screen currently being displayed by the display device.

7. The display control system according to claim 6, wherein the display processing unit terminates the display of the second display screen and displays the first display screen when the priority of the first display screen is higher than the priority of the second display screen.

8. The display control system according to claim 6, wherein the display processing unit displays the first display screen superimposed on the second display screen.

9. The display control system according to claim 6, wherein the display processing unit displays the first display screen and the second display screen side by side.

10. The display control system according to claim 1, wherein when an alarm state indicating an abnormality has occurred in the device is displayed on the display device, and the gesture recognition unit recognizes an operation gesture indicating that the alarm state has been confirmed, the operation reception unit accepts a display operation to cancel the alarm state.

11. A display control system for controlling displays on multiple display devices capable of displaying multiple different display screens related to products within a facility where multiple devices are arranged, A gesture recognition unit recognizes predetermined operation gestures performed by an operator based on images captured by one or more imaging devices that photograph the inside of the facility. An operation receiving unit that receives a display operation, which is an operation related to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit, A display processing unit that displays a display screen corresponding to the display operation on a display device determined based on the position of the worker whose operation gesture is recognized by the gesture recognition unit, A display control system equipped with the following features.

12. In a facility where multiple devices are installed, a computer controls the display on a display device capable of displaying multiple screens. A gesture recognition unit recognizes predetermined operation gestures performed by workers within a predetermined specific range within the shooting range of the aforementioned facility, based on images obtained by the shooting unit that photographs the inside of the aforementioned facility, and An operation receiving unit receives a display operation, which is an operation related to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit. A program designed to function as such.

13. In a facility where multiple devices are installed, a computer controls the display on multiple display devices capable of showing multiple different display screens related to the products. A gesture recognition unit recognizes predetermined operation gestures performed by an operator based on images captured by one or more imaging devices that photograph the inside of the facility. An operation receiving unit that receives a display operation which is an operation related to the display of the display device, corresponding to the operation gesture recognized by the gesture recognition unit, and A display processing unit displays a display screen corresponding to the display operation on a display device determined based on the position of the worker whose operation gesture has been recognized by the gesture recognition unit. A program designed to function as such.

Images