System and method with a virtual switch

Description

Field of the invention

[0001] The invention relates to a system with a camera-based user interface according to the preamble of claim 1. State of the art

[0002] Various types of control devices for technical systems such as vehicles, machines, or electronic devices are known in the state of the art. In addition to conventional control devices with switches, buttons, rotary knobs, and similar mechanical control elements, so-called touch controls with touch- or proximity-sensitive control elements have also been known for several years.

[0003] A separate mechanical, touch-sensitive or proximity-sensitive control element can be provided for each adjustable parameter or for triggering each function of the system.

[0004] Furthermore, devices with gesture control are also known, allowing the user to operate the device by performing gestures in free space (generally at least in the vicinity of the device to be controlled). Image processing systems with one or more cameras, for example, can be used for such gesture control. EP 1 352 301 B1 discloses a method for controlling devices, in which a control element is holographically projected into the action area of an operator, and a movement of the operator toward the virtual control element is detected by a 3D camera.

[0005] DE 10 2014 225 927 A1 discloses a system according to the preamble of claim 1, which is intended to be used to control functions of a motor vehicle. Monitored areas are monitored using a thermal camera. A marking that makes the monitored area of the virtual switch recognizable to the user is projected onto a surface in the passenger compartment or can be designed as a sticker.

[0006] It is also known to provide so-called virtual keyboards as a user interface for computers. These keyboards define a plurality of monitored areas, each of which is assigned to a key on a keyboard. Here, too, the keys are made visible through projection.

[0007] Both projection and sticker designs have disadvantages. Projections can be difficult to see in bright light, and lasers used for this purpose can cause eye damage. The position of stickers or paintwork cannot be changed non-destructively.

[0008] From EP 1505350 A2 a hob with a freely positionable control panel with several control surfaces is known, wherein in an embodiment of the invention the arrangement of the control surfaces on the control panel can be individually arranged by the user and / or selectable functions can be assigned to the control surfaces. Summary of the invention

[0009] The invention is based on the object of providing a system of the generic type in which the position of the virtual buttons can be flexibly selected by the user and which can be used in industry at workstations in production (e.g., assembly or machine workstations) or in logistics (e.g., goods receiving, picking, or packaging workstations). Other areas of application, such as vehicles, household appliances, and / or gaming computers, are outside the claimed scope.

[0010] The object is achieved by a system having the features of claim 1. Advantageous embodiments of the invention emerge from the subclaims.

[0011] The invention relates to a system with at least one camera, a control computer for evaluating the image data captured by the camera and at least one virtual switch, wherein the virtual switch comprises a monitored area in the field of view of the camera and a marking visible to a user for making the monitored area recognizable, and the control computer is designed to detect user intervention in the monitored area and to trigger an action or process when user intervention has been detected.

[0012] The marking on the virtual switch can consist of two elements in particular: A function symbol (e.g., an arrow or a floppy disk) that allows the user to identify the function activated by pressing the virtual switch; and a code in the form of an AR marker, QR code, or similar, which uses image recognition to determine whether the switch is a virtual switch and what function it performs. Furthermore, the detection area can be determined based on the position of the code on the virtual switch.

[0013] According to the invention, the camera is a 3D or time-of-flight (TOF) camera.

[0014] Depending on the circumstances, the action or process can be triggered immediately after the intrusion is detected or when the intrusion has been detected for a specified minimum period of time. Furthermore, depending on the application, any object detected in the monitored area can be considered an intrusion, or the image processing can perform additional processes to distinguish a hand from other objects or even detect a specific hand gesture, triggering the action or process only when a hand or gesture has been detected in the monitored area.

[0015] According to the invention, it is proposed that the marker is permanently attached to a carrier body which can be moved by the user between several selectable positions in the field of view of the camera and the control computer is designed to determine the position of the marker from the image data and to determine a position and / or size of the monitored area depending on the determined position of the marker.

[0016] This allows users to ergonomically select the position of the virtual buttons according to their needs and freely configure their workstation or control console as desired. For example, a left-handed person can position frequently used buttons on the left, while a right-handed person would position the same button on the right. Positioning based on size or user preference is also possible. Unlike projections, the marking cannot be lost due to shadows and, with a suitable printing technique, can be robust and visible in all lighting conditions.

[0017] In the case of a 3D camera, the monitored area is an area in three-dimensional space that is preferably determined by the control computer such that its position is in the immediate vicinity of the marking, preferably above and central to the marking. Here, "above" refers to a direction perpendicular to a main plane of the carrier body.

[0018] The invention can be used for a variety of systems, in particular for manufacturing or logistics workstations (e.g. order picking, goods receipt, shipping), control consoles for machines, systems or vehicles or also for household appliances or gaming computers.

[0019] The carrier body is preferably platelet- or tile-shaped and, in a preferred embodiment of the invention, is made of plastic or metal.

[0020] It is further proposed that the marking comprise a machine-readable symbol or a machine-readable code, and that the control computer be configured to determine the symbol or code of the marking from the image data, determine a function assigned to the virtual switch based on the determined symbol or code, and trigger an action or process corresponding to the function when user intervention in the monitored area of the virtual switch is detected. For example, a marking in the form of a green arrow could be assigned the function "switch to the next step in the workflow." When the user places the carrier body with the visible marking in a specific location, a spatial area in the immediate vicinity of the marking is assigned to this function, and the function "switch to the next step in the workflow" is executed as soon as the user intervenes in this spatial area.

[0021] The support body can be placed loosely on a table or console, or it can comprise means on its rear side for releasably securing the support body to the console or workstation within the camera's field of view. The means for releasably securing the support body can be a magnet or a hook-and-loop fastener. Alternatively or additionally, a console or workstation can be equipped with fastening structures for releasably securing the support body, for example, with recesses or holders for inserting or placing the support body. This prevents accidental displacement of the support body and enables attachment even to inclined or vertical surfaces.

[0022] It is further proposed that the control computer be designed to operate the system in multiple operating modes and select the operating mode based on markings detected from the image data. For example, if a production workstation has "automatic" and "manual" operating modes, the user can place a carrier body with the "automatic" marking in the field of view of the camera, which then detects the marking and switches the workstation to automatic mode. If the marking is removed, the workstation is switched to manual mode. The latter image-based control of the system does not require detection of an intrusion into a monitored area, but only the detection of the presence of the markings.

[0023] In one embodiment of the invention, the system comprises a graphical user interface for a workstation dialog, wherein the control computer is configured to display an overview of functions that can be triggered via the user interface. Functions that the control computer has assigned to a virtual switch are hidden in the overview. The user can then choose whether to control certain functions via the workstation dialog using a mouse or screen touch, or via virtual buttons of the type according to the invention.

[0024] It is further proposed that the control computer be designed to operate the system in a teaching mode in which the user can link a marking to a function.

[0025] A further aspect of the invention relates to a computer-assisted method for creating a virtual switch in a system of the type described above. The method comprises, in particular, the steps of positioning a carrier body with a permanently applied marker in one of several selectable positions in the field of view of the camera by a user, determining the position of the marker from the image data by the control computer; and determining a position and / or size of the monitored area depending on the determined position of the marker by the control computer.

[0026] In an advantageous embodiment of the invention, the marking comprises a machine-readable symbol or a machine-readable code, and the method further comprises the steps of determining the symbol or code of the marking from the image data, determining a function assigned to the virtual switch depending on the determined symbol or code; and triggering an action or process corresponding to the function when user intervention in the monitored area of the virtual switch has been detected.

[0027] Further features and advantages will become apparent from the following description of the figures. The entire description, claims, and figures disclose features of the invention in specific embodiments and combinations. Short description of the characters

[0028] They show: Fig. 1 a system according to a first embodiment of the invention; Fig. 2a section of a workstation according to the first embodiment of the invention with several support elements with markings; and Fig. 3 a flowchart of a method for creating a virtual switching element according to an embodiment of the invention. Description of an embodiment

[0029] Figure 1 shows a system according to a first embodiment of the invention.

[0030] The system is an assembly workstation with a camera 10, a display 12 for displaying a workstation dialogue, a control computer 14 for evaluating the image data captured by the camera 10 and with several virtual switches 16a - 16c.

[0031] The workstation includes a work table 18 for setting up a workpiece carrier, shelves, switches 16a-16c, or storage locations (not shown) for containers with components or for required tools. In addition to the display 12, the camera 10, and the virtual buttons for triggering actions and IT bookings in the control system, the control computer 14 can control any subsystems, for example, conveyor technology for transporting and / or removing workpieces or containers, sensors, additional cameras, lighting, heating or air conditioning, and an actuator for adjusting the height of the work table 18 or chair.

[0032] Each of the virtual switches 16a - 16c comprises, as in Fig. 2shown in more detail, a monitored area 20a - 20c in the field of view of the camera 10 and a marking 22a - 22c visible to a user for identifying the monitored area 20a - 20c. The virtual switch 16a - 16c is therefore an augmented reality object, i.e. a combination of the marking 22a - 22c as a physical structure and the monitored area 20a - 20c as a cuboid structure in a virtual space. The control computer 14 is designed by appropriate software to detect user intervention in the monitored area 20a - 20c and to trigger an action or process when user intervention has been detected. In the exemplary embodiment, the camera 10 is a TOF camera with an integrated 2D camera, which outputs a point cloud as a three-dimensional representation of the workstation on the one hand and a two-dimensional image on the other.To monitor the intervention in the monitored area 20a - 20c, the point cloud is primarily evaluated, whereby the image of the 2D camera 10 can be used additionally to classify the objects in the field of view of the camera 10.

[0033] The marking 22a-22c is permanently affixed or printed on a carrier body 24a-24c, which can be moved by the user between several selectable positions in the field of view of the camera 10. The control computer 14 is designed to determine the position of the marking 22a-22c from the image data and to determine a position of the monitored area 20a-20c depending on the determined position of the marking 22a-22c. The control computer 14 locates the marking 22a-22c using a pattern recognition algorithm in the image of the 2D camera 10 and determines its position in space by overlaying the two-dimensional image with the point cloud.The monitored area 20a - 20c is a cuboid-shaped area in three-dimensional space, the lower side surface of which corresponds to the surface of the carrier body 24a - 24c of the marking 22a - 22c, wherein the carrier body 24a - 24c is arranged substantially centrally to the lower side surface of the carrier body 24a - 24c.

[0034] The marking 22a - 22c comprises, on the one hand, a pictogram or symbol understandable to the user and, on the other hand, a machine-readable code, for example, a QR code. The control computer 14 is designed to determine the code of the marking 22a - 22c from the image data and, depending on the code, to determine a function assigned to the virtual switch 16a - 16c. For this purpose, the control computer 14 comprises an assignment table or database in which each code is assigned to a function. The control computer 14 then triggers an action or process corresponding to the function when user intervention in the monitored area 20a - 20c of the virtual switch 16a - 16c is detected.

[0035] The carrier body 24a - 24c is a plastic plate which, as in Fig. 2 shown, can be inserted into recesses 26 or other fastening structures in the work table 18. In the example from Fig. 2 One of the recesses 26 is empty. In other embodiments of the invention, magnets are embedded in the back of the support body 24a-24c as a means for releasably securing the support body 24a-24c to the workstation within the field of view of the camera 10. If the work table 18 is a steel table, the support body 24a-24c can thus be secured in any desired position.

[0036] The control computer 14 uses the markers 22a-22c not only to define the monitored areas 20a-20c. Rather, the system can operate in multiple operating modes, and the operating mode can be selected by the control system depending on the markers 22a-22c detected from the image data.

[0037] For example, if, as in Fig. 2If, as shown, one of the support bodies 24a, designed as a plate, bears the symbol for an automatic operating mode on one side but is empty on the other side, and the system can be operated in the "Automatic" and "Manual" operating modes, the control computer 14 can interpret the image data such that the system operates in the "Automatic" operating mode, regardless of user intervention, if the automatic symbol is recognized in the image data, and in the "Manual" operating mode otherwise. To switch between the "Automatic" and "Manual" operating modes, the user then only needs to turn the plate over or move it out of the field of view of the camera 10 in some other way. The control computer 14 can directly recognize the symbol for the automatic operating mode or read the associated QR code. In another example, the operating modes can differ in the type of user guidance. e.g.in a more or less detailed workspace dialog, versions that depend on the user's experience ("skill level") or even in a language version of the workspace dialog. In the Fig. 2 In the illustrated embodiment, the virtual switch 16b is assigned to the "Forward" function, and the virtual switch 16c is assigned to the "Light on" and "Light off" operating modes. In this case, too, the control computer 14 can recognize which symbols are displayed or uncovered and select the operating mode accordingly.

[0038] On the display 12, which serves as a graphical user interface for a workstation dialog, the control computer 14 presents an overview of functions that can be triggered via the user interface. Functions that the control computer 14 has assigned to a virtual switch 16a - 16c are hidden from the overview. When the user has positioned the plate assigned to the function, or the carrier body 24a - 24c printed with the marking 22a - 22c, on the workstation, the control computer 14 defines a corresponding virtual switch 16a - 16c for the function as described above, and the function disappears from the overview on the display 12. Conversely, the function reappears in the overview when the user removes the plate and the control computer 14 stops monitoring the corresponding area 20a - 20c.

[0039] In order to assign new functions to virtual buttons, the system can be operated in a teaching mode in which the user can link a marking 22a - 22c to a function.

[0040] Fig. 3shows a flowchart of a computer-assisted method for creating a virtual switch 16a - 16c in a system of the type described above. In a first step, the user positions one of the carrier bodies 24a - 24c in one of several selectable positions in the field of view of the camera 10. In a second step, the position of the marking 22a - 22c is determined by the control computer 14 from the image data of the camera 10. In a third step, the QR code (or AR marker) of the marking 22a - 22c is determined from the image data, and a function assigned to the QR code is read from an assignment table. In a fourth step, the position and / or size of the monitored area 20a - 20c is defined by the control computer 14 depending on the determined position of the marking 22a - 22c, and the virtual switch 16a - 16c thus formed is assigned to the function determined depending on the QR code.

Claims

1. A manufacturing or logistics workplace with at least one camera (10), a control computer (14) for evaluating the image data captured by the camera (10) and for controlling a. a machine, b. a conveyor system for transporting workpieces to and / or from the workplace, or c. a workplace dialog, and with at least one virtual switch (16a - 16c), wherein the virtual switch (16a - 16c) comprises a monitored area (20a - 20c) in the field of view of the camera (10) and a marking (22a - 22c) visible to a user for making the monitored area (20a - 20c) recognizable, and the control computer (14) is configured to detect a user intervention in the monitored area (20a - 20c) and to trigger an action or a process when a user intervention is detected, wherein the process concerns the control of the machine, the conveyor system or the workplace dialog, wherein the marking (22a - 22c) is permanently attached to a carrier body (24a - 24c), which can be moved by the user between several selectable positions in the field of view of the camera (10) and the control computer (14) is configured to determine the position of the marking (22a - 22c) from the image data and to determine a position and / or size of the monitored area (20a - 20c) depending on the determined position of the marking (22a - 22c), characterized in that the camera is a 3D or Time-of-Flight (TOF) camera and the monitored area (20a - 20c) is a cuboid-shaped area in three-dimensional space.

2. A manufacturing or logistics workplace according to claim 1, characterized in that the marking (22a - 22c) comprises a machine-readable symbol or a machine-readable code and / or a symbol that can be read and interpreted by the user, and the control computer (14) is configured to determine the symbol or the code of the marking (22a - 22c) from the image data, to determine a function assigned to the virtual switch (16a - 16c) depending on the determined symbol or code, and to trigger an action corresponding to the function or a signal corresponding to the function when a user intervention in the monitored area (20a - 20c) of the virtual switch (16a - 16c) is detected.

3. A manufacturing or logistics workplace according to claim 1 or 2, characterized in that the carrier body (24a - 24c) comprises means for detachably fastening the carrier body (24a - 24c) to a console or a workplace in the field of view of the camera (10) on its back side.

4. A manufacturing or logistics workplace according to one of claims 1 to 3, characterized in that the means for detachably fastening the carrier body (24a - 24c) comprise a magnet or a hook and loop fastener.

5. A manufacturing or logistics workplace according to one of the preceding claims, characterized by a console or a workplace with fastening structures for detachably fastening the carrier body (24a - 24c).

6. A manufacturing or logistics workplace according to one of the preceding claims, characterized in that the control computer (14) is configured to operate the system in multiple operating modes and to select the operating mode depending on markings (22a - 22c) detected from the image data.

7. A manufacturing or logistics workplace according to one of the preceding claims, characterized by a graphical user interface for a workplace dialog, wherein the control computer (14) is configured to display an overview of functions that can be triggered via the user interface, wherein functions that the control computer (14) has assigned to a virtual switch (16a - 16c) are hidden in the overview.

8. A manufacturing or logistics workplace according to one of the preceding claims, characterized in that the control computer (14) is configured to operate the system in a teaching operating mode in which the user can link a marking (22a - 22c) with a function.

9. A computer-implemented method for creating a virtual switch (16a - 16c) for a manufacturing or logistics workplace with at least one camera (10) and a control computer (14) for evaluating the image data captured by the camera (10) and for controlling a. a machine, b. a conveyor system for transporting workpieces to and / or from the workplace, or c. a workplace dialog, wherein the virtual switch (16a - 16c) comprises a monitored area (20a - 20c) in the field of view of the camera (10) and a marking (22a - 22c) visible to a user for making the monitored area (20a - 20c) recognizable, and the control computer (14) is configured to detect a user intervention in the monitored area (20a - 20c) and to trigger a process when a user intervention is detected, wherein the process concerns the control of the machine, the conveyor system or the workplace dialog, wherein the method comprises the following steps: positioning a carrier body (24a - 24c) with a marking (22a - 22c) permanently attached to it in one of several selectable positions in the field of view of the camera (10) by a user; determining the position of the marking (22a - 22c) from the image data by the control computer (14); and determining a position and / or size of the monitored area (20a - 20c) depending on the determined position of the marking (22a - 22c) by the control computer (14), characterized in that the camera is a 3D or Time-of-Flight (TOF) camera and the monitored area (20a - 20c) is a cuboid-shaped area in three-dimensional space.

10. A method according to claim 9, characterized in that the marking (22a - 22c) comprises a machine-readable symbol or a machine-readable code and / or a symbol that can be read and interpreted by the user, and the method further comprises the following steps: determining the symbol or the code of the marking (22a - 22c) from the image data; determining a function assigned to the virtual switch (16a - 16c) depending on the determined symbol or code; and triggering a signal corresponding to the function when a user intervention in the monitored area (20a - 20c) of the virtual switch (16a - 16c) is detected.