Monitoring and control system for a manufacturing workstation and method for manufacturing a product or sub-products

DE502019012465D1Active Publication Date: 2024-11-28IGZ ING FUR LOGISTISCHE INFORMATIONSSYST MBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
DE502019012465
Authority / Receiving Office
DE · DE
Patent Type
Patents
Current Assignee / Owner
Priority Date
2018-01-29
Filing Date
2019-01-29
Publication Date
2024-11-28
Estimated Expiration
2039-01-29

AI Technical Summary

Technical Problem

Current monitoring and operating systems for manufacturing workstations have limited capabilities in detecting errors and ensuring compliance with the defined sequence of work steps, correct positioning, and alignment of components, leading to increased error rates and operational complexities for workers.

Method used

A monitoring and operating system that utilizes a graphical or acoustic user interface and a sensor arrangement with a camera for contactless guidance and monitoring, allowing for automatic recognition of work steps, gesture control, and data storage, which includes image processing to evaluate movement patterns and object positions, enabling error detection and simplifying worker interactions.

Benefits of technology

Significantly reduces error rates by providing real-time feedback and automatic acknowledgment of work steps, allowing workers to focus on manufacturing while keeping their hands free, and improving ergonomic interactions with the system, even in challenging environments like those with gloves or oily conditions.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader
Need to check novelty before this filing date? Find Prior Art

Description

[0001] The invention relates to a monitoring and operating system for a production workstation according to the preamble of claim 1 and a method for manufacturing a product or partial product according to the preamble of claim 15.

[0002] During production, assembly components are removed from one or more containers according to the assembly instructions or production order and assembled at the production workstation. Other production steps may include manual processing of a workpiece, such as grinding, polishing, drilling, and / or painting. The work steps of the production order are performed by an employee, also referred to below as a worker, at a production workstation.

[0003] The worker receives the work instructions for the current production step via a screen dialog at the workstation. Typically, a previously prepared workpiece is processed. This workpiece is secured at the beginning of the production step using a workpiece carrier so that its position and orientation correspond to the parameters defined for the production step. In addition to assembly, production orders can also include other tasks with definable sub-steps, such as recurring worker movements, material removal, tool use, etc.

[0004] It is common practice to equip production workstations with monitoring and control systems that detect incorrect intervention by a worker in a container. For this purpose, a grid-like arrangement of light barriers is provided above the access opening of the container. However, the possibilities for error detection in such systems are very limited.

[0005] Document WO 2014 / 028959 A1 discloses a system for assisting a worker in a goods handling facility, in which the worker's movements are monitored with a camera. Document US 2006 / 104479 A1 discloses a system for monitoring cashiers at supermarket checkouts.

[0006] The invention is particularly based on the object of providing a monitoring and operating system for a production workstation and a method for monitoring compliance with the defined sequence of work steps in production, through which the error rate in the production process can be significantly minimized. The same applies to compliance with the sequence, as well as to the correct positioning and / or alignment of the components to be attached. Properties of the workpiece that can be detected using image processing technology can also be checked, e.g., whether the cutting was carried out correctly or whether the produced weld seam is in the expected position. Furthermore, operation of the connected system is made considerably easier for the worker, for example by being able to advance or rewind work steps without contact through automatic recognition of the expected work step or using defined gestures.Another advantage is that the worker has their hands free for the actual production work, as the confirmation and feedback of work steps in the production system is fully automated. It is often difficult for workers wearing gloves or working in oily environments to operate the production IT system manually at their workstation. Furthermore, the screen of the production IT system can be positioned ergonomically (e.g., centrally above the workstation) rather than necessarily to the side within the worker's reach.

[0007] The object is achieved by a monitoring and operating system having the features of claim 1 and by a method having the features of claim 15. Advantageous embodiments of the invention emerge from the subclaims.

[0008] The invention relates to a monitoring and operating system for a production workstation for at least one worker for carrying out at least one manual production step, wherein the production workstation comprises a plurality of objects in a changeable configuration or movement state, wherein the system comprises a controller with a graphical or acoustic user interface for reproducing work instructions for the production step and a sensor arrangement for contactless guidance and / or monitoring of a production step carried out by the at least one worker.

[0009] It is proposed that the controller be designed to trigger a process depending on the image data captured by the camera, in particular upon detection of a specific configuration or movement state. The process may, in particular, include switching the display to show the work instructions for the next production step, to display an error message, a special note, or other information. Furthermore, the process may trigger the storage of data in a product database or an acknowledgment or posting of processes, which are then saved. Furthermore, the process may relate to the request for or removal of containers or workpieces and include the triggering of a control signal for conveyor technology.

[0010] The process can also directly affect the control of machines or other devices at the production workstation. Examples include: Tool parameters (torque, pressure, gas or power supply of a welding device, rotational speeds of drills or milling cutters) are set automatically or semi-automatically depending on the detected application. Lighting at the workplace is automatically adjusted or regulated so that the worker (or the camera) has a clear view of the relevant areas. The image information is used to evaluate sensor signals (e.g., brief overheating of a component with a temperature sensor is tolerated because it is recognized that the worker has just switched on a welding torch). When it is detected that the product is finished, the removal of the workpiece carrier via the workpiece carrier conveyor or the workpiece via the conveyor system is automatically triggered.

[0011] In this context, a manual manufacturing step is defined as a manufacturing step that contains at least one sub-step that involves manual intervention by a human worker, for example the assembly of a component, the removal of a component from a container, the use of a hand tool, the testing and / or inspection of a workpiece by the worker.

[0012] Monitorable objects within the meaning of the invention are in particular workpieces, components, workpiece carriers, tools and also the worker.

[0013] In this context, a configuration or movement state refers to both the static spatial configuration of the object and a movement state of the object. A movement state refers in particular to characteristic movements during production, such as inserting a component, screwing, polishing, using a tool, or the like, which can be recognized by the controller through movements or movement patterns of the component, tool, or worker, which can be extracted from the camera data using image processing. However, configuration or movement states can also be recognized and used that are independent of the actual production process, such as a wave or other gesture by the worker, the absence of a tool or component at its storage location, or the like, which allow direct or indirect conclusions to be drawn about the progress of the production process.

[0014] In particular, periodic movement patterns are considered as characteristic configuration or movement states of a production step, which can be easily recognized using image processing. For example, during screwing, polishing, or hammering, periodic movement patterns with characteristic frequencies of 1 - 5 Hz occur. According to the invention, it is proposed to filter out these frequency components using a bandpass filter from a short-time Fourier transform at recording intervals of a few seconds, for example, 2 - 10 seconds, optionally searching for peaks in the frequency spectrum of the filtered data and filtering them out, and evaluating the remaining data based on location and context. The location-dependent evaluation can be based in particular on the spatial relationship to the user's body or the workpiece.For example, a periodic movement on the surface of the workpiece (detected by distance measurement from the depth camera combined with dimensional data from the product database and workpiece positioning) is classified as "polishing," and a periodic movement next to the worker's head is classified as "waving." The duration of the process can also be monitored, and a warning can be issued if the worker exceeds the maximum permitted polishing time. For this purpose, the duration of a detected activity is compared with previously defined extreme values, and a control signal is issued if this limit is exceeded. Alternatively, the system can count how often the complete movement sequence has been repeated, and, for example, switch to the next work instruction after the third repetition.

[0015] If the worker is identified, either through login or image-based methods, personalized gesture recognition can also be implemented, taking into account the worker's preferences and physical characteristics. The same applies to the workstation configuration, which is automatically adapted to the worker, thus ensuring the most ergonomic workplace possible.

[0016] In a particularly advantageous embodiment of the invention, the camera is designed as a depth camera, for example using TOF (Time-of-Flight) technology. Other technologies for 3D or stereo cameras can also be used. For example, a camera system can be used as is known from the field of video games or game consoles. In these camera systems, for example, a pattern consisting of infrared light points is emitted. By distorting the pattern in space, a three-dimensional model can be calculated. This model can be evaluated using statistical analyses or by comparison with known reference patterns. This makes it possible, for example, to recognize the positions and dimensions of objects as well as gestures of employees. Alternatively, three-dimensional models can also be calculated using multiple cameras with different viewing angles in a conventional manner.In general, any sensor that provides the required 3D information (3D point cloud) to the production workstation can be used.

[0017] Especially for the frequently changing configurations of production workstations, which have to be reconfigured regularly, and due to the movement of the worker in three-dimensional space, the three-dimensional resolution of the image data has clear advantages over conventional cameras for two-dimensional images.

[0018] The use of a depth camera offers particular advantages for the use and definition of virtual confirmation buttons. One or more virtual confirmation buttons can be stored via the workstation configuration and / or based on a work plan. These replace physical confirmation buttons or control elements in the work dialog for executing actions within the assembly or production process. This means that if the sensor unit detects that the corresponding virtual spatial cuboid in the room for the corresponding confirmation button has been touched by hand for a definable time, the assigned action is triggered in the production system.

[0019] In a further embodiment of the invention, the sensor arrangement comprises, alternatively or in addition to the depth camera, an RGB camera system and / or further means for digital image processing for automatically recognizing the configuration. In addition, additional image processing and image acquisition processes / functions can be used. For example, the correct color, type and condition of the assembly component or tool can be checked, which leads to greater process reliability. Quality assurance can also be supported by, for example, checking the surface for defective areas and / or comparing the final assembly with a status stored in the system (e.g., whether the component was mounted in the correct location and / or in the correct orientation, and / or whether the adjustment angle of a mounted potentiometer is correct). Labels such as serial numbers and article numbers can be recognized and evaluated.Furthermore, the worker can be identified, for example, through facial recognition or other biometric features. Automatic photo and / or video capture allows these steps to be documented without any additional effort. This documentation can later be used for complaints, research, and traceability.

[0020] In particular, the camera system and the means for digital image processing can be designed to determine information about at least one activity of the worker from image data captured by the camera system, to classify the at least one activity with the aid of the information and to generate at least one control signal depending on the classification.

[0021] According to the invention, the control system is designed to detect and record the completion of a production step, and automatically display the work instructions for the next production step. This eliminates the need for manual switching or scrolling through the work instructions, thus accelerating and simplifying production. Switching the work instructions to the next production step can also be done contactlessly and completely wear-free, thus preventing damage to the hardware from dirty fingers or wear.

[0022] The advantages of the invention are particularly evident in complex manufacturing steps in which the manual manufacturing step comprises several sub-steps, each of which is monitored individually. For example, the sub-steps of a manufacturing step can include at least the removal of a component and the assembly of that component. By combining these sub-steps into a single unit, a logical context can be created that simplifies the control tasks involved in recognizing the sub-steps. The number of sensible and expected sub-steps within a manufacturing step is much smaller than the total number of possible activities of the worker, which simplifies recognition.

[0023] The system can detect the successful execution of substeps, particularly with the help of so-called acknowledgment conditions. Possible acknowledgment conditions are: a specific target image related to the respective assembly position on the workpiece (here, for example, the contour, position, color, orientation of the assembled component are evaluated); component removal from the correct container; use of the correct tool; worker movement for assembly (intervention in the area); recognition of the color / barcode (e.g. QR code), ... of the workpiece carriers used; recognition of the color / barcode (e.g. QR code), ... of the workpieces used; recognition of the color / barcode (e.g. QR code), ... of the component used; no movement in the defined area; manual confirmation by the employee (virtual button / gesture (e.g. "thumbs up")

[0024] In special cases, a combination of several of the above acknowledgment conditions can also be evaluated for acknowledgment recognition.

[0025] In one embodiment of the invention, the worker can override the automatic acknowledgement conditions at any time by manually advancing and manually proceeding to the next step. Switching to previous or future steps is also possible using a gesture (e.g., a swipe gesture). Special gestures for special functions such as displaying a detailed drawing, CAD view, switching on the tool, switching on the test device, reading the test value, photo documentation, or capturing properties of the workpiece or component (e.g., serial number) can also be recorded.

[0026] The inventor further proposes that the substeps include at least the acquisition of production data for a component, in particular identification data. Identification data can include, in particular, a product identifier such as a serial number, a label, or a batch number. Recognition can be achieved via image processing, for example, using barcodes, AR markers, QR codes, or character recognition, or via other sensors such as a reader for RFID tags. By capturing identification data, the manufacturing process can be documented, troubleshooting is facilitated, and process reliability can be improved. In particular, warning signals can be issued via the graphical or acoustic user interface when unsuitable components are detected based on the identification data.

[0027] The detection and identification of manual production steps can be simplified and accelerated if the system includes a database that assigns features recognizable by image processing to specific configuration or movement states. The process of the type described above can then be triggered in a well-defined manner by recognizing a configuration or movement state.

[0028] In a particularly user-friendly embodiment of the invention, the configuration or movement states detected or stored in the database include one or more gestures of the worker. The system can then learn, for example based on a neural network, to recognize and interpret certain gestures of the worker. Such gestures can be hand gestures such as a "thumbs up," holding up a certain number of fingers, waving, making a fist, raising the flat hand, or tapping the hand on the table, or body gestures such as standing up, stretching, or bending. For example, if the system detects a new image pattern that lies within a certain tolerance range to existing image patterns of a defined gesture, the user is asked to confirm the similarity once.

[0029] If this happens, the new image pattern will also be taken into account in the future when recognizing the gesture above (automatic learning).

[0030] According to a particularly intuitive variant of this embodiment, sliding left or right or waving the hand can be assigned to scrolling forward or backward through the work instructions.

[0031] In a particularly advantageous embodiment of the invention, the controller has a teaching operating mode in which the controller analyzes sensor data acquired by the sensor array in a specific configuration or movement state and uses it to learn how to recognize the configuration or movement state. The worker or a representative performs a specific characteristic movement pattern or gesture several times for a specific production step or sub-step until the system reliably recognizes characteristic features of the step or gesture and stores the associated image information in the reference database.

[0032] It is further proposed that the objects comprise one or more workpiece carriers and that the sensor arrangement comprises means for identifying the one or more workpiece carriers and for detecting the position of the one or more workpiece carriers at the production workstation. This allows the workpiece carriers and their position to be reliably detected, and the spatial area selected and monitored for recognizing the gestures or sub-steps can be adjusted and shifted accordingly. The system can independently detect the inserted workpiece carrier and check whether it is the workpiece carrier required by the system. This detection can be achieved, for example, via the shape, color, and / or other features (barcode (e.g., QR code), attached reflector, attached marker (e.g., AR marker), labeling, etc.).In addition, the position and orientation are checked via edge detection using a sensor unit.

[0033] According to a further embodiment of the invention, the objects normally comprise one or, in special cases, several workpieces. It is advantageous if the sensor arrangement comprises means for identifying the one or more workpieces, as well as for detecting the position of the one or more workpieces at the production workstation. This allows the workpieces and their position to be reliably detected, and the spatial area selected and monitored for recognizing the gestures or sub-steps can be adjusted and shifted accordingly. The system can independently recognize the inserted workpiece and check whether it is the workpiece required by the system. This detection can be achieved, for example, via the shape, color, and / or other features (barcode (e.g., QR code), attached reflector, applied marker (e.g., AR marker), labeling, etc.). In addition, the position and orientation are checked.

[0034] According to the invention, the system independently recognizes the tool used and checks whether it is the tool required by the system. This recognition can be achieved, for example, via the shape, color and / or other features (barcode (e.g. QR code), attached reflector, attached marker (e.g. AR marker), labeling, etc.). Identification is also possible based on the tool's original storage position (tool wall holder, tool compartment, etc.). Another option is to recognize the tool based on the worker's posture or movements. In addition, the position and orientation are checked. In addition to recognition via image processing, tool recognition can also be achieved via other methods, in particular by detecting power consumption or a data connection between the tool and the controller.

[0035] The system can also independently recognize the assembly component used and check whether it is the component required by the system. This recognition can be achieved, for example, based on its shape, color, and / or other characteristics (barcode (e.g., QR code), attached reflector, attached marker (e.g., AR marker), labeling, etc.). Identification is also possible based on its original storage position (container, sorting box compartment, wall shelf, etc.). For this purpose, the employee's reaching into the assigned storage container is detected by a sensor unit. In addition, the position and orientation of where the component was mounted on the workpiece are checked.

[0036] The system can independently detect the progress of the assembly process by checking, for example, whether all assembly components have been attached and whether the required sequence of movements was followed. This allows it to check whether all screws have been fully tightened, or whether a screw has been forgotten or only partially tightened. All criteria that can be detected using the acquired image information and corresponding image recognition steps can be checked, such as the completeness, position, and structure of a weld seam, the completeness, position, and structure of holes, etc.

[0037] The system can independently recognize a worker's work steps and check that they have been carried out correctly. To ensure this, the work process is continuously recorded by sensors. If, for example, the workpiece needs to be cleaned / sanded / painted, the system can check whether all areas have been completely processed or can visualize the areas still to be processed or not authorize confirmation of the work step. Conversely, a worker can also be warned if they stay in an area for too long or if areas are being processed that should not be processed in the current step. These warnings can be displayed both acoustically via a warning tone and visually in the workstation dialog. Once the work step has been completed successfully, the system automatically moves on to the next work step without any action from the plant.

[0038] In a particularly advantageous embodiment of the invention, specific work instructions for rectifying defects are automatically displayed to the worker if recurring assembly or processing errors are detected by image recognition.

[0039] Depending on the currently detected situation at the workplace, further interventions in the workplace and / or tool control can be planned. For example, the workplace lighting can be adjusted if glare occurs from highly reflective materials, or if insufficient illumination is detected. Furthermore, the tool parameters can be determined automatically or semi-automatically, i.e., suggested to the operator. Combining the acquired information with data from other sensors (e.g., temperature sensors, vibration sensors, etc.) is also possible.

[0040] A further aspect of the invention relates to a method for manufacturing a product or partial product, which comprises at least one manual manufacturing step performed by a worker. It is proposed that a manual manufacturing step be performed at a production workstation with a monitoring and operating system of the type described above.

[0041] Further features and advantages will become apparent from the following description of the figures. The entire description, the claims, and the figures disclose features of the invention in specific embodiments and combinations. Those skilled in the art will also consider the features individually and combine them into further combinations or subcombinations in order to adapt the invention, as defined in the claims, to their needs or specific areas of application. Fig. 1 shows a schematic plan view of a production workstation with a monitoring and operating system according to the invention; Fig. 2 shows the production workstation from Fig. 1 in a side view; Fig. 3 schematically illustrates the monitoring of a container by a monitoring and operating system according to the invention; Fig. 4 illustrates the monitoring of a container at a production workstation; Fig. 5 shows a container location at the production workstation equipped with different container sizes; Fig. 6 schematically illustrates a production workstation with a virtual confirmation button for triggering actions, e.g., acknowledgment of the work step. Also included are containers that are intervention-monitored; Fig. 7 illustrates the functionality of a virtual confirmation button; Figs. 8a-8 schematically illustrate various gestures of a worker recognized by the monitoring and operating system according to the invention, with gray lines symbolizing the tolerances used in the gesture comparison, which are configurable; Fig.Fig. 9 shows the example of attaching a machine-readable means for identifying a workpiece; and Fig. 10 shows the functionality of the detection of the assembly positions on the workpiece; Fig. 11 shows the sequence of an assembly process. This checks whether the component has been correctly mounted; and Fig. 12 symbolizes a repeated movement sequence of a hand, which can be recognized, for example, as a "polishing" activity, whereby a duration or a number of repetitions can be stored for detection.

[0042] Fig. 1 shows a top view of a production workstation 3. A control system comprises a computer-based controller 2. The control system further comprises a sensor arrangement for monitoring a production process performed by a worker W. In addition to monitoring the work sequence of the worker W, the worker W can also actively influence the process through gestures.

[0043] In the worker's field of vision, a graphical and acoustic user interface designed as screen 2.1 is arranged to display a workstation dialogue for the production step.

[0044] The Fig. 1 The system shown for operating, monitoring, and controlling a manufacturing process at a manufacturing workstation 3 uses a sensor arrangement configured as at least one TOF camera 1. The current situation at the manufacturing workstation 3 is recorded, analyzed, and controlled using the camera system 1. The analysis by the control unit 2 comprises determining information about at least one activity of the worker W from the image data recorded by the camera system 1, classifying the activity using the information from a connected database 2.2, and generating at least one control signal depending on the classification. Based on this analysis, errors, for example, can be detected and signaled to the worker W. Positively detected assembly, movements, and / or removals (e.g., of assembly components 7) lead to the automatic acknowledgment of the current work step.In the case of recurring assembly errors, corrective instructions are automatically displayed. Using gesture recognition, the worker can also actively influence the process, for example, to specifically acknowledge a step, indicate a shortage, display specific work instructions on screen 2.1, or trigger other predefined functions or processes.

[0045] In the field of view of a camera 1 directed at a production workstation 3, there are numerous reference points such as edges and the like that can be identified using simple image processing tools, and the image background is relatively static, so that the silhouette of the worker W can be easily determined.

[0046] The camera system 1 is mounted at the production workstation 3 and connected to a control unit 2. The image material captured by the camera system 1 is evaluated locally, image by image, by the control unit 2 and can then be discarded.

[0047] In the area covered by the camera system 1, the control unit 2 uses the digital evaluation of the image material to detect, among other things, interventions in the containers 4, the gestures of the work, as well as the use of the virtual confirmation buttons 10 and the mounting position 9.2 of the mounted component on the workpiece 6. If a larger or additional area is to be monitored, additional camera systems 1 can be mounted there and connected to the control unit 2.

[0048] According to the invention, the spatial positions of the workpiece 6 and, for example, the tools 14 or the containers 4, and the corresponding sections of the field of view of the camera system 1 are detected by the control unit 2 with the aid of the image processing software and used as the basis for monitoring. The totality of these positions and coordinates is referred to here and below as the configuration. This configuration determines the location of the areas to be monitored. These areas must be completely detectable by the camera system 1, i.e., they must be within the field of view of at least one of the cameras.

[0049] By changing the coordinates of the monitored areas, the system can be flexibly configured without changing the physical structure of the workstation. Changes to the number and / or position of the containers 4, as well as the workpiece 6 (or the workpiece carrier 5) and / or the tools 14 can be adapted to the specific requirements of the current production order. The same applies to the virtual confirmation buttons 10, which can be positioned and activated depending on the situation and trigger different actions. Positioning depending on the current user is also possible. This improves the ergonomics of the workstation, as the operating elements can be adjusted, for example, to the height of the worker W.

[0050] Furthermore, an automatic identification and / or definition of the assigned function of the virtual button is carried out either via the position of the virtual button or the symbol or QR code recognized by image recognition on the "Virtual Button".

[0051] The position of the "virtual buttons" can easily be changed using magnets or similar.

[0052] Fig. 3, Fig. 4 , Fig. 7 and Fig. 10 Show monitored areas 9 above a container 4, or in the case of the virtual confirmation button 10, in front of an area 11 marked with a sticker. Instead of the sticker, a detachably attachable plate or other object can be used to visualize the virtual confirmation button. The plate can be attached magnetically, with Velcro, with brackets, or other structures in the position desired by the user at the production workstation.

[0053] As soon as an object (e.g., the hand of worker W) moves into an actively monitored area 9, this is detected by the control software and reacts depending on the situation by generating an appropriate control signal. Depending on which active area 9 is affected, the event is recognized and classified by the control unit as worker activity. If an intervention is not expected at this time and in this area, or if it is to be classified as an error, this error situation is detected and further processed by the control software. This intervention control can be activated both for container 4 and for areas on the workpiece 6 (assembly positions 9.2), thus preventing incorrect assembly or incorrect removal.Depending on the classification, the control unit generates a control signal, which then generates a feedback signal for the worker W or which is further processed like a user input, for example by switching to the next work step.

[0054] During assembly processes, the duration of the intervention in the monitored area and the before and after situation can also be evaluated.

[0055] Furthermore, it can be checked whether, after the intervention, the component to be assembled is detected by the sensor unit in the corresponding dimension at the position in the dimension stored in the product data, minus defined tolerances at the assembly position. Possible methods include, for example, edge detection in combination with the evaluation of the spatial coordinates of the sensor unit, a target-actual image comparison, or similar methods. See also Fig. 11 .

[0056] Individual work steps are thus recognized by the control software and automatically acknowledged by default when the component is correctly mounted on the workpiece. For manual acknowledgement, the worker performs a defined gesture, which is recognized by the system. Alternatively, the worker can also acknowledge the step using a specially defined virtual confirmation button 10. Using different gestures or virtual confirmation buttons 10, the worker W can thus actively influence the production process. For example, it is possible to trigger a special process, e.g., report a quality defect. Gesture recognition also means that the worker W does not have to return to the operating area of ​​the assembly screen dialog (e.g., touch screen, mouse / keyboard). The worker W can remain in his current work area 8, which minimizes his walking distances and speeds up the entire process.

[0057] To identify the current workpiece 6, the camera system 1 detects an optical marker M1 (e.g., QR code, AR code, barcode) that is attached, for example, to the upper side of the workpiece 6. This contains the recognized ID, which the controller 2 can use (see Fig. 9 ). The optical marker is a means of identifying the workpiece and, if necessary, its positioning and alignment.

[0058] Alternatively, or if automatic identification fails, the ID can also be entered manually in a workstation dialog. The configuration for the production workstation 3 and the corresponding workpiece 6 is loaded into the control system 2. The workpiece 6 must be inserted into a workpiece carrier 5 to enable efficient detection by the camera system 1, as this requires, for example, a defined orientation and / or position of the workpiece 6.

[0059] The container location allocation of the workstation is stored in a configuration. This configuration allows the actual size and position of the individual container locations to be determined depending on the workstation, so that the specific locations can be identified and evaluated using camera system 1 to monitor a worker's removal from a container.

[0060] Typically, the workstation must initially be set up for a production order, i.e., the containers with the required assembly components 7 must be procured and, if necessary, the workpiece carrier 5 must be replaced or reassembled. These steps are also supported by the system by checking whether the correct containers 4 have been placed in the correct locations. Furthermore, the type and orientation of the workpiece carrier 5 are also checked. To reliably position the containers, compartment markings and / or compartment barriers 12 can be attached to the workstation, as shown in Fig. 5 Furthermore, container types and their dimensions can be stored in the configuration so that interventions in the positioned containers can be correctly detected in the subsequent process. With the help of the camera system 1, the position of smaller target containers within a storage space can also be recorded, as shown in Fig. 5 illustrated by a container location with containers of different sizes.

[0061] If the containers 4 are not linked to the storage location (e.g., master data or positioning in a previous process), they can be positioned dynamically by the user. The user is prompted in the dialog to place workpiece carriers, workpieces, tools, and / or containers 14 on the production workstation. Depending on the process, the controller 2 can specify a storage location 12 and, for example, a container 4, or the user can decide on the container used and the selection of the storage location 12.

[0062] By entering (e.g., by scanning) the container ID (e.g., barcode, or similar) after positioning it in an empty location 12, the new container 4 is linked to location 12 in the control and monitoring system. The camera system 1 automatically detects location 12 in which the new container 4 was placed. Alternatively, this assignment of container 4 to a location 12 can also be done manually using the workstation dialog.

[0063] One or more valid containers 4 for the current work step are displayed in the workstation dialog. In the case of multiple removal, for example, it is possible that several or even all assembly positions 9.2 on the workpiece represent a valid destination. Incorrect access to a container 4 or assembly position 9.2 that is not permitted at this time is indicated by an error signal. Correct removal from a container 4 is detected by the control system 2 with the help of the camera system 1 and, if necessary, automatically recorded.

[0064] In the Fig. 8a - Fig. 8c Examples of gestures are shown. An example of a simple gesture is the outstretched hand with the thumb pointing upwards, as shown in Fig. 8a The image from camera system 1 is analyzed, the image of the worker's hand is extracted, and the control software checks whether the corresponding pattern is included. This process can also be used with other gestures, such as those used in Fig. 8b und Fig. 8c are shown. To enable efficient gesture recognition, the area in which gestures are checked can be spatially or temporally restricted (e.g., depending on the current work order). The position at which a gesture was recognized is known to the control system and can be used in the subsequent action (e.g., acknowledgment of the work step at the recognized gesture position if multiple workpieces and / or assembly positions 9.2 are being processed simultaneously).

[0065] Optionally, a time limit can be specified or configured in the software, indicating how long the gesture must be held by the employee for it to be recognized as valid. This can prevent false recognition. Successfully recognized gestures trigger associated actions in the control system (e.g., acknowledgment of a work step). Additional gestures can be taught to the system and linked to actions.

[0066] In addition or as an alternative to gesture recognition, virtual confirmation buttons 10 are configured for user interaction, as shown in Fig. 7are shown. For this purpose, active areas 9.1 are defined in the room where there are no actual physical buttons / switches. The control system detects interventions in these active areas 9.1 and can trigger defined actions (e.g., acknowledgment of an assembly step). These areas can be marked on the physical structure, for example, with a label 11, so that they are easier for the worker W to find. The virtual confirmation buttons 10 can be varied in their position and size - as can the actions associated with their activation. By image recognition of the symbols or QR code on the virtual actuating surfaces, the system automatically detects which function should be triggered by the system. This allows the worker to arrange the buttons ergonomically for themselves. Different actions are also possible depending on the current order status.

[0067] Acknowledgment of the assembly of assembly components can be achieved either by automatic acknowledgment upon detection of the assembly on the workpiece, by means of a virtual confirmation button 10, or by using gestures, e.g., a thumbs-up gesture G1 in a defined area. The gesture can be recognized and interpreted using the camera system 1 and gesture recognition implemented by suitable means for digital image processing. For this purpose, information about at least one activity of the worker W is determined from the image data captured by the camera system 1. The activity is classified using the information from the connected system, and at least one control signal (e.g., acknowledgment) is generated depending on the determined classification.

[0068] In a further embodiment of the invention, the workpiece carriers 5, workpieces 6, tools 14, and / or containers 4 can be positioned on a shelf and / or storage area next to the actual workbench. Here, for example, the removal, i.e., the intervention in the containers on the shelf and / or storage area, is monitored, and the configuration is recognized analogously to the recognition described above.

Claims

1. A monitoring and operating system for a manufacturing workplace (3) for at least one worker (W) for carrying out at least one manual production step, wherein the manufacturing workplace (3) comprises a plurality of objects in a changeable configuration or movement state, wherein the system comprises a control unit (2) with a graphic or acoustic user interface (2.1) for displaying a workplace dialogue for the production step and a sensor arrangement for monitoring a production step carried out by the at least one worker (W), characterised in that the sensor arrangement comprises at least one camera (1) and the control unit (2) is designed to detect at least the completion of the production step, to register it, to automatically display the work instructions for the following production step and to independently recognise the tool used and to check whether it is the tool required by the system, depending on the image data detected by the camera (1).

2. Monitoring and operating system for a manufacturing workplace according to claim 1, characterised in that the camera (1) is designed as a depth camera.

3. Monitoring and operating system according to one of the preceding claims, characterised in that the manual production step comprises a plurality of sub-steps, which are each monitored individually, wherein the sub-steps of a production step comprise at least the removal of a component, the assembly of this component and / or the acquisition of production data, in particular identification data, of this component.

4. Monitoring and operating system according to one of the preceding claims, characterised by a database (2.2) which, by means of image processing, assigns recognisable features to specific configuration or movement states, in particular gestures of the worker (W).

5. Monitoring and operating system according to one of the preceding claims, characterised in that the control unit (2) has a teaching operating mode in which the control unit (2) analyses sensor data detected by the sensor arrangement in a specific configuration or movement state and uses it to learn how to recognise the configuration or movement state.

6. Monitoring and operating system according to one of the preceding claims, characterised in that the objects comprise one or more workpiece carriers and / or workpieces and that the sensor arrangement comprises means for identifying the one or more workpiece carriers and / or workpieces, as well as for detecting the position of the one or more workpiece carriers and / or workpieces at the manufacturing workplace (3).

7. Monitoring and operating system according to one of the preceding claims, characterised in that the manufacturing workplace (3) comprises at least one placement location for at least one container and the sensor arrangement comprises means for identifying the at least one container and / or for detecting of the worker (W) reaching into the container.

8. Monitoring and operating system according to one of the preceding claims, characterised in that the manufacturing workplace includes a conveyor system that can be controlled by the control unit (2) and the process includes triggering a control signal for the conveyor system to request or remove containers or workpieces.

9. Monitoring and operating system according to one of the preceding claims, characterised in that the control unit (2) is designed to detect a duration of a manufacturing activity of the worker (W) and / or a number of complete, repeated motion sequences of the worker from the image data, to compare it with predetermined extreme values and to output a control signal if it is exceeded.

10. Monitoring and operating system according to one of the preceding claims, characterised in that the control unit (2) is designed to identify the worker (W) and to evaluate the acquired image data in accordance with specifications that can be individualised for the worker.

11. Monitoring and operating system according to one of the preceding claims, characterised in that the control (2) is designed to assign the image data to known image patterns and, if the image data have a similarity with a known image pattern that lies within a predetermined tolerance range, to request workers to confirm an assignment to the known image pattern and to automatically carry out the assignment in subsequent operations.

12. Monitoring and operating system according to one of the preceding claims, characterised in that the control unit (2) is designed to recognise assembly or processing errors by image recognition from the image data and to automatically display in specific work instructions for the worker for remedying the defect if a certain assembly or processing error occurs repeatedly.

13. Monitoring and operating system according to one of the preceding claims, characterised in that the control unit (2) is designed to detect the worker reaching into an area defined as a virtual confirmation button from the image data and to trigger the process when the intervention is detected, wherein the detection range of the virtual confirmation button at the manufacturing workplace is equipped with a marking that can be detected by the camera and positioned by the worker, the control unit (2) is designed to detect the position of the marking and to define the detection range of the virtual confirmation button depending on the position of the detected marking.

14. Monitoring and operating system according to claim 12, characterised in that the marking comprises identification information (such as AR marker, QR code, etc.) that can be read from the image data and the control unit (2) is designed to determine the process that can be triggered by the virtual confirmation button depending on the identification information.

15. A method for manufacturing a product or a subproduct, which includes at least one manual manufacturing step carried out by a worker (W), characterised in that the manual manufacturing step is carried out at a manufacturing workplace (3) with a monitoring and operating system according to one of the preceding claims.