Methods and apparatus for functional testing of equipment, use of head-mounted displays, head-mounted displays, and methods and apparatus for visual display of 3D data.

Abstract

In a method for functional testing of equipment (1), it is proposed to create a real-world model (7,8) from the details of a virtual 3D model (39) of the equipment, and to detect its spatial position and posture during functional testing using a 3D measuring device (11). The identified posture and position can be used to make virtual objects (3,4) of the 3D model (39) follow the operations in the real-world model (7,8) in a visual impression (20) in virtual reality.

Description

Technical Field

[0001] The present invention relates to a method for functional inspection of equipment.

Background Art

[0002] Particularly in pharmaceutical equipment, it is actually known that, in many cases, a functional inspection is carried out using cardboard and / or wooden models before starting complex manufacturing.

[0003] The present invention further relates to an apparatus for functional inspection of equipment.

[0004] Regarding the method described above, cardboard and / or wooden models are actually known.

[0005] The present invention further relates to a head-mounted display and its use.

[0006] Head-mounted displays are actually known, for example, as VR (Virtual Reality) goggles, AR (Augmented Reality) and MR (Mixed Reality) goggles, which, for various purposes, are often used in the entertainment industry to create a spatial impression by visually presenting spatially allocated virtual data alone or in combination with a real scene to an individual observer. A head-mounted display is described, for example, in the German-language Wikipedia. According to it, a head-mounted display is characterized, for example, as a visual output device attached to the head. Such a device can be provided to display an image on a screen in front of the eyes or by projection onto the retina, for example, to complement (AR, MR) or replace (VR) the natural visual impression of an observer by an artificially created impression.

[0007] The present invention further relates to a method and a corresponding apparatus for visual display of 3D data.

[0008] In order to display 3D data in a way that can be directly experienced, it is actually known in the entertainment industry to use such methods and devices, particularly those using head-mounted displays as already mentioned above. [Overview of the project] [Problems that the invention aims to solve]

[0009] The fundamental problem underlying this invention is to simplify the functional testing of complex equipment. [Means for solving the problem]

[0010] To solve the above problems, the present invention envisions the features of claim 1. In particular, to solve the above problems, in the method described at the beginning, the present invention proposes representing the equipment as a virtual 3D model from virtual objects, preparing a real model for at least one virtual object, aligning the virtual object with the real model by 3D posture measurement over time, linking at least one virtual object to the virtual object and bringing it into a desired positional relationship with the virtual object, and allowing the user to change the link between the virtual object and at least one virtual object. This makes it possible for the present invention to enable a tactile experience of virtual modeling that enables real functional testing without requiring a complete real image of the equipment under test. As a result, it becomes unnecessary to construct the entire equipment as a real model, and functional testing can be greatly simplified.

[0011] Generally speaking, the aforementioned linking can refer to only a subset of the degrees of freedom of movement for each object or object, or it can force complete fixation. For example, a desired positional relationship may mean tracking only a portion of the movement (e.g., only the X and Y axes, without rotation). This also occurs with holograms and target shoulder rings (glove ports).

[0012] For example, a real-world model might be tilted relative to a glass panel surface, causing the corresponding virtual object to be pulled away from the (virtual) panel surface when linking is activated. Here, it might be possible to create boundary conditions that restrict the alignment of this object relative to the virtual object to only specific degrees of freedom and fix it to the panel's degrees of freedom, thereby ensuring that the virtual object, such as a shoulder ring or glove port, remains within the panel. This would avoid disrupting the visual impression.

[0013] For example, real-world objects that are too far away for a user to touch in a particular test can be omitted. In this case, the idea of ​​activating the linking makes it possible to connect the virtual world to the real world, for example, allowing the details of a 3D model to be experienced tactilely through appropriately positioned real-world models. The idea of ​​deactivating the linking makes it possible to swap virtual objects, and consequently, to reuse a very limited stock of real-world models, for example, in various locations in industrial equipment, especially when structural details are used in multiple ways.

[0014] The individual user who activates or deactivates the linking may be, for example, an individual observer of the (virtual) equipment, particularly a person performing functional inspection or testing of the equipment, or an assistant who maintains the 3D engine or software that generally implements the present invention. The 3D engine, i.e., the graphics engine, may be characterized, for example, as integrated or externally stored program code that is responsible for the calculation of the graphics interface in parallel with the actual program.

[0015] A virtual object may be characterized, for example, as a functional and / or structural part of a 3D model. Examples include static parts such as the shoulder ring or boundary wall of an isolator as a special (pharmaceutical) piece of equipment, or movable parts such as a transfer port or rapid transfer port ("quick change system," abbreviated as RTP, alpha-beta-port system) or airlock, or a functional station, such as a filling station, closing station, or material storage, or a manipulator. This list is not exhaustive; other examples may be advantageously used.

[0016] A virtual object can be characterized, for example, as a rigid body formed by measurement points that are fixedly positioned relative to each other.

[0017] For example, changing a link might involve activating (or starting) and / or deactivating (or ending) the link. This makes it possible to establish or break the spatial connection between virtual objects and virtual bodies in the virtual space. Since virtual bodies are linked to real-world models by 3D pose measurement, and their changes in pose and position are forcibly tracked in the virtual space, this link can thus achieve either - activation - linking the 3D model of the virtual object to reality, or - deactivation - separating the virtual object from reality.

[0018] In a favorable configuration, linking a virtual object to at least one other virtual object may involve enforcing a desired positional relationship between the virtual object and the virtual object's position and / or posture. This creates the impression that the virtual object is working in conjunction with a tactile real-world model. This can be used, for example, to test the feasibility of actual work processes in a 3D model. This enforcement can also be used as a simple means of transferring changes to the real-world model, such as ergonomic improvements, to the virtual object, especially when time constraints are present. This enforcement refers to all degrees of freedom of movement, or a subset of degrees of freedom of movement, and boundary conditions may be taken into consideration in particular.

[0019] Generally, the position of a virtual object or real-world model can be described by three coordinates, for example, a selected point, particularly its center of gravity, center, or other characteristic or special point. The posture of a virtual object or real-world model can be described, for example, by specifying the angle of alignment with respect to a rotation around the selected point referenced by its position, and / or by specifying the position of another point on the virtual object or real-world model that may be fixedly related to the selected point. A pose can be described, for example, by position and posture.

[0020] For example, it is conceivable to set the position and / or orientation of a virtual object to the position and / or orientation of a virtual object. This would enable tracking or linking without deviation.

[0021] Alternatively or additionally, enforcement may be triggered by performing the request. This would enable the exchange of virtual objects and / or real-world models during linking.

[0022] Alternatively or additionally, it can be envisaged that the forcing is carried out continuously, for example repeatedly over time, preferably automatically. Thereby, for example, it becomes possible to make a virtual object follow a virtual body over the entire movement interval.

[0023] In an advantageous configuration, it can be envisaged that in a setup process it is defined which virtual objects the virtual body can be linked to. Thereby, the set of real models can be extended.

[0024] In an advantageous configuration, it can be envisaged that, for example, in the above-mentioned setup process, it is defined how a virtual object can be linked to a virtual body. Thereby, an exact alignment of the virtual body with respect to the virtual object can be defined. For example, the posture of the markers constituting the virtual body can be fixed to a suitable virtual object. Thereby, a new real model can be incorporated later.

[0025] In an advantageous configuration, it can be envisaged that the linkability between a virtual body and at least two virtual objects is adjusted. Thereby, a complex arrangement from a plurality of virtual objects can be reproduced or simulated by selectively docking different virtual objects to reality, for example a real model. In this case, this docking results, for example, from the fact that the connection between the virtual body and the real model is reliably preset.

[0026] In an advantageous configuration, it can be envisaged that the link between a virtual body and a virtual object can be activated or started and / or deactivated or terminated independently of the link between another virtual body and another virtual object. It has been found that it is desirable to deactivate the link in order to avoid image processing artifacts when it is not actually necessary to move the real model.

[0027] In an advantageous configuration, it can be assumed that the number of virtual objects, particularly linkable objects, does not become smaller than the number of virtual bodies. Thereby, the numbers of both may be the same, the number of virtual objects may be larger than the number of virtual bodies, and particularly may be larger than three times. Thereby, it becomes possible to simulate a complex arrangement, for example, an operation process of a manufacturing line or an isolator, without the need to construct the entire facility. This saves the time, materials, and space required for preparation. Furthermore, changes can be implemented more flexibly, the transfer of assembled parts can be performed with less labor, and thus it is not restricted by location. This saves costs.

[0028] In an advantageous configuration, it can be assumed that at least two real models are prepared. Thereby, it becomes possible to simulate an operation process of associating two real models with each other, for example, opening a door.

[0029] In this case, it can be assumed that the corresponding virtual bodies are realized.

[0030] In an advantageous configuration, it can be assumed that at least two real models are brought into a preset spatial relationship with each other by the spatial relationship of at least two virtual objects in the virtual space.

[0031] In this case, it can be assumed that at least two virtual objects are linked to at least two virtual bodies belonging to at least two real models.

[0032] In an advantageous configuration, it can be assumed that at least two real models are forced to be movable relative to each other. Examples of forced guidance include, but are not limited to, a pivotable connection between a door and its frame, for example, an RTP or a rail guide of a vehicle.

[0033] In a favorable configuration, it may be assumed that at least two reality models are movable in a way that restricts their movement relative to each other. Such restrictions may arise, for example, when using isolator gloves, because the hand to which a reality model with markers may be attached can only be inserted into the shoulder ring to which the glove is attached, to the extent that the glove material is stretched to its maximum extent.

[0034] In a favorable configuration, it may be assumed that the user's mobility is limited by at least one reality model during use. This allows for easy testing of whether the operation is feasible within the simulation facility. One example is testing the range of motion of the shoulder ring, which keeps the individual user away from the shoulder ring.

[0035] In a favorable configuration, it may be conceivable to deactivate the linking for a favorably defined or undefined period of time. This would enable the alignment of the reality models, particularly in relation to another reality model to which the virtual object is already linked, thereby aligning the reality models with the virtual world, in particular the virtual 3D model, to which at least one virtual object belongs, and allowing the other reality models to continue to remain aligned with this virtual world.

[0036] In this case, it can be assumed that a deviation will be displayed between at least one virtual object and the virtual object when the linking is inactive. This can be used, for example, to move the real-world model to a desired position and thereby establish a desired relationship with the virtual object.

[0037] In advantageous configurations, the link between a virtual object and at least one other virtual object can be replaced with other links between virtual objects. This allows for the reuse of the real-world model to test other virtual objects in the 3D model, eliminating the need to construct a complete facility. This saves space and time for creating the real-world model and enables functional testing between users in remote or geographically separated locations. Costs for model manufacturing and assembly are also reduced. Furthermore, there are technical advantages; for example, in the virtual model, cross-sectional views can be added or holograms can be faded in to enhance understanding.

[0038] In a favorable configuration, if one link is replaced by another, it can be assumed that the 3D model is isometrically transformed until the virtual object and the other virtual object match at least within an acceptable range. This allows the user to move within the virtual world without changing their position in the real world. This makes it easy to use already constructed real-world structures for further testing without reconstruction. In this case, it becomes easy to adjust the existing real-world model to the position and / or posture of the new virtual object, as described above.

[0039] Such isometric transformations may include, for example, rotation and / or displacement. Thus, this transformation simply corresponds to any change in position in the real world.

[0040] Preferably, only isometric transformations that maintain orientation, i.e., those that do not mirror the image, are permitted. This prevents changes for which there is no corresponding real-world equivalent.

[0041] In a favorable configuration, the orientation of the real-world model may be advantageously changed manually or automatically until the virtual object in question matches at least one other virtual object. This makes it possible to align the real-world model so that the tactile impression from interacting with the real-world model matches the visual impression when observing the virtual object.

[0042] For example, a shoulder ring can be matched to a 3D model by first activating a link with the target virtual object. Then, another shoulder ring, or other parts such as a door or an operable function unit, can be modified as a real-world model so that it matches the corresponding virtual object, and these positioned and / or aligned real-world models can be embedded into the virtual world.

[0043] In this case, it can be assumed that the link between the virtual object and other virtual objects is then activated. This allows the virtual object to subsequently track the movement of the real-world model. As a result, the individual observer of the virtual world receives tactile or sensory information in addition to visual sensory information, giving them the sensation of actually moving or manipulating the virtual object.

[0044] In a favorable configuration, multiple virtual objects may be linked to each virtual object of a 3D model, and each link may be modified independently of the others, particularly activated and / or deactivated. This would allow for the independent configuration of various reality models, such as two shoulder rings, and / or the selection of individual reality models as movable parts of equipment that need to follow virtual objects, while other reality models become available or remain available as reference points in the real world to which the virtual world can dock.

[0045] In a favorable configuration, it can be assumed that the coordinate updates of at least one virtual object will be output. This can be used, for example, to process design data for 3D models. This allows for simple adjustments to the equipment and necessary changes from an ergonomic and / or process economic standpoint, without the need to create a new, complete model in the real world.

[0046] In an advantageous configuration, the equipment may be for the pharmaceutical sector, preferably for filling pharmaceuticals into containers, and / or for use with a protective chamber, preferably an isolator. Here, regulatory requirements and / or ergonomic boundary conditions in the workflow can be easily tested.

[0047] Alternatively or additionally, the above problem can be solved by a method for functional testing of equipment, which represents the equipment as a virtual 3D model from virtual objects, prepares a real-world model for at least one virtual object, aligns the virtual object to the real-world model by 3D orientation measurement over time, links at least one virtual object to the virtual object and brings it into a desired positional relationship with the virtual object, in which case a real-world model corresponding to at least one virtual object is created, identifiable features, particularly markers, are attached for 3D orientation measurement, and the correspondence between the identified features and at least one virtual object is saved. This makes it possible to easily create and incorporate details of equipment that are relevant to testing and for which physical interaction is desired. Identifiable features can be easily used to create virtual objects to be linked to objects.

[0048] This embodiment can be advantageously combined with the embodiments described above. For example, by activating linking, a reality model prepared for use with markers can be easily made available in the method according to the present invention. The markers can be implemented, for example, advantageously as two-dimensional or three-dimensional markers.

[0049] Preferably, the real-world model is created using an additive method, particularly from CAD data or other virtual object data. This allows the 3D model to be realized in as much detail as possible, thereby enabling a tactile experience of even the finest details. Using the real-world model and directly markering it is also an advantageous method for obtaining an even better tactile experience.

[0050] Alternatively, virtual objects can be derived from 3D scans of real-world models. This allows for the direct use of a manufacturer's prototype or sample even when CAD data is unavailable, and / or avoids the hassle of 3D printing complex objects.

[0051] In a favorable configuration, it can be assumed that identifiable features are formed at predetermined positions in the real-world model. This makes it possible to quickly and easily capture or create a correspondence between a virtual object that can be represented by a feature and a virtual object that can record the position of that feature.

[0052] In a favorable configuration, it may be conceivable to measure the position of at least one feature formed on the real-world model. This can be done, for example, using a 3D camera. This measurement makes it possible to attach arbitrary markers, which can simplify the preparation of the real-world model for use.

[0053] In advantageous configurations, it may be assumed that the operator wears gloves and / or hand tracking devices (smart gloves, meta gloves, motion capture gloves, finger tracking devices). Alternatively, direct hand tracking is also possible. Hand detection is beneficial for realistically replicating the actions performed in the virtual world. Isolators or shoulder gloves can also be used, thereby creating a realistic imitation of the resulting physical limitations, for example.

[0054] In this case, it can be assumed that the 3D posture, particularly the position and posture, of one or more fingers and / or one hand and / or one arm may be repeatedly determined. This allows for easy integration of the hands and / or arms that perform operations and / or require collision testing into the virtual world.

[0055] For example, this can be achieved by detection of gloves and / or hand tracking devices, as previously mentioned. This makes it possible to virtually represent realistic operational actions.

[0056] In a favorable configuration, a 3D model represents the shoulder ring, and the actual shoulder ring is adjusted to the shoulder ring's position, particularly in a preceding setup process, and / or it may be assumed that the operator puts their arm through the shoulder ring, preferably with an operating glove attached to the shoulder ring.

[0057] In a favorable configuration, the observer's shooting pose can be assumed to be defined relative to the shoulder ring. This allows the observer's position and posture to be used as a reference for displaying the virtual object.

[0058] In a favorable configuration, at least one virtual object may be assumed to be a transfer port or airlock door. Other details of equipment that need to be operated during use, such as air samplers, agar plates, sampling devices, filling stations, pump bodies, hoses, and generally semi-fixed (e.g., whose movement is restricted by joints or guides) or freely movable equipment components may also be used.

[0059] Generally, real-world models can be classified into those that are movable relative to reference points, such as the boundaries of the facility and / or access points, particularly shoulder rings, and those that are not.

[0060] In an advantageous configuration, it can be assumed that a 3D model has another virtual object, another real-world model is prepared for that virtual object, and that the real-world model is movably positioned relative to another real-world model. This allows individual real-world models to be used as reference points connecting the virtual world and other real-world models, enabling highly accurate positional manipulation within a virtual world aligned with reality.

[0061] In a favorable configuration, the reality model may be assumed to be at least partially immobile and / or at least partially movable with respect to the facility boundaries. This allows the boundaries to be used as reference points or reference surfaces for establishing a correspondence between the virtual world and the real world.

[0062] Examples of at least partially movable components of the equipment that can be advantageously used as a real-world model include the shoulder ring and / or door frame of the transfer port, and / or the door, particularly the transfer port door, or the airlock door, and / or the interlock, and / or the petri dish, air sampler, at least one agar plate, sampling, pump body, and at least one hose, but this list is not exhaustive.

[0063] To address the above challenges, a method has been proposed for functional testing of equipment, particularly as described above, which includes the steps of: preparing CAD data of the equipment; creating at least one real-world model of at least a portion of the CAD data; setting at least one real-world model in a 3D measuring device; and processing at least the 3D measurement data from the 3D measuring device to display a virtual 3D model created from the CAD data. This makes it possible to perform functional testing of complex equipment based on a tactile impression of the real-world model with minimal material costs.

[0064] In advantageous configurations, the field of view of the head-mounted display may be determined, preferably using a 3D measuring device. This makes it possible to embed the individual observer and / or operator into a virtual scene of a virtual object in a CAD model.

[0065] In a favorable configuration, the display of the 3D model can be envisioned as being related to the field of view of the head-mounted display. This allows the 3D model to be observed realistically from the observer's position.

[0066] In a favorable configuration, it can be assumed that changes to at least one real-world model are automatically tracked to the 3D model. This allows changes in the real world to be easily tracked in the virtual world in which the 3D model is defined. This gives the individual observer the impression that they can change the virtual object through changes in the corresponding real-world model—which can be experienced tactilely—for example, linked virtual objects. This makes it possible to perform functional testing of complex industrial equipment, such as pharmaceutical equipment, with minimal use of materials, time, and space.

[0067] In this case, or generally speaking, it can be assumed that modified design data is generated and output from the modified 3D model or modified virtual object. This makes it possible to define the design changes discovered during functional testing.

[0068] In an advantageous configuration, the 3D measuring device may be transported to a fixed measuring structure before setup. This allows the device according to the present invention to be easily transported to a remote location, for example, for on-site functional testing.

[0069] The problem mentioned at the beginning can be solved, either alternatively or additionally, by an apparatus for functional testing of equipment, wherein the equipment exists as a virtual 3D model, and comprises a 3D measuring device, at least one real-world model of the virtual object of the 3D model, a device for automatically incorporating virtual objects detected using the 3D measuring device into the 3D model relative to the real-world model, a device for automatically making the virtual object track the virtual object together with the virtual object, and a device for visually displaying the 3D model, particularly a 3D engine. This provides a means for testing virtually representable equipment that can be experienced through touch.

[0070] In a favorable configuration, it may be envisioned that means are formed for activating and / or deactivating the link between a virtual object and at least one other virtual object. This would allow the operator to easily decide how to connect the virtual world of the 3D model with reality.

[0071] In a favorable configuration, a head-mounted display may be provided to generate a field of view for the 3D model. This would allow for a more natural observer position.

[0072] In an advantageous configuration, a device for generating a field of view of the 3D model may be envisioned to supply the measurements of a 3D measuring device, particularly to the head-mounted display mentioned earlier. This would allow the 3D model to be observed from the position of a human observer.

[0073] In advantageous configurations, it is conceivable that a device could be formed to convert the 3D model to an isometric size relative to the field of view. This would enable the conversion of the 3D model or the virtual position change of the equipment operator.

[0074] The challenges mentioned at the beginning can be solved, either alternatively or additionally, by the use of head-mounted displays, particularly VR, XR, and / or AR goggles, and, advantageously, fixed and / or independently operating 3D measuring devices for creating virtual views of 3D models of equipment within the head-mounted display, in which case individual virtual objects correspond to real-world models detected by the 3D measuring device. This enables a means for tactilely controllable virtual functional testing of the equipment represented as a 3D model.

[0075] This can be used particularly advantageously for functional testing of pharmaceutical equipment, particularly advantageously in methods according to the present invention, especially the methods described above and / or below, and / or in the claims described below, and / or in apparatus according to the present invention, especially the apparatus described above and / or below, and / or in the claims described below. Herein, the present invention can save considerable cost, time, and space because pharmaceutical equipment, particularly control rooms, or control rooms such as RABS (Access Restriction Barrier Systems) or isolators, often have a large spatial extent. This makes conventional structures from cardboard and / or wood cumbersome.

[0076] Preferably, in the present invention, a head-mounted display, such as VR, AR, and / or XR goggles, is used, which includes at least one marker, particularly more than two, for advantageously identifying the shooting pose externally. In many cases, three markers are sufficient to clearly determine the position and posture of the real-world model. However, especially for more complex real-world models, it is beneficial to attach more than three markers. The attachment of markers allows for separate detection and alignment of the shooting pose even with respect to objects outside the field of view. This gives the individual observer a realistic impression of the equipment from their own perspective.

[0077] In this case, the markers may be formed in an active manner. This allows for individual detection of each head-mounted display. This enables easy device swapping without requiring new learning and / or system recalibration. It also allows for the uniform setting of a reference center for multiple encodings. In this case, this uniform center can be used to map virtual objects into the field of view and ultimately embed them without learning for each marker.

[0078] The markers may be formed passively, either as an alternative or additional method. This reduces the resources consumed during operation, thus extending the uptime of the head-mounted display. In this case as well, the markers can be attached to various positions on various head-mounted displays, thereby enabling individual detection. Standardized mounting kits for connecting the goggles and markers allow for embedding with no learning required or minimal effort. Another advantage of passive markers is their light weight, which positively impacts, for example, the comfort of wearing the head-mounted display.

[0079] The problem mentioned at the beginning is, alternatively or additionally, solved by the present invention in a method for the visual display of 3D data, wherein the field of view of a head-mounted display (a display attached to the head), particularly VR and / or XR and / or AR goggles, is intrinsically determined, and information linked to the field of view is displayed within the head-mounted display, in which case the shooting pose of the head-mounted display, which pre-sets the field of view, is advantageously identified repeatedly over time and compensated for and adjusted with the field of view. This makes it possible to spatially accurately embed the field of view into the virtual world, depending on the available computing power of the head-mounted display. The present invention has the advantage that the spatial positional relationship of the observer can be detected even for virtual objects that are not currently in the observer's field of view, as defined by the shooting pose of the head-mounted display.

[0080] Intrinsic decisions can be characterized, for example, by their interaction with the relevant sensors and / or by their alignment with the field of view and / or by the ability to calculate the field of view using tools on the head-mounted display. For example, intrinsic decisions can also be recognized by the fact that they cannot be performed when the head-mounted display is inactive.

[0081] In this case, the field of view of the head-mounted display can be defined, for example, as the observer's field of view corresponding to the current shooting pose of the head-mounted display when the head-mounted display is in use. In this case, the shooting pose can be referred to, for example, as the forward position and posture of the head-mounted display.

[0082] This method may be formed or constituted, for example, as part of a method for functional testing of equipment, as described herein, particularly in the method described herein, and / or as described in the claims below.

[0083] In advantageous configurations, the shooting pose can be identified using a 3D measuring device that is independently formed and / or fixed to the head-mounted display. This would allow the head-mounted display to be detected from all directions at all times, and the field of view to be seamlessly embedded.

[0084] A 3D measuring device may include, for example, at least one or more cameras. Generally speaking, using multiple cameras improves the accuracy of measurements on the one hand, and reduces the likelihood that details will be obscured by other details on the other hand.

[0085] 3D measuring devices are known for spatially detecting the position or posture of real-world models. One method involves creating 2D images from a real-world model in various shooting poses, identifying each model within these images using, for example, attached markers, and then solving a system of equations that describes these images as being of a common real-world model, where the shape, such as the position of individual markers, is treated as an unknown variable, and the image position is treated as an input value. An alternative is to use, for example, structured light, so that its pattern on the real-world model allows for the estimation of the model's posture and position. Methods for measuring signal propagation time are also known.

[0086] In a favorable configuration, the shooting pose may be determined using a measuring device that is linked to and / or independent of the field of view determination. This simplifies the mechanical structure of the apparatus according to the present invention.

[0087] In a favorable configuration, intrinsic determination may be envisioned to be performed using at least one interconnected sensor, particularly a camera and / or motion and / or acceleration and / or posture sensors. This would make known systems available for identifying the field of view and its changes associated with head movement.

[0088] In advantageous configurations, it may be conceivable to measure the shooting pose using active markers on a head-mounted display. Active markers have the advantages of excellent differential indication and easy modification of identification information.

[0089] In advantageous configurations, it may be conceivable to measure the shooting pose using passive markers on a head-mounted display. Passive markers conserve energy during operation, thereby extending the uptime during which the device remains usable.

[0090] In advantageous configurations, it may be conceivable to measure the shooting pose using a fixed measuring device, particularly a fixed camera.

[0091] In advantageous configurations, it is conceivable to measure the intrinsic determination of the shooting pose using a camera linked to a head-mounted display.

[0092] The problem described at the beginning is, according to the present invention, solved alternatively or additionally by a method for visual display of equipment and / or as part of the method described above and / or as described in the subsequent claims, which temporally repeats, intrinsically determines the field of view of a head-mounted display (a display attached to the head), particularly VR and / or XR and / or AR goggles, calculates the airflow, and advantageously displays it visually in the head-mounted display as linked 3D data. In this case, the advantage is that the effect of the work process on the airflow can be directly grasped and / or controlled.

[0093] In controlled environments, airflow is known to be used to prevent the transfer of contaminants to areas requiring particular protection. For example, airflow can also be affected by mobile functional units and / or individual users, and this invention makes it possible to verify this.

[0094] In an advantageous configuration, the head-mounted display may be connected to a fixed processing unit, which is advantageous for transmitting measurement data of the shooting pose and / or image data for the head-mounted display. This allows the computational routines to be offloaded to a higher-capacity fixed unit. In this case, data transmission can be performed, for example, wirelessly or via a wired connection.

[0095] In advantageous configurations, the 3D data may include a 3D model of the equipment and / or AR metadata, particularly regarding the equipment components already mentioned. The use of the 3D model makes it possible to visually represent the equipment in a virtual space as if it were real. The use of AR metadata allows for the additional fading in or display of data such as warnings, notifications, or work instructions, in addition to the image content. This makes it possible, for example, to easily switch to the observer's language or to change the fading in according to the operating status of the equipment. It is also possible to display airflow velocity data as 3D data, particularly as streamlines.

[0096] In advantageous configurations, a head-mounted display, particularly a head-mounted display according to the present invention, can be used to generate a superposition of the real field of view and a virtual representation of 3D data, as described above and / or in the subsequent claims. This enables the application of MR or AR.

[0097] Alternatively or additionally, a head-mounted display, particularly a head-mounted display according to the present invention, can be used to isolate the real environment, for example, as described above and / or in the subsequent claims. This enables VR applications.

[0098] Additionally, in any of the methods described, it may be conceivable to motorize one of the reality models, for example, one of the reality models already mentioned. This would allow for easier and / or more accurate setting of the position and / or posture of the virtual object.

[0099] Alternatively or additionally, in any of the above methods, one reality model, for example, one of the reality models already described, may be advantageously motor-driven and / or automatically adjusted until the deviation of the virtual object's position and / or posture from the corresponding virtual object is advantageously within an acceptable range. This would enable the automatic incorporation of reality models into this method.

[0100] These two configurations can be advantageously combined or used individually, for example, in the shoulder ring described in more detail below.

[0101] Alternatively or additionally, it may be assumed that virtual rays are generated in any of the described methods, and in particular, that it is automatically checked whether the virtual rays are blocked. This would allow the functional test to be performed in a more realistic manner.

[0102] To solve the problems mentioned at the beginning, the present invention further envisions a device for the visual display of 3D data, comprising a head-mounted display provided for determining the linked field of view, a device for detecting the shooting pose of the head-mounted display, and a device for compensating and adjusting the shooting pose with the field of view. This makes it possible to achieve spatially accurate embedding of virtual objects into the virtual world's field of view while keeping the computational power requirements of the head-mounted display low. This could be used, for example, to fade in information and messages to appropriate locations.

[0103] In this case, the device may be formed as part of an apparatus according to the present invention for functional testing of equipment, as described above and / or in the subsequent claims.

[0104] In advantageous configurations, a 3D engine may be formed to visually display 3D data within the field of view. This makes known algorithms for controlling the visual display available.

[0105] In advantageous configurations, a fixed 3D measuring device may be formed for detecting the shooting pose. The fixed 3D measuring device can be configured to cover a wider space, for example, by spatially distributing the corresponding cameras.

[0106] In advantageous configurations, for example, a device may be formed to generate a virtual space for virtual objects detected using the 3D measuring device already described. This makes it possible to detect real-world models and their changes. This can be used, for example, to connect the virtual world and the real world. An example of a 3D measuring device is the Prime camera of OptiTrack (NaturalPoint, Inc., PO Box 2317, Corvallis, OR 97339). X 13 and / or Prime X This combines 13W with OptiTrack's motion capture software, Motive.

[0107] In advantageous configurations, an alternative or additional device for generating a virtual space for visually displaying 3D data may be formed. This makes it possible to prepare a scene for generating a virtual visual impression.

[0108] In a favorable configuration, alternatively or additionally, two virtual spaces, preferably a device for establishing a correspondence between the aforementioned virtual spaces, in particular a device for embedding and virtually displaying virtual objects within the virtual spaces, may be formed. This would provide an easily manageable means for modifying virtual objects by manipulating corresponding real-world models.

[0109] In advantageous configurations, a device may be formed to embed the field of view of a head-mounted display into a virtual space and display it virtually. This makes it possible to provide a natural and faithful visual representation of the virtual world through the observer's eyes, as if the observer were actually visually perceiving the virtual world.

[0110] In advantageous configurations, devices for calculating airflow, particularly for visually displaying airflow, may be formed. This makes it possible to visualize the airflow, especially as streamlines.

[0111] In a favorable configuration, the means for carrying out the method according to the present invention may be formed particularly as described above and / or as described in the subsequent claims, thereby demonstrating a method for realizing the described method.

[0112] In an advantageous configuration of the described apparatus, it may be conceivable that an apparatus be formed that includes a device for motor-driven adjustment of at least one real-world model. This would enable precise and / or automatic and / or remote adjustment.

[0113] Alternatively or additionally, in an advantageous configuration of the described apparatus, it may be envisioned that the apparatus be equipped with a device for advantageously automatically determining the positional and / or attitude deviation of a virtual object from a corresponding virtual object. This would enable computer-assisted support for the capture.

[0114] Alternatively or additionally, in an advantageous configuration of the described apparatus, it may be assumed that the apparatus is equipped with a device for motor-driven adjustment of at least one real-world model until the position and / or posture of the virtual object from the corresponding virtual object is favorably automatically detected to be within an acceptable range. This enables fully automatic or semi-automatic capture of the real-world model.

[0115] Alternatively or additionally, in a favorable configuration of the described apparatus, it may be envisioned that the apparatus be equipped with a device for virtual ray collision inspection. This may enable the simulation of photointerrupters, such as optical grids. For example, the corresponding photointerrupter module can be saved as a virtual object in a 3D model, and rays are automatically generated and monitored based on the orientation and position of the photointerrupter module.

[0116] The present invention will now be described in more detail with reference to examples, but the present invention is not limited to these examples. Further examples can be obtained by combining the features of individual or multiple claims with each other and / or with the features of individual or multiple examples. [Brief explanation of the drawing]

[0117] [Figure 1] This is a highly schematic diagram of a device for functional testing of equipment, equipped with a head-mounted display and a device for visually displaying 3D data. [Figure 2] This figure shows a more realistic, individualized representation of the actual device shown in Figure 1, along with two adjustable shoulder rings and a 3D measuring device. [Figure 3] This is another diagram showing the arrangement in Figure 2, but more realistically, from a side view of the glove. [Figure 4] This is another diagram that shows a more realistic, individual representation of the device shown in Figure 1, along with the movable door. [Figure 5] This figure shows the virtual objects of the 3D model of the device shown in Figure 1 adjusted to achieve the desired positional relationships and establish associations. [Figure 6] This figure shows the real-world model of the device shown in Figure 1 modified to realize the desired positional relationship with the corresponding virtual object and establish the linkage. [Figure 7] This figure shows a device for visually displaying 3D data. [Figure 8]This diagram schematically shows 3D models in various orientations for virtual position changes of an individual observer. [Figure 9] This figure shows a detailed plan view of a 3D model of pharmaceutical equipment. [Figure 10] This diagram schematically illustrates the real space and two virtual spaces for the purpose of explaining the present invention. [Modes for carrying out the invention]

[0118] Figure 1 shows a device for functional testing of equipment, represented as a single unit (1).

[0119] The virtual 3D model 2 of equipment 1 is composed of virtual objects 3 and 4 and is prepared in the first virtual space 5.

[0120] In this case, virtual 3D model 2 is derived from CAD data of industrial equipment, which is not shown in detail in the diagram.

[0121] This equipment is shown only in a very schematic manner using virtual objects 3 and 4 to illustrate the operating principle of the method according to the present invention.

[0122] In fact, this equipment includes more components. A preferred application of the present invention is that the equipment may be designated within a protective chamber or isolator to perform a specific process, for example, transferring / filling pharmaceuticals, or assembling dispensing devices for pharmaceutical formulations.

[0123] The first virtual space 5 contains numerous virtual objects 3 and 4. There may also be another virtual object 63 present.

[0124] Some of these virtual objects 3 and 4 have reality models 7 and 8 that correspond to the real space 6 set up. Another virtual object 63 does not have a reality model set up.

[0125] These reality models 7 and 8 are fixed to special stands 9 so that they remain in desired positions within the real space 6.

[0126] These stands 9 are not part of the virtual 3D model 2, because in this model, virtual objects 3 and 4 are incorporated into other design details, such as glass walls, boundary walls, or tables. However, these glass walls, boundary walls, and tables do not exist in real space 6.

[0127] Furthermore, reality models 7 and 8 are equipped with additional markers 10, which do not have corresponding markers for virtual objects 3 and 4.

[0128] These markers 10 are designated for position and location detection by a 3D measuring device 11 set up in real space 6. The 3D measuring device 11 detects these markers 10 and forms their positions in a second virtual space 12. Therefore, in this second virtual space 12, there are no virtual objects 3 and 4, only the positions of the markers 10, which are grouped into virtual objects 13 and 14 depending on which of the real models 7 and 8 they are attached to.

[0129] Between the first virtual space 5 and the second virtual space 12, a correspondence relationship 15 is established that connects spatial points in the first virtual space 5 to corresponding spatial points in the second virtual space 12, and vice versa.

[0130] Furthermore, in the real space 6, for example, a head-mounted display 16 is positioned on the head of an individual user, which is not shown in detail in the diagram.

[0131] This head-mounted display 16, for example, VR goggles, also has markers 17 for detecting its posture and position in the real space 6 and displaying them in the second virtual space 12, although these are not necessarily represented in the virtual 3D model 2.

[0132] In other words, in the second virtual space 12, there exists a virtual object 18 that represents the head-mounted display 16 using markers 17.

[0133] The head-mounted display 16 generates a field of view 19 that provides the individual user with a three-dimensional visual impression, as is well known. This visual impression 20 is calculated based on the aforementioned correspondence 15 from the measured posture and position of the head-mounted display 16, i.e., the shooting pose, and the posture and position of the virtual objects 3 and 4.

[0134] In this case, virtual objects 3 and 4 are used for the visual impression 20, and as a result, for example, stand 9 and markers 10 and 17 are not displayed.

[0135] In this case, the field of view 19 of the head-mounted display 16 is defined by the position in which the head-mounted display 16 is attached to the user's head and its natural field of view.

[0136] In order to generate a visual impression 20, in Figure 1, field of view data 21, position and posture data 22 from the second virtual space 12, and object data 23 from the first virtual space 5 are processed together by methods known to the public.

[0137] To determine the field of view 19, the head-mounted display 16 may be equipped with its own sensors 46, which are not described in detail in the figure.

[0138] For example, the head-mounted display 16 may be equipped with multiple cameras that detect the individual user's field of view and calculate position, posture, and / or changes in these values ​​based on spatial features such as room corners and edges. For this purpose, it is technically known that the head-mounted display 16 can be equipped with appropriate computing power, thereby eliminating the need for external computing power.

[0139] Since the real-world models 7 and 8 are installed in the real-world space 6, even if the head-mounted display 16 is a VR goggle type and therefore cannot see the real-world models 7 and 8, the individual user in the real-world space 6 can touch and verify these real-world models 7 and 8.

[0140] To achieve this correspondence between tactile and visual impressions 20, the reality models 7 and 8 are initially aligned with the virtual objects 3 and 4 assigned to them.

[0141] Thus, the method according to the present invention makes it possible to activate the link 24 between virtual objects 3 and 4 on the one hand and the corresponding virtual objects 13 and 14 on the other hand. As a result of this link 24, virtual objects 3 and 4 (in this example, virtual object 4) will be linked with virtual objects 13 and 14 (in this example, virtual object 14) based on the correspondence relationship 15.

[0142] Accordingly, when the reality model 8 changes due to the action 25, this action 25 is reflected in the second virtual space 12 by the 3D measuring device 11.

[0143] As already mentioned, direct tracking can also be modified depending on the situation, for example, to follow only specific axes and "fix" certain axes in order to satisfy boundary conditions.

[0144] Thus, through the linking 24, the virtual object 4 will move similarly within the first virtual space 5 according to the correspondence 15.

[0145] As a result, an apparent action 26 occurs in the visual impression 20.

[0146] Therefore, the individual user who generated action 25 through the operation of the real-world model 8 gets the impression that the virtual object 4 is performing the apparent action 26 based on that operation.

[0147] If the user or another person deactivates the link 24, manipulating the real-world model 8 will not change the visual impression 20. This can be determined individually for each object and does not necessarily apply to all objects.

[0148] Furthermore, a head-mounted tracking device 27 is placed in the real space 6. The aforementioned individual user wears this head-mounted tracking device 27 like a glove, thereby performing the aforementioned operations on the real-world models 7 and 8. The markers 28 of the head-mounted tracking device 27 are also displayed as virtual objects 29 in the second virtual space 12 by the 3D measuring device 11, resulting in the appearance of a hand replica 30 in the visual impression 20.

[0149] The head-mounted display 16 can be connected wirelessly or wired, and advantageously to a fixed processing unit, to transmit measurement data of the shooting pose and / or image data of the visual impression 20.

[0150] To generate the visual impression 20, a 3D engine, not shown in detail in the figure, is used. This forms a device for generating a field of view for the 3D model 39. In the first virtual space 6, a device is operated for isolength transformation of the 3D model 39 with respect to the field of view 19, as will be described in more detail below with reference to Figure 8.

[0151] Figure 2 shows the real-world model 7 from Figure 1 viewed from the front, making it appear somewhat more realistic, and Figure 3 shows this arrangement viewed from the side.

[0152] The 3D measuring device 11 is only partially shown in both figures.

[0153] In reality, in the actual space 6, typically three or more cameras 31 are arranged on a special support structure 32.

[0154] The exact position and orientation of camera 31 are known.

[0155] To facilitate on-site assembly, it is conceivable that the assembled support structure 32 may be stored in a container or packaged as a whole and transported to the site of use. The present invention has the advantage that there is no need to transport the model structure of the equipment. Rather, the assembled support structure 32 can be used as a mobile measuring structure.

[0156] The real-world model 7 has two shoulder rings 33, 34 known as isolators for attaching isolator gloves 49. In this equipment, the shoulder rings 33, 34 are often formed of glass walls. Thereafter, the positions of the shoulder rings 33, 34 indicate the positions of the equipment boundaries, such as a protective room or a control room.

[0157] These shoulder rings 33 and 34 can be positioned horizontally and vertically on the stand 9.

[0158] The hand tracking device 27 is connected to either of the shoulder rings 33 or 34 to form an isolator glove 49.

[0159] Each time, the other shoulder rings 33 and 34 may also have a hand tracking device 27 corresponding to the user's other hand. When in use, the user stands in front of the shoulder rings 33 and 34, inserts both arms into the respective shoulder rings 33 and 34, and thereby operates the respective hand tracking devices 27 with both hands.

[0160] In further embodiments, instead of the hand tracking device 27, an optical recognition algorithm is used to detect the user's individual hand posture, position, or shape.

[0161] The shape of the hand can also be detected by special sensors placed on the fingers 35, but this is not explained in detail here.

[0162] For example, a hand tracking device in the form of METAGLOVES "Quantum" from Manus (Floor 9, Kennedyplein 200, NL-5611 ZT Eindhoven) is known and usable.

[0163] Figure 4 shows a more realistic representation of the reality model 8 derived from Figure 1.

[0164] The real-world model 8 can be seen to consist of a movable part 36 and a fixed part 37.

[0165] This figure illustrates a door 47 that can be used, for example, as a transfer port 48 or a rapid transfer port, or in an airlock, as part of the wall of an isolator.

[0166] Details of this door 47 do not need to be discussed in this description of the present invention; the only important point is that this transfer port 48 has a movable part 36 and a fixed part 37.

[0167] This movable part 36 can be grasped and opened by the user to perform the operation 25 described in Figure 1.

[0168] As a result, in the visual impression 20, a change in the virtual object 4, which consists of the corresponding multiple parts, is displayed, and this corresponds to the opening of the door 47.

[0169] This allows the individual user to, for example, verify whether they can reach and operate the door 47 inside the isolator using the shoulder rings 33, 34.

[0170] Figure 5 shows the setup of the linking 24 described above, in an example using shoulder rings 33 and 34.

[0171] This setup is performed to ensure that the shoulder rings 33 and 34 are precisely matched to their virtual counterparts before the individual user begins their planned test.

[0172] In the front row, the real-world model 7 is shown, and behind it, the corresponding virtual object 13 is shown with a dotted line, while the corresponding virtual object 3 is shown with a solid line.

[0173] This diagram has been chosen for simplification. In reality, the real-world model 7, virtual object 3, and virtual object 13 exist in different spaces 6, 12, and 5.

[0174] The projection line 38 between the real-world model 7 and the virtual object 13 symbolizes the 3D measurement performed by the 3D measuring device 11.

[0175] When the request is executed, virtual object 3 is moved to the position corresponding to the position of virtual object 13 according to the correspondence relationship 15.

[0176] To support this process, it may be assumed that both virtual object 13 and virtual object 3 are reproduced within the visual impression 20.

[0177] For the right shoulder ring 34, this process has already been completed in Figure 5.

[0178] As a result, by performing the above request, a desired positional relationship is enforced between the position and posture of the virtual object 3 relative to the virtual object 13.

[0179] In Figure 6, the real-world model 7 is displayed on the front side, and the corresponding virtual object 3 is displayed on the back side.

[0180] The projection line 38 once again demonstrates the effectiveness of the 3D measuring device 11.

[0181] To simplify the display, virtual object 13 is not shown.

[0182] In the visual impression 20, the individual user recognizes that virtual object 3 and the virtual object 13, which is also displayed, do not match.

[0183] The user or assistant can then change the stand 9 so that the virtual object 13 is positioned above the virtual object 3. After that, the linking 24 can be activated. This procedure has the effect that, compared to the procedure shown in Figure 5, activating the linking 24 does not change the virtual object 3, and in particular does not change the other virtual objects 4.

[0184] Since the shoulder rings 33 and 34 remain fixed, there is no need to activate the link 24, and it can be kept permanently inactive.

[0185] By forming individual links 24 between virtual object 14 and virtual object 4, it becomes possible to persistently deactivate individual links 24 for individual virtual objects 13 and 14. This makes it possible to perform such adjustments independently of other virtual objects 3 and 4.

[0186] In this case, the reality model 7 can be motor-driven to adjust, for example, the automatically detected deviation 64 of the position and / or posture between the virtual objects 13, 14 (not shown, see Figure 5) and the corresponding virtual object 3 until it falls within an acceptable range.

[0187] To this end, the illustrated apparatus includes a device for motor-driven adjustment of at least one reality model 7,8; ​​a device for favorably and automatically determining the position and / or attitude deviation 64 between the virtual objects 13,14 and their corresponding virtual objects 3,4; and a device for motor-driven adjustment of at least one reality model 7,8 until the automatically and / or manually detected position and / or attitude deviation 64 between the virtual objects 13,14 and their corresponding virtual objects 3,4 falls within an acceptable range.

[0188] Figure 8 shows a schematic diagram of a 3D model 39 of industrial equipment, such as pharmaceutical equipment.

[0189] The 3D model here exemplifies a wall section 40 with three access ports 41, 42, and 43 arranged within it.

[0190] In this case, each of these access ports 41, 42, and 43 may include, for example, a virtual corresponding to a pair of shoulder rings 33 and 34.

[0191] The left-hand diagram of Figure 8 shows the positions used to match the real-world model 7 of the access port with its virtual counterpart, or to bring it to a desired positional relationship, using the method shown in Figure 6 or Figure 5.

[0192] This state is used to test and verify whether the equipment can access the 3D model 39 through this access port 41.

[0193] Then, when testing other access ports 42, 43, the 3D model 39 can be isoscaled, in particular with respect to the virtual object 13, so that another access port 42 can be matched with the real model 7 or its virtual object 13. This changes the position of the 3D model 39 in the first virtual space 6, or the correspondence 15 between the first virtual space 6 and the second virtual space 12. Then, the equipment can be tested through this access port 42. This situation is reflected in the central image portion of Figure 8.

[0194] The image on the right displays another reality model 44 that can be set in a separate reality space 6 or within the same reality space 6.

[0195] Therefore, this other reality model 44 has additional virtual objects 14 that are generated when the other reality model 44 is set in another (real) space by either the same 3D measuring device 11 or another 3D measuring device 11.

[0196] This method allows two individual users to test the equipment simultaneously, and these users do not need to be in any physical location relative to each other.

[0197] These two individual users can, for example, perform a (virtual) handshake, or confirm an exchange or mutual interaction.

[0198] When the situation in Figure 8 switches from the left side to the center side, the link 24 between access port 41 as a virtual object and virtual objects 13 and 14 of the real-world model 7 is deactivated and replaced by the link 24 between the same virtual object 13 and another access port 42 as a virtual object. When this link 24 is created, a tolerance range is defined within which the link 24 can be accepted without having to modify the virtual object 3.

[0199] Individual users can, for example, verify the filling of pharmaceuticals in a protective room, in this case, an isolator.

[0200] The configuration described here allows for the execution of a method for functional testing of equipment composed of numerous virtual objects 3 and 4 along with a virtual 3D model 39. For each selected virtual object 3 or 4, realistic real-world models 7 and 8 are prepared, each as realistic as possible. For example, these real-world models 7 and 8 are generated as 3D prints from the virtual objects 3 and 4. In this case, the similarity must match so that the visual impression when touching the real-world models 7 and 8 matches the visual impression when viewing the virtual objects 3 and 4.

[0201] The 3D measuring device 11 then repeatedly measures the real-world models 7 and 8 over time in order to appropriately position the corresponding virtual objects 13 and 14 within the second space 12.

[0202] In this case, the real-world models 7 and 8 are marked with markers 10 to enable feature-based identification. When linking 24 is activated, these identified features, i.e., the correspondence 15 between the markers 10 and the virtual objects 3 and 4, are saved to enable the virtual objects 3 and 4 to follow, thereby making it possible to move the virtual objects 13 and 14 together with the virtual objects 3 and 4.

[0203] In this case, virtual objects 3 and 4 may include predetermined positions, such as drilled holes into which markers 10 are attached. This makes it easy to position markers 10 in the positions of real-world models 7 and 8, thereby making it easy to locate and align virtual objects 13 and 14 with respect to virtual objects 3 and 4.

[0204] Alternatively, marker 10 can be arbitrarily applied to the real-world models 7 and 8, and then the formed features, particularly the position of marker 10, can be measured on the real-world models 7 and 8.

[0205] Therefore, the method for functional testing of equipment 1 can begin by first preparing CAD data of the equipment by appropriate design, processing this CAD data into a 3D model 39, forming real models 7 and 8 of selected details from this 3D model 39 or a part of the CAD data, particularly using 3D printing or an alternative manufacturing method, setting these real models 7 and 8 in a 3D measuring device 11, and displaying the 3D model 39 from the observer's position with a visual impression 20.

[0206] If, for example, changes to the 3D model 39 are required during testing according to the procedure shown in Figure 5, these changes to the 3D model 39 can be output as modified design data at the end of the test. This may apply, for example, if it is found that a particular detail of the 3D model 39 is ergonomically unsuitable and requires design modification.

[0207] During the testing process, individual users can manipulate real-world models 7 and 8 as they see fit, thereby confirming the impact of these actions on the visual impression 20 at the level of virtual objects 3 and 4.

[0208] This is made possible by linking 24, which forces virtual objects 3 and 4 to follow the corresponding virtual objects 13 and 14. In the presented embodiment, this is achieved by a device for automatically making virtual objects 3 and 4 follow the corresponding virtual objects 13 and 14, and a 3D engine as a device for visually displaying the 3D model 39.

[0209] Figure 7 schematically shows a head-mounted display 16 equipped with a 3D measuring device 11. This configuration can be used in the configuration shown in Figure 1, but it can also be operated independently of the configuration shown in Figure 1.

[0210] In this head-mounted display 16, the position and orientation of the field of view 19 are repeatedly determined over time, and in the visual impression 20, for example, linked information, such as virtual objects 3 and 4, or other position-related information such as warnings or work instructions, are faded in in the manner described.

[0211] In this case, the field of view 19 is identified using the means inherent to the head-mounted display 16, i.e., an integrated means. As a result, this field of view 19 can only be created with respect to the reality models 7,8 that are within the field of view 19.

[0212] To enable alignment of the head-mounted display 16 with respect to other reality models outside the field of view 19, the posture and position of the head-mounted display 16 are exogenously determined based on markers 10 using a 3D measuring device. This information is then linked to the field of view 19, thereby enabling accurate, nearly uninterrupted, or completely uninterrupted detection of the posture and position of the head-mounted display 16, and ultimately enabling a continuous visual experience in the visual impression 20.

[0213] In isolators, particularly in controlled environments, controlled airflow is often used to control and dissipate potential pollutants, thereby protecting sensitive areas from contamination.

[0214] Furthermore, the present invention makes it possible to calculate such airflow as streamlines 45 and to display its changes in particular during operation 25 as a visual impression 20.

[0215] Figure 9 shows another virtual 3D model 39 used in the present invention. This is a pharmaceutical facility comprising a wall section 40, access ports 41, 42, 43, 50, a measuring station 51 (e.g., for measurement by agar plate and / or particle measurement), a supply section 52 from an automated device, a discharge section 53 to another automated device, a photointerrupter module 54 with a ray 55 for monitoring unexpected and / or unauthorized access to the access ports 41, 42, 43, 50, a sorting pot for preparing stoppers or other packaging or components of a medical dispensing device into the correct position, a material stock 57 for a sorting pot 56 that can be replenished through a pure transfer port 48, for example, a transfer and processing area 58 for processing (filling and closing) containers (e.g., vials), a filling station 59 for filling containers, an individualization station 60 for individualizing bundles of containers supplied by the supply section 52, an installation station 61 for closing containers with the aforementioned stoppers, and a management station 62 for managing the filled and sealed containers.

[0216] Individual components require tactile contact with the individual user during functional testing and are therefore used as real-world models 7 and 8. For example, this is not necessary for the sorting pot 56—a virtual object 63 suffices here. In contrast, it is advantageous to use real-world models 7 and 8 for the filling station 59 and the measuring station 51. These may be actual products rather than 3D prints as already described. This can be simpler for complex structures.

[0217] The photointerrupter module 54 can also exist as a real-world model 7,8, but it may not have any functionality, for example. In the virtual world, a ray 55 is simulated, and it is checked whether the individual user has virtually blocked that ray 55, and a signal is generated if necessary.

[0218] Figure 10 shows another virtual 3D model 39 used in the present invention. Functionally and / or structurally similar or identical parts and functional units to those in prior embodiments are indicated by the same reference numerals and are not described separately. Accordingly, the description relating to prior embodiments applies similarly.

[0219] Figure 10 additionally shows how virtual objects 13 and 14 follow real-world models 7 and 8 based on 3D measurements 68.

[0220] In the setup process, it is defined which virtual objects 13, 14 can be linked to, which are indicated by contours 69 that match the shape of one or more virtual objects 3, 4, 63. As a result, virtual object 14 can be linked to at least two virtual objects 4, 63.

[0221] Virtual object 65 cannot be linked to other virtual objects.

[0222] During the setup process, it is further defined how virtual objects 3, 4, and 63 can be linked to virtual objects 13 and 14. This is represented by the spatial relationships of contours 69 in relation to the posture of marker 10, which corresponds to the positions of markers on the real-world models 7 and 8.

[0223] Linking 66 is deactivated after setup, thereby allowing linking 67 to the virtual object 63 to be activated. When this linking is changed, the virtual object 63 follows the real-world model 8.

[0224] The link 24 between virtual object 13 and virtual object 3 can be activated or terminated independently of the link 66 (or the link 67 which can be selectively activated together) between another virtual object 14 and another virtual object 4,63.

[0225] The number of virtual objects, especially linkable objects 3, 4, 63, and 65, is greater than the number of virtual objects 13 and 14.

[0226] At least two reality models 7 and 8 are prepared, and the corresponding virtual objects 13 and 14 are realized in each.

[0227] At least two reality models 7,8 have a spatial relationship with each other through a relationship in virtual space between at least two virtual objects 3,4, i.e., defined by an additional unlinked virtual object 65. This spatial relationship is mediated by at least two virtual objects 13,14 belonging to at least two reality models 7,8.

[0228] In this case, at least two reality models 7,8 may be compelled to move, or restricted, relatively or absolutely.

[0229] If there are mandatory guidelines or restrictions, the user's mobility is limited by at least one reality model 7,8 during use.

[0230] Therefore, in the method for functional testing of equipment 1, it is proposed to create real models 7 and 8 from the details of a virtual 3D model 39 of the equipment, and to detect their spatial position and posture during functional testing using a 3D measuring device 11. In this case, the identified posture and position can be used to make the virtual objects 3 and 4 of the 3D model 39 follow the operations in the real models 7 and 8 in the visual impression 20 in virtual reality. [Explanation of symbols]

[0231] 1. Equipment for functional testing of equipment 2. Virtual 3D Models 3. Virtual Objects 4. Virtual Objects 5. The First Virtual Space 6. Real space 7. Reality Model 8. Reality Model 9 Stands 10 markers 11 3D measuring device 12. The Second Virtual Space 13 Virtual Objects 14 Virtual Objects 15 Correspondence 16. Head-mounted displays 17 Markers 18 Virtual Objects 19 Field of view 20 Visual Impression 21. Field of View Data 22 Position and Posture Data 23 Object Data 24. Linking 25 operations 26. Apparent behavior 27 Hand Tracking Devices 28 markers 29 Virtual Objects 30 hand copies 31 Camera 32 Support structure 33 Shoulder Ring 34 Shoulder Ring 35 fingers 36 Moving parts 37 Fixed part 38 Projection line 39 3D models 40 Wall 41 Access points 42 Access points 43 Access points 44 Alternative Reality Models 45 Streamline 46 sensors 47 doors 48 Transfer Port 49 Isolator Gloves 50 Other access points 51 Measurement Station 52 Supply section 53 Discharge section 54 Photointerrupter Module 55 Rays of light 56 sorting pots 57 Material Stock 58 Transfer and Processing Area 59 Filling Station 60 Individualization Stations 61 Installation Stations 62 Management Stations 63 Another virtual object 64 deviation 65 Another virtual object 66. Linking 67. Linking 68 3D measurement

Claims

1. A method for functional testing of equipment (1), comprising: representing the equipment as a virtual 3D model (2) from virtual objects (3, 4); preparing real-world models (7, 8) for at least one virtual object (3, 4); aligning virtual objects (13, 14) with the real-world models (7, 8) by 3D posture measurement over time; linking the at least one virtual object (3, 4) to the virtual objects (13, 14) and bringing them to a desired positional relationship; and modifying, in particular activating or starting, and / or deactivating or ending, the link (24) between the virtual objects (13, 14) and the at least one virtual object (3, 4) by the user.

2. The method according to claim 1, characterized in that the linking (24) between the virtual objects (13, 14) and the at least one virtual object (3, 4) includes forcing a desired positional relationship of the position and / or posture of the virtual object (3, 4) with respect to the virtual objects (13, 14), particularly in that position and / or posture, and / or by performing a request and / or continuously.

3. The method according to claim 1 or 2, characterized in that the setup step defines which virtual objects (3, 4) a virtual object (13, 14) can be linked to, and / or the setup step or in the setup step defines how a virtual object (3, 4) can be linked to a virtual object (13, 14).

4. The method according to any one of claims 1 to 3, characterized in that the linkability between virtual objects (13, 14) and at least two virtual objects (3, 4) is adjusted.

5. The method according to any one of claims 1 to 4, characterized in that the link between virtual objects (13, 14) and virtual objects (3, 4) can be activated or initiated and / or deactivated or terminated independently of the link between another virtual object (13, 14) and another virtual object (3, 4).

6. The method according to any one of claims 1 to 5, characterized in that the number of virtual objects (3, 4), in particular linkable virtual objects (3, 4), is not less than or greater than the number of virtual objects (13, 14), in particular greater than three times.

7. The method according to any one of claims 1 to 6, characterized in that at least two reality models (7, 8) are prepared, and in particular, the corresponding virtual objects (13, 14) are realized in each.

8. The method according to any one of claims 1 to 7, characterized in that at least two reality models (7, 8) are brought together by the spatial relationships of at least two virtual objects (3, 4), in particular by the spatial relationships predetermined by at least two virtual objects (13, 14) belonging to the at least two reality models (7, 8).

9. The method according to any one of claims 1 to 8, characterized in that at least two reality models (7, 8) are coerced and / or movable in a manner that restricts each other.

10. The method according to any one of claims 1 to 9, characterized in that the user's mobility is restricted by at least one reality model during use.

11. The method according to any one of claims 1 to 10, wherein the linking is advantageously deactivated for a defined or undefined period of time, and in particular, this is characterized in that a deviation between the at least one virtual object (3, 4) and the virtual objects (13, 14) is displayed when the linking (24) is inactive.

12. The method according to any one of claims 1 to 11, wherein the link (24) between the virtual objects (13, 14) and the at least one virtual object (3, 4) is replaced by another link between the virtual objects (13, 14) and the other virtual object (3, 4).

13. The method according to any one of claims 1 to 12, wherein if the linking (24) is replaced by another linking (24), the 3D model (39) is isometrically transformed, in particular rotated and / or displaced, until the virtual objects (13, 14) and the other virtual objects (3, 4) coincide to at least within an acceptable range.

14. The method according to any one of claims 1 to 13, wherein the orientation of the reality model (7, 8) is advantageously changed manually or automatically until the corresponding virtual object (13, 14) matches the at least one virtual object (3, 4) or the other virtual object, and in particular, the linking (24) between the virtual object (13, 14) and the virtual object (3, 4) or the other virtual object (3, 4) is then activated.

15. The method according to any one of claims 1 to 14, wherein a plurality of virtual objects (13, 14) are linked to each virtual object (3, 4) of the 3D model (39), and each of the links (24) is modified independently of each other, in particular by being activated and / or deactivated.

16. The method according to any one of claims 1 to 15, wherein the coordinate update of the at least one virtual object (3, 4) is output, in particular, for processing the design data of the 3D model (39).

17. The method according to any one of claims 1 to 16, wherein the equipment is for the pharmaceutical field, preferably for filling pharmaceuticals into containers, and / or equipment that works in cooperation with a protective chamber, preferably an isolator.

18. A method for functional testing of equipment 1, the method according to any one of claims 1 to 17, wherein the equipment is represented as a virtual 3D model (2) from virtual objects (3, 4), a real model (7, 8) is prepared for at least one virtual object (3, 4), virtual objects (13, 14) are aligned with the real model (7, 8) by 3D pose measurement over time, the at least one object is linked to the virtual object (13, 14) and brought into a desired positional relationship with the virtual object (13, 14), the real model (7, 8) corresponding to the at least one virtual object (3, 4) is created, preferably by an additive method, identifiable features are attached for 3D pose measurement, and the correspondence (15) between the identified features, in particular markers (10, 17) and the at least one virtual object (3, 4) is stored.

19. The method according to any one of claims 1 to 18, wherein the identifiable features are formed at predetermined positions on the reality model (7, 8).

20. The method according to any one of claims 1 to 19, wherein at least one position of the feature formed on the reality model (7, 8) is measured.

21. The method according to any one of claims 1 to 20, wherein the operator wears gloves and / or a hand tracking device (27), and / or the 3D posture of one or more fingers and / or one hand and / or one arm, preferably the gloves or the gloves and / or the hand tracking device (27) or the hand tracking device (27) is repeatedly determined.

22. The method according to any one of claims 1 to 21, wherein the 3D model (39) represents a shoulder ring, and the actual shoulder rings (33, 34) are adjusted to the position of the shoulder ring, particularly in a preceding setup step, and / or the operator puts their arm through the shoulder rings (33, 34), preferably with an operating glove attached to the shoulder rings (33, 34).

23. The method according to any one of claims 1 to 22, wherein the observer's photographic pose is defined relative to the shoulder rings (33, 34).

24. The method according to any one of claims 1 to 23, wherein the at least one virtual object (3, 4) is a door (47) of a transfer port (48), or has a door (47) of a transfer port (48).

25. The method according to any one of claims 1 to 24, wherein the 3D model (39) has another virtual object (3, 4), another real-world model (7, 8) is prepared for the other virtual object (3, 4), and the real-world model (7, 8) is movably positioned for the other real-world model (44).

26. The method according to any one of claims 1 to 25, wherein the other reality model (44) is at least partially immobile and / or at least partially movable with respect to the boundary of the equipment.

27. A method for functional testing of equipment (1), comprising the steps of: preparing CAD data of the equipment; creating at least one real model (7, 8) of at least a portion of the CAD data; setting the at least one real model in a 3D measuring device (11); and processing at least 3D measurement data from the 3D measuring device (11) to display a virtual 3D model (2) created from the CAD data, the method according to any one of claims 1 to 26.

28. The method according to any one of claims 1 to 27, characterized in that the field of view (19) of the head-mounted display (16) is preferably determined using the 3D measuring device (11).

29. The method according to any one of claims 1 to 28, characterized in that the display of the 3D model (39) is performed in relation to the field of view (19) of the head-mounted display (16).

30. The method according to any one of claims 1 to 29, characterized in that changes to at least one reality model (7, 8) are automatically tracked to the 3D model (39), and in particular, modified design data is generated and output from the modified 3D model (39).

31. The method according to any one of claims 1 to 30, characterized in that the 3D measuring device (11) is transferred to a fixed measuring structure before setting up.

32. An apparatus for functional testing of equipment (1), wherein the equipment exists as a virtual 3D model (2), and comprises a 3D measuring device (11), at least one real-world model (7, 8) of the virtual objects (3, 4) of the 3D model (39), an apparatus for automatically incorporating virtual objects (13, 14) detected by the 3D measuring device (11) with respect to the real-world model (7, 8) into the 3D model (39), an apparatus for automatically making the virtual objects (3, 4) track together with the virtual objects (13, 14), and an apparatus for visually displaying the 3D model (39), particularly a 3D engine.

33. The apparatus according to claim 24, characterized in that means are provided for activating and / or deactivating the link between the virtual objects (13, 14) and the at least one virtual object (3, 4).

34. The apparatus according to claim 24 or 25, characterized in that a head-mounted display (16) is provided for generating a field of view (19) for the 3D model (39).

35. The apparatus according to any one of claims 24 to 26, characterized in that the apparatus for generating a field of view (19) for the 3D model (39) supplies the measured values ​​of the 3D measuring apparatus (11) to the head-mounted display (16) or the head-mounted display (16).

36. The apparatus according to any one of claims 24 to 27, characterized in that it has a device for converting the 3D model (39) to the same length as the field of view (19).

37. Particularly advantageous is the use of a head-mounted display (16), particularly VR, XR and / or AR goggles, for functional testing of pharmaceutical equipment, and a fixed and / or independently operating 3D measuring device (11) for creating a virtual view of a 3D model (39) of the equipment within the head-mounted display (16), wherein individual virtual objects (3, 4) correspond to real-world models (7, 8) detected by the 3D measuring device (11).

38. A head-mounted display (16), particularly a VR, AR and / or XR goggle, having at least one marker (10, 17), in particular two or more markers (10, 17), for advantageously identifying a shooting pose externally.

39. A method for visually displaying 3D data in the manner of any one of claims 1 to 23, comprising: repeatedly determining intrinsically over time the field of view (19) of a head-mounted display (16), particularly VR and / or XR and / or AR goggles; displaying information linked to the field of view (19) within the head-mounted display (16); and advantageously repeatedly identifying and compensating for the field of view (19) of the head-mounted display (16) with a pre-set shooting pose.

40. The method according to any one of claims 1 to 23 or claim 31, characterized in that the aforementioned shooting pose is identified using a 3D measuring device (11) that is formed independently of and / or fixed to the head-mounted display (16), particularly including at least one camera (31).

41. The method according to any one of claims 1 to 23 or claim 31 or 32, characterized in that the aforementioned shooting pose is determined using a measuring device that is linked to and / or independent of the determination of the field of view (19).

42. The method according to any one of claims 1 to 23 or any one of claims 31 to 33, characterized in that the intrinsic determination is carried out using at least one interconnected sensor, particularly a camera (31) and / or motion and / or acceleration and / or posture sensor.

43. The method according to any one of claims 1 to 23 or any one of claims 31 to 34, wherein the shooting pose is measured using active markers (10, 17) on the head-mounted display (16).

44. The method according to any one of claims 1 to 23 or any one of claims 31 to 35, wherein the shooting pose is measured using passive markers (10, 17) on the head-mounted display (16).

45. The method according to any one of claims 1 to 23 or any one of claims 31 to 36, wherein the aforementioned photographic pose is measured using a fixed measuring device, particularly a fixed camera (31).

46. The method according to any one of claims 1 to 23 or any one of claims 31 to 37, wherein the intrinsic determination of the aforementioned shooting pose is measured using a camera (31) linked to the head-mounted display (16).

47. A method as part of a method according to any one of the claims of a method for visual display of equipment and / or a preceding method, comprising repeatedly over time intrinsically determining the field of view (19) of a head-mounted display (16), in particular VR and / or XR and / or AR goggles, calculating airflow, and advantageously displaying it visually in the head-mounted display (16) as linked 3D data.

48. The method according to any one of claims 1 to 23 or any one of claims 31 to 39, wherein the 3D data includes a 3D model of the equipment (39) and / or the 3D data includes AR metadata relating to the equipment or components of the equipment.

49. The method according to any one of claims 1 to 23 or any one of claims 31 to 40, wherein a head-mounted display (16), particularly the head-mounted display (16) according to claim 30, is used to generate a superposition of the real field of view (19) and a virtual display of the 3D data, thereby blocking out the real environment.

50. The method according to any one of claims 1 to 23 or any one of claims 31 to 41, characterized by motor-driven adjustment of a real-world model (7, 8) and / or adjustment of the real-world model (7, 8) until the advantageously automatically detected deviation (64) of the position and / or posture of the virtual objects (13, 14) from the corresponding virtual objects (3, 4) is within an acceptable range.

51. The method according to any one of claims 1 to 23 or any one of claims 31 to 42, characterized in that a virtual ray (55) is generated, and in particular, it is automatically checked whether the virtual ray (55) is blocked.

52. An apparatus for visually displaying 3D data, particularly as part of the apparatus described in any one of claims 24 to 28 and / or for use in the method described in any one of claims 1 to 23 or any one of claims 31 to 43, comprising: a head-mounted display (16) provided for determining a linked field of view (19); a device for identifying a shooting pose of the head-mounted display (16); and a device for compensating and adjusting the shooting pose with the field of view (19).

53. The apparatus according to claim 44, further comprising a 3D engine for visually displaying the 3D data within the field of view (19).

54. The apparatus according to claim 44 or 45, which preferably includes a fixed 3D measuring device (11) for identifying the aforementioned photographic pose.

55. The apparatus according to any one of claims 44 to 46, comprising: a 3D measuring device (11) or an apparatus for generating a virtual space for virtual objects (13, 14) detected using the 3D measuring device (11); and / or an apparatus for generating a virtual space (5, 12) for visually displaying the 3D data; and / or an apparatus for establishing a correspondence (15) between the virtual spaces (5, 12), particularly an apparatus for embedding the virtual objects (13, 14) into the virtual spaces (5, 12) and displaying them virtually.

56. The apparatus according to any one of claims 44 to 47, further comprising a device for embedding the field of view (19) of the head-mounted display (16) into the virtual space (5, 12) and displaying it virtually.

57. The apparatus according to any one of claims 44 to 48, comprising an apparatus for calculating airflow, and more particularly an apparatus for visually displaying the airflow.

58. The apparatus according to any one of claims 24 to 28 or any one of claims 44 to 49, comprising: an apparatus for motor-driven adjustment of at least one reality model (7, 8); and / or an apparatus for advantageously automatically determining the deviation (64) of the position and / or posture of a virtual object (13, 14) from a corresponding virtual object (3, 4); and / or an apparatus for motor-driven adjustment of at least one reality model (7, 8) until the advantageously automatically detected deviation (64) of the position and / or posture of a virtual object (13, 14) from a corresponding virtual object (3, 4) is within an acceptable range.

59. The apparatus according to any one of claims 24 to 28 or any one of claims 44 to 50, comprising an apparatus for collision testing of a virtual ray (55).

60. The apparatus according to any one of claims 44 to 50, comprising means for carrying out the method described in any one of claims 1 to 23 or any one of claims 31 to 43.

Images