Head-mounted display, use of head-mounted display, method and apparatus for functional testing of equipment, and method and apparatus for visual display of 3D data

Abstract

In the head-mounted device (16), it is proposed to form or attach at least one marker (10, 17) on the outside in order to enable the shooting posture of the head-mounted device (16).

Description

Technical Field

[0001] The present invention relates to a head-mounted display and its use.

[0002] Head-mounted displays are actually known, for example, as VR (Virtual Reality) goggles, AR (Augmented Reality) and MR (Mixed Reality) goggles, which are, for various purposes, often in the entertainment industry, to create a spatial impression, spatially assigned virtual data alone or in combination with a real scene, are visually presented to the individual observer. Head-mounted displays are described, for example, in the German-language version of Wikipedia. According to that, a head-mounted display is characterized, for example, as a visual output device attached to the head. Such a device can be provided, for example, to complement (AR, MR) or replace (VR) the natural visual impression of an observer by an artificially created impression, by displaying an image on a screen in front of the eyes or by projection onto the retina.

[0003] The present invention further relates to a method for functional inspection of equipment.

[0004] Especially in pharmaceutical equipment, it is actually known to perform a functional inspection using cardboard and / or wooden models, usually before starting a complicated production.

[0005] The present invention further relates to a device for functional inspection of equipment.

[0006] Regarding the method described above, cardboard and / or wooden models are actually known.

[0007] The present invention further relates to a method and a corresponding device for visual display of 3D data.

[0008] In order to display 3D data in a way that can be directly experienced, the entertainment industry, in particular, is known to use such methods and devices, especially with head-mounted displays as already mentioned above.

[0009] The fundamental problem underlying this invention is to expand the applicability of head-mounted displays.

[0010] To solve the above problems, the features of claim 1 are assumed according to the present invention.

[0011] Accordingly, head-mounted displays, particularly VR, AR, and / or XR goggles, have been proposed that include means for extrinsically identifying the shooting posture and means for intrinsically identifying the shooting posture. The advantage of this is that it is possible to combine the advantages of intrinsically identifying the shooting posture with the advantages of extrinsically identifying the shooting posture.

[0012] Alternatively or additionally, the problem is solved by the features of the independent claim relating to a head-mounted display. Thereafter, the present invention uses a head-mounted display, such as VR, AR, and / or XR goggles, which has at least one marker, in particular more than two, for advantageously identifying the shooting posture. In many cases, three markers are sufficient to clearly determine the position and location of a reality model. However, especially for more complex reality models, it is beneficial to attach more than three markers.

[0013] By attaching markers, separate detection and alignment of the shooting posture are possible even for objects outside the field of view.

[0014] This makes it possible to easily embed the field of view of a head-mounted display into a virtual space of a virtual 3D model through 3D measurement. This gives the observer a realistic impression of the equipment or complex structure from their own perspective. This expands the usability of head-mounted displays.

[0015] In this case, the shooting posture can be determined, for example, by external or internal factors.

[0016] Intrinsic decisions can be characterized, for example, by their interaction with the relevant sensors and / or alignment with the field of view, and / or by the ability to calculate the field of view using tools on the head-mounted display. For example, intrinsic decisions can also be recognized by the fact that they cannot be performed when the head-mounted display is inactive.

[0017] Extrinsic decisions can be characterized, for example, by the fact that they can be made independently or separately from intrinsic decisions. For instance, extrinsic decisions can also be recognized in that they can be made when the head-mounted display is inactive.

[0018] Extrinsic determinations can be made, for example, using the aforementioned markers and / or defined with respect to spatially fixed reference points.

[0019] In this case, the markers may be formed in an active form. This allows for individual detection of each head-mounted display. This enables easy device swapping without requiring new learning and / or system recalibration. It also allows for the uniform establishment of a reference center for multiple encodings. In this case, this uniform center can be used to map virtual objects into the field of view and ultimately embed them without learning for each marker.

[0020] The markers may be formed passively, either as an alternative or additional method. This reduces the resources consumed during operation, thus extending the uptime of the head-mounted display. In this case as well, the markers can be attached to various positions on different head-mounted displays, thereby enabling individual detection. Standardized mounting kits for connecting the goggles and markers allow for embedding with no learning required or minimal effort. Another advantage of passive markers is their light weight, which positively impacts, for example, the comfort of wearing the head-mounted display.

[0021] In the configuration of the present invention, means for intrinsically identifying the shooting posture may be provided. This makes it possible to track a virtual scene based on the intrinsically detected shooting posture.

[0022] In this case, it can be envisioned that means may be formed to compensate for and adjust the imaging posture determined using means for intrinsically determining the imaging posture with another imaging posture that is advantageously determined extrinsically. This would allow the intrinsically determined imaging posture to be compensated for and adjusted by independent measurements, particularly absolute measurements. This would allow for compensation or elimination of errors that may arise from determining the intrinsic imaging posture by integrating detected changes (e.g., from acceleration sensors and / or optical flow) over longer distances or periods.

[0023] In the configuration of the present invention, it is advantageous to be able to input an exogenously identified shooting posture. This makes it easier to achieve a compensatory adjustment value or reference value for intrinsically identifying the shooting posture and synchronizing it with physical reality. For example, this can be done in addition to and / or in addition to compensatory adjustment with the intrinsically identified shooting posture. The present invention achieves a combination of low-latency tracking of a virtual scene based on an intrinsically identified shooting posture and more accurate exogenous identification of the corresponding shooting posture.

[0024] In the configuration of the present invention, it can be assumed that the display means of the virtual scene can be controlled based on the shooting posture, or preferably, the shooting posture specified externally. This enables simple synchronization between the virtual scene and the physical reality.

[0025] To solve the above problems, as a preferred application, an apparatus for reproducing at least one virtual object, comprising a head-mounted display according to the present invention, particularly the head-mounted display described above, or the head-mounted display described in the claims below, and a 3D measuring device for externally detecting at least one marker, is used. Thereby, a system with a redundant detection unit that can easily calibrate or reference the measurement results is described.

[0026] In this case, it can be assumed that the means for compensation adjustment is supplied from the 3D measuring device, and preferably, additionally from the means for internally detecting the shooting posture. This makes it possible to use the physical reality as a reference and thereby recognize the deviation in the internal specification.

[0027] Preferably, the measurement accuracy of the 3D measuring device is higher than the measurement accuracy of the means for internally specifying the shooting posture.

[0028] The problem described at the beginning is solved alternatively or additionally by using a head-mounted display, particularly VR, XR and / or AR goggles, and a 3D measuring device, preferably stationary and / or operating independently of the head-mounted display, for creating a virtual view of the 3D model of the equipment within the head-mounted display. In this case, each virtual object corresponds to the real model detected by the 3D measuring device. Thereby, means for tactilely controllable virtual functional inspection of the equipment represented as a 3D model becomes possible.

[0029] This can be used, in particular advantageously for the functional inspection of pharmaceutical equipment, advantageously in the method according to the invention, in particular the methods described above and / or below, and / or the methods described in the claims below, and / or in the device according to the invention, in particular the devices described above and / or below, and / or the devices described in the claims below. Here, since pharmaceutical equipment, in particular a control room, or a control room such as a RABS (Restricted Access Barrier System) or an isolator generally has a large spatial expansion, significant cost, time, and space savings can be achieved. This complicates conventional structures made of cardboard and / or wood.

[0030] The problem stated at the beginning is, alternatively or additionally, according to the invention, a method for the visual display of 3D data, in which the field of view of a head-mounted display (a display attached to the head), in particular VR and / or XR and / or AR goggles, is determined intrinsically, and information linked to the field of view is displayed within the head-mounted display. In this case, the shooting pose for presetting the field of view of the head-mounted display is advantageously repeatedly specified over time and compensated and adjusted with the field of view. Thereby, with the available computing power of the head-mounted display, it becomes possible to spatially accurately embed the field of view in the virtual world. The invention has the advantage that the spatial positional relationship of the observer can also be detected for virtual objects that are not currently present in the observer's field of view, which is defined by the shooting pose of the head-mounted display.

[0031] In this case, the field of view of the head-mounted display can be given, for example, by the field of view of the observer when the position of the observer's head corresponds to the current shooting pose of the head-mounted display when the head-mounted display is in use. In this case, the shooting pose can be referred to, for example, as the position and orientation in the forward direction of the head-mounted display.

[0032] This method may be formed or constituted, for example, as part of a method for functional testing of equipment, as described in the present invention, particularly the method described herein, and / or in the claims described later.

[0033] In an advantageous configuration, the shooting posture could be determined using a 3D measuring device that is independently formed and / or fixed to the head-mounted display. This would allow the head-mounted display to be detected from all directions at all times, and the field of view to be embedded without interruption.

[0034] A 3D measuring device may include, for example, at least one or more cameras. Generally speaking, using multiple cameras improves the accuracy of measurements on the one hand, and reduces the likelihood that details will be obscured by other details on the other hand.

[0035] 3D measuring devices are known for spatially detecting the position or location of real-world models. One method involves creating 2D images from a real-world model under various shooting orientations, identifying each model within these images using, for example, attached markers, and then solving a system of equations that describes these images as photographs of a common real-world model, where the shape, e.g., the position of individual markers, is treated as an unknown variable, and the image position is treated as an input value. An alternative is to use, for example, structured light, whose pattern on the real-world model allows for the estimation of the real-world model's position and location. Methods for measuring signal propagation time are also known.

[0036] In a favorable configuration, the shooting position may be determined using a measuring instrument that is linked to and / or independent of the field of view determination. This simplifies the mechanical structure of the apparatus according to the present invention.

[0037] In a favorable configuration, intrinsic decisions may be assumed to be made using at least one interconnected sensor, particularly a camera and / or motion and / or acceleration and / or position sensors. This would make known systems available to identify the field of view and its changes with head movement.

[0038] In a favorable configuration, it may be conceivable to measure the shooting posture using active markers on a head-mounted display. Active markers have the advantages of excellent differential indication and easy modification of identification information.

[0039] In advantageous configurations, passive markers on a head-mounted display may be used to measure the shooting posture. Passive markers conserve energy during operation, thereby extending the uptime during which the device remains usable.

[0040] In a favorable configuration, it may be conceivable to measure the shooting posture using a fixed measuring device, particularly a fixed camera.

[0041] In a favorable configuration, it might be conceivable to measure the intrinsic determination of the shooting posture using a camera linked to a head-mounted display.

[0042] The problem described at the beginning is, according to the present invention, solved, alternatively or additionally, by a method for visual display of equipment and / or as part of the methods described in the previously described and / or subsequent claims, which temporally repeats, intrinsically determines the field of view of a head-mounted display (a display attached to the head), particularly VR and / or XR and / or AR goggles, calculates the airflow, and advantageously displays it visually in the head-mounted display as linked 3D data. In this case, the advantage is that the effect of the work process on the airflow can be directly understood and / or controlled.

[0043] In controlled environments, airflow is known to be used to prevent the transfer of contaminants to areas requiring particular protection. The present invention makes it possible to verify that airflow is also affected by, for example, mobile functional units and / or individual users.

[0044] In an advantageous configuration, the head-mounted display may be connected to a fixed processing unit, which is advantageous for transmitting pose measurement data and / or image data for the head-mounted display. This allows the computational routines to be offloaded to a higher-capacity fixed unit. In this case, data transmission can be done, for example, wirelessly or wired.

[0045] In an advantageous configuration, the 3D data may include a 3D model of the equipment and / or also include AR metadata, particularly regarding the equipment components already mentioned. The use of the 3D model makes it possible to visually represent the equipment in a virtual space as if it were real. The use of AR metadata allows for the additional fading in or display of data such as warnings, notifications, or work instructions, in addition to the image content. This makes it possible, for example, to easily switch to the observer's language or to change the fading in depending on the operating status of the equipment. It is also possible to display airflow velocity data as 3D data, particularly as streamlines.

[0046] In an advantageous configuration, a head-mounted display, particularly a head-mounted display according to the present invention, may be used to generate a superposition of the real field of view and a virtual representation of 3D data, for example, as described in the preceding and / or subsequent claims. This enables MR or AR applications.

[0047] Alternatively or additionally, a head-mounted display, particularly a head-mounted display according to the present invention, can be used to isolate the real environment, for example, as described above and / or in the subsequent claims. This enables VR applications.

[0048] Additionally, in any of the methods described, it may be conceivable to motorize one of the reality models, for example, one of the reality models already mentioned. This would allow for easier and / or more accurate setting of the position and / or location of virtual objects.

[0049] Alternatively or additionally, one of the above methods may involve adjusting, advantageously by motor drive and / or automatically, one of the reality models, for example, one of the reality models already mentioned, until the deviation of the position and / or location of a virtual object from its corresponding virtual object is advantageously within an acceptable range. This would enable the automatic incorporation of reality models into this method.

[0050] These two configurations can be advantageously combined or used individually, for example, in the shoulder rings (Schulterringen) which will be described in more detail below.

[0051] To solve the problems mentioned at the beginning, the present invention further envisions a device for the visual display of 3D data, comprising a head-mounted display provided for determining the linked field of view, a device for detecting the shooting posture of the head-mounted display, and a device for compensating and adjusting the shooting posture with the field of view. This makes it possible to achieve spatially accurate embedding of virtual objects into the virtual world's field of view while keeping the computational power requirements of the head-mounted display low. This could, for example, be used to fade in information and messages to appropriate locations.

[0052] In this case, the device may be formed as part of an apparatus according to the present invention for functional testing of equipment, as described above and / or in the subsequent claims.

[0053] In a favorable configuration, a 3D engine may be formed to visually display 3D data within the field of view. This makes known algorithms available for controlling the visual display.

[0054] In an advantageous configuration, a fixed 3D measuring device may be provided for detecting the shooting posture. The fixed 3D measuring device can be configured to cover a wider space, for example, by spatially distributing the corresponding cameras.

[0055] In an advantageous configuration, for example, equipment may be formed to generate a virtual space for virtual objects detected using the 3D measuring device already described. This makes it possible to detect real-world models and their changes. This can be used, for example, to connect the virtual world and the real world. An example of a 3D measuring device is the Prime camera of OptiTrack (NaturalPoint, Inc., PO Box 2317, Corvallis, OR 97339). X 13 and / or Prime X This combines 13W with OptiTrack's motion capture software, Motive.

[0056] In a favorable configuration, alternative or additional equipment may be provided for generating a virtual space for visually displaying 3D data. This would allow for the preparation of a scene for generating a virtual visual impression.

[0057] In a favorable configuration, alternatively or additionally, two virtual spaces may be formed, preferably with devices for establishing the correspondence between the virtual spaces already described, particularly devices for embedding and virtually displaying virtual objects within the virtual spaces. This would provide an easily manageable means for modifying virtual objects by manipulating the corresponding real-world models.

[0058] In an advantageous configuration, a device may be formed to embed the field of view of the head-mounted display into the virtual space and display it virtually. This makes it possible to provide a natural and faithful visual representation of the virtual world through the observer's eyes, as if the observer were actually visually perceiving the virtual world.

[0059] In an advantageous configuration, equipment for calculating airflow, particularly equipment for visually displaying airflow, may be included. This makes it possible to visualize airflow, especially as streamlines.

[0060] To solve the above problems, alternatively or additionally, a method is provided for functional testing of equipment in which the equipment is represented as a virtual 3D model from virtual objects, a real model is prepared for at least one virtual object, the virtual object is aligned with the real model by 3D position measurement over time and iteratively, the at least one virtual object is linked to the virtual object and brought into a desired positional relationship with the virtual object, in which case the link between the virtual object and at least one virtual object is modified by the user. Thus, the present invention enables a tactile experience of virtual modeling that allows for real functional testing without requiring a complete real image of the equipment under test. This eliminates the need to construct the entire equipment as a real model, thus greatly simplifying functional testing.

[0061] Generally speaking, the aforementioned linking can refer to only a subset of the degrees of freedom of movement for each object or object, or it can force complete fixation. For example, a desired positional relationship may mean tracking only a portion of the motion (e.g., only the X and Y axes, without rotation). This also occurs with holograms and targets (globeports).

[0062] For example, a real-world model might be tilted relative to a glass panel surface, causing the corresponding virtual object to be pulled away from the (virtual) panel surface when linking is activated. Here, it might be possible to create boundary conditions that restrict the alignment of this object relative to the virtual object to only specific degrees of freedom and fix it to the panel's degrees of freedom, thereby ensuring that the virtual object, such as a shoulder ring or glove port, remains within the panel. This would avoid visual distortion.

[0063] For example, real-world objects that are too far away for a user to touch in a particular test can be omitted. In this case, the idea of ​​activating the linking makes it possible to connect the virtual world to the real world, for example, allowing the details of a 3D model to be experienced tactilely through a properly positioned real-world model. The idea of ​​deactivating the linking makes it possible to swap virtual objects, and consequently, to reuse a very limited stock of real-world models, for example, in various locations in industrial equipment, especially when structural details are used in multiple ways.

[0064] The individual user who activates or deactivates the linking may be, for example, an individual observer of the (virtual) equipment, particularly one who performs functional inspections or tests of the equipment, or an assistant who maintains the 3D engine or, in general, the software that implements the present invention. The 3D engine, i.e., the graphics engine, may be characterized, for example, as integrated or externally stored program code that is responsible for the calculation of the graphics interface in parallel with the actual program.

[0065] A virtual object can be characterized, for example, as a functional and / or structural part of a 3D model. Examples include static parts such as the shoulder ring or boundary wall of an isolator as a specialized (pharmaceutical) piece of equipment, or movable parts such as a transfer port or rapid transfer port ("quick change system," abbreviated RTP, alpha-beta-port system) or airlock, or a functional station, such as a filling station, closing station, or material storage, or a manipulator. This list is not exhaustive, and other examples may be advantageously used.

[0066] A virtual object can be characterized, for example, as a rigid body composed of measurement points that are fixedly positioned relative to each other.

[0067] For example, changing a link might involve activating (or starting) and / or deactivating (or ending) the link. This makes it possible to establish or break the spatial connection between virtual objects and virtual bodies in virtual space. Since virtual bodies are linked to real-world models by 3D position measurement, and their changes in position and location are forcibly tracked in virtual space, this link can therefore result in either—when activated—the connection of the 3D model of the virtual object to reality, or—when deactivated—the separation of the virtual object from reality.

[0068] In a favorable configuration, linking a virtual object to at least one other virtual object may involve enforcing a desired positional relationship between the virtual object's position and / or location relative to the virtual object. This creates the impression that the virtual object is working in conjunction with a tactile real-world model. This can be used, for example, to test the feasibility of actual work processes in a 3D model. This enforcement can also be used as a simple means of transferring changes to the real-world model, such as ergonomic improvements, to the virtual object, especially when time constraints are present. This enforcement refers to all degrees of freedom of movement, or a subset of degrees of freedom, and boundary conditions in particular may be taken into consideration.

[0069] Generally, the position of a virtual object or real-world model can be described by three coordinates, for example, those of a selected point, particularly the center of gravity, the center, or other characteristic or special point. The position of a virtual object or real-world model can also be described, for example, by specifying the angle of alignment with respect to a rotation around the selected point that its position refers to, and / or by specifying the position of another point on the virtual object or real-world model that may be in a fixed relationship with the selected point. Orientation can be described, for example, by position and location.

[0070] For example, it might be conceivable to set the position and / or location of a virtual object to the position and / or location of a virtual object. This would enable tracking or linking without deviation.

[0071] Alternatively or additionally, enforcement could be triggered by executing a request, thereby enabling the exchange of virtual objects and / or real-world models during linking.

[0072] Alternatively or additionally, coercion may be envisioned to be carried out continuously, for example, repeatedly over time, preferably automatically. This would make it possible, for example, to make a virtual object follow a virtual object throughout the entire movement interval.

[0073] In a favorable configuration, it may be conceivable to deactivate the linking for a favorably defined or unspecified period. This would enable the alignment of reality models, particularly in relation to another reality model to which the virtual object is already linked, thereby aligning the reality model with the virtual world to which at least one virtual object belongs, in particular a virtual 3D model, and allowing the other reality model to continue to remain aligned with this virtual world.

[0074] In this case, it can be assumed that a deviation will be displayed between at least one virtual object and the virtual object when the linking is inactive. This can be used, for example, to move the real-world model to a desired position and thereby establish a desired relationship with the virtual object.

[0075] In a favorable configuration, the link between a virtual object and at least one other virtual object can be replaced with other links between the virtual object and other virtual objects. This allows for the reuse of the real-world model to test other virtual objects in the 3D model, eliminating the need to build a complete facility. This saves space and time for creating the real-world model and enables functional testing between users in remote or geographically separated locations. Costs for model manufacturing and assembly are also reduced. Furthermore, there are technical advantages; for example, the virtual model can be fitted with cross-sectional views or have holograms faded in to enhance understanding.

[0076] In a favorable configuration, if one link is replaced by another, it can be assumed that the 3D model is isometrically transformed until the virtual object and the other virtual object match at least within an acceptable range. This allows the user to move within the virtual world without changing their position in the real world. This makes it easy to use already constructed real-world structures for further testing without reconstruction. In this case, it becomes easy to adjust existing real-world models to the position and / or location of new virtual objects, as described above.

[0077] Such isometric transformations may include, for example, rotation and / or displacement. Thus, this transformation simply corresponds to any change in position in the real world.

[0078] Preferably, only isometric transformations that preserve orientation, i.e., those that do not mirror the image, are permitted. This prevents changes that do not have a corresponding counterpart in the real world.

[0079] In a favorable configuration, the position of the real-world model may be advantageously changed, either manually or automatically, until the virtual object in question coincides with at least one other virtual object. This allows the real-world model to be positioned so that the tactile impression from interacting with the real-world model matches the visual impression when observing the virtual object.

[0080] For example, a shoulder ring can be matched to a 3D model by first activating a link with the target virtual object. Then, another shoulder ring or other parts, such as a door or an operated functional unit, can be modified as a real-world model so that this real-world model matches the corresponding virtual object, and the thus positioned and / or aligned real-world models can be embedded into the virtual world.

[0081] In this case, it can be assumed that the link between the virtual object and other virtual objects is then activated. This would then allow the virtual object to track the behavior of the real-world model. As a result, the individual observer of the virtual world would receive tactile or sensory information in addition to visual sensory information, giving them the sensation of actually moving or manipulating the virtual object.

[0082] In a favorable configuration, multiple virtual objects may be linked to each virtual object in a 3D model, and each link can be modified independently of the others, particularly activated and / or deactivated. This would allow for the independent configuration of various reality models, such as two shoulder rings, and / or the selection of individual reality models as movable parts of equipment that need to follow virtual objects, while other reality models become, or remain, available as reference points in the real world to which the virtual world can dock.

[0083] In a favorable configuration, it can be assumed that the coordinate updates of at least one virtual object will be output. This can be used, for example, to process design data for 3D models. This allows for simple adjustments and necessary changes to the equipment from an ergonomic and / or process economic standpoint, without the need to create a new, complete model in the real world.

[0084] In a favorable configuration, it could be envisioned as equipment for the pharmaceutical sector, preferably for filling pharmaceuticals into containers and / or protective chambers, preferably in cooperation with isolators. Here, regulatory requirements and / or ergonomic boundary conditions in the workflow can be easily tested.

[0085] Alternatively or additionally, the above problem can be solved by a method for functional testing of equipment, which represents the equipment as a virtual 3D model from virtual objects, prepares a real-world model for at least one virtual object, aligns the virtual object to the real-world model by 3D position measurement over time, links at least one virtual object to the virtual object and brings it into a desired positional relationship with the virtual object, in this case creating a real-world model corresponding to at least one virtual object, attaching identifiable features, particularly markers, for 3D position measurement, and saving the correspondence between the identifiable features and at least one virtual object. This makes it possible to easily create and incorporate details of equipment that are relevant to testing and for which physical interaction is desired. Identifiable features can be easily used to create virtual objects to be linked to objects.

[0086] This embodiment can be advantageously combined with the embodiments described above. For example, by activating linking, a reality model prepared for use with markers can be easily made available in the manner according to the present invention. The markers can be implemented, for example, advantageously as two-dimensional or three-dimensional markers.

[0087] Preferably, the real-world model is created using an additive method, particularly from CAD data or other virtual object data. This allows the 3D model to be realized in as much detail as possible, thereby enabling a tactile experience of even the finest details. Using the real-world model and directly markering it is also an advantageous method for obtaining an even better tactile experience.

[0088] Alternatively, virtual objects can be derived from 3D scans of real-world models. This allows for the direct use of a manufacturer's prototype or sample even when CAD data is unavailable, and / or avoids the hassle of 3D printing complex objects.

[0089] In a favorable configuration, it can be assumed that identifiable features are formed at predetermined positions in a real-world model. This makes it possible to quickly and easily capture or create a correspondence between a virtual object that can represent a feature and a virtual object that can record the position of that feature.

[0090] In a favorable configuration, it may be conceivable to measure the position of at least one feature formed on the reality model. This can be done, for example, using a 3D camera. This measurement makes it possible to attach arbitrary markers. This can simplify the preparation of the reality model for use.

[0091] In advantageous configurations, it may be assumed that the operator wears gloves and / or hand tracking devices (smart gloves, meta gloves, motion capture gloves, finger tracking devices). Alternatively, direct hand tracking is also possible. Hand detection is beneficial in realistically replicating the manipulation actions in the virtual world. Isolators or shoulder gloves can also be used, thereby creating a realistic imitation of the physical limitations that may arise, for example.

[0092] In this case, it can be assumed that the 3D positions, particularly the positions and locations, of one or more fingers and / or one hand and / or one arm are repeatedly determined. This allows for easy integration of the hands and / or arms that perform operations and / or require collision testing into the virtual world.

[0093] For example, this can be achieved by detection of gloves and / or hand tracking devices, as previously mentioned. This makes it possible to virtually represent realistic operational actions.

[0094] In a favorable configuration, a 3D model represents the shoulder ring, and a real-world shoulder ring is positioned in its place, particularly during a preceding setup process, and / or it is assumed that the operator puts their arm through the shoulder ring, preferably with an operating glove attached to the shoulder ring.

[0095] In a favorable configuration, the observer's shooting posture may be defined relative to the shoulder ring. This allows the observer's position and location to be used as a reference for displaying the virtual object.

[0096] In a favorable configuration, at least one virtual object may be assumed to be a transfer port or airlock door. Other details of equipment that need to be operated during use, such as air samplers, agar plates, sampling devices, filling stations, pump bodies, hoses, and generally semi-fixed (e.g., whose movement is restricted by joints or guides) or freely movable equipment components may also be used.

[0097] Generally, real-world models can be classified into those that are movable relative to a reference point, such as the boundary of the facility, and / or an access point, particularly a shoulder ring, and those that are not.

[0098] In a favorable configuration, it can be assumed that a 3D model has another virtual object, another real-world model is prepared for that virtual object, and that the real-world model is movably positioned relative to another real-world model. This allows individual real-world models to be used as reference points connecting the virtual world and other real-world models, for highly accurate positional manipulation within a virtual world aligned with reality.

[0099] In a favorable configuration, the reality model may be assumed to be at least partially immobile and / or at least partially movable with respect to the facility boundaries. This allows the boundaries to be used as reference points or reference surfaces for establishing a correspondence between the virtual world and the real world.

[0100] Examples of at least partially movable components of the equipment that can be advantageously used as a real-world model include the shoulder ring and / or door frame of the transfer port, and / or the door, particularly the transfer port door, or the door of the airlock, and / or the interlock, and / or the petri dish, the air sampler, at least one agar plate, the sampler, the pump body, and at least one hose, but this list is not exhaustive.

[0101] To address the above challenges, a method has been proposed, possibly possessing unique inventive qualities, for functional testing of equipment, as described above, which includes the steps of: preparing CAD data of the equipment; creating at least one real-world model of at least a portion of the CAD data; setting at least one real-world model in a 3D measuring device; and processing at least the 3D measurement data of the 3D measuring device to display a virtual 3D model created from the CAD data. This makes it possible to perform functional testing of complex equipment based on a tactile impression of the real-world model with minimal material costs.

[0102] In a favorable configuration, the field of view of the head-mounted display may be determined, advantageously using 3D measuring equipment. This would allow the individual observer and / or operator to be embedded in a virtual scene of a virtual object in a CAD model.

[0103] In a favorable configuration, the display of the 3D model can be assumed to be performed in relation to the field of view of the head-mounted display. This makes it possible to observe the 3D model as if it were real from the observer's position.

[0104] In a favorable configuration, it can be assumed that changes to at least one real-world model are automatically tracked to the 3D model. This makes it easy to track changes in the real world within the virtual world in which the 3D model is defined. This gives the individual observer the impression that they can change the virtual object through changes in the corresponding real-world model—which can be experienced tactilely—for example, linked virtual objects. This makes it possible to perform functional testing of complex industrial equipment, such as pharmaceutical equipment, with minimal use of materials, time, and space.

[0105] In this case, or generally speaking, it can be assumed that modified design data is generated and output from the modified 3D model or modified virtual object. This makes it possible to define the design changes discovered during functional testing.

[0106] In an advantageous configuration, the 3D measuring device may be transported to a fixed measuring structure before setup. This allows the device according to the present invention to be easily transported to remote locations, for example, for on-site functional testing.

[0107] The problem mentioned at the beginning can be solved, either alternatively or additionally, by an apparatus for functional testing of equipment, wherein the equipment exists as a virtual 3D model, and the apparatus comprises a 3D measuring device, at least one real-world model of the virtual object of the 3D model, equipment for automatically incorporating virtual objects detected using the 3D measuring device into the 3D model relative to the real-world model, equipment for automatically making the virtual object track the virtual object together with the virtual object, and equipment for visually displaying the 3D model, particularly a 3D engine. This provides a means for testing virtually representable equipment that can be experienced through touch.

[0108] In a favorable configuration, it may be conceivable that means are formed for activating and / or deactivating the link between a virtual object and at least one other virtual object. This would allow the operator to easily decide how to connect the virtual world of the 3D model with reality.

[0109] In a favorable configuration, a head-mounted display could be assumed to generate a field of view for the 3D model. This would allow for a more natural observer position.

[0110] In an advantageous configuration, it is conceivable that the equipment for generating a field of view of the 3D model would supply the measurements of a 3D measuring device, particularly to the head-mounted display mentioned earlier. This would allow the 3D model to be viewed from the position of a human observer.

[0111] In a favorable configuration, it is conceivable that equipment could be formed to convert the 3D model to an isometric size relative to the field of view. This would allow for the conversion of the 3D model or a virtual change in the position of the equipment operator.

[0112] In an advantageous configuration, the means for carrying out the method according to the present invention may be formed particularly as described in the preceding and / or subsequent claims, thereby demonstrating a method for realizing the described method.

[0113] In an advantageous configuration of the described apparatus, it may be conceivable that an apparatus is formed that includes equipment for motor-driven adjustment of at least one real-world model. This would enable precise and / or automatic and / or remote adjustment.

[0114] Alternatively or additionally, in an advantageous configuration of the described apparatus, it may be envisioned that the apparatus be equipped with a device for advantageously and automatically determining the position and / or positional deviation of a virtual object from its corresponding virtual object. This would enable computer-assisted support for the acquisition.

[0115] Alternatively or additionally, in a favorable configuration of the described apparatus, it may be assumed that the apparatus is equipped with a device for motor-driven adjustment of at least one reality model until the deviation of the virtual object's position and / or location from the corresponding virtual object, favorably automatically detected, is within an acceptable range. This would enable fully automatic or semi-automatic capture of the reality model.

[0116] Alternatively or additionally, in a favorable configuration of the described apparatus, it may be envisioned that the apparatus incorporates equipment for virtual ray collision inspection. This may enable the simulation of photointerrupters, such as optical grids. For example, the corresponding photointerrupter module can be saved as a virtual object in a 3D model, and rays are automatically generated and monitored based on the position and location of the photointerrupter module.

[0117] The present invention will now be described in more detail using examples, but is not limited to these examples. Further examples can be obtained by combining the features of individual or more claims with each other and / or with the individual or more features of the examples. [Brief explanation of the drawing]

[0118] [Figure 1] This figure shows a head-mounted display according to the present invention and its use in a device for visually displaying 3D data. [Figure 2] This is a highly schematic diagram of a device for functional testing of equipment, equipped with a head-mounted display and a device for visually displaying 3D data. [Figure 3] Figure 2 shows a more realistic, individualized representation of the actual device, along with two adjustable shoulder rings and a 3D measuring device. [Figure 4] This is another diagram showing the arrangement shown in Figure 3, but more realistically, from a side view of the glove. [Figure 5] This is another diagram that shows a more realistic, individual representation of the device shown in Figure 2, along with the movable door. [Figure 6] Figure 2 shows the virtual objects of the 3D model of the device shown, adjusted to achieve the desired positional relationships and establish associations. [Figure 7] Figure 2 shows a modified real-world model of the device to achieve the desired positional relationship with the corresponding virtual object and establish the linkage. [Figure 8] This diagram schematically shows 3D models in various orientations for virtual position changes of an individual observer. [Figure 9] This figure shows a detailed plan view of a 3D model of pharmaceutical equipment. [Figure 10] This diagram shows a schematic representation of how the display of the tracked virtual scene is synchronized by compensating and adjusting the intrinsically identified shooting posture and the extrinsically identified shooting posture.

[0119] Figure 1 shows a device for functional testing of equipment, represented as a single unit (1).

[0120] The virtual 3D model 2 of equipment 1 is composed of virtual objects 3 and 4 and is prepared in the first virtual space 5.

[0121] In this case, virtual 3D model 2 is derived from CAD data of industrial equipment, which is not shown in detail in the diagram.

[0122] This equipment is shown only in a very schematic manner using virtual objects 3 and 4 to illustrate the operating principle of the method according to the present invention.

[0123] In fact, this equipment includes more components. A preferred application of the present invention is that the equipment is a pharmaceutical facility designated within a protective chamber or isolator to perform a specific process, such as transferring / filling pharmaceuticals or assembling dispensing devices for pharmaceutical formulations.

[0124] The first virtual space 5 contains numerous virtual objects 3 and 4. There may also be another virtual object 63 present.

[0125] For some of these virtual objects 3 and 4, reality models 7 and 8 corresponding to the real space 6 are installed. For another virtual object 63, a reality model is not installed.

[0126] These reality models 7 and 8 are fixed to special stands 9 so that they remain in desired positions within the real space 6.

[0127] These stands 9 are not part of the virtual 3D model 2, because in this model, virtual objects 3 and 4 are attached to other design details, such as glass walls or boundary walls or tables. However, these glass walls, boundary walls, and tables do not exist in real space 6.

[0128] Furthermore, the reality models 7 and 8 are equipped with additional markers 10, which do not have corresponding markers for the virtual objects 3 and 4.

[0129] These markers 10 are designated for position and location by a 3D measuring device 11 installed in real space 6. The 3D measuring device 11 detects these markers 10 and forms their positions in a second virtual space 12. Therefore, in this second virtual space 12, there are no virtual objects 3 and 4, only the positions of the markers 10, which are grouped into virtual objects depending on which of the real models 7 and 8 they are fixed to.

[0130] Between the first virtual space 5 and the second virtual space 12, a correspondence relationship 15 is established that connects spatial points in the first virtual space 5 to corresponding spatial points in the second virtual space 12, and vice versa.

[0131] Furthermore, in the real space 6, for example, a head-mounted display 16 is positioned on the head of an individual user, which is not shown in detail in the diagram.

[0132] This head-mounted display 16, for example, VR goggles, also has markers 17 that detect its position and location in real space 6 and display it in a second virtual space 12, although these are not necessarily represented in the virtual 3D model 2.

[0133] In other words, in the second virtual space 12, there exists a virtual object 18 that represents the head-mounted display 16 using markers 17.

[0134] The head-mounted display 16 generates a field of view 19 that provides the individual user with a three-dimensional visual impression, as is well known. This visual impression 20 is calculated based on the aforementioned correspondence 15 between the measured position and orientation of the head-mounted display 16, i.e., the shooting posture, and the positions and orientations of the virtual objects 3 and 4.

[0135] In this case, virtual objects 3 and 4 are used for the visual impression 20, and as a result, for example, stand 9 and markers 10, 17 are not displayed.

[0136] In this case, the field of view 19 of the head-mounted display 16 is defined by the position in which the head-mounted display 16 is attached to the user's head and its natural field of view.

[0137] In order to generate a visual impression 20, in Figure 1, field of view data 21, position and location data 22 from the second virtual space 12, and object data 23 from the first virtual space 5 are processed with respect to each other in a manner known to the public.

[0138] To determine the field of view 19, the head-mounted display 16 may be equipped with its own sensors 46, which are not shown in detail in the diagram.

[0139] For example, the head-mounted display 16 may be equipped with multiple cameras that detect the individual user's field of view and calculate position and / or changes in these values ​​based on spatial features such as room corners and edges. For this purpose, it is technically known that the head-mounted display 16 can be equipped with sufficient computing power, thus eliminating the need for external computing power.

[0140] Since the real-world models 7 and 8 are installed in the physical space 6, even if the head-mounted display 16 is a VR goggle type and therefore cannot see the real-world models 7 and 8, the individual user in the physical space 6 can touch and verify these real-world models 7 and 8.

[0141] To achieve this correspondence between tactile and visual impressions 20, the reality models 7 and 8 are initially aligned with the virtual objects 3 and 4 assigned to them.

[0142] Thus, the method according to the present invention makes it possible to activate the link 24 between virtual objects 3 and 4 on the one hand and the corresponding virtual objects 13 and 14 on the other hand. As a result of this link 24, virtual objects 3 and 4 (virtual object 4 in this example) will be linked with virtual objects 13 and 14 (virtual object 14 in this example) based on the correspondence relationship 15.

[0143] Accordingly, when the reality model 8 changes due to the action 25, this action 25 is reflected in the second virtual space 12 by the 3D measuring device 11.

[0144] As already mentioned, direct tracking can also be modified depending on the situation so that only specific axes are followed and specific axes are "fixed" in order to satisfy boundary conditions. Thus, through linking 24, the virtual object 4 will move similarly within the first virtual space 5 according to the correspondence 15.

[0145] As a result, an apparent action 26 occurs in the visual impression 20.

[0146] Therefore, the individual user who generated action 25 through the operation of the real-world model 8 gets the impression that the virtual object 4 is performing the apparent action 26 based on that operation.

[0147] If the user or another party deactivates the link 24, manipulating the reality model 8 will not change the visual impression 20. This can be determined individually for each object and does not necessarily apply to all objects.

[0148] Furthermore, a head-mounted tracking device 27 is placed in the real space 6. The aforementioned individual user wears this head-mounted tracking device 27 like a glove, thereby performing the aforementioned operations on the real-world models 7 and 8. The markers 28 of the head-mounted tracking device 27 are also displayed as virtual objects 29 in the second virtual space 12 by the 3D measuring device 11, resulting in the appearance of a hand replica 30 in the visual impression 20.

[0149] The head-mounted display 16 can be connected wirelessly or wired, and advantageously to a fixed processing unit, to transmit measurement data of the shooting posture and / or image data of the visual impression 20.

[0150] To generate the visual impression 20, a 3D engine, not shown in detail in the figure, is used. This forms the equipment for generating the field of view for the 3D model 39. In the first virtual space 6, equipment is in operation for isometrically transforming the 3D model 39 with respect to the field of view 19, as will be described in more detail below with reference to Figure 8.

[0151] Figure 2 shows the real-world model 7 from Figure 1 viewed from the front, making it appear somewhat more realistic, and Figure 3 shows this arrangement viewed from the side.

[0152] The 3D measuring device 11 is only partially shown in both figures.

[0153] In fact, in real space 6, typically three or more cameras 31 are positioned on a special support structure 32.

[0154] The position and location of this camera 31 are precisely known.

[0155] To facilitate on-site assembly, it is conceivable that the assembled support structure 32 may be stored in a container or packaged as a whole for transport to the site of use. The present invention has the advantage that there is no need to transport the model structure of the equipment. Rather, the assembled support structure 32 can be used as a mobile measuring structure.

[0156] The real-world model 7 has two shoulder rings 33, 34 known as isolators for attaching isolator gloves 49. In this equipment, the shoulder rings 33, 34 are often formed of glass walls. Thereafter, the positions of the shoulder rings 33, 34 indicate the positions of the equipment boundaries, such as a protective room or a control room.

[0157] These shoulder rings 33 and 34 can be positioned horizontally and vertically on the stand 9.

[0158] The hand tracking device 27 is connected to either of the shoulder rings 33 or 34 to form an isolator glove 49.

[0159] Each time, the other shoulder rings 33 and 34 may also have a hand tracking device 27 corresponding to the user's other hand. When in use, the user stands in front of the shoulder rings 33 and 34, inserts both arms into the respective shoulder rings 33 and 34, and thereby operates the respective hand tracking devices 27 with both hands.

[0160] In further embodiments, instead of the hand tracking device 27, an optical recognition algorithm is used to detect the position and shape of the user's hand.

[0161] The shape of the hand can also be detected by special sensors located on the 35th finger, but this is not explained in detail here.

[0162] For example, the METAGLOVES "Quantum" head-mounted tracking device 27 from Manus (Floor 9, Kennedyplein 200, NL-5611 ZT Eindhoven) is known and usable.

[0163] Figure 4 shows a more realistic representation of the reality model 8 derived from Figure 1.

[0164] The real-world model 8 can be seen to consist of a movable part 36 and a fixed part 37.

[0165] This figure illustrates a door 47 that can be mounted on the wall of an isolator, either as a transfer port 48, a rapid transfer port, or in an airlock.

[0166] Details of this door 47 are irrelevant to the description of the present invention; the only important point is that this transfer port 48 has a movable portion 36 and a fixed portion 37.

[0167] This movable part 36 can be grasped and opened by the user to perform the operation 25 described in Figure 1.

[0168] As a result, in the visual impression 20, a change in the virtual object 4, which consists of the corresponding multiple parts, is displayed, and this corresponds to the opening of the door 47.

[0169] This allows the individual user to, for example, verify whether they can reach and operate the door 47 inside the isolator using the shoulder rings 33, 34.

[0170] Figure 5 shows the linking setup 24 described above, using an example with shoulder rings 33 and 34.

[0171] This setup is performed to ensure that the shoulder rings 33 and 34 are precisely matched to their virtual counterparts before the individual user begins their planned test.

[0172] In the front row is the real-world model 7, behind it is the corresponding virtual object 13 shown as a dotted line, and the corresponding virtual object 3 is shown as a solid line.

[0173] This diagram has been chosen for simplification. In reality, the real-world model 7, virtual object 3, and virtual object 13 exist in different spaces 6, 12, and 5.

[0174] The projection line 38 between the real-world model 7 and the virtual object 13 symbolizes the 3D measurement performed by the 3D measuring device 11.

[0175] When the request is executed, virtual object 3 is moved to a position corresponding to the position of virtual object 13 according to the correspondence relationship 15.

[0176] To support this process, it may be assumed that both virtual object 13 and virtual object 3 are reproduced within the visual impression 20.

[0177] For the right shoulder ring 34, this process has already been completed in Figure 5.

[0178] As a result, by performing the above request, a desired positional relationship is enforced between the position and location of the virtual object 3 relative to the virtual object 13.

[0179] In Figure 6, the real-world model 7 is displayed on the front side, and the corresponding virtual object 3 is displayed on the back side.

[0180] The projection line 38 once again demonstrates the effectiveness of the 3D measuring device 11.

[0181] To simplify the display, virtual object 13 is not shown.

[0182] In visual impression 20, the individual user recognizes that virtual object 3 and the similarly displayed virtual object 13 do not match.

[0183] The user or assistant can then change the stand 9 so that the virtual object 13 is positioned above the virtual object 3. After that, the linking 24 can be activated. This procedure has the effect that, compared to the procedure shown in Figure 5, activating the linking 24 does not change the virtual object 3, and in particular does not change the other virtual object 4.

[0184] Since the shoulder rings 33 and 34 remain fixed, there is no need to activate the link 24, and it can be kept permanently inactive.

[0185] By forming individual links 24 between virtual object 14 and virtual object 4, it becomes possible to persistently deactivate individual links 24 for individual virtual objects 13 and 14. This makes it possible to perform such adjustments independently of other virtual objects 3 and 4.

[0186] In this case, the reality model 7 can be motor-driven to adjust, for example, the automatically detected deviation 64 of the position and / or position between the virtual objects 13, 14 (not shown, see Figure 5) and the corresponding virtual object 3 until it falls within an acceptable range.

[0187] To this end, the illustrated apparatus includes a device for motor-driven adjustment of at least one reality model 7,8; ​​a device for advantageously and automatically detecting the position and / or positional deviation 64 between the virtual objects 13,14 and their corresponding virtual objects 3,4; and a device for motor-driven adjustment of at least one reality model 7,8 until the advantageously and automatically detected position and / or positional deviation 64 between the virtual objects 13,14 and their corresponding virtual objects 3,4 falls within an acceptable range.

[0188] Figure 8 shows a schematic diagram of a 3D model 39 of industrial equipment, such as pharmaceutical equipment.

[0189] The 3D model here exemplifies a wall section 40 with three access ports 41, 42, and 43 arranged within it.

[0190] In this case, each of these access ports 41, 42, and 43 may include, for example, a virtual counterpart to a pair of shoulder rings 33 and 34.

[0191] The left-hand diagram of Figure 8 shows the positions used to match the real-world model 7 of the access port with its virtual counterpart, or to bring it to a desired positional relationship, using the method shown in Figure 6 or 5.

[0192] This state is used to test and verify whether the equipment can access the 3D model 39 through this access port 41.

[0193] Then, when testing other access ports 42, 43, the 3D model 39 can be isoscaled, in particular with respect to the virtual object 13, so that another access port 42 can be matched with the real model 7 or its virtual object 13. This changes the position of the 3D model 39 in the first virtual space 6, or the correspondence 15 between the first virtual space 6 and the second virtual space 12. Then, the equipment can be tested through this access port 42. This situation is reflected in the central image portion of Figure 8.

[0194] The image on the right shows another reality model 44 that can be placed in a separate reality space 6 or within the same reality space 6.

[0195] Therefore, this other reality model 44 has additional virtual objects 14 that are generated when the other reality model 44 is placed in another (real) space by either the same 3D measuring device 11 or another 3D measuring device 11.

[0196] This method allows two individual users to test the equipment simultaneously, and these users do not need to be in any physical location relative to each other.

[0197] These two individual users can, for example, perform a (virtual) handshake, or confirm exchanges and mutual interaction.

[0198] When switching from the situation on the left of Figure 8 to the situation in the center, the link 24 between access port 41 as a virtual object and virtual objects 13 and 14 of the real-world model 7 is deactivated and replaced by the link 24 between the same virtual object 13 and another access port 42 as a virtual object. When this link 24 is created, a tolerance range is defined within which the link 24 can be accepted without having to modify the virtual object 3.

[0199] Individual users can, for example, verify the filling of pharmaceuticals in a protective room, in this case, an isolator.

[0200] The configuration described here allows for the execution of a method for functional testing of equipment composed of numerous virtual objects 3 and 4 along with a virtual 3D model 39. For each selected virtual object 3 or 4, realistic real-world models 7 and 8 are prepared, each as realistic as possible. For example, these real-world models 7 and 8 are generated as 3D prints from the virtual objects 3 and 4. In this case, the similarity must match so that the visual impression when touching the real-world models 7 and 8 matches the visual impression when viewing the virtual objects 3 and 4.

[0201] The 3D measuring device 11 then repeatedly measures the real models 7 and 8 over time in order to appropriately position the virtual objects 13 and 14 within the second space 12.

[0202] In this case, the real-world models 7 and 8 are marked with markers 10 to enable feature-based identification. When linking 24 is activated, these identified features, i.e., the correspondence 15 between the markers 10 and the virtual objects 3 and 4, is saved in order to track the virtual objects 3 and 4, thereby making it possible to move the virtual objects 13 and 14 together with the virtual objects 3 and 4.

[0203] In this case, virtual objects 3 and 4 may include predetermined positions, such as drill holes where markers 10 are attached. This makes it easy to position markers 10 in the positions of real-world models 7 and 8, thereby making it easy to locate and align virtual objects 13 and 14 with virtual objects 3 and 4.

[0204] Alternatively, marker 10 can be arbitrarily applied to the real-world models 7 and 8, and then the formed features, particularly the position of marker 10, can be measured on the real-world models 7 and 8.

[0205] Therefore, the method for functional testing of equipment 1 can begin by first preparing CAD data of the equipment by appropriate design, processing this CAD data into a 3D model 39, forming real models 7 and 8 of selected details from this 3D model 39 or a part of the CAD data, particularly using 3D printing or an alternative manufacturing method, placing these real models 7 and 8 on a 3D measuring device 11, and displaying the 3D model 39 from the observer's position with a visual impression 20.

[0206] If, for example, changes to the 3D model 39 are required during testing according to the procedure shown in Figure 5, these changes to the 3D model 39 can be output as modified design data at the end of the test. This may apply, for example, if it is found that a particular detail of the 3D model 39 is ergonomically unsuitable and requires design modification.

[0207] During the testing process, individual users can manipulate the real-world models 7 and 8 as they see fit, thereby observing the impact of these actions on the visual impression 20 at the level of virtual objects 3 and 4.

[0208] This is made possible by linking 24, which forces virtual objects 3 and 4 to follow the corresponding virtual objects 13 and 14. In the presented embodiment, this is achieved by a device for automatically making virtual objects 3 and 4 follow the corresponding virtual objects 13 and 14, and a 3D engine as a device for visually displaying the 3D model 39.

[0209] Figure 7 schematically shows a head-mounted display 16 equipped with a 3D measuring device 11. This configuration can be used in the configuration shown in Figure 1, but it can also be operated independently of the configuration shown in Figure 1.

[0210] In this head-mounted display 16, the position and orientation of the field of view 19 are repeatedly determined over time, and in the visual impression 20, for example, linked information, such as virtual objects 3 and 4, is faded in in the manner described, or other position-related information such as warnings or work instructions.

[0211] In this case, the field of view 19 is detected using the means inherent to the head-mounted display 16, i.e., an integrated means. As a result, this field of view 19 can only be created with respect to the reality models 7,8 that are within the field of view 19.

[0212] To enable alignment of the head-mounted display 16 with respect to other reality models outside the field of view 19, the position and location of the head-mounted display 16 are exogenously determined based on markers 10 using a 3D measuring device. This information is then linked to the field of view 19, thereby enabling accurate, nearly uninterrupted, or completely uninterrupted detection of the position and location of the head-mounted display 16, and ultimately allowing for a continuous visual experience in the visual impression 20.

[0213] In isolators, particularly in controlled environments, controlled airflow is often used to control and disperse potential contaminants, thereby protecting sensitive areas from contamination.

[0214] Furthermore, the present invention makes it possible to calculate such airflow as streamlines and, in particular, to display the changes in streamlines during operation 25 as a visual impression 20.

[0215] Figure 9 shows another virtual 3D model 39 used in the present invention. This is a pharmaceutical facility comprising a wall section 40, access ports 41, 42, 43, 50, a measuring station 51 (for example, for measurements by agar plate and / or particle measurement), a supply section 52 from an automated device, a discharge section 53 to another automated device, a photointerrupter module 54 with a ray 55 for monitoring unexpected and / or unauthorized access to the access ports 41, 42, 43, 50, a sorting pot for preparing stoppers or other packaging or components of a medical dispensing device in the correct position, a material stock 57 for a sorting pot 56 that can be replenished, for example, through a pure transfer port 48, a transport and processing area 58 for processing (filling and sealing) containers (such as vials), a filling station 59 for filling containers, an individualization station 60 for individualizing bundles of containers supplied by the supply section 52, an installation station 61 for closing containers with the aforementioned stoppers, and a management station 62 for managing filled and sealed containers.

[0216] Individual components require tactile contact with the individual user during functional testing and are therefore used as real-world models 7,8. For example, this is not necessary for the sorting pot 56—a virtual object 63 suffices here. In contrast, it is advantageous to use real-world models 7,8 for the filling station 59 and the measuring station 51. These may be actual products rather than 3D prints as already described. This can be simpler for complex structures.

[0217] The photointerrupter module 54 can also exist as a real-world model 7,8, but it may not have any functionality, for example. In the virtual world, a ray 55 is simulated, and it is checked whether the individual user has virtually blocked that ray 55, and a signal is generated if necessary.

[0218] Therefore, the method for functional testing of equipment 1 proposes creating real-world models 7 and 8 from the details of a virtual 3D model 39 of the equipment, and detecting their spatial positions and arrangements during functional testing using a 3D measuring device 11. These specific positions and locations can be used to track virtual objects 3 and 4 of the 3D model 39 in order to display a visual impression 20 in virtual reality by manipulating the real-world models 7 and 8.

[0219] Figure 10 shows a schematic diagram for synchronizing the followed virtual scene. The head-mounted display 16, as described above, has means for intrinsically determining the shooting posture 65. For example, for this purpose, an accelerometer can be configured in a manner known to the present day, or the optical flow in the recorded video sequence can be evaluated. Typically, a method is implemented to detect changes in the shooting posture and integrate them into the shooting posture.

[0220] The intrinsically determined shooting posture 69 is transmitted to the display means 67 of the head-mounted display 16, thereby causing the virtual scene 72 to follow the operation of the head-mounted display 16.

[0221] The shooting posture 70 is determined externally using the 3D measuring device 11. These are then supplied to the compensation adjustment means 66.

[0222] Furthermore, the compensation adjustment means 66 also acquires an intrinsically identified shooting posture 69. If the deviation is too large and / or if there is a temporal or manual specification, the display means 67 generates a synchronization 71 to match or associate the virtual scene 72 with the extriantsically detected shooting posture 70.

[0223] In further embodiments not shown, a 3D measuring device (or part thereof) is attached to a head-mounted display 16. In this case, markers or other identifiers may be fixedly positioned in space as reference points, for example. This also enables the combination of extrinsic measurement of the shooting posture with intrinsic measurement of the shooting posture.

[0224] Therefore, it is proposed that the head-mounted device 16 be provided with at least one marker 10, 17 on its exterior to enable the shooting posture of the head-mounted device 16 (Figure 1). [Explanation of Symbols]

[0225] 1. Equipment for functional testing of equipment 2. Virtual 3D Models 3. Virtual Objects 4. Virtual Objects 5. The First Virtual Space 6. Real space 7. Reality Model 8. Reality Model 9 Stands 10 markers 11 3D measuring device 12. The Second Virtual Space 13 Virtual Objects 14 Virtual Objects 15 Correspondence 16. Head-mounted displays 17 Markers 18 Virtual Objects 19 Field of view 20 Visual Impression 21. Field of View Data 22 Position and location data 23 Object Data 24. Linking 25 operations 26. Apparent behavior 27 Hand Tracking Devices 28 markers 29 Virtual Objects 30 hand copies 31 Camera 32 Support structure 33 Shoulder Ring 34 Shoulder Ring 35 fingers 36 Moving parts 37 Fixed part 38 Projection line 39 3D models 40 Wall 41 Access points 42 Access points 43 Access points 44 Alternative Reality Models 45 Streamline 46 sensors 47 doors 48 Transfer Port 49 Isolator Gloves 50 Other access points 51 Measurement Station 52 Supply section 53 Discharge section 54 Photointerrupter Module 55 Rays of light 56 sorting pots 57 Material Stock 58 Transportation and Processing Areas 59 Filling Station 60 Individualization Stations 61 Installation Stations 62 Management Stations 63 Another virtual object 64 deviation 65 Means for intrinsically detecting the shooting posture 66 Compensation adjustment means 67 Display means 68. Change of position 69 Intrinsically detected shooting posture 70 Shooting posture detected by external factors 71 Synchronization 72 Virtual Scenes

Claims

1. A head-mounted display (16), particularly a VR, AR, and / or XR goggle, comprising means for externally identifying the shooting posture and means for internally identifying the shooting posture.

2. A head-mounted display (16) and / or VR, AR and / or XR goggles, particularly according to claim 1, comprising at least one marker (10, 17), in particular two or more markers (10, 17), for advantageously identifying the shooting posture.

3. The head-mounted display (16) according to claim 1 or 2, wherein means (66) are provided for compensating and adjusting the shooting posture (69) identified using the means (65) for intrinsically identifying the shooting posture with another shooting posture (70) which is advantageously identified extrinsically.

4. The head-mounted display (16) according to any one of claims 1 to 3, which is advantageous in that an externally identified shooting posture (70) can be input, and in particular in addition to an internally identified shooting posture and / or in addition to compensation adjustment with an internally identified shooting posture (69).

5. The head-mounted display (16) according to any one of claims 1 to 4, wherein the means (67) for displaying a virtual scene (72) is controllable based on the shooting posture (70) which is advantageously extrinsically identified or advantageously extrinsically identified.

6. An apparatus for reproducing at least one virtual object (3, 4), comprising a head-mounted display (16) according to any one of claims 1 to 5, and a 3D measuring device (11) for externally detecting at least one marker (10, 17), wherein means (66) for compensation adjustment are supplied from the 3D measuring device (11).

7. Use particularly for functional testing and / or advantageously during operation of pharmaceutical equipment, with a head-mounted display (16), particularly VR, XR and / or AR goggles, and advantageously a fixed and / or independently operating 3D measuring instrument (11) for creating a virtual view of a 3D model (39) of the equipment within the head-mounted display (16), wherein individual virtual objects (3, 4) correspond to real-world models (7, 8) detected by the 3D measuring instrument (11).

8. A method for the visual display of 3D data, comprising: repeatedly determining the field of view (19) of a head-mounted display (16), particularly VR and / or XR and / or AR goggles, over time; displaying information linked to the field of view (19) within the head-mounted display (16); and advantageously repeatedly identifying a shooting posture for the head-mounted display (16) that pre-sets the field of view (19), and compensating and adjusting it with the field of view (19).

9. The method according to claim 8, characterized in that the shooting posture is determined using a 3D measuring device (11) which is formed independently of and / or fixed to the head-mounted display (16), particularly including at least one camera (31).

10. The method according to claim 8 or 9, characterized in that the shooting posture is determined using a measuring instrument that is linked to and / or independent of the determination of the field of view (19).

11. The method according to any one of claims 8 to 10, characterized in that the intrinsic determination is performed using at least one interconnected sensor, particularly a camera (31) and / or motion and / or acceleration and / or position sensor.

12. The method according to any one of claims 8 to 11, wherein the shooting posture is measured using active markers (10, 17) on the head-mounted display (16).

13. The method according to any one of claims 8 to 12, wherein the shooting posture is measured using passive markers (10, 17) on the head-mounted display (16).

14. The method according to any one of claims 8 to 13, wherein the aforementioned shooting posture is measured using a fixed measuring device, particularly a fixed camera (31).

15. The method according to any one of claims 8 to 14, wherein the intrinsic determination of the shooting posture is measured using a camera (31) linked to the head-mounted display (16).

16. A method as part of a method according to any one of claims 8 to 15 for the visual display of equipment, comprising repeatedly over time intrinsically determining the field of view (19) of a head-mounted display (16), particularly VR and / or XR and / or AR goggles, calculating airflow, and advantageously displaying it visually in the head-mounted display (16) as linked 3D data.

17. The method according to any one of claims 8 to 16, wherein the 3D data includes a 3D model (39) of the equipment and / or the 3D data includes AR metadata relating to the equipment or components of the equipment.

18. A method according to any one of claims 8 to 17, using a head-mounted display (16), particularly the head-mounted display (16) according to claim 1, to generate a superposition of a real field of view (19) and a virtual display of the 3D data, or to block out the real environment.

19. An apparatus for visually displaying 3D data, more particularly an apparatus for use in the method described in any one of claims 8 to 18, comprising: a head-mounted display (16) provided for determining a linked field of view (19); a device for identifying the shooting posture of the head-mounted display (16); and a device for compensating and adjusting the shooting posture with the field of view.

20. The apparatus according to claim 19, further comprising a 3D engine for visually displaying the 3D data within the field of view (19).

21. The apparatus according to claim 19 or 20, which preferably includes a fixed 3D measuring device (11) for determining the aforementioned shooting posture.

22. The apparatus according to any one of claims 19 to 21, comprising: a 3D measuring device (11) or a device for generating a virtual space for virtual objects (13, 14) detected using the 3D measuring device (11); and / or a device for generating virtual spaces (5, 6) for visually displaying the 3D data; and / or a device for establishing correspondences (15) between the virtual spaces (5, 6), particularly a device for embedding and virtually displaying the virtual objects (13, 14) in the virtual spaces (5, 6).

23. The apparatus according to any one of claims 19 to 22, further comprising equipment for embedding the field of view (19) of the head-mounted display (16) into the virtual space (5, 6) and displaying it virtually.

24. The apparatus according to any one of claims 19 to 23, comprising an instrument for calculating airflow, and more particularly an instrument for visually displaying the airflow.

25. A method for functional testing of equipment, particularly comprising the method described in any one of claims 8 to 18, the method comprising: representing the equipment as a virtual 3D model (2) from virtual objects (3, 4); preparing real models (7, 8) for at least one virtual object (3, 4); aligning virtual objects (13, 14) with the real models (7, 8) by 3D position measurement over time; linking the at least one virtual object (3, 4) to the virtual objects (13, 14) and bringing them to a desired positional relationship; and modifying, particularly activating or starting, and / or deactivating or ending, the link (24) between the virtual objects (13, 14) and the at least one virtual object (3, 4) by the user.

26. The method according to any one of claims 8 to 18 or claim 25, characterized in that the linking (24) between the virtual objects (13, 14) and the at least one virtual object includes enforcing a desired positional relationship of the position and / or location of the virtual objects (3, 4) with respect to the virtual objects (13, 14), particularly at that position and / or location, and / or by performing a request and / or continuously.

27. The method according to any one of claims 8 to 18 or any one of claims 25 to 26, characterized in that the linking (24) is advantageously deactivated for a defined or undefined period of time, in particular, thereby displaying a deviation between the at least one virtual object and the virtual objects (13, 14) when the linking (24) is inactive.

28. The method according to any one of claims 8 to 18 or any one of claims 25 to 27, wherein the link (24) between the virtual objects (13, 14) and the at least one virtual object (3, 4) is replaced by another link between the virtual objects (13, 14) and the other virtual objects (3, 4).

29. If the linking (24) is replaced by another linking (24), the 3D model (39) is isometrically transformed, in particular rotated and / or displaced, until the virtual objects (13, 14) and the other virtual objects (3, 4) coincide to at least within an acceptable range, the method according to any one of claims 8 to 18 or any one of claims 25 to 27.

30. The method according to any one of claims 8 to 18 or any one of claims 25 to 29, wherein the position of the reality model (7, 8) is advantageously changed manually or automatically until the corresponding virtual object (13, 14) coincides with the at least one virtual object (3, 4) or the other virtual object (3, 4), and in particular, the linking (24) between the virtual object (13, 14) and the virtual object (3, 4) or the other virtual object (3, 4) is then activated.

31. The method according to any one of claims 8 to 18 or any one of claims 25 to 30, wherein a plurality of virtual objects (13, 14) are linked to each virtual object (3, 4) of the 3D model (39), and each of the links (24) is modified independently of each other, in particular by activating and / or deactivating.

32. The method according to any one of claims 8 to 18 or any one of claims 25 to 31, wherein the update of the coordinates of the at least one virtual object (3, 4) is output, in particular, for processing the design data of the 3D model (39).

33. A method according to any one of claims 8 to 18 or any one of claims 25 to 32, which is equipment for the pharmaceutical field for filling a container and / or a protective chamber, preferably in cooperation with an isolator.

34. A method for functional testing of equipment, particularly the method described in any one of claims 25 to 33 and / or the method described in any one of claims 8 to 18 or any one of claims 25 to 33, comprising: representing the equipment as a virtual 3D model (2) from virtual objects (3, 4); preparing real models (7, 8) for at least one virtual object (3, 4); aligning virtual objects (13, 14) to the real models (7, 8) by 3D position measurement over time and iteratively; linking the at least one object to the virtual object (13, 14) and bringing it into a desired positional relationship with the virtual object (13, 14); creating the real models (7, 8) corresponding to the at least one virtual object (3, 4) advantageously by an additive method; attaching identifiable features for 3D position measurement; and saving the identified features, particularly the correspondence (15) between markers (10, 17) and the at least one virtual object (3, 4).

35. The method according to any one of claims 8 to 18 or any one of claims 25 to 34, wherein the identifiable features are formed at predetermined positions on the reality model (7, 8).

36. The method according to any one of claims 8 to 18 or any one of claims 25 to 35, wherein at least one position of the feature formed on the reality model (7, 8) is measured.

37. The method according to any one of claims 8 to 18 or any one of claims 25 to 36, wherein the operator wears gloves and / or a hand tracking device (27), and / or the 3D position of one or more fingers and / or one hand and / or one arm, preferably the gloves or the gloves and / or the hand tracking device (27) or the hand tracking device (27) is repeatedly determined.

38. The method according to any one of claims 8 to 18 or any one of claims 25 to 37, wherein the 3D model (39) represents a shoulder ring, and actual shoulder rings (33, 34) are adjusted to the position of the shoulder ring, particularly in a preceding setup step, and / or the operator puts their arm through the shoulder rings (33, 34), preferably with an operating glove attached to the shoulder rings (33, 34).

39. The method according to any one of claims 8 to 18 or any one of claims 25 to 38, wherein the observer's shooting posture is defined with respect to the shoulder rings (33, 34).

40. The method according to any one of claims 8 to 18 or any one of claims 25 to 39, wherein the at least one virtual object (3, 4) is a door (47) of a transfer port (48) or has a door (47) of a transfer port (48).

41. The method according to any one of claims 8 to 18 or any one of claims 25 to 40, wherein the 3D model (39) has another virtual object (3, 4), another real-world model (7, 8) is prepared for the other virtual object (3, 4), and the real-world model (7, 8) is movably positioned for the other real-world model (44).

42. The method according to any one of claims 8 to 18 or any one of claims 25 to 41, wherein the other reality model (44) is at least partially immobile and / or at least partially movable with respect to the boundary of the equipment.

43. A method for functional testing of equipment, particularly the method described in any one of claims 25 to 42 and / or the method described in any one of claims 8 to 18 or any one of claims 25 to 42, the method comprising the steps of: preparing CAD data of the equipment; creating at least one real model (7, 8) of at least a portion of the CAD data; setting the at least one real model in a 3D measuring device (11); and processing at least 3D measurement data from the 3D measuring device (11) to display a virtual 3D model (2) created from the CAD data.

44. The method according to any one of claims 8 to 18 or any one of claims 25 to 44, characterized in that the field of view (19) of the head-mounted display (16) is preferably determined using the 3D measuring device (11).

45. The method according to any one of claims 8 to 18 or any one of claims 25 to 45, characterized in that the display of the 3D model (39) is performed in relation to the field of view (19) of a head-mounted display (16).

46. The method according to any one of claims 8 to 18 or any one of claims 25 to 45, characterized in that changes to at least one reality model (7, 8) are automatically tracked to the 3D model (39), and in particular, modified design data is generated and output from the modified 3D model (39).

47. The method according to any one of claims 8 to 18 or any one of claims 25 to 46, characterized in that the 3D measuring device (11) is transported to a fixed measuring structure before being set up.

48. The method according to any one of claims 8 to 18 or any one of claims 25 to 47, characterized by motor-driven adjustment of a real-world model (7, 8) and / or adjustment of the real-world model (7, 8) until the advantageously automatically detected deviation (64) of the position and / or location of the virtual objects (13, 14) from the corresponding virtual objects (3, 4) is within an acceptable range.

49. The method according to any one of claims 8 to 18 or any one of claims 25 to 48, characterized in that a virtual ray (55) is generated, and in particular, whether or not the virtual ray (55) is blocked is automatically checked.

50. An apparatus for functional testing of equipment (1), wherein the equipment exists as a virtual 3D model (2), and the apparatus comprises a 3D measuring device (11), at least one real-world model (7, 8) of the virtual objects (3, 4) of the 3D model (39), equipment for automatically incorporating virtual objects (13, 14) detected by the 3D measuring device (11) with respect to the real-world model (7, 8) into the 3D model (39), equipment for automatically making the virtual objects (3, 4) track together with the virtual objects (13, 14), and equipment for visually displaying the 3D model (39), particularly a 3D engine.

51. The apparatus according to claim 50, characterized in that means are provided for activating and / or deactivating the link between the virtual objects (13, 14) and the at least one virtual object (3, 4).

52. The apparatus according to claim 50 or 51, characterized in that a head-mounted display (16) is provided for generating a field of view (19) for the 3D model (39).

53. The apparatus according to any one of claims 50 to 52, characterized in that the device for generating a field of view (19) for the 3D model (39) supplies the measured values ​​of the 3D measuring device (11) to the head-mounted display (16) or the head-mounted display (16).

54. Use of the head-mounted display (16) according to any one of claims 1 to 5 in the method and / or apparatus according to any one of claims 19 to 24 or any one of claims 50 to 53.

55. The apparatus according to any one of claims 18 to 24 or any one of claims 49 to 52, comprising means for carrying out the method according to any one of claims 8 to 18 or any one of claims 25 to 49.

56. Apparatus according to any one of claims 19 to 24 or any one of claims 49 to 55, comprising: an apparatus for motor-driven adjustment of at least one reality model (7, 8); and / or an apparatus for advantageously automatically determining the deviation (64) of the position and / or location of a virtual object (13, 14) from a corresponding virtual object (3, 4); and / or an apparatus for motor-driven adjustment of at least one reality model (7, 8) until the advantageously automatically detected deviation (64) of the position and / or location of a virtual object (13, 14) from a corresponding virtual object (3, 4) is within an acceptable range.

57. The apparatus according to any one of claims 19 to 24 or any one of claims 50 to 56, comprising equipment for collision testing of a virtual ray (55).

Images