Virtual reality-supported diagnostic procedure for a system with computer-based auralization

Abstract

The invention relates to a method (100) for detecting an unintended operating state of a system (10) comprising a plurality of components (12, 13, 14, 17), each configured as a sound source (56, 58), as an object that is at least partially sound-reflecting, and / or as an object that is at least partially sound-absorbing. In the method (100), a first and a second sound image (36, 37) in the system (10) are detected by means of a first and second sound sensor (18.1), respectively. Furthermore, a receiver position (26) within a virtual space (20) in which the system (10) is represented is specified by means of a virtual reality system (30). A virtual sound image (31) is then determined and output to a user (33) and / or an artificial intelligence (50). Based on this, an unintended operating state is detected.The invention also relates to a computer program product (45) with which such a method (100) is implemented and to an evaluation unit (40) equipped with such a computer program product (45). Furthermore, the invention relates to the use of a virtual reality system (30) for inspecting a plant (10).

Description

[0001] The invention relates to a method for detecting an improper operating condition in a system and a computer program product designed for this purpose. The invention also relates to an evaluation unit for implementing the method and a system equipped with such an evaluation unit. Furthermore, the invention relates to the use of a virtual reality system for inspecting a system.

[0002] Patent application DE 10 2016 125 886 A1 discloses a method for calculating an auralization. A first unit is used for auralization in the time domain and a second unit for auralization in the frequency domain. Depending on a test criterion, switching occurs between the first and second units.

[0003] From the doctoral thesis “Physically Base Real-Time Auralization of Interactive Virtual Environments” by Dirk Schröder, RWTH Aachen, a real-time room acoustics simulation is known that is suitable for modeling living spaces, lecture halls, concert halls or subway stations.

[0004] From the patent specification US 11,170,139 B1, a real-time capable acoustic ray tracing method is known in which a first receiver position is determined in a room and a set of rays is emitted from a second position.

[0005] Document DE 10 2016 125 886 A1 discloses a device for efficiently calculating an auralization, comprising a signal processing device that has a first unit for auralization in the time domain and a second unit for auralization in the frequency domain. Switching between the first and second units is possible.

[0006] Patent application DE 10 2017 103 385 A1 describes a method for analyzing structure-borne sound and a suitable device for this purpose. A neural network is constructed and trained in this method so that the structure-borne sound can be analyzed using deep learning. This method can be used to identify a system state of the machine.

[0007] US 2017 / 0 286 572 A1 discloses a physical-digital system with a digital twin. The physical-digital system has at least one sensor linked to the digital twin. The digital twin is designed to simulate the operating behavior of, for example, an aircraft engine based on the sensor signals.

[0008] International patent application WO 2011 / 038 838 A1 discloses an acoustic representation of conditions in an industrial plant. Based on the current state of the industrial plant, an audio profile is computer-aided and converted into an acoustic signal. Signals from a variety of sources are filtered, weighted, aggregated, and / or abstracted using a machine classifier.

[0009] EP 3 968 105 A1 describes a computer-implemented method for simulating a plant. Input data about the plant is supplied to a simulation model. Acoustic data is also supplied to the simulation model. Based on the input data and the acoustic data, a simulation of the plant is initiated using acoustic signals.

[0010] To ensure economical and low-failure operation of systems, especially automation systems, experts are still employed today to inspect the system based on their own sensory impressions, such as acoustic impressions. Such inspections require personal presence. There is a need to make system inspections more efficient and to conduct them remotely. At the same time, there is a need for more meaningful diagnostic options to detect existing improper conditions. The invention is based on the task of providing a suitable option for this.

[0011] The problem is solved by a method according to the invention, which is designed to detect an improper operating state of a system, in particular an automation system. The system can be, for example, a chemical plant, a refinery, a manufacturing plant, a power plant, or an electrolyzer. The system comprises a plurality of components that interact and thereby implement a system process. The components are each designed as a sound source, as an at least partially sound-reflecting object, as an at least partially sound-absorbing object, or as a combination thereof.

[0012] The components are therefore designed to emit sound, absorb sound, interact with the sound, or a combination thereof. During operation of the system, different sound patterns arise depending on a position within the system, so-called location-dependent sound patterns. The method comprises a first step in which the system is made available in an active operating state. An active operating state is understood to be a state in which the system process is at least partially running, starting, or ending, so that sound is emitted. Furthermore, in the first step, a virtual space, for example in a virtual reality system, is made available in which the system is at least partially recreated. The virtual space can be viewed and virtually walked through by a user.

[0013] The method according to the invention further comprises a second step in which a first sound image is recorded by a first sound sensor and a second sound image by a second sound sensor. The first and second sound sensors are arranged in the system and connected to a suitable evaluation unit. In the second step, a sound emitted by at least one first sound source is recorded at their respective positions by means of the first and second sound sensors. The method according to the invention further comprises a third step in which a receiver position within the virtual space provided in the first step is specified. The receiver position can be specified by a user who uses a virtual reality system on which the method according to the invention is carried out. In the third step, a virtual sound image present at the receiver position is also determined.The virtual sound image can be determined based on the sounds detected by the first and second sound sensors. The virtual sound image can be determined taking into account the at least partially sound-reflecting and / or at least partially sound-absorbing properties of objects in the system that are recreated in the virtual space. In particular, the virtual sound image at the receiver position in the virtual space can be determined based on the sounds detected by the first and second sound sensors by calculating sound propagation paths within the system. The virtual sound image corresponds to a sound image that occurs when a user is located at the corresponding position in the system.

[0014] The method according to the invention further comprises a fourth step in which the virtual sound image determined in the third step is output to the user, in particular the user of the virtual reality system, and / or an artificial intelligence. Based on the output virtual sound image present at the receiver position, an improper operating state of the system is detected in the fourth step. The determination and output of the virtual sound image represents a computer-implemented auralization of an acoustic impression, i.e., a sound image, present in the system at the position corresponding to the predetermined receiver position. The virtual sound image can be output as a binaural sound image. The user can detect the improper operating state.Alternatively or additionally, the artificial intelligence can be implemented as a trained neural network that detects deviations from a target sound pattern at the receiver position in the virtual space or at the corresponding position in the system, for example, based on Fourier analysis. Alternatively or additionally, the artificial intelligence can be coupled with an empirical database containing sound patterns of disturbances, which can be used to perform pattern recognition in the virtual sound pattern. If an improper operating condition of the system is detected, a warning can also be issued in the fourth step.

[0015] The method according to the invention allows acoustic sensory impressions, i.e. sound images, to be evaluated as virtual sound images by transferring them to a virtual reality system. Even complex sound images, and thus correspondingly complex virtual sound images, can be represented with sufficient precision. The invention is based, among other things, on the surprising discovery that virtual sound images can be represented in virtual spaces with sufficient realism. In the method according to the invention, a sufficiently low latency between the individual sound sensors is relevant. The overall latency in the output of the virtual sound image, on the other hand, is of secondary importance. The invention is also based, among other things, on the surprising discovery that knowledge of latency differences between individual sound sensors is sufficient to determine and output a sufficiently realistic virtual sound image.Overall, latency resulting from data transmission between the system and the output of the virtual sound image does not limit the quality in terms of realism. This makes the method according to the invention suitable for conducting an acoustically supported inspection, and thus acoustically supported detection of improper operating conditions, remotely, for example, over distances of hundreds or thousands of kilometers. Overall, this enables location-independent detection of improper operating conditions of the system, which is carried out by the user and / or the artificial intelligence.

[0016] In one embodiment of the claimed method, the first sound source can be represented as the first virtual sound source in the virtual space. For this purpose, based on at least the first and second sound images, direction finding, triangulation, or trilateration is used to determine which frequencies of the respective sound images come from which direction, relative to the position of the sound sensors. Accordingly, a sound emission spectrum can be determined for individual components that are sound sources in the system. The respective sound emission spectrum of a sound source can thus be output as a corresponding virtual sound source in the virtual space, specific to each component. As a result, the first virtual sound source can also be inspected separately in the virtual space as a virtual counterpart to a corresponding component of the system. The claimed method thus enables the targeted detection of improper conditions.In addition, the claimed method can also emit support signals which are relevant in terms of...

[0017] Furthermore, at least the third and fourth steps can also be performed for a second sound source. The claimed method is transferable to a plurality of sound sources, allowing for more targeted detection of improper conditions. In the fourth step, the virtual sound image at the receiver position can be determined based on a combination of the first virtual sound source with the second virtual sound source. This can be done, for example, using so-called acoustic ray tracing. The virtual sound image thus corresponds to a superimposed sound image resulting from the corresponding first and second sound sources in the system, taking the receiver position into account. The claimed method can be scaled to complex systems with multiple sound sources in a surprisingly simple manner while conserving computing power, allowing even complex systems to be inspected in virtual space.

[0018] Likewise, in the claimed method, the virtual sound image can be output on a virtual reality headset, allowing the system to be displayed in a virtually walkthrough manner. Virtual reality headsets that are part of a virtual reality system offer a higher degree of realism and are readily available. Furthermore, virtual reality headsets are used in other application areas, such as the entertainment industry. The claimed method utilizes the virtual reality headset as an output medium that is rapidly evolving, so that advances in virtual reality headsets can be quickly utilized by the claimed method.

[0019] Furthermore, a warning can be issued via the virtual reality headset in the fourth step of the process.

[0020] In a further embodiment of the claimed method, an operating data set for the corresponding sound source can be output for a virtual sound source in a display on the virtual reality headset following a selection by the user. Operating data of the component that is a sound source in the system and is simulated as a virtual sound source can be displayed therein. For example, the component can be an electric motor, and its current speed can be displayed as operating data. The user can thus verify the plausibility of the output virtual sound image against the operating data. This further supports the detection of an improper condition. The display of the operating data set can also be embodied as a graphic display superimposed on a visual representation of the system simulated in the virtual space.This could, for example, be a false-color display, also known as a heat map. The graphic display can be used to depict any available parameter that describes the operation of the system, such as acoustic parameters representing a frequency band of a virtual sound source or a volume level at the receiver position. This further simplifies and accelerates the detection of any improper condition of the system.

[0021] Furthermore, at least one component of the system can be a motor, a pump, an agitator, a fan, or another mechanical application. Such components have moving parts and generate operating noises that are characteristic depending on the current operating state, in particular a speed. Accordingly, deviations from this are significant indications of an improper operating state. Alternatively or additionally, at least one component of the system can also be lines through which a process medium can flow, which have an acoustic characteristic when the flow is appropriate. Further alternatively or additionally, at least one component of the system can be a pneumatic module, in which a leak can be acoustically detected using the claimed method. Likewise, the at least one component can be a high-voltage application in which switching operations can be acoustically identified.The claimed method is particularly suitable for detecting improper operating conditions in a wide range of components. Detection can be performed by the user or by artificial intelligence.

[0022] Furthermore, the claimed method can comprise a fifth step in which a virtual sound source corresponding to a component in the system is selected by the user. This can take place, for example, within the virtual space provided by the virtual reality system. This is followed by a sixth step in which a sound image specific to the sound source is determined. The sound source-specific sound image is determined at least based on the first and second sound images acquired in the second step. The sound source-specific sound image comprises an acoustic impression based on the sound emissions caused exclusively by the component corresponding to the selected virtual sound source. The sound source-specific sound image applies specifically to a position corresponding to the predetermined receiver position in the virtual space.The sound source-specific sound image can be determined corresponding to the first virtual sound source. In a further seventh step, the sound source-specific sound image is output in the virtual space as a virtual sound source-specific sound image. In this process, a soundscape is reduced, i.e., its volume is lowered in the output. The soundscape refers to acoustic impressions that are not caused by the selected virtual sound source. This makes it possible to suppress disturbing acoustic impressions in the virtual space and to specifically inspect individual components acoustically. This allows for more precise diagnosis if an improper operating condition exists on the associated component. This further increases the informative value of the claimed method. The fifth, sixth, and seventh steps can also be performed for multiple virtual sound sources.The more sound sensors are used, the more precisely the background noise can be separated from the sound source-specific sound image.

[0023] In one embodiment of the claimed method, the sound source-specific sound image of at least one virtual sound source can be provided to the artificial intelligence to detect an improper operating state of the corresponding sound source in the system, i.e., one of its components. The user can specify an evaluation mode, for example, by making a selection in the virtual space. By selecting the evaluation mode, an algorithm can be specified by which the virtual sound source is to be evaluated. This allows the claimed method to use computationally intensive algorithms that are too computationally complex to run continuously during operation. Alternatively or additionally, the evaluation mode can also be suggested or selected by the artificial intelligence. Overall, the diagnostic options for the claimed method are thus expanded in a way that conserves computing capacity.

[0024] Furthermore, in the claimed method, a comparison data set can be selected which comprises historical operation of the system. The comparison data set is provided to the system represented in the virtual space, so that the historical operation of the system stored therein is at least acoustically recreated and output analogously to the third and fourth steps of the method. Accordingly, based on the comparison data set, a virtual sound image is output at the specified receiver position. When the comparison data set is output in the virtual space, the system can still be walked through. During the method, a switch can be specified between the virtual sound image based on the comparison data set and the virtual sound image based on the sound images captured in the second step. The comparison data set can depict a good condition of the system.The user and / or the artificial intelligence are thus supported in detecting an improper operating condition through a direct comparison of the virtual sound images at the same receiver position. The virtual walkthrough allows meaningful receiver positions to be specified and acoustically examined there. To detect an improper operating condition, a difference can be created between the virtual sound image based on the comparison data set and the virtual sound image based on the acquisition in the second step, thus creating a virtual difference sound image. The improper operating condition can be detected by the user and / or the artificial intelligence by evaluating the virtual difference sound image.

[0025] Furthermore, the first and / or second sound sensor can be designed as a microphone, directional microphone, or sound camera. Microphones, directional microphones, and sound cameras offer increased measurement accuracy and can be coupled to the system's evaluation unit via high-performance communicative data connections. Furthermore, they can also detect sound emissions outside the humanly audible frequency spectrum. This further supports artificial intelligence in its evaluation and detection of an improper operating condition. Damage that causes characteristic sound patterns in the ultrasonic or infrasound spectrum can also be utilized. The diagnostic capabilities of the claimed method are thus further enhanced.

[0026] In a further embodiment of the claimed method, a detected ultrasonic emission or a detected infrasound emission is shifted in its frequency position. The infrasound or ultrasonic emission is shifted in its frequency position so that it lies within the humanly audible spectrum. The thus frequency-shifted ultrasonic or infrasound emission is output as a virtual sound image in the claimed method in the fourth step. For this purpose, the corresponding component, i.e., its representation in virtual space, is selected by the user. The user can also specify a frequency range in which the ultrasonic or infrasound emission is to be output and / or evaluated. The frequency-shifted output of infrasound or ultrasonic emissions represents an acoustic counterpart to a visual representation with false colors, for example, in infrared images.The claimed method is therefore suitable for providing analysable frequency ranges that are inaudible to humans. This further enhances diagnostic possibilities, particularly compared to in-person inspection.

[0027] The problem described above is also solved by a computer program product according to the invention. The computer program product is designed to receive and process measurement data from a plurality of sound sensors. Furthermore, the computer program product is designed to calculate a virtual space with virtual objects that represent components of a system, in particular an automation system, and to recreate this space in cooperation with a suitable device, for example a virtual reality headset. The recreation of the virtual space with the virtual objects includes visual and acoustic recreation. According to the invention, the computer program product is designed to carry out at least one embodiment of the methods outlined above. The features of the underlying method are therefore readily transferable to the computer program product according to the invention.The underlying method can be implemented in a manner that conserves computing power. The computer program product is particularly suitable for receiving measurement data from sound sensors located remotely from the device on which the virtual space is displayed. The computer program product as a whole is suitable for location-independent inspection, in particular acoustic inspection, of the associated system. Furthermore, the computer program product can be monolithic, i.e., executable on a single hardware platform. Alternatively, the computer program product can be modular and comprise a plurality of subprograms that can be executed on separate hardware platforms. The subprograms are interconnected via at least one communicative data connection and interact via this to provide the functionality of the claimed computer program product.The computer program product may be stored in remanent form on a storage device, such as a USB memory, a hard disk or an optical disk.

[0028] In one embodiment of the claimed computer program product, it can be coupled to a digital twin of the system to be simulated. Alternatively, the computer program product can comprise a digital twin of the system. The digital twin is connected to components of the system, for example, a control unit and / or sound sensors, via communicative data connections. Consequently, the digital twin is suitable for detecting a current operating state of the system and tracking it in a digital image of the system. The digital image in the digital twin is designed to simulate an interaction between components of the system and with its environment. The digital twin can, for example, be designed as a digital twin according to the document US 2017 / 286572 A1. The disclosure content of US 2017 / 286572 A1 is incorporated into the present application.Digital twins typically feature precise simulation routines whose performance can be harnessed by the underlying method. The virtual sound image can be determined based on a number of separate simulations of sound propagation, which interact as co-simulations within the meaning of European Patent EP 4 016 216 B1.

[0029] The underlying problem is also solved by an evaluation unit according to the invention. The evaluation unit is designed to receive a plurality of measurement signals from a plurality of sound sensors. Furthermore, the evaluation unit has a data interface via which operating data of a system to be simulated can be received. The monitoring unit is suitable for storing and executing a computer program product that is suitable for evaluating the received measurement data and the operating data. According to the invention, the computer program product is designed according to one of the embodiments outlined above. The evaluation unit can be designed as a master computer, as a programmable logic controller, or as a computer cloud. Alternatively or additionally, the evaluation unit can be coupled to an operator station or belong to the operator station of the system.

[0030] Furthermore, the object is achieved by a system according to the invention, which can be designed, in particular, as an automation system. The system comprises a plurality of components that interact to execute a system process. The components are each designed as a sound source, as an at least partially sound-reflecting object, as an at least partially sound-absorbing object, or a combination thereof. The system also comprises a monitoring unit configured to detect an improper operating state of the system. According to the invention, the monitoring unit of the system is designed according to one of the embodiments described above.

[0031] The object outlined above is equally achieved by a system according to the invention comprising a virtual reality system and an evaluation unit. The virtual reality system has at least one virtual reality headset and is designed for computer-based auralization of a sound image at a predeterminable receiver position within a virtual space. According to the invention, the virtual reality system is used to detect an improper operating state of a system simulated therewith, in particular an automation system. The improper state can be detected regardless of location. Furthermore, a warning can also be output to the user on the virtual reality system. According to the invention, the virtual reality system is connected to an evaluation unit according to one of the embodiments presented above.The features of the evaluation unit, and thus also the features of the underlying method, are thus transferable to the virtual reality system. In particular, the virtual reality system can be configured to at least partially implement at least one of the methods described above.

[0032] The invention is explained in more detail below using individual embodiments in the figures. The figures are to be read as complementary to one another insofar as identical reference numerals in different figures have the same technical meaning. The features of the individual embodiments can also be combined with one another. Furthermore, the features of the embodiments shown in the figures can be combined with the features outlined above. They show in detail: Fig. 1 shows a schematic structure of an automation system on which a first embodiment of the claimed method is carried out; Fig. 2 shows a schematic structure of an automation system on which a second embodiment of the claimed method is carried out; Fig. 3 a schematic structure of an automation system on which a third embodiment of the claimed method is carried out.

[0033] A structure of a plant 10 on which a first embodiment of the claimed method 100 is carried out is shown in Fig. 1 schematically. The system 10 comprises a plurality of components 12 which, during intended operation, interact to carry out a system process 15. One of the components 12 is a motor 14, another component 12 is a fan 13, and yet another component 12 is an agitator 17. The components 12 are each connected to at least one control unit 60 of the system 10 via a communicative data connection 42. The components 12 can be controlled by the control unit 60 using a control program 65 to carry out the system process 15. The system 10 also comprises a plurality of sound sensors 18, each of which is designed as microphones or sound cameras. Among the components 12, the motor 14 and the agitator 17 are designed as sound sources and emit sound emissions that produce a sound 19.The additional components 12 are each designed as at least partially sound-reflecting objects and / or at least partially sound-absorbing objects. The components 12 thus affect sound propagation paths 24, which result for the sound emissions or sounds 19. Depending on a position 11 in the system 10, a location-dependent sound pattern 38 thus results. In a first step 110 of the method 100, the system 10 is provided in an active operating state. An active operating state is understood here to mean an operation in which the system process 15 is at least partially running and a sound emission, i.e., a sound 19, occurs at at least one component 12.

[0034] In the first step 110 of the method 100, a virtual space 20 is also provided in which the system 10 is simulated. In the virtual space 20 there is a digital twin 25 of the system 10, which comprises a plurality of virtual objects 22, each of which simulates a component 12 of the automation system 10. The virtual space 20 is presented to a user 33 via a virtual reality headset 32, which belongs to a virtual reality system 30. The user 33 can thereby specify a receiver position 26 within the virtual space 20, which corresponds to the Fig. 1 corresponds to position 11 in Annex 10. The accessibility is in Fig. 1 symbolized by the arrow 27. The virtual space 20 is provided by a computer program product 45, which is stored executably on an evaluation unit 40 of the system 10.

[0035] The method 100 also includes a second step 120, in which a first sound image 36 is recorded by a first sound sensor 18.1 and a second sound image 37 by a second sound sensor 18.2. The first and second sound images 36, 37 are each an acoustic impression that results at the position of the respective sound sensor 18.1, 18.2 as a function of the sounds 19 and the sound propagation paths 24 in the system 10. In the second step 120, a component 12, namely the engine 14, is determined as the first sound source 56. For this purpose, a triangulation, trilateration, or direction finding is performed using the computer program product 45, which is executed on the evaluation unit 40 of the system 10. Overall, based on the first and second sound images 37, 39, it is determined which sound 19 is emitted separately by the first sound source 56. Likewise, in the second step 120, a component 12, namely the agitator 17, is determined as the second sound source 58.The determination of the second sound source 58 proceeds analogously to the first sound source 56. An emitted sound 19 is also determined for the second sound source 58.

[0036] Furthermore, the method 100 comprises a third step in which the receiver position 26 in the virtual space 20 is specified by the user 33. The receiver position 26 is specified by walking 27 through the virtual space 20 according to a representation by the virtual reality headset. In the third step 130, a virtual sound image 31 is determined that corresponds to a location-dependent sound image 38 that is perceptible at the corresponding position 11 in the system 10. To determine the virtual sound image 31, virtual objects 22 are recreated as the first virtual sound source 28 and as the second virtual sound source 29 in the virtual space 20. Sound propagation paths 24 are calculated in the virtual space 20, and the virtual sound image 31 at the receiver position 26 is thus determined.The virtual objects 22 are also at least partially sound-reflecting and / or at least partially sound-absorbing virtual objects 22 corresponding to the components 12. In the virtual space 20, the corresponding at least partially sound-reflecting and / or at least partially sound-absorbing behavior is simulated to determine the sound propagation paths 24. The virtual sound image 31 is also determined by the computer program product 40, which is at least coupled to the digital twin 25.

[0037] Furthermore, the method 100 comprises a fourth step 140 in which the virtual sound image 31 is output to the user 33. The output to the user 33 occurs via the virtual reality headset 32. The output occurs binaurally and represents an auralization 55. Accordingly, the virtual sound image 31 corresponds to the location-dependent sound image 38, which is perceptible at position 11 in the system 10, which corresponds to the predetermined receiver position 26. The output virtual sound image 31 enables the user 33 to recognize an improper state of the system 10. Likewise, in the fourth step 140, the virtual sound image 31 is output 43 to an artificial intelligence 50, which at least interacts with the computer program product 45.The artificial intelligence 50 is designed as a neural network and is suitable for comparing the virtual sound image 31 with sound images from a database 46, which is connected to the evaluation unit 40 via a communicative data connection 42. To detect the improper operating state, a target sound image from the database 46, which is assigned to an area of ​​the predetermined receiver position 26, is compared with the virtual sound image 31 by the artificial intelligence 50. If an improper operating state of the system 10 is detected, a warning 48 is issued. The warning 48 is transmitted via a communicative data connection 42 between the evaluation unit 40 and the control unit 60, thus enabling a response by the control program 65. Furthermore, in the method 100, the user 33 selects one of the virtual objects 22. The selection is represented by the hand symbol 23.The selection 23 takes place within the virtual space 20. For the selected virtual object 22, operating data 62 for the component 12 of the system 10 that corresponds to the selected virtual object 22 is retrieved. The operating data 62 includes measured values, existing control commands, setpoints and / or other parameters that characterize the operation of the respective component 12. The operating data 62 is displayed to the user 33 in the virtual space 20 as a virtual representation 49, for example as a window floating in the virtual space 20. The operating data 62 associated with the selected component 22 is thus made available to the user 33 for checking the plausibility of the issued warning 48 and / or for further diagnosis in the virtual space. The method 100 allows the virtual space 20 in which the system 10 is recreated to be displayed remotely therefrom. The system 10 is represented by the method 100 according to. Fig. 1 thus inspectable by the user 33 regardless of location.

[0038] A structure of a plant 10 on which a second embodiment of the claimed method 100 is carried out is shown in Fig. 2 is shown schematically. The system 10 comprises a plurality of components 12 which interact during intended operation to carry out a system process 15. One of the components 12 is a motor 14, another component 12 is a fan 13, and yet another component 12 is an agitator 17. The components 12 are each connected to at least one control unit 60 of the system 10 via a communicative data connection 42. The components 12 can be controlled by the control unit 60 using a control program 65 to carry out the system process 15. The system 10 also comprises a plurality of sound sensors 18, each of which is designed as microphones or sound cameras. Among the components 12, the motor 14 and the agitator 17 are designed as sound sources and emit sound emissions that produce a sound 19.The additional components 12 are each designed as at least partially sound-reflecting objects and / or at least partially sound-absorbing objects. The components 12 thus affect sound propagation paths 24, which result for the sound emissions or sounds 19. Depending on a position 11 in the system 10, a location-dependent sound pattern 38 thus results. In a first step 110 of the method 100, the system 10 is provided in an active operating state. An active operating state is understood here to mean an operation in which the system process 15 is at least partially running and a sound emission, i.e., a sound 19, occurs at at least one component 12.

[0039] In the first step 110 of the method 100, a virtual space 20 is also provided in which the system 10 is recreated. In the virtual space 20 there is a digital twin 25 of the system 10, which comprises a plurality of virtual objects 22, each of which recreates a component 12 of the system 10. The virtual space 20 is presented to a user 33 via a virtual reality headset 32, which belongs to a virtual reality system 30. The user 33 can thereby specify a receiver position 26 within the virtual space 20, which corresponds to the Fig. 2 corresponds to position 11 in Annex 10. The accessibility is in Fig. 2 symbolized by the arrow 27. The virtual space 20 is provided by a computer program product 45, which is stored executably on an evaluation unit 40 of the system 10.

[0040] The method 100 also includes a second step 120, in which a first sound image 36 is recorded by a first sound sensor 18.1 and a second sound image 37 by a second sound sensor 18.2. The first and second sound images 36, 37 are each an acoustic impression that results at the position of the respective sound sensor 18.1, 18.2 as a function of the sounds 19 and the sound propagation paths 24 in the system 10. In the second step 120, a component 12, namely the engine 14, is determined as the first sound source 56. For this purpose, a triangulation, trilateration, or direction finding is performed using the computer program product 45, which is executed on the evaluation unit 40 of the system 10. Overall, based on the first and second sound images 37, 39, it is determined which sound 19 is emitted separately by the first sound source 56. Likewise, in the second step 120, a component 12, namely the agitator 17, is determined as the second sound source 58.The determination of the second sound source 58 proceeds analogously to the first sound source 56. An emitted sound 19 is also determined for the second sound source 58.

[0041] Furthermore, the method 100 comprises a third step in which the receiver position 26 in the virtual space 20 is specified by the user 33. The receiver position 26 is specified by walking 27 through the virtual space 20 according to a representation by the virtual reality headset. In the third step 130, a virtual sound image 31 is determined that corresponds to a location-dependent sound image 38 that is perceptible at the corresponding position 11 in the system 10. To determine the virtual sound image 31, virtual objects 22 are recreated as the first virtual sound source 28 and as the second virtual sound source 29 in the virtual space 20. Sound propagation paths 24 are calculated in the virtual space 20, and the virtual sound image 31 at the receiver position 26 is thus determined.The virtual objects 22 are also at least partially sound-reflecting and / or at least partially sound-absorbing virtual objects 22 corresponding to the components 12. In the virtual space 20, the corresponding at least partially sound-reflecting and / or at least partially sound-absorbing behavior is simulated to determine the sound propagation paths 24. The virtual sound image 31 is also determined by the computer program product 40, which is at least coupled to the digital twin 25.

[0042] Furthermore, the method 100 comprises a fourth step 140 in which the virtual sound image 31 is output to the user 33. The output to the user 33 occurs via the virtual reality headset 32. The output occurs binaurally and represents an auralization 55. Accordingly, the virtual sound image 31 corresponds to the location-dependent sound image 38, which is perceptible at position 11 in the system 10, which corresponds to the predetermined receiver position 26. The output virtual sound image 31 enables the user 33 to recognize an improper state of the system 10. Likewise, in the fourth step 140, the virtual sound image 31 is output 43 to an artificial intelligence 50, which at least interacts with the computer program product 45.The artificial intelligence 50 is designed as a neural network and is suitable for comparing the virtual sound image 31 with sound images from a database 46, which is connected to the evaluation unit 40 via a communicative data connection 42. To detect the improper operating state, a target sound image from the database 46, which is assigned to an area of ​​the predetermined receiver position 26, is compared with the virtual sound image 31 by the artificial intelligence 50. If an improper operating state of the system 10 is detected, a warning 48 is issued. The warning 48 is transmitted via a communicative data connection 42 between the evaluation unit 40 and the control unit 60, thus enabling a response by the control program 65.

[0043] Furthermore, in the method 100, in a fifth step 150, the user 33 selects one of the virtual objects 22. The selection is represented by the hand symbol 23. In a further sixth step 160, a sound source-specific sound image 34 present at the predetermined receiver position 26 is determined. The sound source-specific sound image 34 is based on the virtual sound image 31 at the receiver position 26. In the sixth step 160, the sound emitted by the first virtual sound source 28 is isolated. Accordingly, sounds emitted by virtual objects 22 other than the first virtual sound source 28 are classified as background sound 53. The portion of the background sound 53 is output in the output of the sound source-specific sound image 34 using an adjustable acoustic damper 44.The adjustable acoustic damper 44 outputs the background noise 53 at a reduced volume in a seventh step 170 of the method 100. The adjustable acoustic damper 44 also allows the background noise 53 to be completely suppressed in the seventh step 170. In this case, the sound source-specific sound pattern 34 represents a theoretical acoustic impression in the system 10 that would occur if the components 12 not being considered were silent. The sound source-specific sound pattern 34 can also be transmitted to the artificial intelligence 50 to detect an improper operating state. Overall, the method 100 according to... Fig. 2, a diagnosis based on acoustic impressions is possible, which is not possible during a physical inspection of the facility 10. At the same time, the procedure 100 according to Fig. 2 the spectrum of diagnostic options for the user 33 is expanded during a location-independent inspection.

[0044] A structure of a plant 10 on which a first embodiment of the claimed method 100 is carried out is shown in Fig. 3 is shown schematically. The system 10 comprises a plurality of components 12 which, during intended operation, interact to carry out a system process 15. One of the components 12 is a motor 14, another component 12 is a fan 13, and yet another component 12 is an agitator 17. The components 12 are each connected to at least one control unit 60 of the system 10 via a communicative data connection 42. The components 12 can be controlled by the control unit 60 using a control program 65 to carry out the system process 15. The system 10 also comprises a plurality of sound sensors 18, each of which is designed as microphones or sound cameras. Among the components 12, the motor 14 and the agitator 17 are designed as sound sources and emit sound emissions that produce a sound 19.The additional components 12 are each designed as at least partially sound-reflecting objects and / or at least partially sound-absorbing objects. The components 12 thus affect sound propagation paths 24, which result for the sound emissions or sounds 19. Depending on a position 11 in the system 10, a location-dependent sound pattern 38 thus results. In a first step 110 of the method 100, the system 10 is provided in an active operating state. An active operating state is understood here to mean an operation in which the system process 15 is at least partially running and a sound emission, i.e., a sound 19, occurs at at least one component 12.

[0045] In the first step 110 of the method 100, a virtual space 20 is also provided in which the system 10 is recreated. In the virtual space 20 there is a digital twin 25 of the system 10, which comprises a plurality of virtual objects 22, each of which recreates a component 12 of the system 10. The virtual space 20 is presented to a user 33 via a virtual reality headset 32, which belongs to a virtual reality system 30. The user 33 can thereby specify a receiver position 26 within the virtual space 20, which corresponds to the Fig. 3 corresponds to position 11 in Annex 10. The accessibility is in Fig. 3 symbolized by the arrow 27. The virtual space 20 is provided by a computer program product 45, which is stored executably on an evaluation unit 40 of the system 10.

[0046] The method 100 also includes a second step 120, in which a first sound image 36 is recorded by a first sound sensor 18.1 and a second sound image 37 by a second sound sensor 18.2. The first and second sound images 36, 37 are each an acoustic impression that results at the position of the respective sound sensor 18.1, 18.2 as a function of the sounds 19 and the sound propagation paths 24 in the system 10. In the second step 120, a component 12, namely the engine 14, is determined as the first sound source 56. For this purpose, a triangulation, trilateration, or direction finding is performed using the computer program product 45, which is executed on the evaluation unit 40 of the system 10. Overall, based on the first and second sound images 37, 39, it is determined which sound 19 is emitted separately by the first sound source 56. Likewise, in the second step 120, a component 12, namely the agitator 17, is determined as the second sound source 58.The determination of the second sound source 58 proceeds analogously to the first sound source 56. An emitted sound 19 is also determined for the second sound source 58.

[0047] Furthermore, the method 100 comprises a third step in which the receiver position 26 in the virtual space 20 is specified by the user 33. The receiver position 26 is specified by walking 27 through the virtual space 20 according to a representation by the virtual reality headset. In the third step 130, a virtual sound image 31 is determined that corresponds to a location-dependent sound image 38 that is perceptible at the corresponding position 11 in the system 10. To determine the virtual sound image 31, virtual objects 22 are recreated as the first virtual sound source 28 and as the second virtual sound source 29 in the virtual space 20. Sound propagation paths 24 are calculated in the virtual space 20, and the virtual sound image 31 at the receiver position 26 is thus determined.The virtual objects 22 are also at least partially sound-reflecting and / or at least partially sound-absorbing virtual objects 22 corresponding to the components 12. In the virtual space 20, the corresponding at least partially sound-reflecting and / or at least partially sound-absorbing behavior is simulated to determine the sound propagation paths 24. The virtual sound image 31 is also determined by the computer program product 40, which is at least coupled to the digital twin 25.

[0048] Furthermore, the method 100 comprises a fourth step 140 in which the virtual sound image 31 is output to the user 33. In addition, the user 33 changes an acoustic mode of the method 100 via a selection 23 in the virtual space 20. The acoustic mode change 41 adjusts the output of the virtual sound image 31. A frequency range of the virtual sound image 31 is selected which lies in a frequency spectrum 52 inaudible to humans. As a result of the acoustic mode change 41, the portion of the virtual sound image 31 is shifted via a frequency shift 54 ​​into a frequency spectrum 51 audible to humans. Based on the selected portion in the frequency spectrum 52 inaudible to humans and the frequency shift 54, a frequency-shifted sound image 39 is determined and output to the user 33.The frequency-shifted sound image 39 is an acoustic counterpart to a false-color representation, such as in infrared imaging. This allows the frequency spectra 52, which are inaudible to humans and are present in the location-dependent sound image 38, to be directly evaluated by the user as they walk through the virtual space 20. The frequency spectra 52, which are inaudible to humans, are ultrasonic emissions and / or infrasound emissions.

[0049] The virtual sound image 31 and the frequency-shifted sound image 39 are output to the user 33 via the virtual reality headset 32. The output is binaural and represents an auralization 55. Accordingly, the virtual sound image 31 corresponds to the location-dependent sound image 38, which is perceptible at position 11 in the system 10, which corresponds to the predetermined receiver position 26. This applies analogously, taking into account the frequency shift 54, to the frequency-shifted sound image 39. The output virtual sound image 31 and / or the frequency-shifted sound image 39 allows the user 33 to recognize an improper condition of the system 10. Likewise, in the fourth step 140, the virtual sound image 31 and / or the frequency-shifted sound image 39 is output 43 to an artificial intelligence 50, which at least interacts with the computer program product 45.The artificial intelligence 50 is designed as a neural network and is suitable for comparing the virtual sound image 31 and / or the frequency-shifted sound image 39 with sound images from a database 46, which is connected to the evaluation unit 40 via a communicative data connection 42. To detect the improper operating state, a target sound image from the database 46, which is assigned to an area of ​​the predetermined receiver position 26, is compared with the virtual sound image 31 and / or the frequency-shifted sound image 39 by the artificial intelligence 50. If an improper operating state of the system 10 is detected, a warning 48 is issued. The warning 48 is transmitted via a communicative data connection 42 between the evaluation unit 40 and the control unit 60, thus enabling a response by the control program 65.The method 100 allows the virtual space 20 in which the system 10 is recreated to be displayed remotely. The system 10 is represented by the method 100 according to . Fig. 3 can thus be inspected by the user 33 regardless of location. In addition, the analysis of the frequency-shifted sound pattern 39 provides an additional diagnostic option that is not practically feasible during an in-person inspection of the system 10. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] DE 10 2016 125 886 A1 [0002, 0005] US 11,170,139 B1

[0004] DE 10 2017 103 385 A1

[0006] US 2017 / 0 286 572 A1

[0007] WO 2011 / 038 838 A1

[0008] EP 3 968 105 A1

[0009] US 2017 / 286572 A1

[0028] EP 4 016 216 B1

[0028] Cited non-patent literature

[0000] Physically Base Real-Time Auralization of Interactive Virtual Environments” by Dirk Schröder, RWTH Aachen

[0003]

Claims

[1] Method (100) for detecting an improper operating state of a system (10) comprising a plurality of components (12, 13, 14, 17), each of which is designed as a sound source (56, 58), as an at least partially sound-reflecting object and / or as an at least partially sound-absorbing object, comprising the steps: a) providing the system (10) in an active operating state and providing a virtual space (20) in which the system (10) is at least partially recreated; b) detecting a first and a second sound image (36, 37) by means of a first and second sound sensor (18.1) respectively and determining a sound (19) emitted by a first sound source (56); c) specifying a receiver position (26) within the virtual space (20) and determining a virtual sound image (31) at the receiver position (26); d) outputting the virtual sound image (31) to a user (33) and / or an artificial intelligence (50) and detecting an improper operating state based on the virtual sound image (31) present at the receiver position (26). [2] Method (100) according to claim 1, characterized by that in step b) the first sound source (56) is imaged as the first virtual sound source (28) in the virtual space (20). [3] Method (100) according to claim 1 or 2, characterized by that steps a) to c) are carried out for a second sound source (58) and in step d) the virtual sound image (31) is carried out using a combination of the first and second virtual sound sources (56, 58). [4] Method (100) according to one of claims 1 to 3, characterized by that the virtual sound image (31) is output on a virtual reality headset (32) on which the system (10) is displayed in a virtually walkable manner. [5] Method (100) according to one of claims 1 to 4, characterized by that for a virtual sound source (28, 29) in a representation on the virtual reality headset (32) after selection by a user (33) an operating data record (62) of the corresponding sound source (56, 58) is visually output. [6] Method (100) according to one of claims 1 to 5, characterized by that the component (12, 13, 14, 17) of the system (10) is a motor (14), a pump, an agitator (17), a fan (13), or a mechanical application. [7] Method (100) according to one of claims 1 to 6, characterized by that the method (100) further comprises the steps: e) selecting a virtual sound source (28, 29) by a user (33); f) determining a sound source-specific sound image (34) based on at least the first and second sound images (36, 37); g) Outputting the sound source-specific sound image (34) with a reduced background noise. [8] Method (100) according to one of claims 1 to 7, characterized by that the sound source-specific sound image (34) of at least one virtual sound source (28, 29) is provided to the artificial intelligence (50) for detecting an improper operating state of the corresponding sound source (56, 58) in the system (10). [9] Method (100) according to one of claims 1 to 8, characterized by that the first and / or second sound sensor (18, 18.1, 18.2) is designed as a microphone, directional microphone or as a sound camera. [10] Method (100) according to one of claims 1 to 9, characterized by that an ultrasonic emission or an infrasonic emission with a shifted frequency position in the human audible spectrum (51) is emitted. [11] Computer program product (45) designed to receive and process measurement data from a plurality of sound sensors (18, 18.1, 18.2) and to visually and acoustically recreate a virtual space (20), characterized by that the computer program product (45) is configured to carry out a method (100) according to one of claims 1 to 10. [12] Computer program product (45) according to claim 11, characterized by that the computer program product (45) is coupled to a digital twin (25) of a system (10) to be simulated or comprises the digital twin (25). [13] Evaluation unit (40) which is designed to receive measurement signals from a plurality of sound sensors (18, 18.1, 18.2) and is designed to receive operating data (62) of a system (10) to be simulated via a data interface, wherein a computer program product (45) is stored executably on the evaluation unit (40), characterized bythat the computer program product (45) is designed according to claim 11 or 12. [14] Plant (10) which, for carrying out a plant process (15), comprises a plurality of components (12, 13, 14, 17), each of which is designed as a sound source (56, 58), as an at least partially sound-reflecting object and / or as an at least partially sound-absorbing object, characterized by that the system (10) comprises an evaluation unit (40) according to claim 13. [15] System comprising a virtual reality system (30) and an evaluation unit (40), wherein the virtual reality system (30) has a virtual reality headset (32) and is designed for computer-based auralization of a sound image at a predeterminable receiver position (26) within a virtual space and is designed to detect an improper state of a system (10) simulated therewith, characterized bythat the evaluation unit (40) is connected according to claim 14.

Images