System and method for generating interactive audio

Abstract

The attraction system (10) includes a display that operates to present augmented reality and / or virtual reality (AR / VR) images (18) to guests in an interaction space (14). The system (10) includes an audio controller (38) that operates an array of speakers (24) distributed throughout the interaction space (14), and a controller (32) having one or more processors (34). One or more processors (34) are operable to receive data indicating the state of the guest (e.g., the guest's actions, movements or gestures, inputs received from input devices (28)). One or more processors (34) are also operable to adjust the AR / VR images (18) in response to the state of the guest, and to instruct the audio controller (38) to operate the array of speakers (24) to provide interactive audio, based on the state of the guest, so that interactive audio appears to the guest as if it were arising from the dynamic portion of the AR / VR images (18).

Description

Background Art

[0001] This section is for introducing the reader to various aspects of technologies that may be related to various aspects of the technology described and / or claimed below. This discussion is thought to be helpful in showing the reader the background circumstances and promoting a better understanding of the various aspects of the present disclosure. Therefore, it should be understood that these descriptions are not to be regarded as admitting prior art, but should be read from the above viewpoints.

[0002] An amusement park generally includes attractions that provide various experiences to users. For example, an amusement park can include different attractions such as roller coasters, drop towers, and log flumes. Some attractions can include an environment that provides interaction effects useful for providing an immersive experience to users, such as auditory stimuli, tactile stimuli, visual stimuli, and / or other special effects.

Summary of the Invention

[0003] The following shows an overview of some embodiments disclosed in this specification. It should be understood that these aspects are only to show the reader a summary of some of these embodiments and do not limit the scope of the present disclosure. In fact, the present disclosure can include various aspects that may not be shown below.

[0004] In one embodiment, the attraction system includes a display that operates to depict augmented reality and / or virtual reality (AR / VR) images to a guest in an interaction space. The system includes an audio controller configured to operate an array of speakers distributed throughout the interaction space. The system includes a controller having one or more processors, one or more of which are configured to receive data indicating the guest's state, which includes the guest's actions, guest movements, guest gestures, guest facial expressions, guest bodily expressions, user input received from input devices associated with the guest, or a combination thereof. One or more of the processors are also configured to adjust the AR / VR images in response to the guest's state and to instruct the audio controller to operate the array of speakers to provide interactive audio based on the guest's state.

[0005] In one embodiment, a non-temporary computer-readable medium includes instructions that, when executed by the processor, cause the processor to perform an action. The action includes receiving sensor data from a sensor; identifying a first guest interaction with an interactive object based on the sensor data; determining a first interactive audio associated with the interactive object based on the first guest interaction with the interactive object; and instructing an array of speakers to output the first interactive audio that appears to be coming from the interactive object.

[0006] In one embodiment, a method for providing interactive audio includes receiving user input from an input device via a controller that indicates the trajectory of a virtual object in a virtual space. The method also includes determining the trajectory of the virtual object in the physical space based on a mapping between the virtual space and the physical space, via the controller, based on the user input. Furthermore, the method includes including the virtual object moving along the trajectory in the virtual space in an interactive augmented reality and / or virtual reality (AR / VR) image via the controller, based on the user input. The method also includes generating interactive audio in the physical space based on the trajectory of the virtual object in the physical space, via an audio controller.

[0007] A better understanding of these and other features, aspects and advantages of this disclosure will be gained by reading the following detailed description while referring to the attached drawings, which indicate the same parts throughout with the same reference numerals. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of an attraction system according to the embodiments of this disclosure. [Figure 2] This is a schematic diagram of an attraction system, according to the aspects of this disclosure, which includes an interactive environment in which guests can interact with interactive audio. [Figure 3] This is a schematic diagram of an attraction system, according to an aspect of the present disclosure, which includes a virtual environment in which interactive objects (e.g., physical objects) appear to emit interactive audio in response to interaction with guests. [Figure 4] This is a flowchart of the interactive audio generation process according to the aspects of this disclosure. [Modes for carrying out the invention]

[0009] The following describes one or more specific embodiments of this disclosure. For the sake of brevity, not all features of the implementation may be described herein. It should be understood that the development of any such implementation found in any engineering or design project will require numerous implementation-specific decisions to achieve the developer's specific objectives, such as compliance with system-related and business-related constraints, which may vary by implementation. Furthermore, while such development efforts may be complex and time-consuming, they should be understood by those skilled in the art who benefit from this disclosure as routine design, fabrication, and manufacturing activities.

[0010] When describing elements of the various embodiments of this disclosure, the articles “a,” “an,” and “the” mean that there are one, two, or more of these elements. The terms “comprising,” “including,” and “having” are intended to be comprehensive and mean that there may be further elements other than those listed. Furthermore, any reference to “one embodiment” or “a certain embodiment” in this disclosure should not be interpreted as excluding the existence of further embodiments, including the features described.

[0011] This disclosure relates to the field of special effects used in amusement attractions, such as amusement parks. Specifically, this disclosure relates to systems and methods for providing interactive audio that brings an immersive experience to guests of an amusement attraction, associated with interactive objects such as interactive augmented reality / virtual reality (AR / VR) images and / or physical objects. As described herein, interactive audio generally corresponds to dynamic audio outputs that are determined, modified and / or presented within an interactive environment in response to one or more inputs, such as the guest's position, movement, or gestures within the interactive environment. For example, interactive audio, interactive AR / VR images, and interactive objects can be used to immerse guests in an interactive environment such as a rainforest. In this example, the interactive audio may include the sounds of a rainforest, and the interactive AR / VR images that can be provided to guests through AR / VR goggles may include images (e.g., pictures, videos, animations) of rainforest flora and fauna. The interactive audio can be provided to guests through an array of speakers distributed throughout the amusement attraction. For example, interactive audio can be provided so that one or more guests perceive it as being generated by or corresponding to elements of an interactive AR / VR image, or as being generated by or corresponding to physical interactive objects presented or placed within the interactive environment. In some embodiments, an array of speakers can provide guests with binaural audio, meaning that specific sounds are clearly audible to each of the guest's ears. That is, each guest can perceive differences in volume and / or arrival time of each portion of the interactive audio signal received or heard in their left and right ears.For example, if a virtual dog (or a physical dog object or robotic dog element) in an interactive AR / VR image appears to be barking on the left side of the guest in the interactive environment, the barking will be louder in the guest's left ear, and the barking will reach the guest's right ear some time after it reaches the left ear.

[0012] The interactive environment, formed by interactive audio, interactive AR / VR images, and / or interactive objects, can evolve with the guest's actions and movements. For example, a specific action performed by the guest may trigger a change in the interactive audio. The guest within the interactive environment can be continuously tracked by sensors distributed within the environment. These sensors may include cameras, motion sensors, and weight sensors. The sensors can provide sensor data to a controller for processing. The controller can process the sensor data to detect specific movements, gestures, emotions, actions, positions, and / or orientations of the guest. As a result, the controller can determine possible responses for the interactive audio and / or interactive AR / VR images based on the detected guest movements, gestures, emotions, actions, positions, and / or orientations. Specifically, these responses may include changes in the content of the interactive audio, changes in the position of the sound source of the interactive audio (e.g., the perceived sound source), changes in the coordinates of the interactive audio, changes in the content of the interactive AR / VR image, and / or changes in the coordinates (e.g., position and orientation) of the interactive AR / VR image. Interactive audio and interactive AR / VR images can be adjusted in real time based on continuously collected sensor data.

[0013] The controller can instruct a display (e.g., AR / VR goggles or lenses, or a projector and projection surface) to display changes in the content, position, and / or orientation of an interactive AR / VR image in order to provide a calibrated interactive AR / VR image. The controller can also instruct an audio controller to provide calibrated interactive audio. The audio controller can activate specific speakers in an array of speakers to provide specific sounds that, when combined with sounds emitted from all speakers in the interactive environment, form interactive audio heard by guests of the entertainment attraction. Each guest may hear the interactive audio differently depending on their position and orientation relative to the interactive audio. For example, a guest standing to the left of the perceived interactive audio source may hear the interactive audio differently than a guest standing to the right of the perceived audio source. Interactive audio can also be customized according to each guest's preferences. For example, audio may be delivered at a lower volume to guests with high hearing sensitivity, and at a higher volume or modulated frequency to guests with hearing impairments.

[0014] To ensure the correct scaling of interactive audio and interactive AR / VR images within the physical space of the interaction environment (e.g., a room), a mapping system can map the coordinates (e.g., position, orientation, dimensions, etc.) of guests, interactive AR / VR images, and / or interactive objects. In one embodiment, coordinates acquired by a controller can be mapped to coordinates related to the physical space of the interaction environment, which can then be used by an audio controller. For example, the coordinates acquired by the controller may be defined with respect to a specific reference point or coordinate system determined by the controller. On the other hand, the coordinates used by the audio controller may be defined with respect to a different reference point and / or coordinate system. Thus, the mapping performed by the mapping system can transform the coordinates of interactive elements (e.g., coordinates of interactive objects, interactive AR / VR images, and sound sources of interactive audio) from coordinates defined with respect to the reference point and / or coordinate system used by the controller to coordinates defined with respect to the reference point and / or coordinate system used by the audio controller, and vice versa.

[0015] Based on the above, Figure 1 is a block diagram of the attraction system 10. The attraction system 10 may include rides (e.g., roller coasters), navigation areas (e.g., walkways), and performance shows where guests (or multiple guests) 12 can be placed. The attraction system 10 can entertain guests 12 by immersing them in an interactive environment 14 where guests 12 are provided with interactive audio 16, interactive augmented reality / virtual reality (AR / VR) images 18 (e.g., virtual objects on a display), and / or interactive objects 20 (e.g., physical objects) to access. The interactive environment 14 can simulate a specific experience, activity, or scene by providing guests 12 with interactive audio 16, interactive AR / VR images 18, and / or interactive objects 20. For example, the interactive environment 14 can simulate a rainforest by providing interactive AR / VR images 18 and interactive audio 16 related to a rainforest. In another example, the interactive environment 14 can provide one or more guests 12 with the experience of operating a helicopter. In yet another example, the interaction environment 14 can facilitate an activity in which the guest 12 earns points based on successfully targeting a specific part of the virtual environment 14 by firing a virtual projectile.

[0016] The interactive AR / VR image 18 can be provided to the guest 12 through AR / VR goggles 22 worn by the guest 12. In addition to or instead of this, the interactive AR / VR image 18 may also include a projected image (e.g., an image projected onto a wall or film by a projector). The interactive object 20 may include a physical object that the guest 12 can touch, modify (e.g., move, activate), and / or hold. The interactive audio 16 can be provided to the guest 12 through an array of speakers 24 distributed throughout the interaction environment 14.

[0017] The interactive environment 14 can evolve based on the actions of the guest 12. Specifically, it can provide or adjust interaction features 26 (e.g., interactive audio 16, interactive AR / VR image 18, interactive object 20) in response to actions performed by the guest 12 within the interactive environment 14. Furthermore, the guest 12 can react to the interaction features 26 of the interactive environment 14, and this reaction can trigger further changes in the interaction features 26. For example, the interactive AR / VR image 18 may include a representation of a helicopter (e.g., a virtual helicopter) that the guest 12 can remotely control using an input device 28 (e.g., a remote control device with a joystick). The guest 12 can move the joystick in a rough direction instructing the helicopter to move. Thus, the input device 28 can receive user input from the guest 12 and use the user input to control the helicopter's position within the interactive environment 14. The guest 12 can observe the helicopter and notice that its position is changing. As a result, the guest 12 can further adjust the helicopter's position based on its current location (e.g., to achieve a goal, avoid obstacles, etc.).

[0018] An interactive AR / VR image 18 (and / or interactive object 20) can be associated with interactive audio 16. For example, if the interactive AR / VR image 18 includes a helicopter (e.g., a virtual helicopter) flying around the interactive environment 14, the interactive audio 16 can include helicopter sounds. Thus, the guest 12 can hear helicopter sounds while standing within a threshold distance of the helicopter. Since the helicopter is a virtual sound source for helicopter sounds, the interactive audio 16 can become louder the closer it is to the helicopter's position (e.g., the helicopter's position as perceived by the guest 12) and quieter the further it is from the helicopter's position (e.g., the helicopter's position as perceived by the guest 12). The interactive audio 16 can also move along with the associated interactive AR / VR image 18 (and / or interactive object 20). For example, if the helicopter moves to the right of the guest 12, the guest 12 will hear the sound of the helicopter moving to the right. In this way, the user input of the guest 12 can affect the interactive audio 16 (e.g., audio associated with the interactive object 20). The guest 12 in the interactive environment 14 can also react to the interactive audio 16. For example, if the virtual helicopter is too close or too loud for guest 12 to hear and / or see, guest 12 can move the helicopter away or instruct the helicopter to move away by providing user input via input device 28.

[0019] In the example above, guest 12 interacts indirectly with the interactive audio 16 by manipulating an interactive AR / VR image 18 (e.g., a virtual helicopter) associated with the interactive audio 16. Alternatively, guest 12 can also directly influence the interactive audio 16 (e.g., without first manipulating the associated interactive AR / VR image 18 and / or interactive object 20). For example, consider an interactive environment 14 where guest 12 takes on the role of an orchestra conductor. Depending on how guest 12 moves the baton, the music heard by guest 12 (e.g., the interactive audio 16) can change. For example, moving the baton quickly can speed up the music, lifting and holding the baton can pause it, and moving the baton slightly can produce quiet music. Thus, in some embodiments, guest 12's actions can directly influence the interactive audio 16 without necessarily affecting the interactive AR / VR image 18 or interactive object 20.

[0020] Interactive audio 16 can be provided to a guest 12 through speakers 24 (e.g., an array of two or more speakers) distributed within the interaction environment 14. Individual speakers 24 can be positioned to surround the guest 12 within the interaction environment 14. For example, speakers 24 can be placed on the walls, floor, and ceiling of the interaction environment 14 and configured to emit sound in various directions. For instance, a first speaker 24 may emit sound northward, a second speaker 24 westward, and a third speaker 24 upward. Speakers 24 can emit interactive audio 16 in such a way that it is perceived as being heard from a specific sound source. In some embodiments, the positions of the speakers 24 and the interactive audio 16 are configured such that certain parts of the interactive audio 16 are amplified based on constructive interference of sound waves in the physical environment, while other sounds are mitigated or eliminated based on destructive interference.

[0021] For example, interactive audio 16 can be emitted in such a way that it appears to be coming from a dynamic part of the AR / VR image 18 (e.g., a movable element such as a depiction of a virtual bird 70) (e.g., the guest 12 perceives it as such). The sound source can be an interactive object 20, a virtual object provided as part of the interactive AR / VR image 18 (e.g., a virtual helicopter seen through AR / VR goggles 22), or another part of the interactive environment 14. For example, if the interactive environment 14 simulates rain, there can be multiple sound sources, such as a sound source corresponding to raindrops falling on a corrugated iron roof and a sound source corresponding to raindrops falling on an umbrella. In another example, the perceived sound source of the interactive audio 16 can be an interactive object 20, such as a toy dinosaur.

[0022] Each guest 12 within the interaction environment 14 can be positioned at a certain distance and orientation from the apparent and / or actual sound source, thus experiencing the interactive audio 16 in different ways. For example, if the sound source is positioned between two guests 12, the guest on the left will hear the interactive audio 16 associated with the sound source coming from the right, and the guest on the right will hear the interactive audio 16 coming from the left. Depending on the orientation of the guests 12, the interactive audio 16 received by one ear may be louder than the interactive audio 16 received by the other ear. Furthermore, a clear sound may reach one ear of a guest 12 before reaching the other ear. Therefore, the interactive audio 16 provided through the speaker 24 can include binaural audio (e.g., audio including delays and slight variations in the sound received by each ear of the guest 12). In some embodiments, the interactive audio 16 can include binaural audio provided through headphones or earphones of an AR / VR headset.

[0023] In some embodiments, if different audio content of the interactive audio 16 is provided to different guests 12, the guests 12 may also experience the interactive audio 16 in different ways. In some embodiments, the audio content provided to each guest 12 can be modified based on known information about the guest 12 and / or guest preferences. For example, the interactive audio 16 provided to a child may be quieter than the interactive audio 16 provided to an adult guest. In some embodiments, the audio content provided to one guest 12 may be completely different from the audio content provided to another guest 12. For example, consider an interactive environment 14 in which virtual ghouls chase / pursue guests 12. One guest 12 may hear one ghoul chasing them from behind, while another guest 12 may hear a different ghoul. Furthermore, each guest 12 may hear the ghouls sounding closer or further away, depending on how well they are escaping from them.

[0024] In order to provide customized interactive audio 16 for the position of each guest 12, the guest 12 can be tracked via the sensor 30. The sensor 30 can also be used to detect the movements and actions of the guest 12 that can elicit audible and / or visual responses within the interaction environment 14. Specifically, various sensors 30 distributed throughout the interaction environment 14 can collect data indicating the position, orientation, movement, gestures, emotions, and actions of the guest 12, and these can be used to determine possible changes to the interactive audio 16. For example, consider an interaction environment 14 that represents a cave where an echo sound can be heard every time the guest 12 takes a step. Such an interaction environment 14 can include sensors 30 embedded in the floor that detect the pressure applied to a certain area of the floor when the guest 12 places their foot. The data collected by such sensors 30 (and the known positions of such sensors 30) can be used to trigger the emission of an echo sound that appears to originate from or appear as if it were from the area where the foot was placed. In an example of an interaction environment 14 where the guest 12 is conducting a virtual orchestra with a conductor's baton, the sensor 30 (e.g., a camera, one or more accelerometers placed within the baton) can be used to detect specific movements and / or movement patterns of the baton to trigger changes to the content of the interactive audio 16 (e.g., orchestra music). It should be understood that the sensors 30 used within the interaction environment 14 can include various different sensor types including cameras, motion sensors, acceleration sensors, capacitance sensors, laser rangefinders, depth sensors, and the like.

[0025] The controller 32 can receive and process sensor data. Specifically, the controller 32 can process the sensor data to determine the position coordinates of the guest 12 within the interaction environment 14. For example, the position coordinates of the guest 12 may represent the guest's position within the interaction environment 14 (e.g., x, y, and z coordinates), the guest's posture (e.g., standing, squatting, kneeling, or lying down), the guest's eye orientation (e.g., the position of the guest's eyes and the direction they are looking), the guest's ear orientation (e.g., the position of the guest's ears and the direction they are looking), the position and / or orientation of the guest's appendages (e.g., arms, hands, fingers, legs, feet), the position and / or orientation of input devices or interactive objects owned or controlled by the guest, or a combination thereof. In addition to or instead of this, the controller 32 can also process the sensor data to determine the guest 12's actions, movements, gestures, and / or emotions within the interaction environment 14. The position, orientation, movement, gestures, emotions (e.g., facial expressions and / or bodily expressions), actions (e.g., inputs via user devices) and actions of the guest 12 as used herein can be collectively referred to as the state of the guest 12. In some embodiments, the controller 32 may include software such as artificial intelligence software or computer vision software configured to detect and / or classify specific patterns in sensor data. For example, an artificial intelligence algorithm can analyze an image of the guest 12 to detect a specific emotion that the guest 12 is thought to be experiencing, and this can be used to adjust the interactive audio 16. For example, if sensor data indicates that the guest 12 is feeling uncomfortable, the interactive audio 16 can be delivered to the guest 12 at a lower volume. On the other hand, if the sensor data classifies the guest 12 as happy or excited, the interactive audio 16 can be delivered to the guest 12 at a higher volume.

[0026] In addition to interacting with the interaction features 26 of the interaction environment 14 (e.g., interactive objects 20, interactive audio 16, interactive AR / VR images 18) through actions, movements, and gestures, the guest 12 can also interact with the interaction features 26 through an input device 28. In some embodiments, the input device 28 may include electronic devices such as a smartphone, remote control system, tablet, or toy electronic blaster gun. Generally, the input device 28 may include an input mechanism 29 such as a touch screen, push button, joystick, or trigger. The input mechanism 29 can enable the guest 12 to provide user input that allows them to control aspects of the interaction environment 14. For example, the guest 12 can use the joystick on the input device 28 to control a virtual drone in the interaction environment 14 (e.g., an interactive AR / VR image 18). In another example, the input device 28 may be a handheld blaster gun that can be used to fire a virtual projectile when a push button is activated. User input instructions (e.g., pressing a button, selecting an option on or within a touch screen) can be sent from the input device 28 to the controller 32 for processing. In one embodiment, the input device 28 can wirelessly transmit data indicating user input (e.g., user selection, button press instruction, joystick or wheel movement, switch position, trigger activation) to the controller 32 via Bluetooth, Wi-Fi, or another preferred type of wireless communication technology. In another embodiment, the input device 28 can be communicatively coupled to the controller 32 via a cable.

[0027] After processing the sensor data from the sensor 30 and / or the user input data from the input device 28, the controller 32 can generate changes in the interaction environment 14 in response to the received sensor data and / or user input. The changes in the interaction environment 14 can include changes in the coordinates (e.g., position, orientation) and / or content of the interactive AR / VR image 18, and / or changes in the interactive audio 16. As described, the change in the position of the interactive AR / VR image 18 can include a change in the position of a virtual object shown as part of the interactive AR / VR image 18 (e.g., relative to a reference point in the interaction environment 14), and the change in the content of the interactive AR / VR image 18 can include a change in the visual appearance of the virtual object. For example, the interactive AR / VR image 18 can show a virtual object exploding in response to the successful targeting of the virtual object by a virtual projectile fired by the guest 12. The change in the interactive audio 16 can include a change in the content of the interactive audio 16 and / or a change in the position of the sound source of the interactive audio 16. The change in the content of the interactive audio 16 can include a change in the type of audio provided. For example, the change in the content of the interactive audio 16 can include an explosion sound that can be heard in response to the successful targeting of a virtual object or a physical object by a virtual projectile fired by the guest 12 within the interaction environment 14.

[0028] Changes in the interactive audio 16 may include changes in the dimensions and / or coordinates of the three-dimensional volume associated with the interactive audio 16. It should be understood that sound emitted from a sound source is heard within a portion of the three-dimensional (3D) space adjacent to the sound source. Therefore, the interactive audio 16 can be associated with a certain three-dimensional volume (e.g., a portion of three-dimensional space). This three-dimensional volume can change when the interactive audio 16 changes. For example, the quieter the interactive audio 16 becomes, the smaller the 3D volume in which the interactive audio 16 can be perceived. When the position of the sound source of the interactive audio 16 changes, the 3D volume associated with the interactive audio 16 can also move. When the interactive audio 16 is "emitted" in a new direction, the 3D volume associated with the interactive audio 16 can also rotate around the location of the perceived sound source.

[0029] Changes in the interactive audio 16 and / or the interactive AR / VR image 18 can be determined by a software application running on the processor 34 of the controller 32. The controller 32 may include memory 36 and a processor 34. Memory 36 may include volatile memory such as random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM), an optical drive, a hard disk drive, a solid-state drive, or any other non-temporary computer-readable medium including instructions. The processor 34 may be operable to execute such instructions. For example, the processor 34 may include one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more general-purpose processors, or any combination thereof.

[0030] Controller 32 can send commands to audio controller 38 to trigger changes in the interactive audio 16 (e.g., changes in content or changes in the location of the sound source). Audio controller 38 can then command speakers 24 to produce a target audio output. In one embodiment, audio controller 38 can receive the coordinates of the guest 12, the coordinates related to the audio 16, and the content of the interactive audio 16 as input. Based on this information, audio controller 38 can determine the audio output of each speaker 24 to produce the interactive audio 16 containing the desired content. For example, audio controller 38 can command one speaker 24 to produce one type of audio and another speaker 24 to produce another type of audio. It can also command some speakers 24 to be silent for a period of time and others to increase their volume. Audio controller 38 can configure the sound emission from speakers 24 so that each guest 12 in the interaction environment 14 perceives this sound as being emitted from the sound source of the interactive audio 16. Furthermore, the audio controller 38 can be configured to emit sound from the speaker 24 to deliver customized audio to each individual guest 12. For example, interactive audio 16 can be delivered at a lower volume (e.g., loudness) than other guests to guests with higher auditory sensitivity. Sound wave cancellation and boosting can also be considered and / or utilized to achieve the desired result. In other words, the speaker 24 can operate to achieve the desired result by providing a customized soundscape based on the input.

[0031] The audio controller 38 may include a second memory 40 (e.g., audio memory 40) and a second processor 42 (e.g., audio processor 42). The audio memory 40 may include volatile memory such as random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM), an optical drive, a hard disk drive, a solid-state drive, or any other non-temporary computer-readable medium that can store instructions that the audio processor 42 can execute. For example, the audio processor 42 may include one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more general-purpose processors, or any combination thereof.

[0032] In one embodiment, the mapping system 44 can map coordinates acquired by the controller 32 from one coordinate system to another, for example, by mapping coordinates related to the guest 12 to coordinates related to the dynamic portion of the AR / VR image 18. The coordinates acquired by the controller 32 may include the coordinates of the guest's position, the coordinates of the interactive object 20, the coordinates of objects held by the guest 12, the coordinates of virtual objects that are part of the interactive AR / VR image 18, and coordinates related to the interactive audio 16. For example, in one embodiment, the coordinates acquired by the controller 32 can be defined with respect to a specific reference point or coordinate system used by the controller 32. However, in some cases, the reference point or coordinate system used by the controller 32 may not correspond to a reference point or coordinate system in the physical space of the interaction environment 14. For example, the coordinates acquired by the controller 32 can be defined with respect to a coordinate system or reference point in virtual space. In some cases, the controller 32 can use a reference point or coordinate system that corresponds to the changing position of a movable element (e.g., a movable element in physical space, a movable element in virtual space). Therefore, the coordinates obtained by the controller 32 can be mapped to relate or transform them according to a reference point or coordinate system in the physical space of the interaction environment 14.

[0033] For example, consider an augmented reality interactive environment 14 in which a guest 12 launches a virtual projectile, such as a tennis ball, and the guest visualizes (for example, via AR / VR goggles 22) how the virtual tennis ball bounces around in the physical room of the interactive environment 14. The controller 32 can determine the trajectory of the virtual tennis ball. However, the controller 32 may not necessarily have a dimensionally accurate representation of the physical room in which the interactive AR / VR image 18 is presented (for example, a coordinate system relating to one or more guests 12, the physical positioning of physical interactive objects, and the physical position of the speaker 24). Therefore, the mapping system 44 can map the trajectory of the virtual tennis ball (coordinates including a virtual reference point) acquired by the controller 32 to the trajectory in the physical room of the interactive environment 14 (coordinates including a physical reference point). The audio controller 38 can use the mapped trajectory to generate realistic sound effects, for example, when the virtual tennis ball "hits" the floor and walls of the physical room. The mapped trajectory can also be used to provide a realistic visual representation of the virtual tennis ball bouncing in the physical room. Therefore, the mapping system 44 can enable the controller 32 to be used with various different physical environments (e.g., rooms) that function as the interaction environment 14.

[0034] It should be understood that the mapping system 44 may be part of the audio controller 38 or the controller 32, or it may exist outside of the controller 32 and the audio controller 38. The mapping system 44 may send and receive data via wireless communication protocols such as Bluetooth or Wi-Fi, or it may send and receive data via a wired connection. In some embodiments, the controller 32 and the audio controller 38 may be combined into a single device or system. The mapping system 44 may map the physical environment using any of the various types of mapping techniques (e.g., light detection and ranging (LiDAR), Wi-Fi signal-based measurement).

[0035] Although described herein as input devices 28, in some embodiments, at least one of the input devices 28 may also provide an output that contributes to the interactive audio presented to the guest 12. For example, in some embodiments, at least one input device 28 (e.g., a projection emitter, an interactive glove, a game controller, a conductor's baton) may include at least one of the speakers 24 of the interaction environment 14. In such embodiments, the input device 28 may be communicatively coupled to a controller 32 to provide information about the position and / or orientation of the input device 28, as well as information about the inputs received by the input device 28. Alternatively, the input device 28 may be communicatively coupled to an audio controller 38 to receive commands to generate at least a portion of the interactive audio within the interaction environment 14 using at least one speaker 24 of the input device in accordance with this disclosure.

[0036] Figure 2 is a schematic diagram of an attraction system 10 that includes an interaction environment 14 in which a guest 12 interacts with an interactive AR / VR image 18 associated with interactive audio 16. In the illustrated example, the interactive AR / VR image 18 is an image of a virtual bird 70 that the guest 12 sees through AR / VR goggles 22. The virtual bird 70 is associated with interactive audio 16 emitted by speakers 24 distributed in an array within the interaction environment 14. For example, the interactive audio 16 may include the bird song 72 of the virtual bird 70. Thus, as the position and / or orientation of the virtual bird 70 changes, the guest 12 may perceive the bird song 72 as coming from different directions.

[0037] Guest 12 can control the position of a virtual bird 70 using hand gestures so that the virtual bird 70 appears at the position indicated by Guest 12. Specifically, Guest 12 can wear a glove tracked by Sensor 30, and use this glove to detect gestures. For example, by pointing upwards with a gloved hand 74, Guest 12 can command the virtual bird 70 to move upwards. Sensor 30, such as a camera, can capture an image of the gloved hand 74 and / or collect data related to the gloved hand 74. Controller 32 can receive the sensor data and determine the coordinates (e.g., position and orientation) of the gloved hand 74 relative to a reference point in the interaction environment 14. By comparing the coordinates (e.g., position and orientation) related to the gloved hand 74 with previously recorded coordinates, it can determine whether the gesture performed by the hand has changed. If it has changed, the position and orientation of the virtual bird 70 can be adjusted. For example, the controller 32 determines that the virtual bird 70 should move if the position and / or orientation of the gloved hand 74 changes by at least a threshold amount from the previously recorded position and / or orientation. The controller 32 can also identify the new adjusted coordinates (e.g., position and / or orientation) of the virtual bird 70 based on the change in the coordinates of the gloved hand 74. As a result, the controller 32 can send a command to the AR / VR goggles 22 worn by the guest 12 to instruct the AR / VR goggles 22 to display the virtual bird 70 at the adjusted position.

[0038] After the position of the virtual bird 70 has been adjusted, the bird song 72 can be heard as if it is coming from the adjusted position of the virtual bird 70. Thus, the controller 32 can send a command to the audio controller 38 to adjust the interactive audio 16 so that the guest 12 perceives the interactive audio 16 as coming from the adjusted position. Specifically, the command sent to the audio controller 38 may include the coordinates of the guest 12 in the interaction environment 14, the coordinates of the gloved hand 74, the coordinates of the adjusted position of the virtual bird 70, and the coordinates of the three-dimensional volume to be filled with the sound of the bird song 72. Based on the command from the controller 32, the audio controller 38 can identify which speaker 24 on or within the interaction environment 14 should be activated to generate the bird song 72 audio based on the current position of the virtual bird 70. The audio controller 38 can also identify which speaker 24 (or combination of speakers 24) should generate different parts of the bird song 72 audio at what respective volume levels. For example, since Figure 2 shows a hypothetical bird 70 facing left, a speaker 24A located to the right of the interaction environment 14 or within the range of the right side can be instructed to generate bird song audio 72. On the other hand, a speaker 24B located to the left of the interaction environment 14 or within the range of the left side can be muted. In further embodiments, it is possible to emit specific sound waves from different speakers, taking into account that these specific sound waves cancel each other out in certain locations and amplify each other in other locations.

[0039] In one embodiment, the mapping system 44 can map the coordinates (e.g., position and orientation) of a gloved hand 74, the coordinates of a virtual bird 70, and the coordinates of a 3D volume filled with the audio of bird song 72 before transmitting them to the audio controller 38. Specifically, the mapping system 44 can map these coordinates from coordinates defined with respect to a reference point used by the controller 32 to coordinates defined with respect to a reference point in the physical space of the interaction environment 14. In one embodiment, mapping the coordinates may include multiplying the coordinates by a scaling factor, translating the coordinates, and so on.

[0040] Interactive audio 16 can appear to be emitted by a physical interactive object 20, as it may appear to be emitted by a virtual object that is part of an interactive AR / VR image 18. Figure 3 is a schematic diagram of an attraction system 10 that includes a virtual interaction environment 14 in which an interactive object 20 (e.g., a physical object, game piece, game item or weapon, prop, tool, input device) appears to emit interactive audio 16 in response to interaction with a guest 12. In the illustrated embodiment, the interactive object 20 is a teddy bear 80. In some embodiments, there may be no speaker 24 embedded in the teddy bear 80. Instead, the interaction environment 14 may have multiple speakers 24 configured to emit interactive audio 16 that can be perceived as the teddy bear 80 speaking. The teddy bear 80's speech 82 may also be triggered, for example, by the guest 12 picking up the teddy bear 80 (e.g., from a stand 84). Therefore, a sensor 30, such as a camera or motion sensor, can be used to detect whether the guest 12 has picked up the teddy bear 80. In addition to or instead of this, a sensor 30 that detects changes in weight / pressure applied to the top of the stand 84 can also be embedded on or inside the stand 84.

[0041] The controller 32 can analyze sensor data from the sensor 30 to determine whether the guest 12 has picked up the teddy bear 80. If the guest 12 has picked up the teddy bear 80, the controller 32 can trigger the emission of interactive audio 16. Specifically, the controller 32 can send the coordinates of the teddy bear 80 and the guest 12 to the audio controller 38 along with an instruction or command to emit the interactive audio 16. Since the interactive audio 16 can be perceived as being generated by the teddy bear 80, the source and direction of the interactive audio 16 can be determined based on the position coordinates of the guest 12 and the teddy bear 80. As a result, the audio controller 38 can activate a specific speaker 24 located within the interaction environment 14 to generate the interactive audio 16 that appears to be directed away from the teddy bear 80 and towards the guest 12. In one embodiment, a mapping system 44 can map the coordinates of the teddy bear 80 and the guest 12 before they are available to the audio controller 38. Specifically, this mapping may involve transforming coordinates from one coordinate system to another, transforming coordinates from a virtual space constructed by the controller 32 to the physical space of the interaction environment 14, and redefining coordinates for a new reference point.

[0042] It should be understood that different types of interactive audio 16 can be provided based on the type of interaction (e.g., action, movement, emotion, user input, gesture) of the guest 12 with the interactive object 20. For example, the teddy bear 80 may say "hello" in response to being picked up and "goodbye" in response to being placed back on or inside the stand 84.

[0043] Figure 4 is a flowchart of an embodiment of the process 100 for generating interactive audio 16. The reference figures for structural features described later are shown in the various embodiments of Figures 1 to 3. The process 100 includes receiving sensor data from various sensors 30 found within the interaction environment 14 (block 102). As described, the sensor data can indicate various interactions of the guest 12 with the interaction environment 14. Specifically, the sensor data can indicate the guest 12's identity, the guest 12's current location and orientation, as well as the guest 12's movements, actions, gestures, and / or emotions.

[0044] In the embodiment shown in Figure 4, process 100 also includes receiving user input (block 104) (e.g., from input device 28, from sensor 30). As described, in some embodiments, a guest 12 of the attraction system 10 can interact with various interaction features 26 of the interaction environment 14 by providing user input via input device 28. User input provided via input device 28 can control, for example, a portion of the interactive AR / VR image 18. For example, a virtual helicopter can be remotely controlled using an input device 28 such as a smartphone. In this example, the user input may be received through the touch screen of the smartphone and may include user-provided instructions to adjust the position of the virtual helicopter. In another example, a guest 12 can fire a virtual projectile using an input device 28 such as a blaster gun. In this example, the user input may include instructions indicating that a push button has been triggered to fire the virtual projectile.

[0045] In the embodiment shown in Figure 4, process 100 includes identifying the state of guest 12 in the interaction environment 14 and the guest 12's interactions with elements of the interaction environment 14 based on sensor data and / or input data (block 106). The state of guest 12 may include guest 12's coordinates (e.g., position, orientation), movement, gestures, emotions, and / or actions. Identifying guest 12's position can enable audio controller 38 to deliver customized audio to each or all guests 12. Guest 12's position can also be used to provide interactive audio 16 originating from guest 12's position, for example, to amplify the footsteps of a particular guest 12 for the benefit of guest 12 and other guests 12. Furthermore, guest 12's position and orientation can also be used to provide customized audio to guest 12. For example, quieter audio can be provided for a particular guest 12. Guest 12's movement, gestures, emotions, and / or actions can indicate guest 12's interactions with interactive functions 26 of the interaction environment 14. For example, in a specific interaction environment 14, the guest 12 can control the interactive AR / VR image 18 using hand gestures. Alternatively, the guest 12 can provide user input through interactions with the interaction features 26 of the interaction environment 14. For example, the interactive AR / VR image 18 can be controlled using an input device 28.

[0046] In the embodiment shown in Figure 4, process 100 includes determining changes in the interactive audio 16 (block 108) based on the state of the guest 12 and the guest 12's interaction with the interaction features 26 of the interaction environment 14. Changes in the interactive audio 16 can include possible changes in the content of the interactive audio 16, changes in the position and orientation of the sound source of the interactive audio 16, and / or changes in the 3D volume that the interactive audio 16 occupies or traverses. Generally, such changes in the interactive audio 16 can cause the guest 12 to perceive the interactive audio 16 in different ways, such as sounding from a new direction, becoming louder, getting closer or further away, etc.

[0047] In the embodiment shown in Figure 4, process 100 includes mapping the positional coordinates of the guest 12, interactive audio 16, interactive AR / VR image 18, and / or interactive object 20 using a mapping system 44 (block 110). In some embodiments, mapping coordinates may include converting coordinates from one coordinate system to another, defining coordinates relative to a new reference point, or adapting coordinates to a new space. For example, coordinates may be mapped from coordinates defined relative to a reference point used by controller 32 to coordinates defined relative to a reference point used by audio controller 38. This mapping can be applied, for example, when controller 32 determines coordinates in virtual space and audio controller 38 utilizes coordinates defined in the physical space of the virtual environment 14. In some cases, a model (e.g., a trained machine learning model or artificial intelligence model) can be used to convert coordinates in virtual space to suitable coordinates in physical space. In some embodiments, the model may be a digital twin of a physical environment, such as a virtual speaker array which is a digital twin of a physical speaker array, and may be a software-based structure (e.g., a virtual environment) which includes a digital twin of environmental elements.

[0048] In the embodiment shown in Figure 4, process 100 includes causing speaker 24 to generate interactive audio 16 or signaling it to do so (block 112). As described, the audio controller 38 can instruct speakers 24 distributed throughout the interaction environment 14 to emit interactive audio 16 in such a way that some of the interactive AR / VR images 18 and / or interactive objects 20 appear to be emitting interactive audio 16. It should be understood that the interactive audio 16 heard by guest 12 can be a synthesis of sounds generated by various speakers 24. That is, each speaker 24 can be instructed to emit a different sound that, when combined with sounds emitted from other speakers 24, can form interactive audio 16. Furthermore, constructive and destructive interference of sound waves emitted by various speakers 24 can be used to create specific auditory effects within the interaction environment 14, such as quiet areas or loud areas. In addition, speakers 24 can be configured to emit interactive audio 16 in a way that provides custom audio to each guest 12. For example, one guest 12 can listen to one type of audio content, and another guest 12 can listen to a different type of audio content.

[0049] The claimed technologies described herein refer to and apply to tangible objects and specific examples of a practical nature that are not abstract, intangible, or purely theoretical, but which certainly improve the art. Furthermore, if any of the claims appended to the end of this specification contain one or more elements designated as "...means for (performing) (a function)" or "...steps for (performing) (a function)," such elements should be interpreted in accordance with 112(f) of the United States Patent Act. On the other hand, any claim containing elements designated in any other form should not be interpreted in accordance with 112(f) of the United States Patent Act. [Explanation of Symbols]

[0050] 10 Attraction System 12 Guests 14 Interaction environment 16 Interactive Audio 18 Interactive AR / VR Images 20 Interactive Objects 22 AR / VR Goggles 24 speakers 26 Interaction Features 28 Input devices 29 Input Mechanism 30 sensors 32 controllers 34 processors 36 memory 38 Audio Controllers 40 processors 42 memory 44 Mapping Systems

Claims

1. It is an attraction system, A display configured to depict augmented reality and / or virtual reality (AR / VR) images to a guest in an interactive space, An audio controller configured to operate an array of speakers distributed throughout the entire interaction space, A controller including one or more processors, The one or more processors are provided with, The system receives data indicating the state of the guest, including the guest's actions, movements, gestures, facial expressions, and physical expressions, user input received from an input device associated with the guest, or a combination thereof. The AR / VR image is adjusted in response to the state of the guest. The audio controller is instructed to operate the speakers in the array to provide interactive audio based on the state of the guest. An attraction system configured in such a way.

2. The attraction system according to claim 1, wherein the AR / VR image includes an image displayed on or by an AR / VR lens, an image projected by a projector, or a combination thereof.

3. The attraction system according to claim 1, comprising a virtual model stored on or within the controller, wherein the virtual model includes a digital twin of the speaker array.

4. The attraction system according to claim 1, comprising a mapping system configured to map coordinates associated with the guest from a first coordinate system to a second coordinate system associated with the dynamic portion of the AR / VR image, wherein the interactive audio appears to the guest as if it were originating from the dynamic portion of the AR / VR image.

5. The attraction system according to claim 1, comprising a mapping system configured to map coordinates associated with the dynamic portion of the AR / VR image from a first coordinate system to a second coordinate system associated with the physical location of the guest, wherein the interactive audio appears to the guest as if it were originating from the dynamic portion of the AR / VR image.

6. The attraction system according to claim 1, wherein the one or more processors of the controller are configured to perform mapping between a first coordinate system and a second coordinate system, the first coordinate system being associated with the interaction space and the second coordinate system being associated with a virtual model.

7. The first coordinate system is, The position coordinates of the guest in the interaction space, including one or more of the guest's position, the guest's posture, the position or orientation of at least one of the guest's eyes, the position or orientation of at least one of the guest's ears, and the position or orientation of at least one of the guest's appendages, The position coordinates of the speaker in the array within the interaction space, The attraction system according to claim 6, including the following:

8. The attraction system according to claim 6, wherein the first coordinate system is associated with the controller, and the second coordinate system is associated with the audio controller.

9. The attraction system according to claim 1, wherein the input device associated with the guest includes at least one speaker in the array, the input device is communicably coupled to the controller to provide at least a portion of the data indicating the state of the guest, and the input device is communicably coupled to the audio controller to receive commands to provide at least a portion of the interactive audio via the at least one speaker of the input device.

10. The attraction system according to claim 1, wherein providing the interactive audio includes changing the sound source position of the interactive audio.

11. The attraction system according to claim 1, wherein the audio controller is configured to instruct the array of speakers to generate the interactive audio such that the guest perceives the interactive audio as originating from a physical interactive object in the interaction space.

12. The attraction system according to claim 1, wherein the interactive audio heard by the first guest is different from the interactive audio heard by the second guest.

13. A non-temporary computer-readable medium containing instructions for providing interactive audio, wherein the instructions, when executed by a processor, Receiving sensor data from a sensor, Based on the aforementioned sensor data, the first interaction of the guest with the interactive object is identified, Based on the guest's first interaction with the interactive object, a first interactive audio associated with the interactive object is determined; Commanding an array of speakers to output the first interactive audio which appears to originate from the interactive object, A non-temporary computer-readable medium configured to cause the processor to perform an operation including the following.

14. When the aforementioned instruction is executed by the processor, Based on the sensor data, identify the guest's second interaction with the interactive object, Based on the guest's second interaction with the interactive object, a second interactive audio associated with the interactive object is determined. Commanding the speaker array to output the second interactive audio, A non-temporary computer-readable medium according to claim 13, configured to cause the processor to perform an operation including the above.

15. The instruction that identifies the first interaction of the guest with the interactive object is, when executed by the processor, Based on the aforementioned sensor data, the first coordinates of the guest and the second coordinates of the interactive object are detected, wherein the first coordinates indicate the position, orientation, or both of the guest, and the second coordinates indicate the position, orientation, or both of the interactive object. In order to identify the first interaction of the guest with the interactive object, the first coordinates of the guest and the second coordinates of the interactive object are compared with previously obtained coordinates of the guest and the previously obtained coordinates of the interactive object, A non-temporary computer-readable medium according to claim 13, configured to cause the processor to perform an operation including the above.

16. The instruction that identifies the first interaction of the guest with the interactive object is, when executed by the processor, Mapping the coordinates of the guest and the interactive object from a first coordinate system defined with respect to a physical reference point to a second coordinate system defined with respect to a virtual reference point. A non-temporary computer-readable medium according to claim 13, configured to cause the processor to perform an operation including the above.

17. A method for generating interactive audio, The controller receives user input from an input device indicating the trajectory of a virtual object in the virtual space, Based on the user input, the controller determines the trajectory of the virtual object in the physical space based on the mapping between the virtual space and the physical space, Based on the user input, the controller is used to include the virtual object moving in the virtual space according to the trajectory in an interactive augmented reality and / or virtual reality (AR / VR) image. via an audio controller, generating interactive audio in the physical space based on the trajectory of the virtual object in the physical space, A method that includes this.

18. The controller is used to instruct the speaker to emit the interactive audio such that the interactive audio appears at a first time point as originating from a first point in the trajectory of the virtual object, The controller is used to instruct the speaker to emit the interactive audio such that the interactive audio appears at a second time point as originating from a second point in the trajectory of the virtual object, The method according to claim 17, including the method described in claim 17.

19. The method according to claim 17, comprising mapping the trajectory of the virtual object from a coordinate space used by the controller to a coordinate space used by the audio controller via the controller.

20. Providing the aforementioned interactive audio means The controller is used to activate a first speaker set to emit a first portion of the interactive audio at a first time point, The controller is used to activate a second speaker set, different from the first speaker set, to emit the second portion of the interactive audio at a second time point. The method according to claim 17, including the method described in claim 17.

Images